OpenTelemetry Collector on Kubernetes Environment

The OpenTelemetry Collector supports receiving and exporting Metrics, Logs, and Traces on the Kubernetes environment.

Basic OpenTelemetry Deployment on Kubernetes Environments

When deploying OpenTelemetry Collector on a Kubernetes Environment, refer to Basic OpenTelemetry Integration on Kubernetes, a sample helm-values.yaml file for configuring OpenTelemetry to export logs, metrics, and traces to Kloudfuse.

Ensure that you use either https or http correctly, in conjunction with tls being enabled or disabled.

Basic OpenTelemetry Integration on Kubernetes
image:
  repository: "otel/opentelemetry-collector-contrib"

config:
  exporters:
    logging:
      verbosity: basic
    otlphttp:
      tls:
        insecure: true  # add only if you're using insecure communication
      metrics_endpoint: https://<REPLACE KFUSE ADDRESS>/ingester/otlp/metrics

  extensions:
    health_check: {}
    memory_ballast:
      size_in_percentage: 40

  processors:
    batch:
      timeout: 10s
    resourcedetection:
      detectors:
        - env
        - eks
        - ec2
        - gcp
        - aks
        - azure
      override: false
      timeout: 2s

  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: ${env:MY_POD_IP}:4317
        http:
          cors:
            allowed_origins:
              - http://*
              - https://*
          endpoint: 0.0.0.0:4318

  service:
    extensions:
      - health_check
    pipelines:
      metrics:
        exporters:
          - otlphttp
        processors:
          - batch
          - resourcedetection
        receivers:
          - otlp

  telemetry:
    metrics:
      address: ${MY_POD_IP}:8888

mode: daemonset

service:
  enabled: true

nameOverride: otelcol

ports:
  metrics:
    enabled: true
  otlp:
    enabled: true
  otlp-http:
    enabled: true

presets:
  kubernetesAttributes:
    enabled: true

resources: {}
yaml

Install in Default Namespace

Install the otel repo in a default namespace.

helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
helm upgrade --install opentelemetry-collector open-telemetry/opentelemetry-collector -f values.yaml

Kubernetes and System-Level Metrics

Kloudfuse APM Services integrate with Kubernetes and System Metrics to show related infrastructure metrics. See Service Detail Infrastructure.

You can export these metrics using either Datadog agent, or the OpenTelemetry collector.

When using the OpenTelemetry Collector, you must update it with additional configurations. Specifically, enable hostmetrics receivers, and add additional kf_metrics_agent resource attributes. See Enable Additional Attributes for System-Level Metrics.

Enable Additional Attributes for System-Level Metrics
...
command:
  extraArgs: [--feature-gates=receiver.kubeletstats.enableCPUUsageMetrics]
...
config:
  ...
  processors:
    batch:
      timeout: 10s
    resource:
      attributes:
      - key: kf_metrics_agent
        value: "otlp"
        action: upsert
    resourcedetection:
      detectors:
        - env
        - eks
        - ec2
        - gcp
        - aks
        - azure
      override: false
      timeout: 2s
  receivers:
    hostmetrics:
      collection_interval: 30s
      scrapers:
        cpu:
          metrics:
            system.cpu.utilization:
              enabled: true
        memory:
          metrics:
            system.memory.utilization:
              enabled: true
        disk:
        filesystem:
          metrics:
            system.filesystem.utilization:
              enabled: true
    kubeletstats:
      collection_interval: 30s
      auth_type: "none"
      endpoint: "http://${env:K8S_NODE_NAME}:10255"
      extra_metadata_labels:
        - k8s.volume.type
      metric_groups:
        - node
        - pod
        - container
        - volume
...
service:
  pipelines:
   ...
    metrics:
      exporters: [otlphttp]
      processors: [batch, resource, resourcedetection]
      receivers: [otlp, kubeletstats, hostmetrics]
yaml