OpenTelemetry Collector on Kubernetes Environment

The OpenTelemetry Collector supports receiving and exporting Metrics, Logs, and Traces on the Kubernetes environment.

Basic OpenTelemetry Deployment on Kubernetes Environments

When deploying OpenTelemetry Collector on a Kubernetes Environment, refer to Basic OpenTelemetry Integration on Kubernetes, a sample helm-values.yaml file for configuring OpenTelemetry to export logs, metrics, and traces to Kloudfuse.

Ensure that you use either https or http correctly, in conjunction with tls being enabled or disabled.

Basic OpenTelemetry Integration on Kubernetes
image:
  repository: "otel/opentelemetry-collector-contrib"
config:
  exporters:
    debug:
      verbosity: basic
    otlphttp:
      tls:
        insecure: true # add only if you're using insecure communication
      logs_endpoint: https://<REPLACE KFUSE ADDRESS>/ingester/otlp/v1/logs
      metrics_endpoint: https://<REPLACE KFUSE ADDRESS>/ingester/otlp/metrics
      traces_endpoint: https://<REPLACE KFUSE ADDRESS>/ingester/otlp/traces
  extensions:
    health_check: {}
  processors:
    batch:
      timeout: 10s
    resourcedetection:
      detectors:
        - env
        - eks
        - ec2
        - gcp
        - aks
        - azure
      override: false
      timeout: 2s
  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: ${env:MY_POD_IP}:4317
        http:
          cors:
            allowed_origins:
              - http://*
              - https://*
          endpoint: 0.0.0.0:4318
  service:
    extensions:
      - health_check
    pipelines:
      logs:
        exporters:
          - otlphttp
        processors:
          - batch
          - resourcedetection
        receivers:
          - otlp
      metrics:
        exporters:
          - otlphttp
        processors:
          - batch
          - resourcedetection
        receivers:
          - otlp
      traces:
        exporters:
          - otlphttp
        processors:
          - batch
          - resourcedetection
        receivers:
          - otlp
    telemetry:
      metrics:
        address: ${MY_POD_IP}:8888
mode: daemonset
service:
  enabled: true
nameOverride: otelcol
ports:
  metrics:
    enabled: true
  otlp:
    enabled: true
  otlp-http:
    enabled: true
presets:
  kubernetesAttributes:
    enabled: true
resources: {}

Install in Default Namespace

Install the otel repo in a default namespace.

helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
helm upgrade --install opentelemetry-collector open-telemetry/opentelemetry-collector -f values.yaml

Kubernetes and System-Level Metrics

Kloudfuse APM Services integrate with Kubernetes and System Metrics to show related infrastructure metrics. See Service Detail Infrastructure.

You can export these metrics using either Datadog agent, or the OpenTelemetry collector.

When using the OpenTelemetry Collector, you must update it with additional configurations. Specifically, enable hostmetrics receivers, and add additional kf_metrics_agent resource attributes. See Enable Additional Attributes for System-Level Metrics.

Enable Additional Attributes for System-Level Metrics
...
command:
  extraArgs: [--feature-gates=receiver.kubeletstats.enableCPUUsageMetrics]
...
config:
  ...
  processors:
    batch:
      timeout: 10s
    resource:
      attributes:
      - key: kf_metrics_agent
        value: "otlp"
        action: upsert
    resourcedetection:
      detectors:
        - env
        - eks
        - ec2
        - gcp
        - aks
        - azure
      override: false
      timeout: 2s
  receivers:
    hostmetrics:
      collection_interval: 30s
      scrapers:
        cpu:
          metrics:
            system.cpu.utilization:
              enabled: true
        memory:
          metrics:
            system.memory.utilization:
              enabled: true
        disk:
        filesystem:
          metrics:
            system.filesystem.utilization:
              enabled: true
    kubeletstats:
      collection_interval: 30s
      auth_type: "none"
      endpoint: "http://${env:K8S_NODE_NAME}:10255"
      extra_metadata_labels:
        - k8s.volume.type
      metric_groups:
        - node
        - pod
        - container
        - volume
...
service:
  pipelines:
   ...
    metrics:
      exporters: [otlphttp]
      processors: [batch, resource, resourcedetection]
      receivers: [otlp, kubeletstats, hostmetrics]