OTel Collector on Kubernetes

The OpenTelemetry Collector Contrib distribution can be deployed on Kubernetes as a DaemonSet using the opentelemetry-collector Helm chart. It collects traces, metrics, logs, Kubernetes events, and infrastructure objects and forwards them to Kloudfuse over OTLP HTTP.

Prerequisites

  • Helm installed and configured against your cluster.

  • The external hostname of your Kloudfuse cluster.

  • If ingestion authentication is enabled, an API key — see Ingestion Authentication with API Key.

  • OTel Collector Contrib version 0.130.0 or later (required for k8s_leader_elector extension used in system-level metrics).

Install the Collector

Add the OTel Helm repository and install:

helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
helm repo update
helm upgrade --install opentelemetry-collector open-telemetry/opentelemetry-collector \
  -f helm-values.yaml \
  --namespace otel \
  --create-namespace

Basic Configuration

The following helm-values.yaml configures the OTel Collector to receive OTLP data and export metrics to Kloudfuse. Use https or http for the endpoint scheme depending on whether TLS is enabled on your Kloudfuse cluster:

image:
  repository: "otel/opentelemetry-collector-contrib"

config:
  exporters:
    debug:
      verbosity: basic
    otlphttp:
      tls:
        insecure: true          # remove if using https
      metrics_endpoint: https://<kloudfuse-hostname>/ingester/otlp/metrics
      headers:
        Kf-Api-Key: <token>     # omit if auth is not enabled

  extensions:
    health_check: {}

  processors:
    batch:
      timeout: 10s
    resourcedetection:
      detectors: [env, eks, ec2, gcp, aks, azure]
      override: false
      timeout: 2s

  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: ${env:MY_POD_IP}:4317
        http:
          cors:
            allowed_origins:
              - http://*
              - https://*
          endpoint: 0.0.0.0:4318

  service:
    extensions: [health_check]
    pipelines:
      metrics:
        receivers: [otlp]
        processors: [batch, resourcedetection]
        exporters: [otlphttp]

mode: daemonset

service:
  enabled: true

ports:
  metrics:
    enabled: true
  otlp:
    enabled: true
  otlp-http:
    enabled: true

presets:
  kubernetesAttributes:
    enabled: true

resources: {}
yaml
The memory_ballast extension was removed in OTel Collector Contrib 0.97.0 and later. Remove it from any existing configurations — it will cause a startup failure with unknown type: "memory_ballast".
When running multiple OTel Collector DaemonSets on the same cluster (for example, one for logs and one for metrics), the default hostPorts (4317, 4318, 8888, etc.) conflict because only one DaemonSet pod can bind a given hostPort per node. Disable the ports that are not needed for each collector: 
ports:
  otlp:
    enabled: false
  otlp-http:
    enabled: false
  metrics:
    enabled: false
yaml

Logs

To add a logs pipeline, set the logs_endpoint on the exporter and add a logs pipeline to the service:

config:
  exporters:
    otlphttp:
      logs_endpoint: https://<kloudfuse-hostname>/ingester/otlp/v1/logs
      metrics_endpoint: https://<kloudfuse-hostname>/ingester/otlp/metrics
      traces_endpoint: https://<kloudfuse-hostname>/ingester/otlp/traces

  service:
    pipelines:
      logs:
        receivers: [otlp]
        processors: [batch, k8sattributes, resourcedetection]
        exporters: [otlphttp]
      metrics:
        receivers: [otlp]
        processors: [batch, k8sattributes, resourcedetection]
        exporters: [otlphttp]
      traces:
        receivers: [otlp]
        processors: [batch, k8sattributes, resourcedetection]
        exporters: [otlphttp]
yaml

Traces

To add a traces pipeline, set the traces_endpoint and add a traces pipeline:

config:
  exporters:
    otlphttp:
      traces_endpoint: https://<kloudfuse-hostname>/ingester/otlp/traces

  service:
    pipelines:
      traces:
        receivers: [otlp]
        processors: [batch, k8sattributes, resourcedetection]
        exporters: [otlphttp]
yaml

Kubernetes and System-Level Metrics

Kloudfuse APM Services correlate APM data with Kubernetes infrastructure metrics on the Infra tab of a service detail page.

Enable the hostmetrics, kubeletstats, and k8s_cluster receivers to collect node, pod, and cluster-level metrics alongside application telemetry. The k8s_cluster receiver uses the k8s_leader_elector extension to prevent duplicate cluster metrics in multi-Collector deployments (requires OTel Collector Contrib 0.130.0+).

command:
  extraArgs: [--feature-gates=receiver.kubeletstats.enableCPUUsageMetrics]

config:
  extensions:
    health_check: {}
    k8s_leader_elector/k8scluster:
      auth_type: serviceAccount
      lease_name: k8sclusterlease
      lease_namespace: default   # namespace where the lease resource will be created

  processors:
    batch:
      timeout: 10s
    resource:
      attributes:
        - key: kf_metrics_agent   (1)
          value: "otlp"
          action: upsert
    resourcedetection:
      detectors: [env, eks, ec2, gcp, aks, azure]
      override: false
      timeout: 2s
    transform/k8spodphase:       (2)
      metric_statements:
        - context: datapoint
          statements:
            - 'set(attributes["phase"], "Pending")   where metric.name == "k8s.pod.phase" and value_int == 1'
            - 'set(attributes["phase"], "Running")   where metric.name == "k8s.pod.phase" and value_int == 2'
            - 'set(attributes["phase"], "Succeeded") where metric.name == "k8s.pod.phase" and value_int == 3'
            - 'set(attributes["phase"], "Failed")    where metric.name == "k8s.pod.phase" and value_int == 4'
            - 'set(attributes["phase"], "Unknown")   where metric.name == "k8s.pod.phase" and value_int == 5'

  receivers:
    hostmetrics:
      collection_interval: 30s
      scrapers:
        cpu:
          metrics:
            system.cpu.utilization:
              enabled: true
        memory:
          metrics:
            system.memory.utilization:
              enabled: true
        disk:
        filesystem:
          metrics:
            system.filesystem.utilization:
              enabled: true
    kubeletstats:
      collection_interval: 30s
      auth_type: serviceAccount
      endpoint: "https://${env:K8S_NODE_NAME}:10250"
      insecure_skip_verify: true
      node: '${env:K8S_NODE_NAME}'
      collect_all_network_interfaces:
        pod: true
        node: true
      extra_metadata_labels:
        - k8s.volume.type
      metric_groups: [node, pod, container, volume]
      metrics:
        k8s.container.cpu.node.utilization:
          enabled: true
        k8s.pod.cpu.node.utilization:
          enabled: true
        k8s.container.memory.node.utilization:
          enabled: true
        k8s.pod.memory.node.utilization:
          enabled: true
    k8s_cluster:
      k8s_leader_elector: k8s_leader_elector/k8scluster
      collection_interval: 30s
      auth_type: serviceAccount
      allocatable_types_to_report: [cpu, memory, pods]
      metrics:
        k8s.node.condition:
          enabled: true
      node_conditions_to_report: [Ready, MemoryPressure]
      resource_attributes:
        container.id:
          enabled: false
        k8s.container.status.last_terminated_reason:
          enabled: true

  service:
    extensions: [health_check, k8s_leader_elector/k8scluster]
    pipelines:
      metrics:
        receivers: [otlp, kubeletstats, hostmetrics, k8s_cluster]
        processors: [resource, resourcedetection, transform/k8spodphase, batch]
        exporters: [otlphttp]
yaml
1 kf_metrics_agent: otlp enables correlation of host metrics with APM services in the Kloudfuse UI.
2 transform/k8spodphase converts the numeric pod phase value from k8s_cluster into a human-readable string label.

Kubernetes Events

Kloudfuse ingests Kubernetes events through a dedicated logs pipeline using the k8s_events receiver.

Run the Collector as a Deployment (not a DaemonSet) when collecting Kubernetes events. Running in DaemonSet mode results in duplicate events. This is a known upstream limitation — see GitHub issue #42266.

The events pipeline uses a separate logs endpoint and a resource processor to tag events as OTel-sourced:

config:
  exporters:
    otlphttp/k8sevents:
      logs_endpoint: https://<kloudfuse-hostname>/ingester/otlp/k8s_events   (1)

  receivers:
    k8s_events: {}

  processors:
    resource/k8s_events:
      attributes:
        - key: kf_events_agent
          value: "otlp"
          action: upsert

  service:
    pipelines:
      logs/k8s_events:                          (2)
        receivers: [k8s_events]
        processors: [resource/k8s_events]
        exporters: [otlphttp/k8sevents]
yaml
1 Kloudfuse treats data sent to /ingester/otlp/k8s_events as Kubernetes events, not log records.
2 This must be a separate pipeline from the main logs pipeline — do not add k8s_events to an existing pipeline.

Grant the Collector service account read access to events:

rules:
  - apiGroups: [""]
    resources: [events]
    verbs: ["get", "list", "watch"]
yaml

Verify Events Are Arriving

Kubernetes events collected via the OTel k8s_events receiver are stored in the Logs store, not the Events store. Query them via the Loki API using the kf_events_agent label:

{kf_events_agent="otlp"}

To filter by namespace or event reason, combine labels:

{kf_events_agent="otlp", k8s_namespace_name="<namespace>"}
OTel Kubernetes events appear in Logs (tagged kf_events_agent=otlp), not in the Events explorer. This differs from the Datadog Agent path, where Kubernetes events are stored in the Events store under source="kubernetes".

Kubernetes Infrastructure Objects

Kloudfuse can ingest Kubernetes infrastructure objects (pods, nodes, deployments, etc.) using the k8sobjects receiver. Objects appear in the Kloudfuse UI under Infrastructure > Kubernetes.

Each data type requires its own pipeline. Do not reuse an existing logs pipeline for objects. Exporter names must be prefixed with otlphttp/.
Kloudfuse drops objects that have not been updated within 15 minutes. To prevent stale data, configure a pull interval shorter than 15 minutes on each object type.

Step 1: Configure Exporter

exporters:
  otlphttp/k8sobjects:
    logs_endpoint: https://<kloudfuse-hostname>/ingester/otlp/k8s_objects
yaml

Step 2: Configure Receiver

receivers:
  k8sobjects:
    objects:
      - name: pods
        interval: 5m
        mode: pull
      - name: deployments
      - name: replicasets
      - name: services
      - name: nodes
      - name: jobs
      - name: cronjobs
      - name: daemonsets
      - name: statefulsets
      - name: persistentvolumes
      - name: persistentvolumeclaims
      - name: roles
      - name: rolebindings
      - name: clusterroles
      - name: clusterrolebindings
      - name: serviceaccounts
      - name: ingresses
      - name: namespaces
yaml

Step 3: Configure Processors

processors:
  resource:
    attributes:
      - key: kf_infra_agent   (1)
        value: "otlp"
        action: upsert
  resourcedetection:
    detectors: [env, gcp]     # replace with your environment: eks, aks, ec2, azure
    timeout: 2s
    override: false
  k8sattributes:
    extract:
      metadata:
        - k8s.namespace.name
        - k8s.deployment.name
        - k8s.statefulset.name
        - k8s.daemonset.name
        - k8s.cronjob.name
        - k8s.job.name
        - k8s.node.name
        - k8s.pod.name
        - k8s.pod.uid
        - k8s.pod.start_time
        - k8s.replicaset.name
yaml
1 kf_infra_agent: otlp enables filtering by OTel-sourced objects in the Kloudfuse UI.

Step 4: Configure Pipeline

service:
  pipelines:
    logs/k8sobjects:
      receivers: [k8sobjects]
      processors: [k8sattributes, resource, resourcedetection]
      exporters: [otlphttp/k8sobjects]
yaml

Step 5: RBAC Permissions

Grant the Collector service account read access to all collected object types:

rules:
  - apiGroups: [""]
    resources: [namespaces, nodes, pods, services, persistentvolumeclaims, persistentvolumes, serviceaccounts]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["apps"]
    resources: [daemonsets, deployments, replicasets, statefulsets]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["rbac.authorization.k8s.io"]
    resources: [clusterroles, clusterrolebindings, roles, rolebindings]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["networking.k8s.io"]
    resources: [ingresses]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["batch"]
    resources: [jobs, cronjobs]
    verbs: ["get", "list", "watch"]
yaml
Missing RBAC rules can silently prevent object collection without visible errors in the Collector logs.

Step 6: Leader Election

Prevent duplicate collection in multi-Collector deployments:

extensions:
  leader_election: {}

service:
  extensions: [leader_election]
yaml

Complete Example

extensions:
  leader_election: {}

receivers:
  k8sobjects:
    objects:
      - name: pods
        interval: 5m
        mode: pull
      - name: deployments
      - name: replicasets
      - name: services
      - name: nodes
      - name: jobs
      - name: cronjobs
      - name: daemonsets
      - name: statefulsets
      - name: persistentvolumes
      - name: persistentvolumeclaims
      - name: roles
      - name: rolebindings
      - name: clusterroles
      - name: clusterrolebindings
      - name: serviceaccounts
      - name: ingresses
      - name: namespaces

processors:
  resource:
    attributes:
      - key: kf_infra_agent
        value: "otlp"
        action: upsert
  resourcedetection:
    detectors: [env, gcp]
    timeout: 2s
    override: false
  k8sattributes:
    extract:
      metadata:
        - k8s.namespace.name
        - k8s.pod.name
        - k8s.pod.uid
        - k8s.node.name
        - k8s.deployment.name
        - k8s.replicaset.name
        - k8s.statefulset.name
        - k8s.daemonset.name
        - k8s.job.name
        - k8s.cronjob.name

exporters:
  otlphttp/k8sobjects:
    logs_endpoint: https://<kloudfuse-hostname>/ingester/otlp/k8s_objects

service:
  extensions: [leader_election]
  pipelines:
    logs/k8sobjects:
      receivers: [k8sobjects]
      processors: [k8sattributes, resource, resourcedetection]
      exporters: [otlphttp/k8sobjects]
yaml

After restarting the Collector, open the Kloudfuse UI at Infrastructure > Kubernetes and confirm that pods, nodes, deployments, and namespaces appear. Use the filter kf_infra_agent=otlp to view only OTel-sourced objects.

External References