OTel Kubernetes Integration

Overview

The OTel Collector on Kubernetes can be run as a DaemonSet (one pod per node) and collects:

  • Traces — received from instrumented pods over OTLP gRPC (4317) or OTLP HTTP (4318)

  • Metrics — node CPU, memory, and disk via kubeletstats; pod and container metrics via k8s_cluster

  • Logs — container log files tailed from each node

  • Kubernetes events — pod scheduling, node conditions, and deployment changes via the k8sevents receiver

  • Infrastructure objects — pod, deployment, and node metadata via the k8sobjects receiver

All telemetry is exported to Kloudfuse over OTLP HTTP. The Helm chart manages RBAC, service accounts, and ConfigMap generation automatically.

Prerequisites

Before you begin, ensure the following requirements are in place:

Requirement Details

Helm

Helm installed and configured against your cluster.

Kloudfuse hostname

The external hostname of your Kloudfuse cluster.

API key

If ingestion authentication is enabled, an API key — see Ingestion Authentication with API Key.

OTel Collector Contrib

OTel Collector Contrib version 0.130.0 or later (required for k8s_leader_elector extension used in system-level metrics).

Install the Collector

Add the OTel Helm repository and install:

helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
helm repo update
helm upgrade --install opentelemetry-collector open-telemetry/opentelemetry-collector \
  -f helm-values.yaml \
  --namespace otel \
  --create-namespace

Basic Configuration

The following helm-values.yaml configures the OTel Collector to receive OTLP data and export metrics to Kloudfuse. Use https or http for the endpoint scheme depending on whether TLS is enabled on your Kloudfuse cluster:

image:
  repository: "otel/opentelemetry-collector-contrib"

config:
  exporters:
    debug:
      verbosity: basic
    otlphttp:
      tls:
        insecure: true          # remove if using https
      metrics_endpoint: https://<kloudfuse-hostname>/ingester/otlp/metrics
      headers:
        Kf-Api-Key: <token>     # omit if auth is not enabled

  extensions:
    health_check: {}

  processors:
    batch:
      timeout: 10s
    resourcedetection:
      detectors: [env, eks, ec2, gcp, aks, azure]
      override: false
      timeout: 2s

  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: ${env:MY_POD_IP}:4317
        http:
          cors:
            allowed_origins:
              - http://*
              - https://*
          endpoint: 0.0.0.0:4318

  service:
    extensions: [health_check]
    pipelines:
      metrics:
        receivers: [otlp]
        processors: [batch, resourcedetection]
        exporters: [otlphttp]

mode: daemonset

service:
  enabled: true

ports:
  metrics:
    enabled: true
  otlp:
    enabled: true
  otlp-http:
    enabled: true

presets:
  kubernetesAttributes:
    enabled: true

resources: {}
yaml
The memory_ballast extension was removed in OTel Collector Contrib 0.97.0 and later. Remove it from any existing configurations — it will cause a startup failure with unknown type: "memory_ballast".
When running multiple OTel Collector DaemonSets on the same cluster (for example, one for logs and one for metrics), the default hostPorts (4317, 4318, 8888, etc.) conflict because only one DaemonSet pod can bind a given hostPort per node. Disable the ports that are not needed for each collector: 
ports:
  otlp:
    enabled: false
  otlp-http:
    enabled: false
  metrics:
    enabled: false
yaml

Logs

To add a logs pipeline, set the logs_endpoint on the exporter and add a logs pipeline to the service:

config:
  exporters:
    otlphttp:
      logs_endpoint: https://<kloudfuse-hostname>/ingester/otlp/v1/logs
      metrics_endpoint: https://<kloudfuse-hostname>/ingester/otlp/metrics
      traces_endpoint: https://<kloudfuse-hostname>/ingester/otlp/traces

  service:
    pipelines:
      logs:
        receivers: [otlp]
        processors: [batch, k8sattributes, resourcedetection]
        exporters: [otlphttp]
      metrics:
        receivers: [otlp]
        processors: [batch, k8sattributes, resourcedetection]
        exporters: [otlphttp]
      traces:
        receivers: [otlp]
        processors: [batch, k8sattributes, resourcedetection]
        exporters: [otlphttp]
yaml

Traces

To add a traces pipeline, set the traces_endpoint and add a traces pipeline:

config:
  exporters:
    otlphttp:
      traces_endpoint: https://<kloudfuse-hostname>/ingester/otlp/traces

  service:
    pipelines:
      traces:
        receivers: [otlp]
        processors: [batch, k8sattributes, resourcedetection]
        exporters: [otlphttp]
yaml

Kubernetes and System-Level Metrics

Kloudfuse APM Services correlate APM data with Kubernetes infrastructure metrics on the Infra tab of a service detail page.

Enable the hostmetrics, kubeletstats, and k8s_cluster receivers to collect node, pod, and cluster-level metrics alongside application telemetry. The k8s_cluster receiver uses the k8s_leader_elector extension to prevent duplicate cluster metrics in multi-Collector deployments (requires OTel Collector Contrib 0.130.0+).

command:
  extraArgs: [--feature-gates=receiver.kubeletstats.enableCPUUsageMetrics]

config:
  extensions:
    health_check: {}
    k8s_leader_elector/k8scluster:
      auth_type: serviceAccount
      lease_name: k8sclusterlease
      lease_namespace: default   # namespace where the lease resource will be created

  processors:
    batch:
      timeout: 10s
    resource:
      attributes:
        - key: kf_metrics_agent   (1)
          value: "otlp"
          action: upsert
    resourcedetection:
      detectors: [env, eks, ec2, gcp, aks, azure]
      override: false
      timeout: 2s
    transform/k8spodphase:       (2)
      metric_statements:
        - context: datapoint
          statements:
            - 'set(attributes["phase"], "Pending")   where metric.name == "k8s.pod.phase" and value_int == 1'
            - 'set(attributes["phase"], "Running")   where metric.name == "k8s.pod.phase" and value_int == 2'
            - 'set(attributes["phase"], "Succeeded") where metric.name == "k8s.pod.phase" and value_int == 3'
            - 'set(attributes["phase"], "Failed")    where metric.name == "k8s.pod.phase" and value_int == 4'
            - 'set(attributes["phase"], "Unknown")   where metric.name == "k8s.pod.phase" and value_int == 5'

  receivers:
    hostmetrics:
      collection_interval: 30s
      scrapers:
        cpu:
          metrics:
            system.cpu.utilization:
              enabled: true
        memory:
          metrics:
            system.memory.utilization:
              enabled: true
        disk:
        filesystem:
          metrics:
            system.filesystem.utilization:
              enabled: true
    kubeletstats:
      collection_interval: 30s
      auth_type: serviceAccount
      endpoint: "https://${env:K8S_NODE_NAME}:10250"
      insecure_skip_verify: true
      node: '${env:K8S_NODE_NAME}'
      collect_all_network_interfaces:
        pod: true
        node: true
      extra_metadata_labels:
        - k8s.volume.type
      metric_groups: [node, pod, container, volume]
      metrics:
        k8s.container.cpu.node.utilization:
          enabled: true
        k8s.pod.cpu.node.utilization:
          enabled: true
        k8s.container.memory.node.utilization:
          enabled: true
        k8s.pod.memory.node.utilization:
          enabled: true
    k8s_cluster:
      k8s_leader_elector: k8s_leader_elector/k8scluster
      collection_interval: 30s
      auth_type: serviceAccount
      allocatable_types_to_report: [cpu, memory, pods]
      metrics:
        k8s.node.condition:
          enabled: true
      node_conditions_to_report: [Ready, MemoryPressure]
      resource_attributes:
        container.id:
          enabled: false
        k8s.container.status.last_terminated_reason:
          enabled: true

  service:
    extensions: [health_check, k8s_leader_elector/k8scluster]
    pipelines:
      metrics:
        receivers: [otlp, kubeletstats, hostmetrics, k8s_cluster]
        processors: [resource, resourcedetection, transform/k8spodphase, batch]
        exporters: [otlphttp]
yaml
1 kf_metrics_agent: otlp enables correlation of host metrics with APM services in the Kloudfuse UI.
2 transform/k8spodphase converts the numeric pod phase value from k8s_cluster into a human-readable string label.

Kubernetes Events

Kloudfuse ingests Kubernetes events through a dedicated logs pipeline using the k8s_events receiver.

Run the Collector as a Deployment (not a DaemonSet) when collecting Kubernetes events. Running in DaemonSet mode results in duplicate events. This is a known upstream limitation — see GitHub issue #42266.

The events pipeline uses a separate logs endpoint and a resource processor to tag events as OTel-sourced:

config:
  exporters:
    otlphttp/k8sevents:
      logs_endpoint: https://<kloudfuse-hostname>/ingester/otlp/k8s_events   (1)

  receivers:
    k8s_events: {}

  processors:
    resource/k8s_events:
      attributes:
        - key: kf_events_agent
          value: "otlp"
          action: upsert

  service:
    pipelines:
      logs/k8s_events:                          (2)
        receivers: [k8s_events]
        processors: [resource/k8s_events]
        exporters: [otlphttp/k8sevents]
yaml
1 Kloudfuse treats data sent to /ingester/otlp/k8s_events as Kubernetes events, not log records.
2 This must be a separate pipeline from the main logs pipeline — do not add k8s_events to an existing pipeline.

Grant the Collector service account read access to events:

rules:
  - apiGroups: [""]
    resources: [events]
    verbs: ["get", "list", "watch"]
yaml

Verify Events Are Arriving

Kubernetes events collected via the OTel k8s_events receiver are stored in the Logs store, not the Events store. Query them via the Loki API using the kf_events_agent label:

{kf_events_agent="otlp"}

To filter by namespace or event reason, combine labels:

{kf_events_agent="otlp", k8s_namespace_name="<namespace>"}
OTel Kubernetes events appear in Logs (tagged kf_events_agent=otlp), not in the Events explorer. This differs from the Datadog Agent path, where Kubernetes events are stored in the Events store under source="kubernetes".

Kubernetes Infrastructure Objects

Kloudfuse can ingest Kubernetes infrastructure objects (pods, nodes, deployments, etc.) using the k8sobjects receiver. Objects appear in the Kloudfuse UI under Infrastructure > Kubernetes.

Each data type requires its own pipeline. Do not reuse an existing logs pipeline for objects. Exporter names must be prefixed with otlphttp/.
Kloudfuse drops objects that have not been updated within 15 minutes. To prevent stale data, configure a pull interval shorter than 15 minutes on each object type.

Step 1: Configure Exporter

exporters:
  otlphttp/k8sobjects:
    logs_endpoint: https://<kloudfuse-hostname>/ingester/otlp/k8s_objects
yaml

Step 2: Configure Receiver

receivers:
  k8sobjects:
    objects:
      - name: pods
        interval: 5m
        mode: pull
      - name: deployments
      - name: replicasets
      - name: services
      - name: nodes
      - name: jobs
      - name: cronjobs
      - name: daemonsets
      - name: statefulsets
      - name: persistentvolumes
      - name: persistentvolumeclaims
      - name: roles
      - name: rolebindings
      - name: clusterroles
      - name: clusterrolebindings
      - name: serviceaccounts
      - name: ingresses
      - name: namespaces
yaml

Step 3: Configure Processors

processors:
  resource:
    attributes:
      - key: kf_infra_agent   (1)
        value: "otlp"
        action: upsert
  resourcedetection:
    detectors: [env, gcp]     # replace with your environment: eks, aks, ec2, azure
    timeout: 2s
    override: false
  k8sattributes:
    extract:
      metadata:
        - k8s.namespace.name
        - k8s.deployment.name
        - k8s.statefulset.name
        - k8s.daemonset.name
        - k8s.cronjob.name
        - k8s.job.name
        - k8s.node.name
        - k8s.pod.name
        - k8s.pod.uid
        - k8s.pod.start_time
        - k8s.replicaset.name
yaml
1 kf_infra_agent: otlp enables filtering by OTel-sourced objects in the Kloudfuse UI.

Step 4: Configure Pipeline

service:
  pipelines:
    logs/k8sobjects:
      receivers: [k8sobjects]
      processors: [k8sattributes, resource, resourcedetection]
      exporters: [otlphttp/k8sobjects]
yaml

Step 5: RBAC Permissions

Grant the Collector service account read access to all collected object types:

rules:
  - apiGroups: [""]
    resources: [namespaces, nodes, pods, services, persistentvolumeclaims, persistentvolumes, serviceaccounts]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["apps"]
    resources: [daemonsets, deployments, replicasets, statefulsets]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["rbac.authorization.k8s.io"]
    resources: [clusterroles, clusterrolebindings, roles, rolebindings]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["networking.k8s.io"]
    resources: [ingresses]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["batch"]
    resources: [jobs, cronjobs]
    verbs: ["get", "list", "watch"]
yaml
Missing RBAC rules can silently prevent object collection without visible errors in the Collector logs.

Step 6: Leader Election

Prevent duplicate collection in multi-Collector deployments:

extensions:
  leader_election: {}

service:
  extensions: [leader_election]
yaml

Complete Example

extensions:
  leader_election: {}

receivers:
  k8sobjects:
    objects:
      - name: pods
        interval: 5m
        mode: pull
      - name: deployments
      - name: replicasets
      - name: services
      - name: nodes
      - name: jobs
      - name: cronjobs
      - name: daemonsets
      - name: statefulsets
      - name: persistentvolumes
      - name: persistentvolumeclaims
      - name: roles
      - name: rolebindings
      - name: clusterroles
      - name: clusterrolebindings
      - name: serviceaccounts
      - name: ingresses
      - name: namespaces

processors:
  resource:
    attributes:
      - key: kf_infra_agent
        value: "otlp"
        action: upsert
  resourcedetection:
    detectors: [env, gcp]
    timeout: 2s
    override: false
  k8sattributes:
    extract:
      metadata:
        - k8s.namespace.name
        - k8s.pod.name
        - k8s.pod.uid
        - k8s.node.name
        - k8s.deployment.name
        - k8s.replicaset.name
        - k8s.statefulset.name
        - k8s.daemonset.name
        - k8s.job.name
        - k8s.cronjob.name

exporters:
  otlphttp/k8sobjects:
    logs_endpoint: https://<kloudfuse-hostname>/ingester/otlp/k8s_objects

service:
  extensions: [leader_election]
  pipelines:
    logs/k8sobjects:
      receivers: [k8sobjects]
      processors: [k8sattributes, resource, resourcedetection]
      exporters: [otlphttp/k8sobjects]
yaml

After restarting the Collector, open the Kloudfuse UI at Infrastructure > Kubernetes and confirm that pods, nodes, deployments, and namespaces appear. Use the filter kf_infra_agent=otlp to view only OTel-sourced objects.

Verify Signals Are Arriving

  1. Confirm the Collector DaemonSet pods are running:

    kubectl get pods -n kloudfuse -l app.kubernetes.io/name=opentelemetry-collector

    You should see one pod per schedulable node.

  2. Check the Collector logs for export errors:

    kubectl logs -n kloudfuse -l app.kubernetes.io/name=opentelemetry-collector | grep -i "error\|warn\|failed"

  3. In the Kloudfuse UI, click Infrastructure > Kubernetes. Your cluster nodes and pods should appear within a few minutes of the Collector starting. Use the filter kf_infra_agent=otlp to confirm the OTel Collector is the data source.

  4. To verify metrics, open Metrics Explorer and search for k8s.node.cpu.usage. Filter by k8s.cluster.name to scope to your cluster.

  5. To verify traces, click APM > Services. Instrumented services should appear once they receive traffic.

Troubleshooting

Collector Pods Not Starting

If pods are in CrashLoopBackOff or Pending:

  1. Check pod events for configuration errors:

    kubectl describe pod -n kloudfuse <collector-pod>

  2. Validate the Collector configuration by checking the logs immediately after startup — configuration parse errors are printed on startup.

  3. Confirm the Helm values file sets the correct otlphttp endpoint pointing to your Kloudfuse cluster.

No Data Appearing in Kloudfuse

If pods are running but no data appears:

  1. Confirm the otlphttp exporter endpoint is reachable from the cluster:

    kubectl exec -it <collector-pod> -n kloudfuse -- wget -qO- https://<kloudfuse-hostname>/ingester/otlp/v1/logs

    A 4xx response indicates the endpoint is reachable. A connection timeout indicates a network or firewall issue.

  2. Check whether a NetworkPolicy blocks egress from the Collector pods to the Kloudfuse hostname on port 443.

  3. Confirm the Kf-Api-Key header value in the exporter matches an active key in Kloudfuse under Administration → API Keys (if authentication is enabled).

Kubernetes Events Not Appearing

If metrics and traces appear but events are missing:

  1. Confirm the k8sevents receiver is included in the logs/k8sevents pipeline in your Helm values.

  2. Confirm the Collector service account has permission to list and watch events:

    kubectl auth can-i list events --as=system:serviceaccount:kloudfuse:<collector-sa>

    If this returns no, update the ClusterRole to include events in its resource list.

External References