Datadog Kubernetes Integration

Overview

The Datadog Agent can run as a DaemonSet (one pod per node) with Cluster Agents(one or two replicas). Together they collect:

  • Node-level metrics — CPU, memory, disk, and network from each node via the node agent

  • Kube-state metrics — Kubernetes object state (deployments, pods, nodes) via the Cluster Agent

  • Container logs — collected from all running containers and forwarded over HTTPS

  • Kubernetes events — pod scheduling, node conditions, and deployment changes via the Cluster Agent

  • Application traces — received from instrumented pods over port 8126 and forwarded to the Kloudfuse APM ingester

Prerequisites

Before you begin, ensure the following requirements are in place:

Requirement Details

Datadog Agent version

Version 7.41 or higher required when repointing an existing agent from Datadog HQ to Kloudfuse.
Check the image tag: kubectl describe pod -n <namespace> | grep Image — tag should be 7.41 or later (for example, gcr.io/datadoghq/agent:7.41.0).

Helm chart version

Datadog Helm chart version 3.1.10 or higher required.
Check: helm list -n <namespace-where-agent-is-installed>

Install the Datadog Agent

Add the Datadog Helm repository:

helm repo add datadog https://helm.datadoghq.com
helm repo update

Create a datadog-values.yaml with your Kloudfuse hostname. This configuration enables logs, traces, metrics, and orchestrator exploration:

datadog:
  apiKey: <ingester-api_key>
  dd_url: https://<kloudfuse-hostname>   (1)
  logsEnabled: true
  logs:
    enabled: true
    containerCollectAll: true
    containerCollectUsingFiles: true
    autoMultiLineDetection: true
  kubeStateMetricsEnabled: false
  kubeStateMetricsCore:
    enabled: true
    ignoreLegacyKSMCheck: true
  orchestratorExplorer:
    enabled: true
  apm:
    enabled: true
    portEnabled: true
  dogstatsd:
    port: 8125
    useHostPort: true
  processAgent:
    enabled: true
  prometheusScrape:
    enabled: true
    version: 1

clusterAgent:
  replicaCount: 1
  enabled: true
  datadog_cluster_yaml:
    use_v2_api:
      events: true
      series: true
      service_checks: true
    orchestrator_explorer:
      orchestrator_dd_url: "https://<kloudfuse-hostname>"   (1)
  admissionController:
    enabled: false

agents:
  useConfigMap: true
  customAgentConfig:
    use_v2_api:
      events: true
      series: true
      service_checks: true
    logs_config:
      logs_dd_url: https://<kloudfuse-hostname>
      logs_no_ssl: false
      use_http: true
      auto_multi_line_detection: true
      use_v2_api: false
    apm_config:
      enabled: true
      apm_non_local_traffic: true
      apm_dd_url: "https://<kloudfuse-hostname>/ingester"
    orchestrator_explorer:
      orchestrator_dd_url: "https://<kloudfuse-hostname>"   (1)
    metadata_providers:
      - name: host
        interval: 300
yaml
1 Replace <kloudfuse-hostname> with your Kloudfuse cluster hostname.

Deploy the agent:

kubectl create namespace datadog-agent
helm upgrade --install datadog-agent -f datadog-values.yaml datadog/datadog \
  -n datadog-agent \
  --version 3.65.0

Dual Shipping

The Datadog Agent supports dual shipping — sending telemetry to multiple endpoints. This can be either Kloudfuse and another observability system or multiple instances of Kloudfuse.

Add the following to your existing Helm values file instead of replacing dd_url.

Agent version 7.41 or higher and Helm chart version 3.1.10 or higher are required.

Under the agents.customAgentConfig key:

agents:
  customAgentConfig:
    dd_url: "https://<kloudfuse-hostname>"
    additional_endpoints:
      "https://<alternate-hostname>":       # additional (Kloudfuse) endpoint
        - <api_key>
    use_v2_api:
      series: true
    logs_config:
      logs_dd_url: https://<kloudfuse-hostname>
      logs_no_ssl: false
      use_http: true
      use_v2_api: false
      additional_endpoints:
        - api_key: <api_key>
          Host: <kloudfuse-hostname>                 # additional (Kloudfuse) endpoint
          Port: 443
          use_compression: true
    apm_config:
      apm_dd_url: "https://<kloudfuse-hostname>/ingester"
      additional_endpoints:
        "https://<kloudfuse-hostname>":              # additional (Kloudfuse) endpoint
          - <api_key>
    process_config:
      process_dd_url: "https://<kloudfuse-hostname>/ingester"   # original (primary) endpoint
      additional_endpoints:
        "https://<kloudfuse-hostname>":              # additional (Kloudfuse) endpoint
          - <api_key>
    metadata_providers:
      - name: host
        interval: 300
yaml

The <kloudfuse-hostname> entries are the original destination the Agent already ships to — another observability system, or your first Kloudfuse instance; keep them set to the endpoints from your existing single-destination configuration. The additional_endpoints entries add the Kloudfuse destination: replace <kloudfuse-hostname> with your Kloudfuse cluster hostname and <api_key> with an ingestion API key.

Under the clusterAgent.datadog_cluster_yaml key:

clusterAgent:
  datadog_cluster_yaml:
    dd_url: "https://<kloudfuse-hostname>"       # original (primary) endpoint
    additional_endpoints:
      "https://<kloudfuse-hostname>":           # additional (Kloudfuse) endpoint
        - <api_key>
    use_v2_api:
      series: true
    process_config:
      process_dd_url: "https://<kloudfuse-hostname>/ingester"       # original (primary) endpoint
      additional_endpoints:
        "https://<kloudfuse-hostname>":                  # additional (Kloudfuse) endpoint
          - <api_key>
    orchestrator_explorer:
      orchestrator_dd_url: "https://<kloudfuse-hostname>"  # original (primary) endpoint
      additional_endpoints:
        "https://<kloudfuse-hostname>":                  # additional (Kloudfuse) endpoint
          - <api_key>
yaml

Apply the changes:

helm upgrade datadog-agent -f datadog-values.yaml datadog/datadog -n <namespace>

Configure Logs

Log collection from all containers is enabled by the base values file above. To exclude specific containers from log collection:

datadog:
  env:
    - name: DD_CONTAINER_EXCLUDE_LOGS
      value: "name:<container-name>"
yaml

To apply custom parsing or masking rules before forwarding:

agents:
  customAgentConfig:
    logs_config:
      processing_rules:
        - type: mask_sequences
          name: mask_credit_cards
          replace_placeholder: "[MASKED]"
          pattern: \b\d{4}[- ]?\d{4}[- ]?\d{4}[- ]?\d{4}\b
yaml

To override log settings for a specific pod, use Autodiscovery annotations:

annotations:
  ad.datadoghq.com/<CONTAINER_NAME>.logs: >
    [{"source": "my-app", "service": "my-service"}]
yaml

Configure Events

The Cluster Agent collects Kubernetes events (pod scheduling, node conditions, deployment rollouts) via the Kubernetes API. To forward process-level and container lifecycle events, add the following:

clusterAgent:
  datadog_cluster_yaml:
    process_config:
      events_dd_url: "https://<kloudfuse-hostname>/ingester"

agents:
  customAgentConfig:
    process_config:
      events_dd_url: "https://<kloudfuse-hostname>/ingester"
      container_collection:
        enabled: false
yaml

Verify Events Are Arriving

Kubernetes events collected by the Datadog Cluster Agent are stored in the Events store. Query them in the Kloudfuse UI or via the /events-query GraphQL API:

{
  events(
    durationSecs: 300,
    filter: { and: [{eq: {name: "@source", value: "kubernetes"}}] },
    timestamp: "<ISO-8601-timestamp>",
    limit: 20
  ) {
    id title text severity source eventType host
    labels { name value }
    timestamp
  }
}
graphql

Events from a specific pod appear with labels kube_namespace and kube_name. The eventType field is kubernetes_apiserver for events collected by the Cluster Agent.

Datadog Agent Kubernetes events appear in the Events store (source="kubernetes"). This differs from the OTel Collector path, where Kubernetes events are stored in Logs (tagged kf_events_agent=otlp).

Configure Metrics

Kube-State-Metrics

Kube-state-metrics is enabled by the base values file above. Set kubeStateMetricsEnabled: false to avoid running the legacy KSM check alongside the core check.

DogStatsD Custom Metrics

Configure application pods to send custom metrics to the agent on the node’s host IP:

env:
  - name: DD_AGENT_HOST
    valueFrom:
      fieldRef:
        fieldPath: status.hostIP
  - name: DD_ENTITY_ID
    valueFrom:
      fieldRef:
        fieldPath: metadata.uid
yaml

Prometheus Scraping

Prometheus autodiscovery is enabled by the base values file above. Add the following annotations to any pod that exposes a Prometheus endpoint:

prometheus.io/path: /metrics
prometheus.io/port: "9090"
prometheus.io/scrape: "true"
yaml

For explicit per-container configuration using Autodiscovery annotations:

ad.datadoghq.com/<CONTAINER_NAME>.check_names: '["openmetrics"]'
ad.datadoghq.com/<CONTAINER_NAME>.init_configs: '[{}]'
ad.datadoghq.com/<CONTAINER_NAME>.instances: >
  [{"openmetrics_endpoint": "http://%%host%%:%%port%%/metrics"}]
yaml

High-Cardinality Tags

The Datadog Agent defaults to low cardinality, tagging metrics only at the host level. Setting cardinality to orchestrator enables granular pod- and container-level tags:

datadog:
  env:
    - name: DD_CHECKS_TAG_CARDINALITY
      value: "orchestrator"
yaml

See Datadog — Tag Cardinality for details.

Custom Tags

Add custom tags to all metrics collected by the agent. Each tag must be in key:value format:

datadog:
  tags:
    - custom_tag_name:custom_tag_value
yaml

Verify Metrics Are Arriving

  1. Confirm both the node agent and Cluster Agent pods are running:

    kubectl get pods -n datadog-agent

    You should see one datadog-agent- pod per node and one datadog-agent-cluster-agent- pod.

  2. Check the agent status for active checks and any errors:

    kubectl exec -it <datadog-agent-pod> -n datadog-agent -- agent status

  3. In the Kloudfuse UI, click Infrastructure in the top navigation, then select Kubernetes. Your cluster nodes should appear within a few minutes of the agent starting.

  4. To confirm host metrics are flowing, open Metrics Explorer, search for kubernetes.cpu.usage.total, and filter by kube_cluster_name.

Troubleshooting

Agent Pods Not Ready

If agent pods are in CrashLoopBackOff or Pending:

  1. Check pod events for resource constraints or image pull errors:

    kubectl describe pod -n datadog-agent <agent-pod>

  2. Confirm the datadog-values.yaml has a valid dd_url pointing to your Kloudfuse cluster.

  3. Confirm the Helm chart version is 3.1.10 or higher:

    helm list -n datadog-agent

Metrics Not Appearing in Kloudfuse

If pods are running but no metrics appear:

  1. Inspect the agent logs for outbound connection errors:

    kubectl logs -n datadog-agent <agent-pod> | grep -i "error\|warn\|failed"

  2. Confirm dd_url is set correctly in the agent configuration:

    kubectl exec -it <agent-pod> -n datadog-agent -- agent config | grep dd_url

  3. Verify the agent’s API key matches a valid ingestion key in Kloudfuse under Administration → API Keys.

Logs Not Appearing

If metrics appear but container logs do not:

  1. Confirm datadog.logsEnabled: true and logs.enabled: true are set in your Helm values.

  2. Check the log agent section of the agent status output for collection errors:

    kubectl exec -it <agent-pod> -n datadog-agent -- agent status | grep -A 20 "Logs Agent"

  3. Confirm logs_dd_url in agents.customAgentConfig.logs_config matches your Kloudfuse hostname on port 443.

External References