Datadog Agent Trace Integration

The Datadog Agent acts as a trace collection proxy for applications instrumented with Datadog SDKs (dd-trace-*).

Applications send spans to the agent over the local network; the agent batches and forwards them to Kloudfuse.

No re-instrumentation is needed if your applications already use Datadog SDKs.

Overview

The Datadog Agent exposes a trace listener on port 8126. Applications configured with DD_AGENT_HOST and DD_TRACE_AGENT_PORT send spans to this listener using the Datadog APM wire protocol.

The agent then forwards the spans to Kloudfuse over HTTPS.

Prerequisites

Before you begin, ensure the following requirements are in place:

Requirement Details

Datadog Agent

Datadog Agent DaemonSet running in your cluster. See Datadog Kubernetes Integration for agent installation.

Application instrumentation

Applications instrumented with a Datadog SDK (dd-trace-java, ddtrace, dd-trace-go, or equivalent).

Helm values

The Datadog Agent Helm values configured to forward traces to Kloudfuse — set datadog.site and datadog.apiKey to your Kloudfuse cluster values.

Configure Applications to Send Traces

Environment Variables

Every application pod that should emit traces needs these environment variables:

Variable Value

DD_AGENT_HOST

The IP address of the host node — use the Downward API (see below)

DD_TRACE_AGENT_PORT

8126

DD_SERVICE

A stable logical name for your service (e.g., payments-api)

DD_ENV

Deployment environment (e.g., production, staging)

DD_VERSION

Application version — used for version-based trace filtering

Kubernetes Pod Spec

Use the Downward API to expose the host node’s IP as DD_AGENT_HOST:

env:
  - name: DD_AGENT_HOST
    valueFrom:
      fieldRef:
        fieldPath: status.hostIP
  - name: DD_TRACE_AGENT_PORT
    value: "8126"
  - name: DD_SERVICE
    value: payments-api
  - name: DD_ENV
    value: production
  - name: DD_VERSION
    value: "1.4.2"
yaml
A Kubernetes Service name is not used here because the Datadog Agent runs as a DaemonSet — one pod per node. A ClusterIP Service would round-robin across all agent pods in the cluster, so your app pod on Node A could send traces to the agent on Node B. status.hostIP resolves to the IP of the node the app pod is scheduled on, ensuring traces are always sent to the agent on the same node.

Language-Specific Notes

The Datadog SDK reads DD_AGENT_HOST and DD_TRACE_AGENT_PORT automatically for all supported languages. No additional code changes are needed in most cases.

Java — Use JAVA_TOOL_OPTIONS to attach the tracer JAR without modifying the application entrypoint:

env:
  - name: JAVA_TOOL_OPTIONS
    value: "-javaagent:/agent/dd-java-agent.jar"
  - name: DD_AGENT_HOST
    valueFrom:
      fieldRef:
        fieldPath: status.hostIP
  - name: DD_TRACE_AGENT_PORT
    value: "8126"
yaml

Mount the agent JAR using an init container or a shared volume, or bake it into the application image.

Python — Use ddtrace-run to wrap the application command:

command: ["ddtrace-run", "python", "app.py"]
yaml

Go — Import and start the tracer at the beginning of main():

import "gopkg.in/DataDog/dd-trace-go.v1/ddtrace/tracer"

func main() {
    tracer.Start()
    defer tracer.Stop()
    // ...
}
go

The Go tracer reads DD_AGENT_HOST and DD_TRACE_AGENT_PORT from the environment automatically.

DogStatsD Custom Metrics

If your application also emits custom metrics via DogStatsD, configure the DogStatsD port alongside the trace settings:

env:
  - name: DD_AGENT_HOST
    valueFrom:
      fieldRef:
        fieldPath: status.hostIP
  - name: DD_DOGSTATSD_PORT
    value: "8125"
yaml

DogStatsD metrics are forwarded to the Kloudfuse metrics ingester through the same agent.

Verify Traces Are Arriving

  1. Generate traffic through your instrumented application — make a request to any instrumented endpoint.

  2. In the Kloudfuse UI, click APM in the top navigation.

  3. Select Services from the drop-down and set the time range to the last 15 minutes.

  4. Your service name (from DD_SERVICE) should appear in the list.

  5. Click the service name, then select the Traces tab to see individual trace records.

If no data appears within two minutes of generating traffic, proceed to the troubleshooting steps below.

Troubleshooting

No Traces Appearing

  1. Confirm the Datadog Agent pod on the same node as your application is running and healthy:

    kubectl get pods -n datadog -o wide

    Identify the agent pod on the same node as your application pod.

  2. Check the agent’s trace listener is accepting connections from your pod:

    kubectl exec -it <app-pod> -- nc -zv $DD_AGENT_HOST 8126

    A successful connection prints Connection to <ip> 8126 port [tcp] succeeded.

  3. Inspect the agent logs for trace ingestion errors:

    kubectl logs -n datadog <agent-pod> | grep -i "trace\|error\|warn"

  4. Confirm DD_AGENT_HOST resolves correctly inside the application pod:

    kubectl exec -it <app-pod> -- env | grep DD_

Agent Host Not Reachable

If nc fails to connect to port 8126:

  • Verify the Datadog Agent DaemonSet is running on the node your pod is scheduled on.

  • Check that no NetworkPolicy blocks traffic from application pods to the DaemonSet on port 8126. The required policy allows egress from your application namespace to the datadog namespace (or wherever the agent runs) on port 8126.

    apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dd-agent-traces
  namespace: <app-namespace>
spec:
  podSelector: {}
  policyTypes:
    - Egress
  egress:
    - ports:
        - protocol: TCP
          port: 8126
    yaml

  • If the DaemonSet uses host networking (hostNetwork: true), confirm the node’s firewall allows intra-node traffic on port 8126.

Service Not Appearing in APM

If the agent is running and receiving spans but the service does not appear in Kloudfuse APM:

  • Confirm DD_SERVICE is set on the application pod — without it the SDK defaults to a generated name derived from the process binary.

  • Check that the agent is successfully forwarding to Kloudfuse: inspect agent logs for HTTP errors on the outbound ingester request.

  • Verify the agent’s datadog.apiKey in the Helm values matches an active ingestion API key in Kloudfuse under Administration → API Keys.

High Cardinality Causing Throttling

If trace ingestion is throttled (visible in the Kloudfuse Usage Dashboard):

  • Avoid using per-request identifiers (user IDs, UUIDs, order numbers) as span names. Use templated names (GET /users/{id}) and store the variable data in span tags.

  • Check for unbounded tag cardinality — resource attributes (http.url, http.route) that contain full URLs with query strings inflate the number of unique series. Strip query parameters before setting the attribute.

See Cardinality Management for a full reference.

Kloudfuse Reference