Tracing Architecture Integration

APM tracing gives you a complete picture of how a single request moves through your system — which services it touched, how long each step took, and where errors or latency originated. Unlike metrics and logs, traces require changes to your application code. This page explains how tracing fits into the integration pipeline and what components are involved.

Overview

Tracing works by injecting a small instrumentation library into each service. The library records the start and end of every significant operation (a database query, an outbound HTTP call, a message queue publish) as a span, tags it with service name and context labels, and links related spans across service boundaries into a single trace.

Tracing data flow — instrumented services emit spans to a collection agent
Tracing data flow
The New Relic agent (Service D in the diagram) is the exception to the two-stage pattern. It sends spans directly to the Kloudfuse ingester using the New Relic collector protocol — bypassing kf-agent and the Datadog Agent entirely. No additional collection agent is required. See New Relic Trace Integration for configuration details.

No trace data is visible in Kloudfuse until instrumentation is added. Metrics and logs can flow without touching application code; traces cannot.

Key Concepts

Spans

A span is a single timed operation within a service. Every span records:

  • A name (for example, http.request, db.query, kafka.produce)

  • A start time and duration

  • A status (success or error)

  • Attributes — key-value pairs describing the operation (HTTP method, SQL statement, queue name)

  • A trace ID and parent span ID — linking the span to its place in the overall request

Traces

A trace is the complete journey of a single request across all the services it touched. It is assembled from individual spans that share the same trace ID. The trace forms a tree: the root span is the entry point (typically the first HTTP request), and child spans are nested operations within and across services.

Kloudfuse displays traces in the APM Trace Explorer as a flame graph, showing the relative timing and nesting of every span.

Context Propagation

For spans from different services to be linked into a single trace, the trace ID must be passed between services as each request crosses a service boundary. This is called context propagation.

Both Datadog and OTel SDKs handle context propagation automatically for common protocols:

  • HTTP — trace context is injected into request headers (traceparent for W3C, x-datadog-trace-id for Datadog).

  • gRPC — context is injected into gRPC metadata.

  • Message queues — context is embedded in message attributes (Kafka, RabbitMQ, SQS, depending on the library).

As long as all services in a call chain use compatible SDKs and propagation formats, spans assemble into a complete trace automatically.

Integration Components

Tracing involves three components: the SDK in your code, a collection agent, and the Kloudfuse ingester.

Instrumentation SDK

The SDK is a library added to your application. It intercepts calls to common frameworks and clients (HTTP servers, database drivers, message queue clients) and emits spans without requiring per-call code changes. This is called automatic instrumentation.

You can also add manual instrumentation to create custom spans for business logic that the automatic instrumentation does not cover.

Kloudfuse supports two SDK families:

  • OpenTelemetry SDKs — open-standard libraries available for all major languages. Spans are emitted over OTLP (HTTP or gRPC) to a collector.

  • Datadog SDKs (dd-trace-*) — Datadog-compatible libraries. Spans are emitted over the Datadog trace protocol to the Datadog Agent.

Both families produce traces that appear identically in Kloudfuse APM. You can mix OTel and Datadog SDKs across services in the same environment — traces from both are correlated by trace ID.

Collection Agent

The collection agent runs in your cluster and receives spans from application pods. It buffers, compresses, and forwards spans to the Kloudfuse ingester. Applications send spans to the agent over the local network — no direct internet access from application pods is required.

Two agents support trace collection:

kf-agent (OTel-based)

kf-agent is a Kubernetes-native collector deployed as a Kubernetes Deployment. It exposes OTLP endpoints for applications instrumented with OTel SDKs:

Because kf-agent runs as a Deployment (not a DaemonSet), all pods in the cluster share a single set of agent pods regardless of which node they run on. This minimizes resource overhead for trace collection.

See OTel Kubernetes Integration for installation and configuration.

Datadog Agent

The Datadog Agent DaemonSet also accepts traces from applications instrumented with Datadog SDKs. Application pods send traces to the agent on the host node using the DD_AGENT_HOST environment variable (which resolves to the node’s host IP).

The Datadog Agent trace listener runs on port 8126. Applications using Datadog SDKs configure this via the DD_AGENT_HOST and DD_TRACE_AGENT_PORT environment variables.

See Datadog Kubernetes Integration for Datadog Agent setup.

Kloudfuse Ingester

The ingester receives spans from the collection agent over HTTPS and routes them to the Traces Store. Kloudfuse accepts spans in two formats:

  • OTLP HTTP Protobuf (POST /ingester/v1/traces) — for kf-agent and OTel Collectors

  • Datadog APM wire protocol — for Datadog Agent

Both formats produce traces queryable with the same APM views and TraceQL.

Choosing Between OTel and Datadog SDKs

Consideration OTel SDK Datadog SDK

Standard

Open standard (CNCF); vendor-neutral

Datadog proprietary; maps to OTel conventions internally

Automatic instrumentation coverage

Broad; most major frameworks covered

Broad; especially strong for JVM and Python

Agent

kf-agent or any OTel Collector

Datadog Agent

Manual instrumentation API

OTel Tracer API (standardized)

dd-trace API

Mixed environments

Compatible with Datadog via context propagation headers

Compatible with OTel via W3C traceparent header

If you are migrating from Datadog to Kloudfuse, Datadog SDK instrumentation works without changes — just repoint the Datadog Agent at Kloudfuse.

If you are starting fresh, OTel SDKs are recommended for new services because they are vendor-neutral and avoid lock-in.

Language Instrumentation Guides

Step-by-step instrumentation for each supported language:

For migrating existing Datadog-instrumented services to OTel:

Tracing Outside Kubernetes

Docker

For Docker Compose environments, run kf-agent or the Datadog Agent as an additional service in your Compose file. Application containers send traces to the agent container by service name.

See Docker Integration for Docker setup including the agent service definition.

AWS Lambda

The Datadog Lambda Extension Layer collects APM traces from Lambda functions. The extension runs inside the Lambda execution environment alongside your function code and forwards traces directly to Kloudfuse without a separate agent.

See Lambda Extension Layer for setup.

Verifying Traces Are Arriving

After deploying the agent and instrumenting at least one service:

  1. Trigger a request through your application — for example, make an HTTP call to an instrumented endpoint.

  2. In the Kloudfuse UI, click APM in the top navigation.

  3. Select Traces from the drop-down and set the time range to the last 15 minutes.

  4. The trace for your request should appear. Each row represents one trace; click it to open the flame graph view.

If no traces appear, check:

  • The agent pod is running and its logs show no connection errors to the Kloudfuse ingester.

  • The application is configured with the correct agent host and port.

  • The instrumentation library is loaded — for JVM agents, check that -javaagent: is present in the JVM flags.