APM Architecture, Concepts, and Workflow

A trace is the complete journey of a single request across every service it touched. It begins at the entry point (frontend, API gateway, or similar), is assembled from individual spans that share the same trace ID, and enables end-to-end latency and performance analysis.

The trace forms a tree: the root span is the entry point (typically the first HTTP request), and child spans are nested operations within and across services. Kloudfuse displays a trace in the Traces Explorer’s trace detail as a flame graph and waterfall, showing the relative timing and nesting of every span.

A span is a single timed operation within a service — a database query, an outbound HTTP call, a function call you instrumented manually. Every span records:

A service is a logically grouped, independently deployable unit of software that handles part of a transaction and exposes a network endpoint — for example, payment-service, auth-service, or inventory-db. Services generate spans, send traces, and are the primary unit of observability in APM Services. Exactly what distinguishes one service from another (so that, say, the same code running in two environments counts as two services) is configurable — see Service identity.

For spans from different services to be linked into a single trace, the trace ID must be passed between services as each request crosses a service boundary. This is called context propagation.

Both Datadog and OTel SDKs handle context propagation automatically for common protocols:

As long as all services in a call chain use compatible SDKs and propagation formats, spans assemble into a complete trace automatically.

The SDK is a library added to your application. It intercepts calls to common frameworks and clients (HTTP servers, database drivers, message queue clients) and emits spans without requiring per-call code changes. This is called automatic instrumentation.

You can also add manual instrumentation to create custom spans for business logic that the automatic instrumentation does not cover.

Kloudfuse supports two SDK families:

Both families produce traces that appear identically in Kloudfuse APM. You can mix OTel and Datadog SDKs across services in the same environment — traces from both are correlated by trace ID.

The collection agent runs in your cluster and receives spans from application pods. It buffers, compresses, and forwards spans to the Kloudfuse ingester. Applications send spans to the agent over the local network — no direct internet access from application pods is required.

Two agents support trace collection:

The ingester receives spans from the collection agent over HTTPS and routes them to the Traces Store. Kloudfuse accepts spans in two formats:

Both formats produce traces queryable with the same APM views and TraceQL.

For Docker Compose environments, run kf-agent or the Datadog Agent as an additional service in your Compose file. Application containers send traces to the agent container by service name.

See Docker Integration for Docker setup including the agent service definition.

The Datadog Lambda Extension Layer collects APM traces from Lambda functions. The extension runs inside the Lambda execution environment alongside your function code and forwards traces directly to Kloudfuse without a separate agent.

See Lambda Extension Layer for setup.

Start at APM > Services > Service List (Service List). The table is sorted by request volume by default and gives you, per service, requests/s, p50/p99 latency, max latency, error %, and Apdex for the selected time range. Scan it for the two signals that matter most day to day: a service with a rising Error %, or one whose p99 Latency has drifted away from its usual value. If you don’t yet know what "usual" looks like for a service, widen the time range to a day or a week first — a service that’s always noisy at this percentile isn’t the same problem as one that just changed.

Use the facet panel on the left (span type, language, version, and any Infrastructure/Cloud/Additional labels) to narrow the table to a subset of services — for example, only the services in one Kubernetes namespace or one cloud account. See Filtering traces for the full list of filterable attributes (the same facets apply to Services, Databases, and Traces).

Database-backed calls (Postgres, Redis, and so on) get their own row in the Databases tab of the same page, so a slow query shows up as its own line item instead of being folded into the calling service’s numbers.

For a topology-level view of the same health data, switch to APM > Services > Service Map (Service Map). Nodes are sized by traffic and colored by error rate, so a problem service — or an unhealthy dependency several hops away from the one you actually own — is visually obvious even before you read a number.

Don’t wait to eyeball this table, though: put alerts on the services that matter most, and let Kloudfuse tell you when a number moves instead of checking on a schedule.

Once the Service List, Service Map, or an alert points you at a suspect service, click it to open the Service detail page. This is where you narrow from "this service is unhealthy" to "this is the specific call, deployment, or dependency causing it." Work through these tabs roughly in this order:

From any of these tabs, switch to the Traces tab (or go directly to APM > Traces > Explorer) to find an individual slow or failing request. Filter by service_name and, if you already suspect it, span_type or an endpoint, then open a trace to see its flame graph and waterfall — this tells you exactly which span, on which host, took the time or returned the error, down to the stack trace and correlated logs.

When the traces list is too large to eyeball, or no single trace explains the pattern, reach for K-Lens instead of scrolling through dozens of traces one at a time. Rather than inspecting individual requests, K-Lens compares the slow or erroring population of spans against everything else and tells you which attribute — a pod, an availability zone, a database name, a service version, a Kubernetes node — is over-represented in the problem population. This turns "p99 is elevated" into "it’s these three pods in us-west1-a" in one step, without you having to guess which attribute to filter by first. See Using K-Lens as part of an investigation for a worked example of exactly this scenario.

Rather than waiting to notice a problem in the Service List, put an alert on it. The fastest path is from the service itself: on the Service detail page, click No APM alerts * (or * next to an existing alert count) to open the alert wizard already scoped to that service. You can also start from Alerts > New Alert and pick APM as the alert type — see Create an APM alert for the full field-by-field reference.

The wizard has four steps:

Service-level objectives (SLOs) complement threshold/anomaly alerts for services where you want to track compliance against a target over a rolling window (say, "99.9% of requests under 500ms over 30 days") rather than react to individual spikes. Configure one from the No SLO + control on the Service detail page; once configured, SLO burn appears in the SLOs tab. Use SLOs for the handful of services with an actual reliability commitment, and threshold/anomaly alerts for everything else — an SLO alert on every service just becomes noise.

When an APM alert fires, don’t stop at the metric that tripped it — pivot to the underlying request. From the alert, or from the service it fired on, use the Logs tab to see log lines from the same service and time window, and the Traces tab to find the individual requests responsible. If the alert doesn’t point at an obvious single cause, use K-Lens the same way you would in a manual investigation: select the time window the alert covers on the heatmap and let it rank the attributes for you.

Because Kloudfuse correlates traces, logs, and infrastructure metrics by shared labels (service name, pod, trace ID), you can move between "the alert fired," "here is a failing request," and "here is the log line and pod it happened on" without re-entering filters by hand.