Metrics for APM and Traces

Instead of showing individual spans Service List, Service detail, and Service Map show aggregated numbers for

  • request rate

  • latency

  • error rate

Kloudfuse computes those numbers from two families of metrics it derives from your span data.

  • edge_latency_

  • service_latency_

This page explains what each family measures, how a span becomes a metric, and why edge metrics carry more labels (and cost more) than service metrics.

Two metric families, two derivations

Metric family What it measures

edge_latency_*

A call between two services — a client span (the caller) matched to the server span it invoked, or a client span matched to a database call. This is what draws the edges (and their latency) on the Service Map. It powers the Downstream/Upstream tabs on Service detail.

service_latency_*

A service’s handling of a request — derived from each SERVER or CONSUMER span’s duration, independent of who called it. This powers the Service List’s request rate, latency, and error rate columns, and the Span Names tab.

Both families are histograms with the same bucket boundaries, so p50/p90/p99 latency will always be at the same scale for both metrics.

Bucket boundaries are (5ms, 10ms, 25ms, 50ms, 100ms, 250ms, 500ms; 1s, 2.5s, 5s, 10s, 20s, 30s, 60s, 120s, 300s)

edge_latency additionally tracks _min/_max per bucket; service_latency does not.

service_latency_* explicitly excludes database spans — a downstream database call only shows up as an edge, not as its own "service."

Service identity

Every span is tagged with a service_hash — a value generated from attributes to define what counts as "the same service". Two spans with the same service_name but different has values (for example, a different kube_namespace or cloud_account_id) show up as distinct rows in the Service List and distinct nodes on the Service Map. This means you can sometime see the same service name appear more than once.

By default, the identity attributes are:

global:
  serviceIdentityLabels:
    - kf_platform
    - availability_zone
    - cloud_account_id
    - kube_cluster_name
    - kube_namespace
    - project
    - region
    - service_name
yaml

service_name is always included in the hash first with the remaining attributes are hashed in sorted order. An attribute that is absent or explicitly UNKNOWN on a given span is skipped rather than hashed as an empty value. A span missing one identity attribute will not get a wildly different hash from otherwise-identical spans.

Plan the identity attribute list carefully. Adding attributes narrows what counts as "the same service" — useful for separating environments or tenants sharing a service name, but it also multiplies the number of distinct services (and edges) Kloudfuse tracks. See Trace Attribute Cardinality for how to reason about that trade-off.

How an edge is formed

An edge is a matched pair of spans across a service boundary: a CLIENT or PRODUCER span (the outbound call) and the SERVER or CONSUMER span it invoked, joined by span ID — the client span’s ID equals the server span’s parent span ID. A client span that calls a database is treated as a complete edge on its own, with the database identified as the "server" side.

Matching occurs as spans stream in: a client span is buffered briefly, waiting for its corresponding server span to arrive (they may come from different services, different agents, and arrive slightly out of order). Root spans — the entry point of a trace, with no client span above them — are recorded as edges immediately, with no client side.

Dangling edges

Not every client span gets a matching server span, and not every server span has a client span: the downstream service might not be instrumented, the client-side span might be dropped by sampling, or the request might simply be in flight when the buffering window closes. Kloudfuse doesn’t discard these — it still emits an edge metric after a short idle timeout (60 seconds by default), filling in the missing side so the edge is still visible and countable:

  • Client span arrived, server never did (typically an uninstrumented downstream, or an external HTTP call) — if the call was a database call, the missing side is filled in with the database’s identity. Otherwise, the missing server-side labels are marked UNKNOWN, the downstream service name is taken from the client span’s peer-service attribute when present, and the span name is normalized to EXTERNAL — this keeps an uninstrumented downstream from creating one distinct span name per unique call, which would otherwise be a cardinality problem.

  • Server span arrived, client never did — the missing client-side labels are marked UNKNOWN. If the server span is the trace’s root span, its service name is recorded as EXTERNAL (an external caller reached this service directly); otherwise it’s marked UNKNOWN (the caller was likely dropped by sampling or lost in transit).

This is why you may see EXTERNAL or UNKNOWN nodes on the Service Map or Downstream/Upstream tabs — they represent real, measured calls whose other end Kloudfuse couldn’t observe, not an error in the pipeline.

Why edge metrics cost more than service metrics

edge_latency_* carries a label for every identity attribute twice — once for the server side and once with a client_ prefix for the caller — plus the service identity attributes described above.

service_latency_* carries a single, flat set of labels (span type, error, source, service hash, and the identity attributes) since it describes one span, not a relationship between two.

Combined with the fact that an edge exists per pair of services rather than per service, edge_latency_* is structurally higher-cardinality than service_latency_*.

This is why widening the service identity attribute list has a larger effect on edge metrics than you might expect: every identity attribute you add is duplicated across both sides of every edge. See Trace Attribute Cardinality before adding identity or custom attributes broadly.

For the exact PromQL used to compute latency, request rate, error rate, and Apdex from these metrics, see APM Metrics API Reference.