APM Metrics API Reference

This is a reference for the underlying Prometheus and GraphQL queries that power the APM UI — useful if you are building your own dashboards or alerts against the same data APM uses, or debugging why a number in the UI looks the way it does. If you just want to use the screens themselves, see Services and Traces instead. For how these metrics are generated from raw span data, see Metrics for APM and Traces.

Overview

The APM system derives two Prometheus metric families from raw span data: edge_latency_* and service_latency_*. Together they power the RED metrics (Rate, Errors, Duration) and dependency/service graph shown throughout APM, and are queryable with PromQL. See Metrics for APM and Traces for the full explanation of how each is derived and why there are two.

Key metrics types

Both families expose the same suffixes:

  • _count: total number of requests

  • _sum: sum of all latencies (for average calculations)

  • _bucket: histogram buckets for percentile calculations

  • max / _min: maximum/minimum latency observed (edge_latency* only — see Metrics for APM and Traces)

Edge latency vs. service latency

Metric family Powers

service_latency_*

A service’s own handling of a request, independent of caller — the Service List columns and the Span Names tab on Service detail. Excludes database spans.

edge_latency_*

A call between two services (or a call to a database) — the Service Map’s edges, and the Downstream/Upstream tabs on Service detail.

Use service_latency_* for any query scoped to one service’s own numbers; use edge_latency_* whenever the query needs to distinguish caller from callee (a client_service_hash/client_service_name label, or a database call).

Common labels

  • service_hash: unique identifier for a service. The attributes used for hash calculation are configurable; default: ["kf_platform", "availability_zone", "cloud_account_id", "kube_cluster_name", "kube_namespace", "project", "region", "service_name"]

  • service_name: human-readable service name

  • client_service_hash: hash of the calling service (edge_latency_* only — for dependency tracking)

  • client_service_name: name of the calling service (edge_latency_* only)

  • span_type: type of span (for example, db for database calls); only present on edge_latency_*, since service_latency_* excludes database spans

  • error: boolean indicating if the request resulted in an error

  • le: histogram bucket boundaries (for percentile calculations)

Service List page queries

The Service List’s columns are powered by service_latency_* — see Edge latency vs. service latency.

P99 latency calculation

Description: Calculates the 99th percentile latency for all services.

histogram_quantile(0.99,
  sum(rate(service_latency_bucket[${stepInMs}ms]))
  by (service_hash, service_name, le)
)
promql

Parameters:

  • stepInMs: time window for rate calculation

P95 latency calculation

histogram_quantile(0.95,
  sum(rate(service_latency_bucket[${stepInMs}ms]))
  by (service_hash, service_name, le)
)
promql

P90 latency calculation

histogram_quantile(0.90,
  sum(rate(service_latency_bucket[${stepInMs}ms]))
  by (service_hash, service_name, le)
)
promql

P75 latency calculation

histogram_quantile(0.75,
  sum(rate(service_latency_bucket[${stepInMs}ms]))
  by (service_hash, service_name, le)
)
promql

P50 latency (median) calculation

histogram_quantile(0.50,
  sum(rate(service_latency_bucket[${stepInMs}ms]))
  by (service_hash, service_name, le)
)
promql

Average latency

sum by (service_hash, service_name) (rate(service_latency_sum[${stepInMs}ms]))
/
sum by (service_hash, service_name) (rate(service_latency_count[${stepInMs}ms]))
promql

Maximum latency

service_latency_* does not track max/_min — use edge_latency* (excluding database calls) for the closest equivalent:

max(max_over_time(edge_latency_max{span_type!="db"}[${stepInMs}ms]))
by (service_hash, service_name)
promql

Minimum latency

min(min_over_time(edge_latency_min{span_type!="db"}[${stepInMs}ms]))
by (service_hash, service_name)
promql

Request count

round(sum by (service_hash, service_name)
  (increase(service_latency_count[${stepInMs}ms]))
)
promql

Requests per second

sum by (service_hash, service_name)
  (rate(service_latency_count[${stepInMs}ms]))
promql

Error rate

sum by (service_hash, service_name) (rate(service_latency_count{error="true"}[${stepInMs}ms]))
/
sum by (service_hash, service_name) (rate(service_latency_count[${stepInMs}ms]))
promql

APDEX score

(sum by (service_hash, service_name) (increase(service_latency_bucket{le="1.0"}[${stepInMs}ms]))
 + sum by (service_hash, service_name) (increase(service_latency_bucket{le="0.5"}[${stepInMs}ms])))
/
(2 * sum by (service_hash, service_name) (increase(service_latency_count[${stepInMs}ms])))
promql

Service detail page queries

When viewing a specific service, queries are filtered by service_hash. The Span Names and Deployments tabs (grouping by span_name/endpoint or service_version) use service_latency_*, since they describe the service’s own handling of a request:

Service P99 latency over time

histogram_quantile(0.99,
  sum(rate(service_latency_bucket{service_hash="${serviceHash}"}[${rateIntervalSeconds}]))
  by (${property}, le)
)
promql

Parameters:

  • serviceHash: the specific service’s hash

  • property: grouping property (for example, endpoint, version)

  • rateIntervalSeconds: rate calculation window

Service request rate

sum by (${property})
  (rate(service_latency_count{service_hash="${serviceHash}"}[${rateIntervalSeconds}]))
promql

Service error rate

sum by (${property}) (rate(service_latency_count{service_hash="${serviceHash}",error="true"}[${rateIntervalSeconds}]))
/
sum by (${property}) (rate(service_latency_count{service_hash="${serviceHash}"}[${rateIntervalSeconds}]))
promql

Downstream/Upstream dependencies (client services)

The Downstream and Upstream tabs distinguish caller from callee, so they use edge_latency_* instead — for analyzing which services call the current service:

histogram_quantile(0.99,
  sum(rate(edge_latency_bucket{client_service_hash="${serviceHash}"}[${rateIntervalSeconds}]))
  by (service_hash, service_name, le)
)
promql

Trace list page queries

Trace queries are handled through GraphQL rather than Prometheus metrics.

GraphQL queries

Get services list

query GetServices {
  services(
    filter: {
      attributeFilter: {
        eq: { key: "${customerFilterKey}", value: "${customerFilterValue}" }
      }
    }
    durationSecs: ${durationSecs}
    kfSource: "${kfSource}"
    service: { kfType: "${spanTypeFilter}" }
    timestamp: "${endTime}"
  ) {
    name
    distinctLabels
    labels
    hash
    kfType
  }
}
graphql

Get traces

{
  traces(
    durationSecs: ${durationSecs}
    filter: ${buildTracesFilter(...)}
    limit: ${limit}
    pageNum: ${pageNum}
    timestamp: "${endTime}"
    sortField: "${sortBy}"
    sortOrder: ${sortOrder}
  ) {
    traceId
    span {
      spanId
      parentSpanId
      startTimeNs
      endTimeNs
      attributes
      durationNs
      name
      service {
        name
        labels
        hash
        distinctLabels
      }
      statusCode
      method
      endpoint
      rootSpan
    }
    traceMetrics {
      spanCount
      serviceExecTimeNs
    }
  }
}
graphql

Get SLOs

{
  listSLOs {
    id
    name
    type
    service {
      name
      hash
      distinctLabels
      kfType
      labels
    }
    goodEventsSLIQuery
    totalEventsSLIQuery
    matchers
    latencyThreshold
    objective
    description
    timeWindow
    alertUid
    contactPoints
  }
}
graphql

Database-specific queries

Database calls have no service of their own, so they only appear as edge_latency_* edges (service_latency_* excludes them entirely). Queries filter by span_type="db":

Database P99 latency

histogram_quantile(0.99,
  sum(rate(edge_latency_bucket{span_type="db"}[${stepInMs}ms]))
  by (service_hash, service_name, le)
)
promql

Database request count

round(sum by (service_hash, service_name)
  (increase(edge_latency_count{span_type="db"}[${stepInMs}ms]))
)
promql

Common query parameters

Time windows

  • stepInMs: step size in milliseconds for instant queries

  • rateIntervalSeconds: rate interval in seconds format (for example, 5m, 1h)

  • durationSecs: total duration in seconds for the query window

Filters

  • selectedFacetValuesByName: key-value pairs for filtering by service attributes

  • customerFilter: customer-specific filtering

  • spanTypeFilter: filter by span type (db, http, and so on)

Aggregation

  • sumBy: fields to group by in aggregations (typically includes service identifiers)

Usage examples

Example 1: Get P99 latency for a specific service

curl -X POST http://api.example.com/prometheus/api/v1/query \
  -d 'query=histogram_quantile(0.99, sum(rate(service_latency_bucket{service_hash="abc123"}[5m])) by (le))'
bash

Example 2: Get all services via GraphQL

curl -X POST http://api.example.com/graphql \
  -H "Content-Type: application/json" \
  -d '{
    "query": "{ services(durationSecs: 3600, timestamp: \"2024-01-01T00:00:00Z\") { name hash } }"
  }'
bash

Example 3: Calculate service error rate over last hour

curl -X POST http://api.example.com/prometheus/api/v1/query \
  -d 'query=sum by (service_name) (rate(service_latency_count{error="true"}[1h])) / sum by (service_name) (rate(service_latency_count[1h]))'
bash

Notes

  1. All latency values in Prometheus metrics are in milliseconds.

  2. GraphQL timestamps use ISO 8601 format.

  3. service_latency_* already excludes database operations, so no span_type filter is needed on service-level queries; database-specific queries filter edge_latency_* by span_type="db" instead — see Database-specific queries.

  4. APDEX thresholds are typically 0.5s (satisfied) and 1.0s (tolerable).

  5. Rate intervals should be at least 4x the scrape interval for accuracy.