K-Lens

K-Lens answers "what’s different about these spans?" It checks every attribute of the spans in a selected region of traffic and compares their value distribution against a baseline (everything else in view), then ranks the attributes by how different they are. This is usually much faster than manually filtering the Traces Explorer one attribute at a time when you don’t yet know which attribute is the culprit.

Select APM, then K-Lens under Traces.

When to reach for K-Lens

K-Lens is for the investigations where you know something is wrong but don’t yet know why — you have a symptom (a latency spike, a cluster of errors, a metric that moved), but filtering the Traces Explorer requires already knowing which attribute to filter on. Reach for it when:

  • A service-level metric (p99 latency, error rate) moved, but the cause could be any of dozens of pods, endpoints, or downstream calls, and checking each by hand would take longer than the investigation deserves.

  • The problem is intermittent or affects only a subset of traffic, so a single trace you happen to open might not be representative of the pattern.

  • You want to confirm or rule out a hypothesis quickly — for example, "is this really isolated to one availability zone?" — rather than build a query to test it.

If you already know which service, endpoint, or attribute is responsible — because an alert told you, or a dashboard already isolated it — go straight to that service’s Service detail page or a filtered view in the Traces Explorer instead. K-Lens earns its keep specifically when the which attribute question is still open.

The span heatmap

K-Lens plots every span in the current time range and filters as a point on a duration-vs-time heatmap: time along the x-axis, span duration along the y-axis. Four summary tiles above the heatmap report the time window, total spans, and the P50, P95, and P99 duration for whatever is currently selected (the full window until you make a selection).

Use the Filter traces bar, the Errors only checkbox, and the Span type drop-down above the heatmap the same way you would in the Traces Explorer to narrow the population before you compare it — see Filtering traces for the available facets.

Select a region to analyze

Click and drag a rectangle over the part of the heatmap you want to investigate — for example, the band of unusually slow spans during a latency spike, or a cluster of points that appears only during a specific time window.

Once you release the drag, the summary tiles update to show the selection's span count and P50/P95/P99 alongside the baseline (everything outside the selection), and two actions become available:

  • Analyze — compare the selection against the baseline attribute by attribute; see Analyze.

  • View spans — open the selected spans as a trace list; see View spans.

You can also use Zoom in to re-render the heatmap to just the selected time range, Clear selected to discard the selection, or Reset map to return to the full window.

Analyze

Selecting Analyze compares the selection against the baseline across every attribute on the span — service, span name, pod, node, availability zone, cloud account, Kubernetes namespace, service version, database system/name, peer service, error status, and more. For each attribute, K-Lens reports what share of the selection’s values differ from what you’d expect given the baseline’s distribution, so the attribute list is effectively ranked by how strongly it explains the selection.

The attribute most over-represented in the selection is usually your root cause candidate — for example, if pod_name shows one or two pods accounting for 80% of the selection while they’re a small fraction of the baseline, those pods are almost certainly where the problem lives. Use the Filter attributes box to jump to a specific attribute, and enable Show absent values to include attributes that are missing entirely from one side of the comparison.

How to read the results

  • Start at the top of the list, not with the attribute you expect. The ranking already accounts for how strongly each attribute explains the selection versus the baseline — an attribute you didn’t think to check (a database name, a Kubernetes node, an availability zone) frequently outranks the one you assumed was responsible.

  • Look for concentration, not just presence. An attribute where the selection’s top value also appears frequently in the baseline isn’t very explanatory — you want a value that’s common in the selection and rare or absent in the baseline. A pod name at 80% of the selection but 2% of the baseline is a strong signal; one at 30% of the selection and 25% of the baseline probably isn’t the cause.

  • Cross-check two or three ranked attributes together, not just the first one. If both pod_name and kube_node point at the same handful of pods/nodes, that’s corroborating evidence, not two separate findings — it usually means the underlying issue is at the node level (resource pressure, a bad node) rather than something about the pods themselves. If they point at unrelated things, treat the top-ranked one as the primary lead and the others as secondary.

  • A result dominated by error=true or a specific span_status_code tells you the selection is mostly failed requests rather than merely slow ones — pivot to reading the error message and stack trace on one of those spans instead of continuing to look for an infrastructure cause.

  • If nothing ranks strongly, the selection may be too broad (spanning multiple unrelated causes) or too narrow (not enough spans to establish a pattern). Reselect a tighter or wider region on the heatmap and analyze again.

Using K-Lens as part of an investigation

K-Lens is most useful as the middle step in a debugging session, not the first or the last — see Debugging a specific problem for the full workflow it fits into. A typical sequence:

  1. Notice the symptom. An alert fires, or the Service List shows a service’s p99 latency has climbed. Say `checkout-service’s p99 latency has roughly doubled over the last hour, but Requests/s and Error % look normal.

  2. Rule out the obvious first. Check the service’s Deployments tab — no recent rollout — and Downstream/Upstream — no single dependency stands out. The slowdown looks spread across the service rather than tied to one deployment or one call it makes.

  3. Open K-Lens and select the slow band. On the duration-vs-time heatmap, filtered to service_name=checkout-service, drag a selection over the elevated band of points from the last hour (the visibly "higher" cluster compared to the rest of the day).

  4. Analyze. The ranked attributes show kube_node accounting for over 70% of the selection on just two nodes, versus under 5% of the baseline — and availability_zone shows the same two nodes are both in one zone. Two corroborating attributes pointing at the same small set of nodes is a strong lead: this looks like a node- or zone-level resource problem, not an application bug.

  5. Confirm with View spans, then pivot to infrastructure. Use View spans to open a handful of the slow requests and check their flame graph for where the time is actually going (a slow span on the affected nodes, not a downstream call). Then check those nodes' resource metrics in Service detail's Infra tab, or directly in Infrastructure, to confirm CPU/memory/disk pressure.

  6. Close the loop. If this is a recurring pattern, put an alert on the service metric that first caught your eye, so the next occurrence pages someone instead of waiting to be noticed on a dashboard.

The point of the exercise is that K-Lens turned "latency is up, for some reason" into "it’s these two nodes in one zone" in a couple of clicks, without requiring you to already suspect nodes or zones as the cause.

View spans

Selecting View spans instead opens the spans that fall inside your selection as a Traces list, using the same view as APM Traces. From there you can switch to any of the other Traces Explorer view modes (Timeseries, Table, Top List, Pie Chart, Stat), or open an individual trace’s flame graph and waterfall for full detail.