Metrics User Guide

Table of Contents

General approach
A query returns "No data"
A dashboard or query is slow
Choosing a precompute feature
A chart’s values look wrong for the metric’s type
Deciding what’s safe to drop
See also

The other pages in this section document what each screen does; this one is about combining them to answer a specific question — why does this query return nothing, why is this dashboard slow, which of two precompute features to reach for, and what’s actually safe to drop.

General approach

Check the label space before you write the query. If you’re not sure a filter or group-by will match anything, Cardinality Explorer answers that without running a query against the data itself.
Prefer a field filter you can verify over one you’re guessing at. The Explorer’s filter value field autocompletes from what’s actually present on the selected metric — if a value you expect doesn’t appear in that list, it doesn’t exist on this metric, and the query will return nothing no matter how the rest of it is built.
Know which precompute feature you’re reaching for before you build it. Derived Metrics, Metric Shaping Rules, and Recording Rules all produce a new metric on a schedule, but solve different problems — see Choosing a precompute feature below before building any of them.

A query returns "No data"

Confirm the metric itself is reporting: clear every filter and operation, select just the metric, and check whether the raw, unfiltered query returns anything in the current time range. If it doesn’t, the source has stopped emitting this metric — check the collection agent, not the query.
If the unfiltered metric has data, add filters back one at a time rather than all at once, to find which one eliminates the result. A filter value that looks right but doesn’t match anything usually means a label name or value that’s slightly different from what you typed — use the filter’s autocomplete list rather than typing a remembered value from another metric.
If a by grouping is involved, confirm the label you’re grouping by is actually present on this metric — Cardinality Explorer lists every label a specific metric carries, which is faster than trial-and-error in the query builder.
Widen the time range. A metric that reports on an unusual interval, or a source that’s only intermittently active, can look like "no data" in a narrow window and have data just outside it.

A dashboard or query is slow

Open Cardinality Explorer, select the metric the slow query targets, and check Show series count on its labels. A query grouping by a high-cardinality label (a pod name, a request ID, a build hash) touches every one of those series on every refresh — that’s almost always the actual cost.
If the query legitimately needs to read that many series every time it’s viewed, that’s exactly what Derived Metrics is for — precompute it once per interval instead of on every dashboard load. See Choosing a precompute feature to confirm it’s the right one of the three.
If the high-cardinality label itself is never actually useful to filter or group by — a label the collector added automatically that nobody queries on — reduce the source metric’s cardinality permanently with a metric shaping rule instead of precomputing around it.
Before shaping or dropping anything, check Metric Usage for what currently references the metric, so you know what else the change affects.

Choosing a precompute feature

All three features evaluate a query on a schedule and store the result as a new metric, but they answer different questions:

Derived Metrics: "This expression is expensive or repeated often — cache it." Configured in the UI, additive — the source metric is untouched.
Metric Shaping Rules: "This metric has more series than I’ll ever need — permanently reduce it." Configured in the UI, and by default replaces the source metric going forward; the pre-shaping series cannot be recovered.
Recording Rules: The same idea as either of the above, but configured in custom_values.yaml and applied with a Helm upgrade rather than through the UI — the option to reach for when the rule needs to ship as part of your Kloudfuse configuration rather than be managed by whoever has UI access.

If you’re not sure which one fits, ask whether you need the original series to keep flowing (derived metric, additive) or whether you’re deliberately giving them up for good (shaping rule, destructive) — that distinction decides it in almost every case.

A chart’s values look wrong for the metric’s type

Confirm the metric’s actual type — see Metric Types. A counter charted without rate() or increase() shows an ever-climbing line that’s technically correct but rarely what you meant to see; wrap it in one of those functions first.
If the metric is a histogram and the chart looks flat or nonsensical, confirm you’re aggregating with a histogram-aware function (histogram_quantile() and friends — see Miscellaneous functions) rather than avg or sum directly on the bucket series.
If the metric came through a shaping rule, its output type may differ from the source’s — an aggregation other than sum on a counter produces a gauge, not a counter, and needs _over_time functions instead of rate(). See Output metric types and how to query them.

Deciding what’s safe to drop

Open Metrics Usage and set Status to Unused to list metrics nothing currently references.
Check Last queried on each candidate before dropping it — a metric with zero references but recent ad hoc query activity is still in active use by someone, just not through saved content.
For a metric that does have references, click into it to open the reference drawer and see exactly which dashboards, alerts, SLOs, derived metrics, or shaping rules depend on it before you touch it.
Once you’ve confirmed a metric or label is safe to reduce, use Metrics Cardinality Explorer to see which of its labels are actually driving series count, then act with a metric shaping rule.