PromQL Best Practices

The fastest PromQL queries on Kloudfuse read the fewest series and let pre-aggregated rollup tables answer as much as possible. This page explains how Kloudfuse evaluates PromQL, where the common performance traps are, and how to avoid them.

How Kloudfuse evaluates PromQL

Kloudfuse stores metrics in Apache Pinot and translates each PromQL expression into one or more Pinot SQL queries. Two mechanisms shape the cost of every query:

  • Rollup tables — raw data is pre-aggregated into five resolution tiers (5 m, 10 m, 30 m, 1 h, 4 h). The query engine picks the coarsest tier whose step size fits the request, so a dashboard with a 30-minute step reads far less data than one at 1-minute resolution.

  • Dynamic splits — long-range queries are automatically split into chunks (2 h, 6 h, 12 h, 1 d, 7 d, …) and executed in parallel, newest-first, to improve cache locality.

Both mechanisms work without any query changes. The practices below ensure your queries are shaped in a way that lets these optimisations apply cleanly.

Prefer equality matchers over regex

Every label matcher in a selector determines how many series the rest of the expression has to process. Matcher types have very different costs:

Matcher How Kloudfuse resolves it Cost

label="value" (equality)

Direct columnar lookup

Cheapest

label=~"a|b|c" (simple alternation)

Expanded to OR’d equality checks

Cheap

label=~".+" (non-empty)

Converted to an EXISTS check

Cheap

label=~"complex.*pattern" (general regex)

RE2 regex applied to every label value

Expensive

label=~".*" (match-all)

Dropped — equivalent to no filter, scans everything

Most expensive

Use equality matchers whenever you know the exact value. When you need to match several known values, an alternation regex (=~"prod\|staging") is nearly as fast as equality because it is rewritten to IN checks internally. Reserve general regex for cases where enumeration is genuinely impractical.

# Fast — equality lookup
up{job="prometheus", env="prod"}

# Fast — alternation rewritten to IN checks
up{job=~"prometheus|node-exporter"}

# Slow — RE2 scan across all job label values
up{job=~"prom.*"}

Control cardinality before it controls you

Kloudfuse enforces a default series limit of 5,000 results per query. When a query returns more distinct label combinations than the limit, results are silently truncated and a warning is returned:

query results hit series cardinality limit of 5000 and were truncated.
Consider adding filter or space aggregations to the query.

The fix is always the same: narrow the selector or aggregate earlier.

Narrow the selector by adding label matchers that exclude series you do not need:

# Before: returns every container in the cluster
container_cpu_usage_seconds_total

# After: scoped to one namespace
container_cpu_usage_seconds_total{namespace="payments"}

Aggregate earlier with sum, avg, or count and a by clause so the engine never materialises the full per-series result:

sum by (namespace) (rate(container_cpu_usage_seconds_total[5m]))

If a metric is inherently high-cardinality — for example, per-request latency histograms broken down by endpoint and user — consider using Metric Shaping to pre-aggregate it into a lower-cardinality recording rule before querying it.

Aggregate with by, not without

When you aggregate with a by clause Kloudfuse can push the aggregation into the storage layer and avoid streaming raw series to the query engine. Using without disables this pushdown and forces the engine to do the work in memory:

# Pushdown-eligible — specify the labels you want to keep
sum by (namespace, pod) (rate(http_requests_total[5m]))

# Not pushdown-eligible — specify the labels you want to drop
sum without (instance, __replica__) (rate(http_requests_total[5m]))

Prefer by whenever you know which labels matter for the result. The without form is useful when you genuinely do not know which labels are present, but that uncertainty comes with a measurable cost.

Size rate windows to the query step

The range window in rate(), increase(), and the *_over_time functions determines how far back each evaluation point looks. A common mistake is setting it much larger than the step, which causes adjacent evaluation points to overlap heavily and read the same data repeatedly.

The Prometheus convention — and Kloudfuse’s metric shaping default — is to use a rate window of roughly 4× the evaluation step:

Step Rate window

15 s

1 m

1 m

4 m or 5 m

5 m

20 m or [5m] on a rollup table

1 h

4 h

 
# Appropriate for a 1-minute dashboard step
rate(http_requests_total{job="api"}[5m])

# Oversized for a 1-minute step — each point looks back 1 hour
# and re-reads 60× the data it needs
rate(http_requests_total{job="api"}[1h])

For counters, prefer rate over increase in dashboards. Both use the same underlying rollup data, but rate produces a per-second value that stays comparable when the step changes.

Let automatic rollups answer long-range queries

Kloudfuse pre-aggregates raw metrics into rollup tiers at 5 m, 10 m, 30 m, 1 h, and 4 h. The query engine selects the coarsest rollup that still satisfies the requested step. No query changes are required — the selection is automatic.

To take full advantage of this:

  • Set the step to match the granularity you actually need. A 24-hour dashboard panel rarely needs 1-minute resolution; a 30-minute step hits the 30 m rollup table and reads 48 data points instead of 1,440.

  • Keep range windows consistent with the rollup tier. A [5m] window on a query with a 1 h step is evaluated against the 1 h rollup table, where each stored value already represents 1 h of data. The window will contain at most one sample, which is usually not what the query intends. Align the window to the tier: use [1h] or [2h] when the step is 1 h.

  • Avoid sub-minute steps for historical queries. Raw data older than the rollup retention period is only available in aggregated form. Requesting a 15-second step on a 7-day range cannot be satisfied by rollups and forces a full raw-data scan.

Bound the step and time range

Kloudfuse enforces a hard limit of 11,000 data points per timeseries per query. A request that would exceed this returns an HTTP 400 error:

exceeded maximum resolution of 11000 points per timeseries

The formula is (end − start) / step > 11000. For a 7-day range the minimum step is:

7 days = 604,800 seconds → minimum step = 604,800 / 11,000 ≈ 55 seconds

In practice, set the step to at least 1 minute for hour-scale ranges and at least 5 minutes for day-scale ranges. For dashboards, link the step to the panel width — a 1,000-pixel-wide panel does not benefit from more than 1,000 data points.

The 50,000,000 sample limit applies across all series combined. A query that matches 10,000 series over 5,000 steps would require 50,000,000 samples and will be rejected. Reducing either the series count or the step count keeps you within the limit.

Be deliberate about subquery cost

A subquery (func(<expr>[<range>:<step>]) evaluates its inner expression at every <step> interval across <range>. The number of inner evaluations is range / step, and each evaluation is itself a full query:

# Inner expression evaluated every 1 minute over 1 hour = 60 evaluations
max_over_time(rate(http_requests_total[5m])[1h:1m])

# Inner expression evaluated every 5 minutes over 1 hour = 12 evaluations
max_over_time(rate(http_requests_total[5m])[1h:5m])

When the step is omitted, Kloudfuse uses a default of 1 minute. Always specify an explicit step that matches the granularity you need to avoid inadvertently multiplying the query cost by 60 or more.

Subquery nesting is limited to four levels. Deeply nested subqueries that wrap histogram_quantile or multiple aggregation layers count toward this limit.

Pre-aggregate high-cardinality metrics with Metric Shaping

When a metric is inherently high-cardinality — user IDs, endpoints, request IDs — no selector trick will bring it below the series limit. The right answer is Metric Shaping: a Kloudfuse feature that generates recording rules which pre-aggregate the metric at ingest time.

Metric Shaping rules run at a configurable interval and produce a new metric with a lower-cardinality label set. The generated metric name follows the pattern kfuse_metric_shaping:<table>:<rule_id>:<interval>. Dashboard queries then target the shaped metric instead of the raw one.

Use Metric Shaping when:

  • A metric consistently hits the cardinality warning even after adding matchers.

  • The same high-cardinality aggregation appears in multiple dashboards or alert rules.

  • A percentile (p95, p99) over a high-cardinality histogram is too slow to evaluate at query time.