Aggregation operators

Vector aggregations combine the series produced by a range aggregation into fewer series, exactly as in PromQL. Use by (label, …​) to keep the listed labels as grouping dimensions, or without (label, …​) to drop them.

avg

Averages the values of the input series, per group when by (…​) is given. Use it to compare typical per-stream levels across a dimension — for example, the average logging rate of each source.

Syntax

avg by (<labels>) (<metric expression>)
none

Parameters

Parameter Required Description

by (<labels>)

Optional

Keeps only the listed labels as grouping dimensions.

without (<labels>)

Optional

Groups by every label except the listed ones.

Example

Compare the average per-stream logging rate of the ZooKeeper and Kafka clusters.

avg by (source) (rate({source=~"zookeeper|kafka"}[5m]))
Expected output
source Value

kafka

0.09556

zookeeper

0.2436

avg weighs every input series equally, regardless of how many lines each contributed.

bottomk

Returns the k input series with the smallest values, keeping their labels. Use it to find the quiet outliers — services logging suspiciously little are often as interesting as the noisy ones.

Syntax

bottomk(<k>, <metric expression>)
none

Parameters

Parameter Required Description

<k>

Required

How many series to return.

Example

Find the two quietest of these five sources by log volume over the last five minutes.

bottomk(2, sum by (source) (
  count_over_time({source=~"zookeeper|kafka|busybox|filebeat|catalog-service"}[5m])
))
Expected output
source Value

catalog-service

92

filebeat

9

bottomk only ranks series that exist; a service logging nothing at all produces no series — detect that with absent_over_time.

count

Counts how many series the inner expression produced, per group when by (…​) is given. Note the two levels of counting: count_over_time counts log lines within a stream, while count counts the resulting series — useful for questions like how many distinct streams are logging.

Syntax

count by (<labels>) (<metric expression>)
none

Parameters

Parameter Required Description

by (<labels>)

Optional

Keeps only the listed labels as grouping dimensions.

without (<labels>)

Optional

Groups by every label except the listed ones.

Example

Count how many distinct log streams each source produced in the last five minutes — a quick way to see fleet size per component.

count by (source) (count_over_time({source=~"zookeeper|kafka|busybox"}[5m]))
Expected output
source Value

busybox

34

kafka

176

zookeeper

109

A stream is a unique combination of all label values, so pods, hosts, and containers each contribute their own stream.

max

Returns the largest value among the input series, per group when by (…​) is given. A common companion to sum on dashboards: the total tells you how much, the max tells you whether one member dominates.

Syntax

max by (<labels>) (<metric expression>)
none

Parameters

Parameter Required Description

by (<labels>)

Optional

Keeps only the listed labels as grouping dimensions.

without (<labels>)

Optional

Groups by every label except the listed ones.

Example

Find the largest per-stream line count per log level for Grafana over the last five minutes.

max by (level) (count_over_time({source="grafana"}[5m]))
Expected output
level Value

debug

8,020

error

31,974

info

33,570

warn

4

To see which series holds the maximum, use topk(1, …​) — it keeps the series labels.

min

Returns the smallest value among the input series, per group when by (…​) is given. Use it to find the quietest member of a fleet or the lower bound of a metric across streams.

Syntax

min by (<labels>) (<metric expression>)
none

Parameters

Parameter Required Description

by (<labels>)

Optional

Keeps only the listed labels as grouping dimensions.

without (<labels>)

Optional

Groups by every label except the listed ones.

Example

Find the lowest per-stream line count among all Grafana streams in the last five minutes.

min(count_over_time({source="grafana"}[5m]))
Expected output
Value

1

min/max pick one value per group; to rank several series use bottomk/topk.

sort_desc

Sorts the input series by value in descending order — largest first. The natural choice for leaderboard-style panels where the biggest contributors should top the list.

Syntax

sort_desc(<metric expression>)
none

Example

List these sources' five-minute log volumes from loudest to quietest.

sort_desc(sum by (source) (
  count_over_time({source=~"zookeeper|kafka|busybox"}[5m])
))
Expected output
source Value

zookeeper

9,298

kafka

5,758

busybox

371

To limit the list as well as order it, use topk instead.

sort

Sorts the input series by value in ascending order. Sorting affects presentation only — the series and values are unchanged — and applies to instant queries, where results are a flat list.

Syntax

sort(<metric expression>)
none

Example

List these sources' five-minute log volumes from quietest to loudest.

sort(sum by (source) (
  count_over_time({source=~"zookeeper|kafka|busybox"}[5m])
))
Expected output
source Value

busybox

356

kafka

5,635

zookeeper

8,973

Use sort_desc for descending order.

stddev

Computes the population standard deviation of the input series values, per group when by (…​) is given. Use it to quantify imbalance across a fleet — a high deviation in per-stream log counts means a few members are much noisier than the rest.

Syntax

stddev by (<labels>) (<metric expression>)
none

Parameters

Parameter Required Description

by (<labels>)

Optional

Keeps only the listed labels as grouping dimensions.

without (<labels>)

Optional

Groups by every label except the listed ones.

Example

Measure how unevenly log volume is distributed across Grafana’s streams in the last five minutes.

stddev(count_over_time({source="grafana"}[5m]))
Expected output
Value

3,591.81

This aggregates across series at one instant; stddev_over_time aggregates one stream’s samples across time.

stdvar

Computes the population variance of the input series values — the square of stddev — per group when by (…​) is given.

Syntax

stdvar by (<labels>) (<metric expression>)
none

Parameters

Parameter Required Description

by (<labels>)

Optional

Keeps only the listed labels as grouping dimensions.

without (<labels>)

Optional

Groups by every label except the listed ones.

Example

Compute the variance of per-stream log counts across Grafana’s streams in the last five minutes.

stdvar(count_over_time({source="grafana"}[5m]))
Expected output
Value

14,372,873.41

For a value in the same unit as the inputs, use stddev.

sum

Adds the values of all input series into one series, or one series per group with by (…​). sum is the most common wrapper around a range aggregation — it collapses per-stream detail into totals along the dimension you care about.

Syntax

sum by (<labels>) (<metric expression>)    (also: sum without (<labels>) (...))
none

Parameters

Parameter Required Description

by (<labels>)

Optional

Keeps only the listed labels as grouping dimensions.

without (<labels>)

Optional

Groups by every label except the listed ones.

Example

Total Grafana’s log volume per level over the last five minutes, collapsing all per-stream series into four rows.

sum by (level) (count_over_time({source="grafana"}[5m]))
Expected output
level Value

debug

8,339

error

69,542

info

83,775

warn

10

Without by or without, all labels are dropped and a single series is returned.

topk

Returns the k input series with the largest values, keeping their labels. topk answers ranking questions directly — the noisiest sources, the busiest namespaces — without pulling the full series list.

Syntax

topk(<k>, <metric expression>)
none

Parameters

Parameter Required Description

<k>

Required

How many series to return.

Example

Rank the three chattiest of these five sources by log volume over the last five minutes.

topk(3, sum by (source) (
  count_over_time({source=~"zookeeper|kafka|busybox|filebeat|catalog-service"}[5m])
))
Expected output
source Value

busybox

327

kafka

5,340

zookeeper

8,352

In range queries topk is evaluated at every step, so the membership of the top set can change over time.