Aggregation operators

Aggregation operators combine input series into fewer series. Use by (label, …​) to keep the listed labels as grouping dimensions, or without (label, …​) to drop them.

avg

Averages the values of the input series per group. Every series weighs equally regardless of how it was produced.

Syntax

avg by (<labels>) (<expr>)    (also: avg without (<labels>) (...))
none

Parameters

Parameter Required Description

by (<labels>)

Optional

Keep only the listed labels as grouping dimensions.

without (<labels>)

Optional

Group by every label except the listed ones.

Example

Apply avg to the goroutine gauges of the Kloudfuse query services, grouped by service name.

avg by (app_kubernetes_io_name) (
  go_goroutines{app_kubernetes_io_name=~".*query-service"}
)
Expected output
app_kubernetes_io_name Value

trace-query-service

15.42

logs-query-service

97.5

rum-query-service

11.34

events-query-service

15.5

query-service

313.35

Without by or without, all labels are dropped and a single series is returned.

bottomk

Returns the k input series with the smallest values, keeping their labels. The quiet outliers are often as interesting as the noisy ones.

Syntax

bottomk(<k>, <expr>)
none

Parameters

Parameter Required Description

<k>

Required

How many series to return.

Example

Find the two Kloudfuse services running the fewest goroutines.

bottomk(2, sum by (app_kubernetes_io_name) (
  go_goroutines{app_kubernetes_io_instance="kfuse"}
))
Expected output
app_kubernetes_io_name Value

analytics-service

11

recorder

14

bottomk ranks only series that exist; detect fully missing series with absent.

count_values

Groups the input series by their sample value and counts how many series hold each value, writing the value itself into a new label. Use it for discrete-valued metrics — versions, capacities, states — to see the distribution at a glance.

Syntax

count_values("<label>", <expr>)
none

Parameters

Parameter Required Description

<label>

Required

The label that will carry each distinct value.

Example

Count how many node-capacity series report each distinct CPU capacity value.

count_values("cores", kubernetes_state_node_cpu_capacity)
Expected output
cores Value

96

31

10000

1

48

8

1

4

16

159

High-cardinality float metrics explode into one group per distinct value — reserve count_values for discrete values.

count

Counts how many series the inner expression produced, per group. Useful for fleet-size questions: how many pods, how many targets.

Syntax

count by (<labels>) (<expr>)    (also: count without (<labels>) (...))
none

Parameters

Parameter Required Description

by (<labels>)

Optional

Keep only the listed labels as grouping dimensions.

without (<labels>)

Optional

Group by every label except the listed ones.

Example

Apply count to the goroutine gauges of the Kloudfuse query services, grouped by service name.

count by (app_kubernetes_io_name) (
  go_goroutines{app_kubernetes_io_name=~".*query-service"}
)
Expected output
app_kubernetes_io_name Value

rum-query-service

116

events-query-service

195

trace-query-service

199

query-service

206

logs-query-service

200

Without by or without, all labels are dropped and a single series is returned.

group

Collapses each group to a single series with value 1. Use it to deduplicate label combinations before counting or joining, when the values themselves do not matter.

Syntax

group by (<labels>) (<expr>)    (also: group without (<labels>) (...))
none

Parameters

Parameter Required Description

by (<labels>)

Optional

Keep only the listed labels as grouping dimensions.

without (<labels>)

Optional

Group by every label except the listed ones.

Example

Apply group to the goroutine gauges of the Kloudfuse query services, grouped by service name.

group by (app_kubernetes_io_name) (
  go_goroutines{app_kubernetes_io_name=~".*query-service"}
)
Expected output
app_kubernetes_io_name Value

logs-query-service

1

rum-query-service

1

events-query-service

1

query-service

1

trace-query-service

1

Without by or without, all labels are dropped and a single series is returned.

max

Returns the largest value among the input series per group — the worst-case detector.

Syntax

max by (<labels>) (<expr>)    (also: max without (<labels>) (...))
none

Parameters

Parameter Required Description

by (<labels>)

Optional

Keep only the listed labels as grouping dimensions.

without (<labels>)

Optional

Group by every label except the listed ones.

Example

Apply max to the goroutine gauges of the Kloudfuse query services, grouped by service name.

max by (app_kubernetes_io_name) (
  go_goroutines{app_kubernetes_io_name=~".*query-service"}
)
Expected output
app_kubernetes_io_name Value

events-query-service

21

trace-query-service

21

query-service

3,285

logs-query-service

1,507

rum-query-service

16

Without by or without, all labels are dropped and a single series is returned.

min

Returns the smallest value among the input series per group — the floor of a fleet.

Syntax

min by (<labels>) (<expr>)    (also: min without (<labels>) (...))
none

Parameters

Parameter Required Description

by (<labels>)

Optional

Keep only the listed labels as grouping dimensions.

without (<labels>)

Optional

Group by every label except the listed ones.

Example

Apply min to the goroutine gauges of the Kloudfuse query services, grouped by service name.

min by (app_kubernetes_io_name) (
  go_goroutines{app_kubernetes_io_name=~".*query-service"}
)
Expected output
app_kubernetes_io_name Value

trace-query-service

12

query-service

13

rum-query-service

9

events-query-service

12

logs-query-service

10

Without by or without, all labels are dropped and a single series is returned.

quantile

Computes the given quantile (0 to 1) across the values of the input series, per group. This is a cross-series statistic at one instant — for a quantile of one series over time, use quantile_over_time; for histogram data, use histogram_quantile.

Syntax

quantile(<q>, <expr>)    (also: quantile by (<labels>) (<q>, ...))
none

Parameters

Parameter Required Description

<q>

Required

The quantile, between 0 and 1.

Example

Find the 90th-percentile per-series goroutine count across all Kloudfuse service pods.

quantile(0.9, go_goroutines{app_kubernetes_io_instance="kfuse"})
Expected output
Value

375

Three quantile tools, three scopes: quantile across series, quantile_over_time across time, histogram_quantile across histogram buckets.

stddev

Computes the population standard deviation of the input values per group — a direct measure of imbalance across a fleet.

Syntax

stddev by (<labels>) (<expr>)    (also: stddev without (<labels>) (...))
none

Parameters

Parameter Required Description

by (<labels>)

Optional

Keep only the listed labels as grouping dimensions.

without (<labels>)

Optional

Group by every label except the listed ones.

Example

Apply stddev to the goroutine gauges of the Kloudfuse query services, grouped by service name.

stddev by (app_kubernetes_io_name) (
  go_goroutines{app_kubernetes_io_name=~".*query-service"}
)
Expected output
app_kubernetes_io_name Value

query-service

754.22

rum-query-service

0.9981

events-query-service

1.55

logs-query-service

240.33

trace-query-service

1.39

Without by or without, all labels are dropped and a single series is returned.

stdvar

Computes the population variance of the input values per group — the square of stddev.

Syntax

stdvar by (<labels>) (<expr>)    (also: stdvar without (<labels>) (...))
none

Parameters

Parameter Required Description

by (<labels>)

Optional

Keep only the listed labels as grouping dimensions.

without (<labels>)

Optional

Group by every label except the listed ones.

Example

Apply stdvar to the goroutine gauges of the Kloudfuse query services, grouped by service name.

stdvar by (app_kubernetes_io_name) (
  go_goroutines{app_kubernetes_io_name=~".*query-service"}
)
Expected output
app_kubernetes_io_name Value

logs-query-service

57,671.00

trace-query-service

1.975

query-service

568,808.12

events-query-service

2.404

rum-query-service

1.039

Without by or without, all labels are dropped and a single series is returned.

sum

Adds up the values of all input series, per group when by (…​) is given. The most common aggregation: totals along the dimension you care about, per-stream detail collapsed.

Syntax

sum by (<labels>) (<expr>)    (also: sum without (<labels>) (...))
none

Parameters

Parameter Required Description

by (<labels>)

Optional

Keep only the listed labels as grouping dimensions.

without (<labels>)

Optional

Group by every label except the listed ones.

Example

Apply sum to the goroutine gauges of the Kloudfuse query services, grouped by service name.

sum by (app_kubernetes_io_name) (
  go_goroutines{app_kubernetes_io_name=~".*query-service"}
)
Expected output
app_kubernetes_io_name Value

rum-query-service

1,317

query-service

64,560

trace-query-service

3,066

logs-query-service

19,501

events-query-service

3,022

Without by or without, all labels are dropped and a single series is returned.

topk

Returns the k input series with the largest values, keeping their labels — ranking questions answered directly.

Syntax

topk(<k>, <expr>)
none

Parameters

Parameter Required Description

<k>

Required

How many series to return.

Example

Rank the three Kloudfuse services running the most goroutines.

topk(3, sum by (app_kubernetes_io_name) (
  go_goroutines{app_kubernetes_io_instance="kfuse"}
))
Expected output
app_kubernetes_io_name Value

ingester

100,406

trace-transformer

96,123

query-service

64,244

In range queries the top set is re-evaluated at every step, so membership can change over time.