How Metrics Rollup Works

Table of Contents

Overview
Benefits
Drawbacks

Overview

To improve query performance and reduce loading times, Kloudfuse computes and aggregates metrics data at multiple resolution granularities directly from the data stream. Depending on the query’s step size and time range, Kloudfuse automatically selects the most appropriate resolution — using rolled up data for longer time ranges and raw data for shorter ones.

Starting with version 3.5.3, metrics rollup is enabled by default.
Starting with version 4.4.0, metrics are rolled up at two fixed resolutions: 5 minutes and 30 minutes. The query service automatically picks the best resolution based on the query’s step size and time range.
To disable rollup or set rollup retention, see Configure metrics rollup.

To learn how Kloudfuse creates and uses rolled up metrics, see Metrics rollup processes.
For an in-depth look at the various components of our implementation, see Elements of metrics rollup.

For the more general discussion of this feature, read these sections:

Benefits
Drawbacks

Benefits

The primary benefit of this approach is a reduced I/O cost, as Kloudfuse samples aggregate metrics instead of raw values. These pre-calculated aggregates improve query performance, and quicker calculation means dashboards and graphs load faster. Additionally, it is relatively inexpensive to increase retention times for these aggregated metrics.

Consider the number of metrics that your system processes regularly. The following image plots a selection of metrics that a Kloudfuse cluster monitors:

A selection of metrics monitored by a Kloudfuse cluster

Selected metrics from a Kloudfuse cluster

In situations where the raw data stream has intervals of 15 or 30 seconds, compare the number of records that each query processes with the number of records when using pre-aggregated metrics data at the 5 minute and 30 minute rollup resolutions. When the data stream is at 15 or 30 seconds, using rolled up (pre-aggregated) metrics at 5 minutes reduces the number of records retrieved by a factor of 20 or 10, respectively. At 30 minutes, the number of records retrieved drops by a factor of 120 or 60, respectively.

Disk access counts for raw metrics vs. rolled up metrics
Query Duration	Number of stored records
	1 metric				200 metrics
	Raw data		Rolled up data		Raw data		Rolled up data
	15s	30s	5 min	30 min	15s	30s	5 min	30 min
6h	1,440	720	72	12	288,000	144,000	14,400	2,400
2d	11,520	5,760	576	96	2,304 K	1,152 K	115,200	19,200
7d	40,320	20,160	2,016	336	8,064 K	4,032 K	403,200	67,200
2w	80,640	40,320	4,032	672	16,128 K	8,064 K	806,400	134,400
1mo=30d	172,800	86,400	8,640	1,440	34,560 K	17,280 K	1,728 K	288,000
1y=365.25d	2,104 K	1,052 K	105,192	17,532	420,768 K	210,384 K	21,038 K	3,506 K

Drawbacks

In addition to some storage overhead that may potentially lead to adding disks when you plan to retain large amounts of historical data, metrics rollup uses more in-memory resources than working with raw metrics alone.