Retention Architecture
See Implementing Retention Policies for how to configure retention policies; this page covers why the mechanism is built the way it is — specifically, why retention is decided at write time, before data ever reaches Kafka, rather than being applied later as a filter.
Why retention is decided before the Kafka write, not after
A record’s retention policy determines two things that are both cheaper to decide once, up front, than to re-derive repeatedly downstream: which Kafka partitions it’s allowed to land on, and how long it survives once it’s in Pinot. Both the ingester (metrics, events, traces, and RUM) and logs-parser (logs) decide both of these before the record is written to Kafka, using the retention policy already attached to the record by a relabel rule — see Configure Relabel Rules.
global.retentionPolicy in the Helm values is the single source of truth for every stream’s default and custom policies — logs, metrics, events, traces, rum, and logs_views, plus the shared expireRoundUp setting. Not every key in that list goes through the mechanism this page describes, though — see the logs_views exception below.
One topic, one contiguous partition group per policy
Both the ingester and logs-parser split a stream’s Kafka topic into as many equal-sized, contiguous partition groups as there are installed retention policies for that stream — the default policy plus however many custom policies are configured, in the order they were installed. The default policy always occupies group 0 (partitions 0..k-1); the first custom policy occupies the next group (k..2k-1); and so on. Within a record’s own group, a hash of a partition key picks the specific partition, the same way an unpartitioned topic would.
This is why Configure Kafka Partitions requires the topic’s partition count to be evenly divisible by the number of installed policies: a remainder would leave some policies with a larger or smaller group than others, or none at all.
Ingester: metrics, events, traces, and RUM
For every stream the ingester handles — metrics, events, traces, and rum in global.retentionPolicy terms — it attaches a retention policy to a record via a relabel rule, then writes two Kafka message headers: the policy name and the stream type. A Kafka Balancer (RetentionPolicyBalancer) reads those headers back off the message, looks up the policy’s installation index among that stream’s configured policies, and restricts the normal partition balancer to only the partitions in that policy’s group.
Logs-parser: logs
Logs follow the same grouping logic, implemented independently in logs-parser rather than shared with the ingester, because logs reach Kafka through a different write path. A relabel rule sets the reserved __kf_retention label on each parsed record (Logs Stream shows the pattern); logs-parser reads that label, looks up its installation index the same way the ingester does, and computes the destination partition with the same hash-within-a-range approach — close enough to the ingester’s own partitioning that the two stay interchangeable from Pinot’s point of view, without either service calling the other.
One difference: the logs topic reserves additional partitions beyond the policy groups, set by hydrationPartitions and used exclusively for replaying archived logs during hydration — see Hydrate From Archive. Those partitions are never selected for ordinary ingestion traffic.
The expiry column: retention’s other half
Partition grouping controls where a record lands in Kafka; it doesn’t by itself make Pinot forget old data. Both writers also stamp an expiry column on every record: the ingester for metrics, events, traces, and RUM, and logs-parser for logs. The expiry is the record’s timestamp plus its policy’s retention duration, rounded up to the deployment’s expireRoundUp interval for better dictionary compression in Pinot. Pinot’s retention machinery uses that column, together with each table’s configured retention, to age out expired data independent of which partition it came from.
The unit Pinot ages out is a segment, not a row. Pinot deletes a segment once every record in it has expired; it never deletes individual rows from a segment. A record can therefore stay stored past its own expiry until the rest of its segment expires too. Keeping each policy’s records on their own partitions means a segment holds records from only one policy, so a long-retention record never holds back the deletion of short-retention ones.
Realtime to offline: segment handoff
Once data is in Kafka, Pinot realtime servers consume it — each server assigned a subset of the topic’s partitions, the same partitions the ingester or logs-parser just grouped by retention policy — and build in-memory segments. When a realtime segment is complete, Pinot hands it off to become an immutable offline segment, the standard Apache Pinot realtime-to-offline flow; retention policy doesn’t change this handoff, it only ever affects which Kafka partition a record arrived on and how its expiry is computed.
The logs_views exception
Scheduled Views keep their results in their own Pinot table, kf_logs_views, fed from the kf_logs_views_topic Kafka topic. Unlike the other keys in global.retentionPolicy, logs_views doesn’t split that topic into partition groups: there are no custom policies for Scheduled Views, so every row lands in the same partitions.
Retention still works by expiry, the same way as the other streams:
-
Each Scheduled View row carries its own
expirytime: the time it was written plus the Scheduled Views retention, 14 days by default. Pinot stores it as received. See Interval and retention. -
Pinot deletes a completed segment once every row in it has expired, as described in The expiry column.
-
A segment whose rows carry no expiry falls back to time-based retention: the
defaultpolicy inglobal.retentionPolicy.logs_views, or the table’s own retention (14 days) if that isn’t set.
Next steps
-
Implementing Retention Policies — configure default and custom retention policies, Kafka partitions, and the relabel rules that attach a policy to a record
-
How Relabel Rules Work — the ingester-level relabeling mechanism used to attach a retention policy to metrics, events, and traces
-
Log Parsing Configuration — the logs-parser relabeling mechanism used to attach a retention policy to logs
-
Hydrate From Archive — how the logs topic’s reserved hydration partition is used to replay archived logs