Retention User Guide

This page covers how to size Kloudfuse for the retention policies you configure: how many Kafka partitions a topic needs, how large the Kafka and Pinot volumes need to be, and how custom policies affect memory.

Sizing Kafka for Retention

A topic’s partition count has to satisfy two constraints:

  • It must be divisible by the number of retention policies for its stream (the default policy plus each custom policy).

  • It must have enough partitions for Pinot to parallelize across.

The chart’s install-time check validates these topics:

  • kf_metrics_topic

  • kf_metrics_rollup_topic

  • kf_events_topic

  • kf_traces_topic

  • kf_traces_errors_topic

  • kf_logs_topic

See Configure Kafka Partitions for which stream each topic is validated against.

Determine how many partitions each policy needs based on its expected volume, then multiply by the number of policies.

Example: You have a logs stream with a default policy and two custom policies (prod, staging). Each policy needs 4 partitions of realtime throughput:

global:
  kafkaTopics:
    - name: kf_logs_topic
      partitions: 12   # 4 partitions × (1 default + 2 custom)
      hydrationPartitions: 2 (1)
      replicationFactor: 2
      minInSyncReplicas: 1
yaml
1 hydrationPartitions is sized separately and isn’t divided among policies. See Configure Kafka Partitions for that distinction.

Sizing Kafka PVCs

Kafka’s PVC size doesn’t depend on how long Pinot retains your data, because Kafka is a short-lived buffer in front of Pinot. The chart caps each Kafka partition at 10 GiB (log.retention.bytes: 10737418240).

Kafka PVC size = (total partitions across all topics) × 10 GiB × replicationFactor ÷ number of broker replicas × 1.10 (1)
1 An additional 10% headroom for activity other than buffering.

Example: 6 topics at 1 partition each, plus kf_logs_topic with 12 partitions and 2 hydration partitions, all with replicationFactor: 2, on 3 brokers:

total partitions = 6 + 12 + 2 = 20 partitions
PVC per broker   = 20 partitions × 10 GiB
                 × 2 (1)
                 ÷ 3 brokers
                 × 1.10 (2)
                 ≈ 147 GiB per broker
1 Replication factor
2 Additional headroom

Two things to keep in mind about this formula:

  • It scales with partition count, not with retention. Adding a custom policy raises the Kafka PVC requirement even though it doesn’t change how much data Pinot stores.

  • The 10 GiB is a ceiling on how large a partition is allowed to grow. A low-volume custom policy’s partitions can stay well below 10 GiB in practice.

Sizing Pinot storage

Pinot’s storage grows with retention duration. The correct size for each component depends on ingestion volume:

Component How it is used

realtime server

Peak unflushed segment volume: roughly your highest-throughput stream’s ingestion rate × how long a segment stays open before completion.

offline server

Completed segments served from local disk. If deep storage isn’t configured, this holds your entire retention window for every table; if deep storage is configured, this is a working-set cache and can be smaller.

controller

Holds segment metadata and, without deep storage, a copy of each offline segment.

minion

Scratch space for background tasks (segment generation, purge); not retention-sensitive.

Sizing memory

Logs-parser and Ingester

Both track retention, but they don’t need more memory to track custom policies.

Kafka Brokers

Kafka’s throughput comes from the OS page cache, not from JVM heap. More partitions add some off-heap overhead. Unless you see OutOfMemoryError or sustained GC pressure on the brokers, you shouldn’t need to adjust the broker’s heapOpts.

Pinot Realtime Server

Each partition that is actively being written to keeps a segment open in memory until that segment completes. More partitions means more segments held in memory, and custom policies multiply the partition count, which raises the Java heap required. Higher label and facet cardinality also requires more memory.

Pinot controller and broker

A custom policy increases the total number of segments for the same volume. A segment is committed on either row count or time. When a segment hits either threshold, it is sealed and can no longer be updated:

  • Row count: the segment holds the maximum number of rows.

  • Time: the segment has been open for the maximum amount of time without being sealed.

More policies lower the per-partition row rate, so a segment takes longer to reach the row count threshold, if it reaches it at all. This means segments are more likely to be sealed by the time threshold.

As a result, infrequently used policies produce more segments for the same volume of data. More segments means the Pinot controller and broker need more memory and CPU for all processing. This affects all streams, not just the ones that are infrequently used.

Pinot offline server

Its memory pressure depends on total data volume, the number of queries, and the volume of data those queries read.

Custom Policy Checklist

  • Add the policy under global.retentionPolicy.<stream>.custom, at the end of the list. Configure Additional Retention Policies explains why reordering is destructive.

  • Size the stream’s kafkaTopics entries so partitions is divisible by the stream’s number of retention policies.

  • Recompute the broker PVC size with the new partition count, and resize kafka-kraft.broker.persistence.size (or kafka.broker.persistence.size if you deploy the legacy Kafka chart) before you apply the partition count changes.

  • If the new policy has a longer retention window, reassess the Pinot realtime and offline PVC sizes.

  • If the new policy increases a stream’s partition count, check Pinot realtime server heap and controller and broker segment-count overhead before rolling it out.

  • Add relabel rules that assign records to the new policy. See Configure Relabel Rules.

Next steps