Pinot Graceful Termination

Table of Contents

Disable Pinot Graceful Termination
Enable Pinot Graceful Termination
Settings
Heap Usage During Shutdown
Termination Time

Starting in release 4.3.0, Pinot realtime servers can commit in-progress (consuming) segments that have made enough progress when a pod stops gracefully — during a rolling upgrade, a restart, or a node drain. On restart, the server resumes from the committed data instead of re-reading it from the ingest stream: no re-ingestion of already-consumed data, and no restart-induced catch-up pressure on ingestion delay.

Graceful termination applies only to Pinot realtime servers (pinot-server-realtime). Starting in release 4.4.0, it is enabled by default for the main pinot instance. In release 4.3.0, and for isolated Pinot instances such as pinot-logs or pinot-metrics in any release, it is disabled by default.

Disable Pinot Graceful Termination

To disable graceful termination, add the following to your Helm values file:

pinot:
  server:
    realtime:
      forceCommitOnShutdown:
        enabled: false

yaml

Then apply the change:

helm upgrade --install kfuse oci://us-east1-docker.pkg.dev/mvp-demo-301906/kfuse-helm/kfuse \
  -n kfuse \
  --version <VERSION> \ (1)
  -f custom-values.yaml

bash

1	Replace `<VERSION>` with a valid Kloudfuse release value. See Release Notes for the latest release.

When graceful termination is disabled, a stopping realtime server does not commit its consuming segments, and on restart it re-reads that data from the ingest stream.

Enable Pinot Graceful Termination

To enable graceful termination in release 4.3.0, add the following to your Helm values file:

pinot:
  server:
    realtime:
      forceCommitOnShutdown:
        enabled: true

yaml

If your cluster uses stream isolation, each isolated Pinot instance has its own settings and is disabled by default in every release. Add the same block under each isolated instance that you want to enable, for example pinot-logs or pinot-metrics:

pinot-logs:
  server:
    realtime:
      forceCommitOnShutdown:
        enabled: true

yaml

Then apply the change:

helm upgrade --install kfuse oci://us-east1-docker.pkg.dev/mvp-demo-301906/kfuse-helm/kfuse \
  -n kfuse \
  --version <VERSION> \ (1)
  -f custom-values.yaml

bash

1	Replace `<VERSION>` with a valid Kloudfuse release value. See Release Notes for the latest release.

Settings

The forceCommitOnShutdown block accepts the following settings:

Setting Default Description

Setting	Default	Description
`enabled`	`true` (4.4.0 and later), `false` (4.3.0)	Commit consuming segments when the realtime server stops gracefully. Isolated Pinot instances default to `false`.
`minRowsPercent`	`25`	Commit only segments that have reached at least this percentage of their target row count. Smaller segments are not committed and are re-read from the ingest stream on restart, which avoids creating many tiny segments.
`timeoutMs`	`300000`	The maximum time, in milliseconds, that the server waits for the commits to complete before it continues shutting down. The default is 5 minutes.

enabled

true (4.4.0 and later), false (4.3.0)

Commit consuming segments when the realtime server stops gracefully. Isolated Pinot instances default to false.

minRowsPercent

25

Commit only segments that have reached at least this percentage of their target row count. Smaller segments are not committed and are re-read from the ingest stream on restart, which avoids creating many tiny segments.

timeoutMs

300000

The maximum time, in milliseconds, that the server waits for the commits to complete before it continues shutting down. The default is 5 minutes.

Heap Usage During Shutdown

When graceful termination is enabled, a realtime server builds all eligible consuming segments at the same time during shutdown. A server that hosts many consuming partitions can therefore need significantly more heap memory while it stops. If the server runs out of memory during the commit, it stops ungracefully and re-reads the data from the ingest stream on restart, which is the outcome this feature is designed to avoid.

Before you enable graceful termination on a large deployment, make sure the realtime servers have heap headroom, and consider raising timeoutMs so that all segments can finish building in time. Contact Kloudfuse support for sizing guidance.

Termination Time

Starting in release 4.3.0, Pinot realtime server pods allow up to 21.5 minutes to terminate. When graceful termination is enabled, the commit timeout (timeoutMs, 5 minutes by default) is added on top, for a total of 26.5 minutes with the default settings. The commit runs before the server drains in-flight queries. See Longer Pinot Server Pod Termination.

These windows are ceilings, not typical durations. A server normally finishes committing and draining well before the timeout and exits on its own. In the worst case — if it does not finish in time — the pod uses the full window and does not serve queries during that time, until it is terminated at the timeout.