Segment Merge Troubleshooting

This page assumes you’ve read Segment Merge Architecture and Configure Segment Merge. Start with the controller log: it records, for every run, which segments the controller skipped and why. The other checks use the Pinot controller API through a port forward:

kubectl port-forward -n kfuse svc/pinot-controller 9000:9000

Read the controller log

On every run, scheduled or on demand, the controller logs two lines per table: how many segments it considered and why it skipped the others, then how many merge tasks it created. Search the controller logs for them:

kubectl logs -n kfuse pinot-controller-0 | grep KfMergeTaskGenerator

With more than one controller replica, check each pinot-controller pod: scheduled runs log on the lead controller for the table, and on-demand runs on the controller that received the request.

Segment eligibility for table: kf_metrics_rollup_REALTIME — 490 candidates, 0 eligible, exclusions: {LATEST_IN_PARTITION=3, TOO_LARGE=11, OUTSIDE_MERGE_WINDOW=476}
Generated 0 KfMergeTask tasks for table: kf_metrics_rollup_REALTIME
text

candidates counts the table’s sealed segments, eligible the ones the run could merge, and exclusions the skipped segments by reason. In this example, nearly every segment is older than the scheduled one-day window, so the run merges nothing; an on-demand run with a wider window would. The reasons:

Reason Meaning What to do

LATEST_IN_PARTITION

The latest sealed segment of its partition.

Nothing. It’s never merged.

IN_BUFFER

The segment ended in the last hour.

Nothing. A later run merges it.

TOO_LARGE

The segment is already as large as the target size.

Nothing. It doesn’t need merging.

OUTSIDE_MERGE_WINDOW

The segment is outside the run’s mergeNewerThan or mergeOlderThan window.

Merge older data with an on-demand run and a wider window.

EXPIRING_SOON

The segment’s rows expire within minRemainingLifetime (7 days by default).

For a table with short retention, lower minRemainingLifetime; see Runs create no tasks for a table.

RUNNING_TASK

A running merge, delete, or object storage conversion task is working on the segment.

Wait for the task to finish.

FILTER_MARKER_MISMATCH

An earlier rollup merge split the segment by resolution, and this run has no resolution filter, or one that doesn’t keep the segment’s resolution.

See Rollup merges skip segments or mix resolutions.

MISSING_DOWNLOAD_URL

The segment’s upload to deep storage failed, and the controller hasn’t repaired it yet.

Nothing. The segment becomes eligible once the controller’s retry uploads it.

SIZE_UNKNOWN

The segment’s size isn’t recorded in its metadata.

The segment stays unmerged until its size is recorded.

OBJECTSTORE_CONVERTED

The segment was converted to object storage before conversion recorded its full size.

The segment stays unmerged.

SHOULD_NOT_MERGE

The segment’s metadata excludes it from merging.

Nothing.

The controller also logs these lines when it skips a whole table or group:

  • Skip KfMergeTask for non-REALTIME table or Skip KfMergeTask for upsert/dedup table: segment merge doesn’t run on offline tables, or on tables with upsert or dedup enabled.

  • Skipping KfMergeTask for table: …​ Invalid KfMergeTask row filter config: the table’s resolution filter is invalid, so scheduled runs merge nothing on this table. See Rollup merges skip segments or mix resolutions.

  • Skipped N merge group(s) for table: …​ FAILED/TIMED_OUT 1 or more times: see A merge failed.

Scheduled merges never run

If the controller log has no Segment eligibility line for the table after the scheduled time, check the table group in your custom-values.yaml:

  • schedule is empty. The chart ships with schedule: "", so merges run only on demand until you set a Quartz cron expression. See Enable scheduled merging.

  • The expression doesn’t match when you expect. A Quartz expression starts with seconds, not minutes, and one of the two day fields must be ?. The schedule uses the controller’s time zone, which is normally UTC. See Write a schedule.

  • mergeTask.enabled is false for the table group, which turns off every merge of its tables.

If runs are logged and tasks are generated but never start, check that the minions are running (pinot.minion.enabled: false removes them, and with them every merge), and that merges aren’t paused:

curl "http://localhost:9000/tasks/KfMergeTask/state"

STOPPED means merges are paused; see Pause all merges to resume them.

Runs create no tasks for a table

If the log shows Generated 0 KfMergeTask tasks for a table, read the exclusions in the line before it; see Read the controller log. The common causes:

  • Most segments are OUTSIDE_MERGE_WINDOW. Scheduled runs only consider data from the last day. Merge older data with an on-demand run.

  • Most segments are EXPIRING_SOON. The table’s retention is 7 days or less, and its rows carry an expiry time, so every segment expires within the 7-day minimum remaining lifetime. Set minRemainingLifetime below the table’s retention under mergeTask.extraTaskConfigs, for example minRemainingLifetime: "3d" for 7-day retention.

  • Eligible segments, but no tasks. A group is only merged if it holds at least 2 segments and at least half the target size, and a skipped segment ends the group in front of it. A partition with little data, or with eligible segments scattered between skipped ones, can form no group large enough. This resolves itself as more data arrives.

  • Failed groups. A group whose merge failed is skipped until cleanup; see A merge failed.

An on-demand request returns HTTP 400

No task is generated for table means the request found nothing to merge within its window. It doesn’t mean the whole table is merged. Check, in order:

  1. Merges are still running. Segments that are still being merged are skipped, and the output of a merge can only be merged again once its task is done. Wait until taskstates lists no job in progress, then send the request again.

  2. The window is too narrow. Segments outside mergeNewerThan and mergeOlderThan aren’t considered. Widen mergeNewerThan, set it to "" to remove the limit, or run successive windows.

  3. The skip reasons. Read the exclusions the request logged; see Read the controller log.

Other HTTP 400 messages name the problem:

  • does not have KfMergeTask configured; refusing adhoc generation: the table’s mergeTask.enabled is false, so the controller refuses every merge of it.

  • Invalid KfMergeTask row filter config for table: the request’s filterColumn, filterValues, or filterDropValues is invalid, for example a value listed both to keep and to drop.

A request for a table that doesn’t exist returns HTTP 404 with Failed to find table.

A merge failed

A failed merge leaves its input segments untouched, so no data is lost. Later runs skip the failed group, and log Skipped N merge group(s) …​ FAILED/TIMED_OUT 1 or more times, until you clean up its task records.

  1. Find the failed jobs:

    curl "http://localhost:9000/tasks/KfMergeTask/taskstates"

  2. Check the minion logs for the cause. Two messages have a known fix:

    • No row of the N input segments matches row filter: a rollup merge found no row of a resolution it keeps. Check the resolution settings in Merge the metrics rollup table.

    • CRC of input segment …​ changed since task generation: another task changed an input segment while the merge waited. The merge stopped safely; after cleanup, the next run merges the updated segments.

      A rollup row whose resolution is in neither the kept list nor filterDropValues also fails its task; add the resolution to the right list.

  3. Clean up the task records so a skipped group can be retried:

    curl -X PUT "http://localhost:9000/tasks/KfMergeTask/cleanup"

  4. Run the merge again, on schedule or on demand.

Minions drop running merges

A minion that runs more merges than its CPU limit allows can lose its cluster session and drop its running merges. N concurrent merges use up to N + 3 cores. Lower pinot.minion.mergeTask.numConcurrentTasksPerInstance, or raise the minion’s CPU limit, until N + 3 fits. See Resource use.

Each running merge also needs temporary disk of about three times the target segment size.

To reduce load on the minions without stopping merges, lower tableMaxNumTasks for the table group.

Rollup merges skip segments or mix resolutions

With rollupSplitByResolution: false, an on-demand rollup request that doesn’t include the resolution filter combines rows of different resolutions into one segment as they are, and skips segments that an earlier filtered merge already split: the controller log counts them as FILTER_MARKER_MISMATCH. Set pinot.metrics.mergeTask.rollupSplitByResolution: true, or include the filter in every rollup request. See Merge the metrics rollup table.

Pause all merges

To stop merging while you investigate, pause every merge on every table. Running merges stop gracefully, and queued ones wait until you resume:

curl -X PUT "http://localhost:9000/tasks/KfMergeTask/stop"
curl -X PUT "http://localhost:9000/tasks/KfMergeTask/resume"

Stopping or aborting a merge is always safe: a merge either replaces its inputs completely, or leaves them untouched.