Configure Segment Merge

This page explains how to enable scheduled merging through Helm and how to run a merge on demand. See Segment Merge Architecture for how merges select and replace segments, and Segment Merge Troubleshooting when merges don’t run as expected.

Prerequisites

Merges run on Pinot minions, which are enabled by default. By default, Kloudfuse runs one minion per node: global.numNodes, or the number of query nodes (global.kfRoles.query.numNodes) when node role isolation is enabled. To run a fixed number of minions instead, set replicaCount in your custom-values.yaml:

pinot:
  minion:
    enabled: true    (1)
    replicaCount: 2  (2)
yaml
1 enabled: false removes the minions, and with them every merge.
2 replicaCount: the number of minions. Unset, or 0, runs one minion per node, or per query node with role isolation.

Each running merge uses up to 4 CPU cores and temporary disk of about three times the target segment size; see Resource use.

Enable scheduled merging

Merging is enabled for every table by default, but runs only on demand: the chart ships with an empty schedule. To merge in the background, set a Quartz cron schedule on the table group in your custom-values.yaml. For example, to merge metrics segments every hour:

pinot:
  metrics:
    mergeTask:
      enabled: true               (1)
      schedule: "0 0 */1 * * ?"   (2)
      tableMaxNumTasks: "10"      (3)
      rollupSplitByResolution: true  (4)
yaml
1 enabled: false turns merging off for the table group, including on-demand runs.
2 schedule: a Quartz cron expression. "" (the default) disables scheduled runs; on-demand runs still work.
3 tableMaxNumTasks: the most merge tasks one run creates for a table. Lower it to reduce minion load.
4 rollupSplitByResolution: the metrics group includes kf_metrics_rollup. true, the default since 4.4.0, makes its merges write each resolution to its own segment and delete the rows of legacy resolutions. See Merge the metrics rollup table before you schedule metrics merges.

Apply the change with a Helm upgrade:

helm upgrade --install kfuse oci://us-east1-docker.pkg.dev/mvp-demo-301906/kfuse-helm/kfuse \
  -n kfuse \
  --version <VERSION> \ (1)
  -f custom-values.yaml
bash
1 Replace <VERSION> with a valid Kloudfuse release value. See Release Notes for the latest release.

The controller picks up a new schedule without a restart. Scheduled runs only merge data from the last day; to merge older data, see Run a merge on demand.

Write a schedule

A Quartz cron expression has six fields, separated by spaces, plus an optional seventh for the year:

Field Values Example in 0 0 */1 * * ?

Seconds

0-59

0: at second 0

Minutes

0-59

0: at minute 0

Hours

0-23

*/1: every hour

Day of month

1-31

*: every day

Month

1-12 or JAN-DEC

*: every month

Day of week

1-7 or SUN-SAT

?: no specific day

matches every value, /n every n-th value, a-b a range, and a,b a list. One of the two day fields must be ?, because you can’t set both. Unlike a Unix cron, the expression starts with seconds. Some examples:

  • 0 0 */1 * * ?: every hour, on the hour.

  • 0 */30 * * * ?: every 30 minutes.

  • 0 0 1-5 * * ?: every hour from 01:00 to 05:00, to merge off-peak.

The schedule uses the controller’s time zone, which is normally UTC. See How scheduled runs execute for what happens at each run.

Table groups

Each block below configures merging for the listed tables:

Helm key Tables Target size

pinot.metrics.mergeTask

kf_metrics, kf_metrics_rollup, kf_metrics_staging

1 GiB

pinot.logs.mergeTask

kf_logs

2 GiB

pinot.logs_views.mergeTask

kf_logs_views

1 GiB

pinot.traces.mergeTask

kf_traces, kf_traces_errors

2 GiB

pinot.events.mergeTask

kf_events

2 GiB

pinot.rum.mergeTask

The kf_rum_* tables

2 GiB

pinot.llm_traces.mergeTask, pinot.llm_evals.mergeTask, pinot.genai_traces.mergeTask, pinot.genai_evaluations.mergeTask

The LLM observability tables

1 GiB
A table whose retention is 7 days or less merges nothing by default when its rows carry an expiry time, because every segment expires within the 7-day minimum remaining lifetime. To merge such a table, set minRemainingLifetime below the table’s retention under mergeTask.extraTaskConfigs, for example minRemainingLifetime: "3d" for 7-day retention.

Run more merges in parallel

To run more merges in parallel on each minion, raise pinot.minion.mergeTask.numConcurrentTasksPerInstance (default 1). Keep the total CPU use, N + 3 cores for N concurrent merges (see Resource use), below the minion’s CPU limit, or the minion can lose its cluster session and drop its running merges.

Run a merge on demand

An on-demand (ad hoc) run merges a table immediately, with settings that apply to that run only. Use it to merge data older than the scheduled window, for example a backlog after you first enable merging.

  1. Forward the Pinot controller port:

    kubectl port-forward -n kfuse svc/pinot-controller 9000:9000

  2. Start a run. The taskConfigs override the table’s merge settings for this run only:

    curl -X POST "http://localhost:9000/tasks/execute" -H "Content-Type: application/json" -d '{
  "taskType": "KfMergeTask",
  "tableName": "kf_metrics",
  "taskConfigs": {
    "mergeNewerThan": "30d",
    "mergeOlderThan": "1d",
    "tableMaxNumTasks": "50"
  }
}'

    • mergeNewerThan: merge only segments with data newer than this period. "" removes the limit.

    • mergeOlderThan: merge only segments older than this period. 1d keeps the run away from the data that scheduled runs cover.

    • tableMaxNumTasks: the most merge tasks this run creates.

  3. Wait for the run to finish: curl "http://localhost:9000/tasks/KfMergeTask/taskstates" lists no job in progress. Then repeat the same request to merge the next batch. Segments that are still being merged are skipped, and the output of a merge can only be merged again once its task is done, so a request sent too early can find nothing to do.

  4. Stop when a request returns HTTP 400 with No task is generated while no merge job is in progress. That means no segment is left to merge within this request’s window, not that the whole table is merged. See A request returns No task is generated.

The table’s mergeTask.enabled must be true; otherwise the controller refuses the request with HTTP 400 and the message does not have KfMergeTask configured; refusing adhoc generation.

Merge the metrics rollup table

Since Kloudfuse 4.4.0, rollup merges split segments by resolution and delete the rows of legacy resolutions by default; see Rollup table resolutions. This is controlled by pinot.metrics.mergeTask.rollupSplitByResolution, which is true by default. The chart turns it into these merge settings on kf_metrics_rollup, for both scheduled and on-demand runs:

  • filterColumn: resolution.

  • filterValues: the resolutions to keep: global.metrics.rollupResolutions (300,1800 by default), followed by global.metrics.legacyRollupIntervalSecs (300 by default) if it isn’t already listed. Rollup rows written before 3.5.3 carry no resolution of their own and are read as the legacy interval, so the merge always keeps it. You can’t set these values by hand.

  • filterDropValues: the resolutions whose rows the merge deletes. By default, the legacy resolutions 600, 3600, and 14400 that the merge doesn’t keep: 600,3600,14400 with the default settings, or 3600,14400 with legacyRollupIntervalSecs: 600. To choose others, set pinot.metrics.mergeTask.rollupExtraTaskConfigs.filterDropValues to a comma-separated list, which replaces the default. To delete nothing, set it to "". The Helm upgrade fails if the list includes a kept resolution, the legacy interval included.

Rows of a resolution in filterDropValues are deleted permanently. If your deployment wrote rollups at 600 seconds, set global.metrics.legacyRollupIntervalSecs: 600 before any rollup merge runs. This applies if you set global.metrics.legacyRollupIntervalSecs: 600 when you upgraded to 3.5.3, or set the global.metrics.rollupIntervalSecs key, removed in 4.4.0, to 600. Otherwise the legacy interval is 300, 600 is in the default drop list, and the merge deletes your 600-second rollup history. The chart can’t detect this. If you’re not sure, set filterDropValues: "" until you have checked.

A row whose resolution is in neither list, for example 1200, 7200, or 28800 on a deployment that used a 600-second interval, fails its task and leaves the data untouched. To delete those rows, list them in filterDropValues together with the default drops you want to keep deleting, for example "1200,3600,7200,14400,28800".

An on-demand run on the rollup table uses these settings, so it needs no filter in the request:

curl -X POST "http://localhost:9000/tasks/execute" -H "Content-Type: application/json" -d '{
  "taskType": "KfMergeTask",
  "tableName": "kf_metrics_rollup",
  "taskConfigs": {
    "mergeNewerThan": "30d",
    "mergeOlderThan": "1d",
    "tableMaxNumTasks": "10"
  }
}'

With the switch off, a request can still split one run: add "filterColumn": "resolution", "filterValues" listing global.metrics.rollupResolutions followed by global.metrics.legacyRollupIntervalSecs if it isn’t already listed (for example "300,1800" with the default legacy interval of 300, or "300,1800,600" with 600), and optionally "filterDropValues", which must never list a kept resolution or the legacy interval. A merge without the filter skips segments that an earlier filtered merge already split, so once you split segments this way, include the filter in every later rollup request, or turn the switch on.

Monitor and control merges

Use these controller endpoints, with the port forward from Run a merge on demand:

Request Effect

curl "http://localhost:9000/tasks/KfMergeTask/taskstates"

Lists merge jobs and their states.

curl -X PUT "http://localhost:9000/tasks/KfMergeTask/stop"

Pauses all merges, on every table. Running merges stop gracefully, and queued ones wait until you resume.

curl -X PUT "http://localhost:9000/tasks/KfMergeTask/resume"

Resumes paused merges.

curl -X PUT "http://localhost:9000/tasks/KfMergeTask/cleanup"

Removes finished task records. A group of segments whose merge failed is skipped by later runs; clean up to let it retry.

Stopping or aborting a merge is always safe: a merge either replaces its inputs completely, or leaves them untouched.