Data Scrubbing Architecture
A scrub job deletes matching rows from Pinot by rewriting the sealed segments that hold them, one segment at a time.
Components
A scrub job passes through six components. Zapper coordinates the job; the Pinot minions are the only component that deletes rows.
| Component | Role |
|---|---|
Admin › Scrubbing page |
Where you preview, create, and cancel jobs in the Kloudfuse UI. |
Zapper |
The backend service ( |
Pinot broker |
Runs Zapper’s row counts for the preview, at creation, and for the final recount. |
configdb |
Holds one record per job and table ( |
Pinot controller |
Finds new jobs in configdb and creates one deletion task per segment to rewrite. |
Pinot minion |
Runs each task: rewrites one segment without the matching rows, then records the segment, rows deleted, and any failure in configdb. |
When RBAC is enabled (RBACEnabled, on by default), creating or canceling a job requires the admin role. When audit logs are enabled (EnableAuditLogs, on by default), Zapper writes an audit log entry for every job it creates or cancels.
Tables scrubbed for each stream
A job targets only one stream. These are the Pinot tables it scrubs for each stream.
| Stream | Pinot tables |
|---|---|
Logs |
|
Metrics |
|
APM |
|
Events |
|
RUM |
|
kf_metrics_rollup, kf_traces_errors, and kf_rum_vitals are scrubbed in 4.4.0 and later. Earlier releases leave their rows in place.
|
How a scrub job runs
-
Preview. While you edit the job, Zapper counts the rows that match your filters and time range through the Pinot broker. The preview counts the rows of the stream’s main table:
kf_metricsfor Metrics,kf_tracesfor APM, every RUM table for RUM. -
Create. When you confirm the job, Zapper counts the matching rows of each table of the stream again and writes one record per table with matching rows, in Created status, with that count as the job’s total.
-
Select segments. The Pinot controller looks for newly Created jobs for each table. For that job, it picks the table’s sealed segments that overlap the job’s time window, oldest first, and creates tasks for them.
-
Rewrite segments. A minion will run each task on one segment. It downloads the segment, finds the rows that match the job’s filters and builds a new segment without them. The new segment will replace the original in place. The minion then marks the segment as done.
-
Decide the status. Every 30 seconds, Zapper checks each running job. Once a table’s record has received no progress for 15 minutes, Zapper sets its final status; see How job status is decided.
A task that fails leaves its segment unchanged and unmarked, so it can be retried on a later run.
Which rows a job deletes
A row is deleted when it matches every filter of the job and its timestamp falls inside the job’s time window: after the start, and up to and including the end. Filters match the labels of logs, metrics, and events, and the attributes of spans and RUM events. All filters must match; there is no OR.
If the job has no label filter, it deletes every row of the stream in its time range.
Which segments a job processes
For each table, the controller only processes segments that:
-
are sealed. Rows that are still in a consuming segment, usually the last few minutes to hours of data, are never scrubbed, even when they fall inside the time window;
-
overlap the job’s time window;
-
hold at least one row;
-
aren’t already marked as done;
-
aren’t being merged by a segment merge or converted to object storage. The controller picks them up on a later run.
The controller runs, by default, 1 scrubbing task processing one segment per minute, across all tables. Narrowing the time window reduces the number of processed segments.
How job status is decided
| Status | Meaning |
|---|---|
Created |
Zapper recorded the job. No segment has been processed yet. |
In progress |
Minions have reported progress. |
Failed |
No progress for 15 minutes, and at least one segment task failed. |
Incomplete |
No progress for 15 minutes and no failed task, but fewer rows were deleted than were counted at creation; for example, rows still in a consuming segment. |
Completed |
No progress for 15 minutes, no failed task, and every row counted at creation was deleted. |
Cancelled |
An administrator canceled the job. |
Zapper checks the conditions in the order shown: a failed task makes the job Failed even if rows remain. Before it marks a job Completed, Zapper counts the matching rows once more. If new rows have arrived in the time window, Zapper leaves the job In progress, but the controller only picks up Created jobs.
RUM session replays
Session replay recordings are files in the replay object store bucket, not rows in Pinot. When a RUM job deletes views of sessions that have a replay, Kloudfuse can delete those recordings too:
-
The minion that scrubs a
kf_rum_viewssegment publishes the application and session ID of each deleted view whose session has a replay to thekf_rum_expired_sessions_topicKafka topic. -
Zapper’s RUM deletion consumer reads the topic and deletes every object under
<application_id>/<session_id>/in the replay bucket, on S3, GCS, or Azure Blob Storage.
Both steps are off by default. They run only when
-
view-based session replay retention is enabled on Pinot (
pinot.rum.viewBasedSessionReplayRetention.enabled: true) -
Zapper consumer is enabled (
zapper.config.rumDeletion.enabled: true) on a cluster with RUM and session replay storage configured.
Otherwise, a RUM job deletes the Pinot rows and leaves replay files in the bucket. A failure to publish to Kafka does not fail the scrub task.