Data Scrubbing Troubleshooting

A job’s final status is set 15 minutes after its last progress, so wait that long before you troubleshoot it.

Where to look

Search these in the Logs Explorer, over the time since the job was created:

Component Logs Explorer query What it shows

Zapper

kube_namespace="kfuse" and source="zapper" and "Successfully inserted job into database"

The job was created. Replace the phrase with No items to delete, ignoring to find tables without matching rows, or Job is not progressing to find when Zapper set a final status.

Pinot controller

kube_namespace="kfuse" and source="pinot-controller" and "deletion job"

Found deletion job with ID: <job> in 'created' status on every run, which names the job the controller is working on, and any error reading the configdb. Replace the phrase with tasks configs for table to see how many tasks each run created.

Pinot minions

kube_namespace="kfuse" and source="pinot-minion" and "Updated job for"

Updated job for <table> : segments deleted: …, items deleted: …, failures: … for each progress report. Replace the phrase with KfDelete to see task errors.

To see a job’s progress, check its Progress column on the Scrubbing page; see Track a scrub job.

To check scrub tasks in Pinot, forward the controller port, then list the task states:

kubectl port-forward -n kfuse svc/pinot-controller 9000:9000
curl "http://localhost:9000/tasks/KfDeleteTask/taskstates"

Creating a job fails

The error message names the problem:

  • Admin access required for mutation operations: your role isn’t admin. Ask an administrator to create the job.

  • job name can only contain alphanumeric characters, hyphens, and underscores: rename the job. The Scrubbing page also requires 3 to 50 characters.

  • failed to count items in <table>: the count query on that Pinot table failed or timed out after 60 seconds. Check that the Pinot brokers and servers are healthy, narrow the time window, or replace a broad =~ filter with exact ones.

  • A message about a duplicate key: a job with the same name already exists for this stream. Use another name.

The preview fails or shows no data

  • The preview fails or never loads. The count queries time out after 60 seconds. Narrow the time window, or use exact filters instead of regular expressions.

  • No data matches. Check the filters on the stream’s page with the same time range: a label name or value that differs in case, or a filter on a facet rather than a label, matches nothing. Confirm scrubbing stays disabled until the preview finds matching data.

A job stays Created

A job in Created status has received no progress from the minions. If it stays that way for 15 minutes, Zapper ends it, usually as Incomplete. Check, in order:

  1. The controller can read the configdb. Search the controller logs with the query from Where to look. If they show Error fetching deletion job from PostgreSQL at every run, followed by Caused by: org.postgresql.util.PSQLException: This connection has been closed., its connection to the configdb was closed and is never reopened, so no scrub job runs on that controller. See Scrub jobs stop after the configdb connection closes.

  2. Another job holds the table. Each table works on one job at a time. If the controller logs name a different job in Found deletion job with ID, wait for that job to reach its final status, then create this one again. See Run one job per stream at a time.

  3. The controller creates tasks. Finished generating 0 tasks configs in the controller logs at every run means no sealed segment overlaps the time window with rows left to process, typically because all of the window’s data is still in consuming segments. Wait until it seals, then create the job again.

  4. The minions run tasks. Check that the pinot-minion pods are running, and that scrub tasks aren’t paused:

    curl "http://localhost:9000/tasks/KfDeleteTask/state"

    STOPPED means scrub tasks are paused; see Pause and resume scrub tasks.

A job ends Incomplete

Incomplete means the job made no progress for 15 minutes, no segment task failed, and rows counted at creation remained. The common causes:

  • The window included consuming data. Rows that were still in a consuming segment are counted in the job’s total, but never deleted. This is the usual cause when the job’s window reaches into the last few hours, and the job deleted most of its rows. Once that data has sealed, create a new job with the same filters and window. Its total only counts the rows that remain, and although it processes every segment in the window again, segments with no matching row left are kept unchanged.

  • The job never started, or stopped making progress. With few or no rows deleted, the controller or minions didn’t run its tasks; check the causes in A job stays Created, then create a new job.

  • One segment took longer than 15 minutes. Progress is recorded only for finished segments, so a segment whose rewrite takes longer than the 15-minute limit ends the job before it finishes. Create a new job for the same window.

An Incomplete job is never resumed. Cancel it if you don’t want it listed as open, and create a new job for what it left.

A job ends Failed

Failed means at least one segment task failed during the job. A failed task leaves its segment unchanged and is retried at the next run, so the job may still have deleted everything: check the job’s Progress column and the stream’s page for remaining data.

To find the cause, search the minion logs in the Logs Explorer for the task error, with kube_namespace="kfuse" and source="pinot-minion" and "KfDelete". A message that has a known meaning:

  • filters on a field but table <table> has no KV index to evaluate it on segment <segment>; refusing to record a no-op purge as success: the segment can’t evaluate the job’s label filter, so the task refuses to report it as scrubbed. The task fails on every retry; report it to Kloudfuse support with the table and segment names.

If the remaining data is still there once the cause is fixed, create a new job for it.

A job stays In progress

When the deleted count reaches the total, Zapper counts the matching rows again before it marks the job Completed. If it still finds some, for example data that arrived late with timestamps inside the window, it leaves the job In progress. The controller doesn’t process a job in that status, so it stays there, at or near 100%. Cancel the job, then create a new one with the same filters and window to delete the rows that remain.

Data remains after a job completes

  • The data is in another stream. A job scrubs one stream. A value that appears in log lines can also be in span attributes or events; scrub each stream.

  • The data is at the window’s edge. The window excludes its start time. Widen the window slightly.

  • The data was ingested after the job. Rows ingested after the job was created, for example from a source that is still sending the value, aren’t covered. Stop the source; see Prevent the next scrub.

  • RUM session replays still play. Replay files are only deleted when RUM replay deletion is enabled; see RUM session replays. Zapper exposes zapper_rum_deletion_* metrics, including zapper_rum_deletion_errors_total by error type, for the consumer that deletes them.

  • The data is in an archive. Scrubbing doesn’t change archives in your object store; see Archiving Logs and Hydration.

Pause and resume scrub tasks

To stop all scrub tasks, on every table, while you investigate, use the port forward from Where to look:

curl -X PUT "http://localhost:9000/tasks/KfDeleteTask/stop"
curl -X PUT "http://localhost:9000/tasks/KfDeleteTask/resume"

A segment that a minion is already rewriting is still replaced. A job that is paused for more than 15 minutes ends Incomplete; create it again after you resume. To stop one job instead, cancel it; see Cancel a scrub job.