Data Scrubbing Known Issues
- Scrub jobs stop after the configdb connection closes
- Data in consuming segments isn’t scrubbed
- Ended jobs are never resumed
- Each table runs one job at a time
- A job can end Failed after its retries succeeded
- The job total differs from the preview
- Large windows are slow
- Filters are limited to label matches
- The job list shows recent jobs only
- Data outside Pinot isn’t scrubbed by default
Each issue lists the releases it affects, if any, and a workaround.
Scrub jobs stop after the configdb connection closes
Affects: releases before 4.2.2.
The Pinot controller and minions held a single, long-lived connection to the configdb. When PostgreSQL, or a proxy or firewall in between, closed an idle connection, it was never reopened: the controller logs Error fetching deletion job from PostgreSQL with Caused by: org.postgresql.util.PSQLException: This connection has been closed. at every run, and no scrub job is processed. New jobs stay Created, then end Incomplete without deleting anything.
Workaround: restart the controllers and minions, which opens new connections, then create the job again:
kubectl rollout restart statefulset pinot-controller pinot-minion -n kfuse
The connection can close again after the restart. Release 4.2.2 and later use a connection pool that replaces closed connections.
Data in consuming segments isn’t scrubbed
A job only rewrites sealed segments. Rows that are still in a consuming segment when the job runs, usually the most recent few minutes to hours of data, are not deleted, but they are counted in the job’s total, so the job ends Incomplete.
Workaround: wait until the data has sealed before you scrub it, or create a second job for the same window later. See Wait until the data is sealed.
Ended jobs are never resumed
Once Zapper sets a job’s final status, the controller never processes it again. This applies to Incomplete and Failed jobs, and to a job that stays In progress because new matching rows were found after it deleted its total. See A job stays In progress.
Workaround: create a new job with the same filters and window. It deletes only the matching rows that remain.
Each table runs one job at a time
Jobs aren’t queued. Each Pinot table works on one job at a time, but the minions record their progress against every Created job of that table. A second job created while the first is still running can absorb the first job’s progress and end Completed early, before its own rows were deleted.
Workaround: start a job for a stream only when the previous job for that stream has reached its final status. Jobs for different streams don’t affect each other.
A job can end Failed after its retries succeeded
A job ends Failed if any segment task failed during the job, even when the retry of that task then succeeded and every matching row was deleted.
Workaround: check the job’s Progress column and the stream’s page for remaining data. See A job ends Failed.
The job total differs from the preview
The preview counts the rows of the stream’s main table, while the job’s total counts every table of the stream. In 4.4.0 and later, the total of a Metrics job includes the rows of kf_metrics_rollup, and that of an APM job the rows of kf_traces_errors. The total is usually larger than the preview’s count, and the progress percentage is based on the total.
Large windows are slow
By default, scrubbing processes about one segment per minute in total, across all tables, and every sealed segment that overlaps the window is rewritten, even if few of its rows match. A job over weeks of a busy table can take many hours.
Workaround: narrow the window, or split it into consecutive jobs. See Keep the time window narrow and Speed up large jobs carefully.
Filters are limited to label matches
The Scrubbing page offers =, !=, and =~ filters on labels, or on attributes for APM and RUM, and every filter must match. There is no OR, no negated regular expression, and no full-text search of log messages. For RUM, Zapper only accepts = on the application (application.id).
Workaround: to delete the union of two conditions, create one job for each, one after the other.
The job list shows recent jobs only
The Scrubbing page lists the 100 most recent job records. A job has one record for each Pinot table of its stream, up to six for RUM, so the list can show fewer than 100 jobs, and older jobs drop off it. Record the names of the jobs you run, for example in the ticket that requested them, if you need to keep track of them longer.
Data outside Pinot isn’t scrubbed by default
A scrub job only deletes rows from Pinot tables. It doesn’t change archives in the object store, and only deletes RUM session replay files when replay deletion is enabled; see RUM session replays.