Data Scrubbing Best Practices

A scrub job can’t be undone, and it only deletes data in sealed segments.

Read the preview before you confirm

The preview runs the same filters and time range as the job. Check the counts are in the range you expect, and open the matching data on the stream’s page (the Open in link, for example Open in Logs; Metrics has no such link) to look at more than the sample. A count much larger than you expect usually means a filter is broader than intended, or missing.

Never confirm a job with no label filter unless you mean to delete every row of the stream in the time range.

Prefer exact filters to regular expressions

An = filter on a specific label value deletes only what you name. A =~ regular expression can match more values than you intend, and != matches everything else that has the label. When you need a regular expression, look at the matching values on the stream’s page first.

Combine filters to narrow a job: all filters must match, so adding a filter, for example kube_namespace next to kube_service, can only reduce what the job deletes.

Keep the time window narrow

Every sealed segment that overlaps the window is downloaded and rewritten, even if it holds only a few matching rows. A window of a few hours finishes in minutes; a window of several weeks on a busy table can take many hours.

To delete data from a long period, split it into consecutive jobs rather than one large job.

Wait until the data is sealed

Rows that are still in a consuming segment aren’t scrubbed. When the data you need to delete is recent, wait until its segments have sealed, or create a second job for the same window later.

Run one job per stream at a time

Start a new job for a stream only once the previous job for that stream has reached its final status. The minions record progress against every Created job of a table, so a second job can take over the first job’s progress and end Completed before its own rows are deleted. Jobs for different streams don’t share progress, but they do share the scrub task limit, so they run one segment at a time between them by default.

Audit Trail will use Job Names

Use unique names that explain why data was deleted, for example a ticket or request ID: gdpr-req-4711 or INC-2048-leaked-token. Don’t put sensitive values in the name.

Verify the result

When the job is Completed, query the same filters and time range on the stream’s page. Then check the other streams: a secret written to a log line can appear in span attributes or event payloads. Removing data from Kloudfuse doesn’t remove it from the source or from archives.

Prevent the next scrub

Scrubbing removes data after the fact. If the data should never have been ingested, filter it out at the source: either mask or drop it at the collector (Keeping Sensitive Data Out of Kloudfuse), or with log parsing rules (Log Parsing). Otherwise, new data keeps arriving and you need to scrub it again.

Speed up large jobs carefully

Each running task downloads and rebuilds whole segments, which requires additional CPU, memory, and temporary disk on the minion. This can cause competition for resources with segment merges.