Log Parsing Examples

Relates to: A dissect or grok pattern isn’t extracting fields.

Use case: A team onboards a legacy internal proxy that logs pipe-delimited key:value pairs instead of JSON, and wants user, action, and status as separate facets so they can filter failed logins by user in the Explorer. They adapt a tokenizer copied from another team’s comma-delimited source, updating the field names but not the delimiter character.

A source logs pipe-delimited fields, but the tokenizer was written assuming commas:

Against the line user:alice|action:login|status:success, this doesn’t fail — it produces two facets, with the second one silently absorbing everything after the first colon:

There’s no error because dissect doesn’t require a full-line match: it matches as far as it can, and once the , it’s looking for never appears, the rest of the pattern is simply never applied. Fix the delimiter to match the actual line:

Generic pattern — any line structured as repeated key:value pairs joined by one consistent delimiter follows this shape; substitute your own field names and delimiter character:

Relates to: A promoted facet does not show up as a label and Relabel, Transform priority order.

Use case: A team instruments a payments service that logs plain-text request lines, extracts the caller’s IP as a facet with grok so it’s visible per-line, and wants to promote it to a label so dashboards can filter and group by client IP without digging into individual lines. They reach for relabel — the first action-capable function they find in the configuration reference — without realizing it runs in an earlier pipeline stage than Grammar.

Given the line checkout request from 10.1.2.3, this extracts client_ip with grok, then tries to promote it to a client label with relabel:

client_ip is extracted correctly, but no client label is ever created:

No error, no warning. relabel always runs in a fixed stage before Grammar, regardless of where it sits in the config list — from relabel’s point of view, `client_ip simply doesn’t exist yet. Changing the keyword to transform — nothing else — fixes it:

Generic pattern — promoting any Grammar-extracted facet to a label always takes this shape, with transform after the extraction step, never relabel:

Relates to: Scope every function with conditions.

Use case: An SRE team wants to cut ingestion noise from a chatty nginx sidecar’s /healthz liveness probe, which fires every few seconds with no diagnostic value. They write a dropLogLine rule matching the literal text, intending it only for that one source — but since they were only looking at nginx’s logs while writing it, they never add a source scope, and the rule applies to every source in the pipeline.

The intent is to drop noisy nginx health-check lines. This condition matches on message text alone, with no source scoping:

This drops the intended nginx line (GET /healthz from payments-service healthz-checker) — but it also drops an unrelated line from a completely different source that merely happens to mention the word: incident report: customer mentioned healthz endpoint flakiness in ticket #4821. Both come back "Event was dropped by the pipeline".

Adding a #source matcher scopes the condition to only the source it was written for:

The nginx line is still dropped, and the unrelated line (source support-bot) now passes through untouched.

Generic pattern — any condition meant to apply to one source needs an explicit #source matcher alongside the condition that expresses the actual intent; never rely on the content match alone to imply the scope:

Relates to: A wide JSON payload has no facets at all and JSON auto-parsing facets.

Use case: A billing-events service emits JSON events that can carry a lot of optional metadata — line items, discounts, customer attributes. Ordinary events have a dozen or so fields, but a fully-populated enterprise-plan event can carry 60 or more. The team expects Kloudfuse to facet all of it the way it does for their other, narrower JSON sources, and only notices something’s wrong when this specific source turns out to have zero searchable facets in the Explorer, even though ingestion volume looks completely normal.

JSON auto-parsing extracts at most 50 facets from a single log line by default. With nothing else about the payload changed:

Crossing the ceiling doesn’t drop the facets past 50 — it drops all of them. Two fixes, depending on intent:

Generic pattern — either template applies as-is to any JSON source that occasionally exceeds the default ceiling; pick based on whether you need every field or only a known few:

Relates to: Too many configured rules will slow down the pipeline.

Use case: A platform team has been onboarding new tenants onto a shared Kloudfuse deployment for over a year. Every onboarding runbook includes a step to add a relabel rule tagging that tenant’s logs with a tenant_id label, so the rule count has grown by one with every new tenant — unnoticed, past 2,000. Someone eventually notices logs-parser running hotter than it used to and starts investigating, assuming the cause is rising log volume rather than the accumulated rule count.

Deploying 2,000 relabel rules to a running logs-parser — each scoped to a source that never appears, so none of them ever actually fire — raised steady-state CPU roughly 28x over its pre-change baseline, with zero change in log volume. The cost comes entirely from evaluating conditions, not from anything matching. CPU returned to baseline within minutes of rolling the config back.

Unlike the other examples, this one doesn’t show up in a single log line — it’s a sustained-load effect, visible in the platform’s own container_cpu_usage metric. See the troubleshooting guide for the exact queries.

Generic pattern — this is the shape of rule that tends to accumulate one-per-tenant until it’s a performance problem; the fix is consolidating many of these into fewer, more general rules, not any single rule’s logic: