Log Parsing Troubleshooting

This page assumes you’ve read Log Parsing Architecture and Log Parsing Configuration — it’s a set of diagnostic workflows for when the pipeline isn’t doing what the configuration seems to say it should.

Validating the Parser pipeline

Before debugging a particular function, first check that data is reaching logs-parser and passing through it. If it isn’t, it might be a config block that never handles the logs you want.

logs_parser_msgs_read_total

Shows the number of raw messages logs-parser has read from its Kafka input topic. A single message may contain multiple log lines. This metric is a rough indicator of activity and is not an exact log count. Use it to verify that the pod is processing, not to count logs. If the metric remains unchanged, the pod is not retrieving data from Kafka.

logs_parser_events_read_total{source, remapped}

A count of individual log events after unpacking, categorized by source (the originating agent or payload type — datadog, fluent-bit, otlp, and so on) and indicating whether the event was already remapped. To check a source, filter to source="<your-source>" and verify the rate is non-zero before concluding a conditions block or pattern is the problem. If the rate is zero but logs_parser_msgs_read_total is normal, it usually means the source label is incorrect — check the source value the agent sends against what your conditions block expects.

logs_parser_stage_events_processing_failure_total{source, stage}

The number of events that failed processing, identified by the pipeline stage where the failure occurred (for example tokenize) and the source. When volume looks normal, this metric distinguishes between "my rule doesn’t match anything" (silent, no failure recorded) and "my rule is actively erroring" (failures recorded here, specific to a stage).

A few queries to start with:

# Is the pod actually consuming anything?
sum(rate(logs_parser_msgs_read_total[5m]))

# Is the volume of this particular source flowing?
sum by (source) (rate(logs_parser_events_read_total{source="<your-source>"}[5m]))

# At what stage is anything currently failing, and which source is it?
sum by (source, stage) (rate(logs_parser_stage_events_processing_failure_total[5m]))
promql

When the first query returns a flat result, the problem occurs before parsing — the pod, Kafka, or the ingester. If the second query is flat but the first is not, events from the source are not reaching this pipeline as expected — confirm the source label sent by the agent matches what your conditions blocks expect. If the third query shows a non-zero rate for a configured stage, a rule is causing errors, not just failing to match.

Too many configured rules will slow down the pipeline

Every function in kf_parsing_config.config uses CPU to process a log line, even without a match, since its conditions block is still checked. Too many rules (more work per event) or high log volume can overload logs-parser. A configuration that works fine at low volume can be overwhelmed at higher volume, with nothing in the config itself having changed.

  • A sustained rise in container_cpu_usage for the logs-parser container, correlated with a config deploy, is the earliest and most sensitive sign of a problem. For example, adding 2,000 rules scoped to a source that never appears can raise CPU usage 28x, even though log volume stays the same — the extra cost comes from evaluating conditions, not from matching logs.

    max(container_cpu_usage{kube_namespace="<namespace>", kube_container_name="logs-parser"})
    promql

  • Compare the read rate of logs-parser with the corresponding write rate from the ingester:

    sum(rate(ingester_kafka_writer_write_count[5m])) - sum(rate(logs_parser_msgs_read_total[5m]))
    promql

    When the result stays close to zero, logs-parser is keeping up. If it steadily rises, production is outpacing consumption and the Kafka backlog is growing. This is a delayed signal — CPU usage usually rises before this gap appears.

What to do about it

  • Combine conditions wherever possible. Each function adds a per-line evaluation cost — reduce checks by merging functions or adding an early function that excludes irrelevant sources.

  • When a source’s volume increases, re-check its rules against the current line count rather than just the rule logic.

A dissect or grok pattern isn’t extracting fields

  1. Use an exact line from a genuine log event (not retyped) with the tokenizer or pattern in a dissect debugger or grok debugger. The most common cause is an extra space, a different quote character, or a varying delimiter. Detecting this in a debugger is faster than repeated deployments.

  2. Check the function’s conditions block — see Functions and conditions. A correct pattern scoped to the wrong source, or using the wrong matcher, will not cause an error; the function simply won’t fire.

  3. If the function reading the extracted field is a relabel rule, change it to transform. relabel always runs before Grammar, regardless of its position in the config list — it will never see a facet dissect or grok produced. See Relabel, Transform priority order.

  4. If a field’s value contains more text than expected, check the literal delimiter in the tokenizer. A mismatch (wrong punctuation, a missing space) can cause dissect to include text meant for the next field.

A promoted facet does not show up as a label

  1. Ensure the promotion is written as transform, not relabel. relabel runs before Grammar/JSON auto-parsing, even if it’s not first in the config list, so a relabel rule using facet_to_label_map cannot see extracted facets — only transform runs late enough to do this. See Relabel, Transform priority order and Transform and fingerprinting.

  2. For a facet, sourceLabels must have the @ prefix (not # or bare, which read labels).

  3. The source facet is not eliminated by promotion — if you’re using the absence of the original facet to confirm the promotion worked, check for the presence of the new label instead.

A wide JSON payload has no facets at all

A single log line is automatically broken into at most 50 facets when parsed from JSON. This is a hard limit, not a truncation: a JSON object with 51 fields produces zero facets, not the first 50. If facets vanish completely rather than just becoming sparser, check the typical payload’s field count — if it exceeds 50, that’s the cause. Raise the limit with skipAutoFacet — see JSON logs — or select only the facets you need with whitelistFacets, which keeps the line below the limit regardless of field count.

A log’s timestamp looks incorrect

Kloudfuse takes into account three timestamp sources — for a complete description of the selection and bounds-checking logic, see Log Timestamp Handling. As for the parsing-specific starting points:

  • When the timestamp is in the future, the usual cause is a parsing bug, not a clock fault — see Future timestamps are almost always bugs. Examples include:

    • Extracting the wrong substring

    • Interpreting a field as epoch seconds when it’s actually milliseconds

    • Applying the timezone incorrectly

  • If you’ve added a custom setTimestamp function or a dissect/grok pattern that extracts a timestamp field, confirm the field’s format is actually one setTimestamp’s `formats list recognizes. If the system doesn’t recognize a format, it falls back silently to the agent or ingest timestamp rather than erroring.

  • If the timestamp is falling back to ingest time more often than expected, check whether timestamp.<source> or timestamp_for_all_sources is set to "message" (the default is "agent") — a source whose lines lack reliably parseable timestamps will fall back on every line that fails the bounds check.

A config change doesn’t seem to take effect

  1. Confirm the logs-parser rollout from your Helm upgrade actually completed — there’s no hot-reload, so a change sits inert until the pods restart with the new values.

  2. Check the YAML indentation of the config block itself. kf_parsing_config.config is a literal block-scalar string (|-) containing a YAML list; an indentation mistake inside that string is invisible to Helm’s own YAML validation (the outer document is still valid YAML) but breaks parsing of the inner pipeline config.

  3. Confirm you’re looking at logs ingested after the rollout. Logs already written before the change was deployed were parsed under the old configuration and won’t be retroactively reparsed.

  4. If the change is a conditions-scoped rule, re-check the condition against the actual logs you’re inspecting — see A dissect or grok pattern isn’t extracting fields above; a change that "doesn’t work" is frequently a change that’s deployed correctly but scoped to the wrong logs.

References