Log Parsing Configuration

Remap is the pipeline’s first stage — it maps fields from the incoming payload (JSON, msgpack, or proto, depending on your agent) onto Kloudfuse’s internal fields: message, timestamp, source, and initial labels/facets.

__kf_agent is a reserved matcher for the sending agent, and supports datadog, fluent-bit, fluentd, kinesis, gcp, otlp, and filebeat. All Remap fields must be given in JSONPath notation.

Relabel works on the labels Remap (or Logs Remap) produced — it can add, drop, rewrite, or derive a label. It’s loosely inspired by Prometheus relabeling (a regex against one or more source labels, an action, a target), but the action set below is this pipeline’s own, not an exact match to Prometheus’s.

Every pipeline function — relabel, transform, parser (dissect/grok), and the others listed under Other pipeline functions — is a list item under config, with its arguments under args and, optionally, a list of conditions that determine whether it runs for a given log line. remap always runs first, ahead of everything else. Beyond that, each function executes in a fixed stage, and your config list’s order doesn’t change which stage a function runs in — it only orders functions relative to each other within the same stage. relabel always runs before Grammar (parser, JSON auto-parsing) and transform always runs after it, so a relabel rule can never see a facet parser extracts no matter which one you write first — see Relabel, Transform priority order. A function with no conditions always runs. Each condition is a matcher / value / op triplet:

Kloudfuse auto-detects facets and a timestamp from the message using a heuristic, which won’t always be precise. Define a custom pattern to extract fields exactly instead.

Given this log line:

this dissect tokenizer:

produces the facets sourceIp: 10.12.0.35, requestMethod: GET, responseCode: 503, and contentLength: 21. Limit the tokenizer to a single source by using a conditions matcher on #source, so it applies only to the logs it’s meant for:

Add another - parser: dissect: …​ item, with its own conditions, for a second source — each function in the config list is independently scoped by its own conditions block; nothing groups them by source implicitly.

Kloudfuse automatically detects JSON messages and turns each field into a facet, flattening nested objects with _ — {"location": {"city": "SF"}} becomes the facet location_city. A field that is an array — such as "aliases": ["johndoe", "johnny"] — is extracted as a single string facet.

By default, a single log line can generate up to 50 facets. This is a strict ceiling, not a truncation to the first 50: a JSON object with 51 fields results in zero facets. Raise the ceiling with skipAutoFacet:

Once ingested, query the extracted facets from the Logs Explorer the same way as any other facet — see Logs Search Syntax.

relabel always runs before Grammar and JSON auto-parsing, and transform always runs after — regardless of where either appears in your config list. For this reason, transform, not relabel, is used when something depends on a facet that Grammar or JSON auto-parsing extracts. The most common application is facet_to_label_map, which promotes a facet extracted by Grammar into a label — for example, promoting the facet eventSource into the label source:

Any other action in Relabel’s action table — drop, keep, label_map, and so on — functions the same way here. The difference between transform and relabel determines when the rule applies relative to extraction, not the action’s capabilities. If a drop or keep condition depends on a facet extracted by Grammar, it must be written using transform, since relabel runs too early to see that facet, regardless of its position in the config list.

At some point in the pipeline, the internal Kf-Parse stage produces the log’s fingerprint by dividing the message into its static template and dynamic values — so that ts=2023-02-01T22:51:33Z caller=logging.go:29 method=Authorise result=false took=9.775µs becomes the fingerprint ts=<v_0> caller=<v_1> method=<v_2> result=<v_3> took=<v_4> plus the facets ts, caller, method, result, and took. Storing the template once and the variable values separately is what keeps Kloudfuse’s highly repetitive log traffic both fully indexed and storage-efficient — see Log Fingerprints for where this shows up in the Explorer. Kf-Parse itself isn’t configurable.

For more information on how Kloudfuse selects and validates a log’s timestamp, see Log Timestamp Handling.

Beyond remap, relabel/transform, and dissect/grok, the pipeline supports these additional functions — each a list item under config, exactly like the examples above:

A further set of low-level tokenization primitives — resetPos, truncate, tokenize, rebuild, tokenizedKvExtract, tokenizedTsExtract, dissectKVOptimize, and tokenizedNewr — exist for advanced custom parsing scenarios beyond what dissect and grok cover. They’re implementation details rather than something most configurations need; contact your Kloudfuse representative if dissect or grok don’t cover a format you’re trying to parse.