Remap Concepts

Overview

Remap normalizes each agent’s raw payload into a common set of fields and labels. This page describes how the engine resolves values, how the editor is laid out, and what each mapping section controls. For the procedure, see Edit and Save a Remap Mapping.

Input paths and first-match-wins

Each normalized field is backed by an ordered list of input paths. The engine walks the list from the top and uses the first path that resolves to a value in the payload. Paths are JSONPath expressions rooted at the event, written with a $. prefix:

$.message
$.log
$.msg
text

Order determines priority, so rearranging the list changes which value wins. Paths may be nested — $.kubernetes.labels.app and $.log.body both resolve.

Your paths are additive. Each agent’s default rows cannot be removed, so your paths run alongside them, never in place of them. Adding one can change which value wins, but never empties a field.

Two kinds of rows appear in every list:

Default rows

Marked with a DEFAULT badge. These are the extraction paths built into the engine for that agent. You can reorder them to change their priority, but you cannot edit or remove them.

Custom rows

Paths you add yourself. These can be edited, reordered, and removed.

A USED badge marks the row that actually resolved against the sample currently loaded in the Input pane.

Input path means a JSONPath into the raw payload — a place to read a value from. It is unrelated to the source field, which names the application that emitted the log. The source field is itself populated by input paths, like any other field.

Configuration is per agent

Remap rules are scoped to the agent that sent the event. The tab bar above the rules pane selects which agent you are editing — Datadog, Filebeat, Fluent Bit, FluentD, GCP, Kinesis, and OTLP. Each agent has its own paths and its own engine defaults, and editing one does not affect the others.

The configuration is global: it applies to all data from that agent across the cluster, not per-tenant or per-agent-instance.

Page layout

The page is a three-pane flow, left to right, with resizable dividers:

Input

A raw payload from the selected agent. Use Load a live sample to pull recent real events from your cluster, or switch to Raw and supply your own JSON. Values that currently feed the normalized output are highlighted, so you can see which parts of the payload the rules actually consume.

Rules

The editor itself — the mapping sections described below, for the selected agent.

Output

The normalized result the engine produces for the payload in Input, split into Fields and Labels. This is computed by the real remap engine, not a client-side approximation, so it reflects what production will do.

The Structured / Raw toggle in the page header switches both the Input and Output panes between a browsable tree/table rendering and the underlying JSON text.

Edit the input by hand in Raw mode

In Structured view the Input pane is a read-only tree. Switch to Raw and it becomes an editable text box: you can paste a payload, or edit the loaded sample in place. The Output pane re-resolves as you type, so you can see immediately how a change affects the normalized result.

This is the way to test a payload you do not have in live traffic — a field that only appears on error, a shape a new shipper will send, or an edge case you want to confirm before it reaches production. Rather than waiting for a matching event, load any sample, switch to Raw, and edit it into the shape you want to check.

The text must be valid JSON. While it is not, the pane shows an Invalid JSON warning and the Output pane stops resolving; it recovers as soon as the text parses again.

Editing the payload only changes what you are testing against. It never modifies the saved configuration or any ingested data — the Input pane is a scratch pad, and nothing you type there is persisted.

The Output pane is read-only in both views; Raw renders the engine’s normalized result as JSON.

Hovering a path row highlights the value it resolves to in the Input pane, and the output entry it produces. Use this to trace a mapping end to end when a payload has several similar-looking fields.

Core fields

The Core section maps the three fields you control directly.

Field Description

message

The log message text.

source

The application that emitted the log. See source.

timestamp

Event time, as epoch milliseconds. If no path resolves, the ingest time is used instead.

Log level

Log level appears in the Output pane but has no editable path list. The engine derives it automatically from the payload and defaults to info when it cannot. It is shown so you can see the result, but it is not user-mappable.

Custom timestamp formats

Below the timestamp path list, Custom timestamp formats accepts a comma-separated list of Go time layouts, for example 2006/01/02 15:04:05. Supply these only when your timestamps are in a format the engine does not already recognize. Standard formats such as RFC 3339 and epoch values need no configuration.

Cloud and Kubernetes metadata

Two collapsible sections — Cloud metadata and Kubernetes — map structured infrastructure attributes. Each entry writes a fixed, standardized label name, shown beside the field as -> tag <name>. Because the output name is fixed, these labels stay consistent across agents, which is what makes cross-source filtering work.

Section Field Label written

Cloud metadata

Cluster

kube_cluster_name

Availability zone

availability_zone

Instance type

instance_type

Project

project

Account ID

cloud_account_id

Kubernetes

Namespace

kube_namespace

Host

host

Pod

pod_name

Container

kube_container_name

Docker image

docker_image

Service

kube_service

Each section header shows how many of its fields have engine defaults for the selected agent, or no defaults when the agent supplies none. Like core fields, these accept ordered, first-match-wins input path lists.

Custom labels

The Custom labels section promotes arbitrary payload values to labels, so they become available for filtering and grouping in the Logs explorer.

A custom label is just a path. The engine names the resulting label after the last segment of that path — $.kubernetes.labels.app produces a label named app — and the derived name is shown read-only beside the input. There is no rename.

If the path resolves to an object rather than a scalar, it fans out: each immediate child becomes its own label, named by its own key. Fan-out is one level deep only.

For example, given this payload:

{
  "kubernetes": {
    "labels": {
      "app": "checkout",
      "tier": "web",
      "meta": { "team": "payments" }
    }
  }
}
json
Path you configure Labels produced

$.kubernetes.labels.app

app=checkout

$.kubernetes.labels

app=checkout, tier=web

$.kubernetes.labels.meta.team

team=payments

The second row fans out: each immediate child of labels becomes its own label. The nested meta object produces no label at all, because fan-out does not recurse — to get team, point a path directly at it, as in the third row.

Avoid custom label names that collide with a standard label the engine already writes — level, host, pod_name, kube_namespace, kube_service, and the metadata names in the table above. source is protected: the engine’s own value wins over a colliding custom label. level is not — a custom label named level is written as a tag, and the editor gives you no warning.

Adding labels increases cardinality. Promote fields with bounded value sets; avoid request IDs, trace IDs, user IDs, and other high-cardinality values.

Next steps