Datadog Agent Log Processing

The Datadog Agent applies log processing rules before forwarding log records to Kloudfuse. Rules are configured either globally in datadog.yaml (to apply to all log sources) or per source in conf.d/<source>.d/conf.yaml.

Overview

Log processing rules are set in log_processing_rules and support three operations:

Rule type Effect

mask_sequences

Replace regex matches in the log body with a fixed placeholder string

exclude_at_match

Drop log records whose body matches the regex — matched records are not forwarded

include_at_match

Forward only log records whose body matches the regex — all other records are dropped

Rules in conf.d/<source>.d/conf.yaml apply only to that source. Rules in datadog.yaml under logs_config.processing_rules apply to every log source the agent collects.

Mask Sensitive Data

Use log_processing_rules with type: mask_sequences to replace sensitive values before logs leave the agent. Each rule applies a regular expression to the raw log line and replaces every match with replace_placeholder. Because masking happens in the agent, the original value never reaches Kloudfuse or crosses the network.

log_processing_rules can be set per log source in a check config file, or globally in datadog.yaml to apply to every log source the agent collects.

datadog.yaml — mask card numbers, card security codes, and credentials across all log sources
logs_config:
  processing_rules:
    - type: mask_sequences
      name: mask_card_security_codes
      replace_placeholder: "$1$2[CSC REDACTED]"
      pattern: "(?i)\\b(cvv2?|cvc2?|cid|pin_?block|track[12]|service_code)([\"']?\\s*[:=]\\s*[\"']?)[;%]?[^\\s,;\"'&}]+"   (1)
    - type: mask_sequences
      name: mask_credentials
      replace_placeholder: "$1$2[SECRET REDACTED]"
      pattern: "(?i)\\b(password|passwd|secret(?:_?key)?|api[_-]?key|x-api-key|auth(?:oriz(?:ation|ed))?_?token|access_?token|bearer)([\"']?\\s*[:=]\\s*[\"']?)[^\\s,;\"'&}]+"
    - type: mask_sequences
      name: mask_card_numbers
      replace_placeholder: "[CARD REDACTED]"
      pattern: "\\b(?:4\\d{3}|5[1-5]\\d{2}|2(?:22[1-9]|2[3-9]\\d|[3-6]\\d\\d|7[01]\\d|720)|3[47]\\d{2}|30[0-5]\\d|3(?:09|[689]\\d)\\d|6(?:011|5\\d\\d|4[4-9]\\d|22\\d)|35(?:2[89]|[3-8]\\d)|62\\d\\d)(?:[ -]?\\d){9,15}\\b"   (2)
    - type: mask_sequences
      name: mask_emails
      replace_placeholder: "***@***.***"
      pattern: "\\b[A-Za-z0-9._%+\\-]+@[A-Za-z0-9.\\-]+\\.[A-Za-z]{2,}\\b"
yaml
1 The key and separator are captured and echoed back through $1$2 (Agent 7.17+), so only the value is replaced and a JSON line stays valid JSON. Requiring an explicit : or = keeps prose such as "the password was reset" untouched. Keep these key-anchored rules before the card-number rule: they mask the whole value of a keyed field — including a card number embedded in track2 — so the broad card rule below cannot fragment it. Extend the keyword lists to the field names your applications actually use.
2 Anchored on the issuing-network prefixes of the major card brands, then 13–19 digits with an optional single space or dash between any two of them — so contiguous numbers and every grouped layout (4-4-4-4, Amex 4-6-5, Diners 4-6-4, 13-digit Visa 4-3-3-3) all match, while 16-digit numbers starting with other digits do not. Maestro ranges are not included. It runs last, after the key-anchored rules above. A card-shaped value that is not under one of those keys and is longer than 19 digits — such as raw Track 2 discretionary data — is not caught by this rule, so treat raw track/PAN dumps as data the application should not emit in the first place.

To add rules for a single log source, put them under log_processing_rules in that source’s check config (conf.d/<source>.d/conf.yaml). A source’s log_processing_rules are applied in addition to the global logs_config.processing_rules; the global rules continue to apply to every source, so do not duplicate them in source configs.

Before you deploy a pattern

Patterns are compiled with the Go regexp engine (RE2). It has no lookahead or lookbehind, so a pattern that works in Python or PCRE can be rejected here — and an invalid pattern in the global list stops the logs agent from starting. No logs are collected from that host until it is fixed; metrics and traces are unaffected. agent status reports it as Invalid processing rules. Validate every pattern against the same engine before rolling it out, for example with regexp.MustCompile in a short Go test, and run the rule against a sample of real log lines.

The card-number rule above will also mask other long numeric identifiers that begin with a valid card prefix — trace and span IDs, order numbers, and similar. For cardholder data that is the safe direction, but it can break log-to-trace correlation. log_processing_rules has no per-rule condition, so if that matters, scope the rule to the log sources of services that handle payments rather than setting it globally. (Do not reach for exclude_at_match here: it drops the whole record, it does not exempt it from masking.)

The agent’s own APM obfuscator (apm_config.obfuscation.credit_cards, on by default) only recognizes a card number when it is the entire value of a span tag, and it skips error.msg, error.stack, and sql.query by design. To cover those, add apm_config.replace_tags rules for the same keys.

Masking matched card and credential values in the agent reduces the amount of sensitive data sent to the observability platform. Compliance scope still depends on the complete data flow and the requirements of the applicable framework. For how this fits with ingest authentication, access policies, and audit logging on the Kloudfuse side, see Keeping Sensitive Data Out of Kloudfuse.

Add Tags to Logs

Tags set on a log source are attached to every log record collected by that source. Set them in the check config file:

conf.d/my-app.d/conf.yaml — add tags to a specific log source
logs:
  - type: file
    path: /var/log/my-app/*.log
    service: my-service
    source: my-app
    tags:
      - env:production
      - team:platform
yaml

To add tags to all metrics, logs, and events collected by the agent, set them globally in datadog.yaml:

datadog.yaml — add tags to all collected data
tags:
  - env:production
  - team:platform
  - region:us-east-1
yaml

Global tags are merged with any source-level tags, so both sets are forwarded to Kloudfuse.

Parse JSON Logs

The Datadog Agent automatically detects and parses log bodies that are valid JSON. When a log body is a JSON object, each top-level key is promoted to a log attribute and is available as a queryable label in Kloudfuse — no additional configuration is required.

For example, if a log record arrives with this body:

{"level":"error","msg":"connection refused","service":"payments","trace_id":"abc123"}
json

The agent extracts the following attributes, each of which becomes a searchable label:

Attribute key Value

level

error

msg

connection refused

service

payments

trace_id

abc123

JSON auto-detection is always active when logs_enabled: true. If your logs are not being parsed, confirm the log body is valid JSON and that no wrapping text precedes the opening {.

Filter Out Unwanted Logs

Use type: exclude_at_match to drop log records whose body matches a regex. Matched records are dropped before forwarding to Kloudfuse:

conf.d/my-app.d/conf.yaml — drop health check and readiness probe logs
logs:
  - type: file
    path: /var/log/my-app/*.log
    service: my-service
    source: my-app
    log_processing_rules:
      - type: exclude_at_match
        name: exclude_healthchecks
        pattern: "GET /(healthz|readyz|ping)"
      - type: exclude_at_match
        name: exclude_debug
        pattern: "\\[DEBUG\\]"
yaml

To forward only logs that match a pattern and drop everything else, use type: include_at_match. Only records matching the pattern are forwarded:

conf.d/my-app.d/conf.yaml — forward only ERROR and FATAL logs
logs:
  - type: file
    path: /var/log/my-app/*.log
    service: my-service
    source: my-app
    log_processing_rules:
      - type: include_at_match
        name: errors_only
        pattern: "ERROR|FATAL|CRITICAL"   (1)
yaml
1 Only log records whose body contains ERROR, FATAL, or CRITICAL are forwarded. All other records are dropped.

External References