Datadog Agent Log Processing

The Datadog Agent applies log processing rules before forwarding log records to Kloudfuse. Rules are configured either globally in datadog.yaml (to apply to all log sources) or per source in conf.d/<source>.d/conf.yaml.

Overview

Log processing rules are set in log_processing_rules and support three operations:

Rule type Effect

mask_sequences

Replace regex matches in the log body with a fixed placeholder string

exclude_at_match

Drop log records whose body matches the regex — matched records are not forwarded

include_at_match

Forward only log records whose body matches the regex — all other records are dropped

Rules in conf.d/<source>.d/conf.yaml apply only to that source. Rules in datadog.yaml under logs_config.processing_rules apply to every log source the agent collects.

Mask PII Data

Use type: mask_sequences to replace sensitive values before logs leave the agent. Each rule applies a regex pattern and replaces any match with the value of replace_placeholder.

conf.d/my-app.d/conf.yaml — mask PII in a specific log source
logs:
  - type: file
    path: /var/log/my-app/*.log
    service: my-service
    source: my-app
    log_processing_rules:
      - type: mask_sequences
        name: mask_passwords
        replace_placeholder: "***"
        pattern: "password=[^\\s&]+"
      - type: mask_sequences
        name: mask_emails
        replace_placeholder: "***@***.***"
        pattern: "\\b[A-Za-z0-9._%+\\-]+@[A-Za-z0-9.\\-]+\\.[A-Za-z]{2,}\\b"
      - type: mask_sequences
        name: mask_credit_cards
        replace_placeholder: "****-****-****-****"
        pattern: "\\b\\d{4}[\\s\\-]?\\d{4}[\\s\\-]?\\d{4}[\\s\\-]?\\d{4}\\b"
yaml

To apply masking globally across all log sources, add the same rules under logs_config in datadog.yaml:

datadog.yaml — mask PII across all log sources
logs_config:
  processing_rules:
    - type: mask_sequences
      name: mask_passwords
      replace_placeholder: "***"
      pattern: "password=[^\\s&]+"
    - type: mask_sequences
      name: mask_emails
      replace_placeholder: "***@***.***"
      pattern: "\\b[A-Za-z0-9._%+\\-]+@[A-Za-z0-9.\\-]+\\.[A-Za-z]{2,}\\b"
yaml
mask_sequences applies regex substitution on the raw string value of the log body before any parsing occurs. Test patterns against representative log samples before enabling in production.

Add Tags to Logs

Tags set on a log source are attached to every log record collected by that source. Set them in the check config file:

conf.d/my-app.d/conf.yaml — add tags to a specific log source
logs:
  - type: file
    path: /var/log/my-app/*.log
    service: my-service
    source: my-app
    tags:
      - env:production
      - team:platform
yaml

To add tags to all metrics, logs, and events collected by the agent, set them globally in datadog.yaml:

datadog.yaml — add tags to all collected data
tags:
  - env:production
  - team:platform
  - region:us-east-1
yaml

Global tags are merged with any source-level tags, so both sets are forwarded to Kloudfuse.

Parse JSON Logs

The Datadog Agent automatically detects and parses log bodies that are valid JSON. When a log body is a JSON object, each top-level key is promoted to a log attribute and is available as a queryable label in Kloudfuse — no additional configuration is required.

For example, if a log record arrives with this body:

{"level":"error","msg":"connection refused","service":"payments","trace_id":"abc123"}
json

The agent extracts the following attributes, each of which becomes a searchable label:

Attribute key Value

level

error

msg

connection refused

service

payments

trace_id

abc123

JSON auto-detection is always active when logs_enabled: true. If your logs are not being parsed, confirm the log body is valid JSON and that no wrapping text precedes the opening {.

Filter Out Unwanted Logs

Use type: exclude_at_match to drop log records whose body matches a regex. Matched records are dropped before forwarding to Kloudfuse:

conf.d/my-app.d/conf.yaml — drop health check and readiness probe logs
logs:
  - type: file
    path: /var/log/my-app/*.log
    service: my-service
    source: my-app
    log_processing_rules:
      - type: exclude_at_match
        name: exclude_healthchecks
        pattern: "GET /(healthz|readyz|ping)"
      - type: exclude_at_match
        name: exclude_debug
        pattern: "\\[DEBUG\\]"
yaml

To forward only logs that match a pattern and drop everything else, use type: include_at_match. Only records matching the pattern are forwarded:

conf.d/my-app.d/conf.yaml — forward only ERROR and FATAL logs
logs:
  - type: file
    path: /var/log/my-app/*.log
    service: my-service
    source: my-app
    log_processing_rules:
      - type: include_at_match
        name: errors_only
        pattern: "ERROR|FATAL|CRITICAL"   (1)
yaml
1 Only log records whose body contains ERROR, FATAL, or CRITICAL are forwarded. All other records are dropped.

External References