Datadog Agent Processing

Overview

The Datadog Agent provides log processing rules and check-level configuration to reshape telemetry data before it is forwarded to Kloudfuse. This page covers common log and metric processing patterns using log_processing_rules for logs, and OpenMetrics check configuration and DogStatsD mapper profiles for metrics.

Unlike a pipeline-based collector, Datadog Agent configuration is split across two file types:

  • datadog.yaml — global agent settings that apply to all collected data

  • conf.d/<source>.d/conf.yaml — per-integration or per-log-source settings

Each example below shows both files where relevant, with comments indicating which file each block belongs to.

Log Processing

Mask PII Data

Use log_processing_rules with type: mask_sequences to replace sensitive values before logs leave the agent. Each rule applies a regex pattern and replaces any match with the value of replace_placeholder.

log_processing_rules can be set per log source in a check config file, or globally in datadog.yaml to apply to all log sources.

conf.d/my-app.d/conf.yaml — mask PII in a specific log source
logs:
  - type: file
    path: /var/log/my-app/*.log
    service: my-service
    source: my-app
    log_processing_rules:
      - type: mask_sequences
        name: mask_passwords
        replace_placeholder: "***"
        pattern: "password=[^\\s&]+"
      - type: mask_sequences
        name: mask_emails
        replace_placeholder: "***@***.***"
        pattern: "\\b[A-Za-z0-9._%+\\-]+@[A-Za-z0-9.\\-]+\\.[A-Za-z]{2,}\\b"
      - type: mask_sequences
        name: mask_credit_cards
        replace_placeholder: "****-****-****-****"
        pattern: "\\b\\d{4}[\\s\\-]?\\d{4}[\\s\\-]?\\d{4}[\\s\\-]?\\d{4}\\b"
yaml

To apply masking globally across all log sources, add the same rules under logs_config in datadog.yaml:

datadog.yaml — mask PII across all log sources
logs_config:
  processing_rules:
    - type: mask_sequences
      name: mask_passwords
      replace_placeholder: "***"
      pattern: "password=[^\\s&]+"
    - type: mask_sequences
      name: mask_emails
      replace_placeholder: "***@***.***"
      pattern: "\\b[A-Za-z0-9._%+\\-]+@[A-Za-z0-9.\\-]+\\.[A-Za-z]{2,}\\b"
yaml
Test patterns against representative log samples. mask_sequences applies regex substitution on the raw string value of the log body before any parsing occurs.

Add Tags to Logs

Tags set on a log source are attached to every log record collected by that source. Set them in the check config file:

conf.d/my-app.d/conf.yaml — add tags to a specific log source
logs:
  - type: file
    path: /var/log/my-app/*.log
    service: my-service
    source: my-app
    tags:
      - env:production
      - team:platform
yaml

To add tags to all metrics, logs, and events collected by the agent, set them globally in datadog.yaml:

datadog.yaml — add tags to all collected data
tags:
  - env:production
  - team:platform
  - region:us-east-1
yaml

Global tags are merged with any source-level tags, so both sets are forwarded to Kloudfuse.

Parse JSON Logs

The Datadog Agent automatically detects and parses log bodies that are valid JSON. When a log body is a JSON object, each top-level key is promoted to a log attribute and is available as a queryable label in Kloudfuse — no additional configuration is required.

For example, if a log record arrives with this body:

{"level":"error","msg":"connection refused","service":"payments","trace_id":"abc123"}
json

The agent extracts the following attributes, each of which becomes a searchable label:

Attribute key Value

level

error

msg

connection refused

service

payments

trace_id

abc123

If your logs are not being parsed automatically, confirm that JSON auto-detection is enabled in datadog.yaml:

datadog.yaml — confirm JSON auto-detection is enabled
logs_config:
  auto_multi_line_detection: false   (1)
  use_http: true
yaml
1 auto_multi_line_detection is for multi-line aggregation and is separate from JSON parsing. JSON detection is always active when logs_enabled: true.

Filter Out Unwanted Logs

Use log_processing_rules with type: exclude_at_match to drop log records whose body matches a regex. Records that match the pattern are dropped and not forwarded to Kloudfuse:

conf.d/my-app.d/conf.yaml — drop health check and readiness probe logs
logs:
  - type: file
    path: /var/log/my-app/*.log
    service: my-service
    source: my-app
    log_processing_rules:
      - type: exclude_at_match
        name: exclude_healthchecks
        pattern: "GET /(healthz|readyz|ping)"
      - type: exclude_at_match
        name: exclude_debug
        pattern: "\\[DEBUG\\]"
yaml

To keep only logs that match a pattern and drop everything else, use type: include_at_match. Only records matching the pattern are forwarded:

conf.d/my-app.d/conf.yaml — forward only ERROR and FATAL logs
logs:
  - type: file
    path: /var/log/my-app/*.log
    service: my-service
    source: my-app
    log_processing_rules:
      - type: include_at_match
        name: errors_only
        pattern: "ERROR|FATAL|CRITICAL"   (1)
yaml
1 Only log records whose body contains ERROR, FATAL, or CRITICAL are forwarded. All other records are dropped.

Metric Processing

Add Tags to All Metrics

Tags set in datadog.yaml are attached to every metric, log, and event collected by the agent. Use this to add environment, region, or team labels that apply across your entire infrastructure:

datadog.yaml — add global tags to all metrics
tags:
  - env:production
  - team:platform
  - region:us-east-1
yaml

To add tags only to metrics collected by a specific OpenMetrics endpoint, set them in the check config:

conf.d/openmetrics.d/conf.yaml — add tags to a specific metrics source
instances:
  - openmetrics_endpoint: http://my-app:8080/metrics
    namespace: my_app
    metrics:
      - ".*"
    tags:
      - env:production
      - team:platform
yaml

Drop High-Cardinality Labels from Metrics

When collecting metrics from a Prometheus or OpenMetrics endpoint, use exclude_labels to remove label keys that create an unbounded number of time series — such as request IDs, user IDs, or raw URLs. The listed labels are stripped from every metric collected by that check instance:

conf.d/openmetrics.d/conf.yaml — drop high-cardinality labels
instances:
  - openmetrics_endpoint: http://my-app:8080/metrics
    namespace: my_app
    metrics:
      - ".*"
    exclude_labels:
      - request_id
      - user_id
      - trace_id
yaml

To keep only a specific set of labels and drop all others, use labels_mapper in combination with exclude_labels, or define a metric_tags allowlist:

conf.d/openmetrics.d/conf.yaml — keep only specific labels, drop the rest
instances:
  - openmetrics_endpoint: http://my-app:8080/metrics
    namespace: my_app
    metrics:
      - ".*"
    exclude_labels:
      - request_id
      - user_id
      - pod_id
      - container_id
    tags:
      - env:production   (1)
yaml
1 Static tags added here are always present on the metrics regardless of what labels the endpoint exposes.

Rename Metrics

For OpenMetrics endpoints, rename individual metrics by providing a mapping in the metrics list. Each entry maps the scraped metric name to the desired output name:

conf.d/openmetrics.d/conf.yaml — rename specific metrics
instances:
  - openmetrics_endpoint: http://my-app:8080/metrics
    namespace: my_app
    metrics:
      - http_server_duration: http.server.request.duration   (1)
      - process_cpu_usage: process.cpu.utilization
      - ".*"                                                 (2)
yaml
1 The scraped metric http_server_duration is forwarded to Kloudfuse as my_app.http.server.request.duration (the namespace is prepended automatically).
2 Collect all remaining metrics under their original names.

For DogStatsD metrics, use dogstatsd_mapper_profiles in datadog.yaml to rename metrics by matching a pattern:

datadog.yaml — rename DogStatsD metrics
dogstatsd_mapper_profiles:
  - name: my_app
    prefix: "my_app."
    mappings:
      - match: "my_app.http_server_duration"
        name: "my_app.http.server.request.duration"
      - match: "my_app.process_cpu_*"
        name: "my_app.process.cpu.$1"
        match_type: wildcard
yaml

Drop Metrics by Name or Pattern

For OpenMetrics endpoints, use exclude_metrics to drop metrics by exact name or regex pattern. Each entry is a drop condition — any metric whose name matches is not forwarded to Kloudfuse:

conf.d/openmetrics.d/conf.yaml — drop metrics by exact name
instances:
  - openmetrics_endpoint: http://my-app:8080/metrics
    namespace: my_app
    metrics:
      - ".*"
    exclude_metrics:
      - go_gc_duration_seconds
      - process_open_fds
      - process_max_fds
yaml

To drop all metrics whose names match a regex pattern:

conf.d/openmetrics.d/conf.yaml — drop metrics matching a pattern
instances:
  - openmetrics_endpoint: http://my-app:8080/metrics
    namespace: my_app
    metrics:
      - ".*"
    exclude_metrics:
      - "go_.*"                   (1)
      - "process_runtime_.*"
yaml
1 Drops all metrics whose names start with go_. Any metric matching any pattern in exclude_metrics is dropped.

To forward only metrics that match a specific pattern and drop everything else, list only the desired pattern in metrics rather than using ".*":

conf.d/openmetrics.d/conf.yaml — collect only metrics matching a pattern
instances:
  - openmetrics_endpoint: http://my-app:8080/metrics
    namespace: my_app
    metrics:
      - "http_.*"    (1)
yaml
1 Only metrics whose names start with http_ are collected. All other metrics exposed by the endpoint are ignored.

External References