Traces Relabel Rules

Traces relabel rules run on every incoming span before Kloudfuse stores it. Use them to drop spans you do not need, such as health checks, to clean up span names and labels, and to send spans to a retention policy. For the rule fields, actions, and evaluation order, see How Relabel Rules Work.

The same rules also run separately on span errors: the error events recorded on a span. A rule that drops a span does not drop its errors unless the rule also matches the error’s labels.

Configure the rules

Add the rules to ingester.config.traces.relabelConfigs in your custom-values.yaml file:

ingester:
  config:
    traces:
      relabelConfigs:
        - source_labels: [span_name]
          regex: "GET /(health|ready|live)z?"
          action: drop
yaml

Then apply the change:

helm upgrade --install kfuse oci://us-east1-docker.pkg.dev/mvp-demo-301906/kfuse-helm/kfuse \
  -n kfuse \
  --version <VERSION> \ (1)
  -f custom-values.yaml
bash
1 Replace <VERSION> with a valid Kloudfuse release value. See Release Notes for the latest release.

Labels the rules see

Rules see a span’s low-cardinality attributes, with attribute names normalized to use underscores (service.name becomes service_name). These include:

  • Kloudfuse span labels: service_name, span_name, span_kind, span_type, span_status_code, error, and peer_service.

  • Kubernetes and cloud labels, such as kube_cluster_name, kube_namespace, and region.

  • OpenTelemetry resource attributes, such as service_namespace, service_version, deployment_environment, host_name, container_name, and cloud_provider.

  • Labels added by an ingestion API key. Rules can match these labels but cannot change them.

High-cardinality span attributes, such as request IDs and URLs, are not visible to the rules. To make another attribute available, add its original name to ingester.config.traces.lcAttributesConfig.

Examples

Each example shows a single rule. You can list as many rules as you need; they run in order.

Drop health check spans

Stop storing spans for health and readiness probes:

- source_labels: [span_name]
  regex: "GET /(health|ready|live)z?"
  action: drop
yaml

Drop spans from a service in one environment

Stop storing spans from load-generator in the staging environment:

- source_labels: [service_name, deployment_environment]
  regex: "load-generator;staging"
  action: drop
yaml

Rewrite a span name

Strip a span_ prefix from span names, so span_checkout becomes checkout:

- source_labels: [span_name]
  regex: "span_(.*)"
  target_label: span_name
  replacement: "${1}"
yaml

To rewrite the name only for one service, add service_name to source_labels and match it in regex, for example regex: "my-service;span_(.*)".

Add a label from other labels

Create a service_span label, such as checkout-charge, from the service name and the span name:

- source_labels: [service_name, span_name]
  regex: "(.*);(.*)"
  target_label: service_span
  replacement: "${1}-${2}"
yaml

Assign a retention policy

Keep spans from the prod cluster under the prod retention policy. The policy must be defined in global.retentionPolicy.traces.custom; see Configure relabel rules:

- source_labels: [kube_cluster_name]
  regex: "prod"
  action: retention
  retention_policy: prod
yaml

Check the result

Spans dropped by a rule appear in the Consumption Tracking dashboard with the drop reason relabel. Dropping a span can leave gaps in a trace: its child spans are still stored.