Relabel Rules
During ingestion, the Kloudfuse platform supports the rewriting the labels for Metrics, Events, Logs, and Traces data. This page describes configuration of relabel rules for Metrics, Events, and Traces. For configuration of relabel rules for Logs, see Parsing logs.
The configuration follows the Prometheus relabel config format, see https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config for more details. You can use helm chart values in the relevant relabel_configs section of the custom_values.yaml file.
ingester:
config:
aggregator:
metric:
# relabelConfigs - Configure relabel rules for metric labels.
# Format follows Prometheus relabel_config syntax.
# Ref: https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config
relabelConfigs: []
events:
# relabelConfigs - Configure relabel rules for events labels.
# Format follows Prometheus relabel_config syntax.
# Ref: https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config
relabelConfigs: []
traces:
# relabelConfigs - Configure relabel rules for traces labels.
# Format follows Prometheus relabel_config syntax.
# Ref: https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config
relabelConfigs: []Common Example of Relabel Rules
The examples below only shows a single relabel rule for each scenario. Multiple rules can be be provided, and they are evaluated in order. Specifically, each rule is evaluated against the current state of the labels after application of the previous rules.
Add New Label Based on Current Label Value
In a common scenario, you can enrich some metrics with new labels, depending on the existing label values. For example, for the label aws_account with value 123456, you can add aws_account_name label with AWS_account_name_for_123456 value by inserting the following section at the end of the custom_values.yaml file.
ingester:
config:
aggregator:
metric:
# relabelConfigs - Configure relabel rules for metric labels.
# Format follows Prometheus relabel_config syntax.
# Ref: https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config
relabelConfigs:
- ...
- regex: "123456"
replacement: AWS_account_name_for_123456
source_labels:
- aws_account
target_label: aws_account_name
- ...Run the helm upgrade to apply the changes.
Replace Label Value Based on Current Label Value
In a common scenario, you can update the label value, based on regex grouping. Consider the following example: A label span_name always comes with prefix span_. You can add a regex pattern that strips the prefix span_. Note that in the example below, the replacement value supports group references, e.g., ${1}.
ingester:
config:
aggregator:
metric:
# relabelConfigs - Configure relabel rules for metric labels.
# Format follows Prometheus relabel_config syntax.
# Ref: https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config
relabelConfigs:
- ...
- regex: "span_(.*)"
replacement: ${1}
source_labels:
- span_name
target_label: span_name
- ...Run the helm upgrade to apply the changes.
Conditionally Replace Label Value
By utilizing source_labels and regex you can conditionally replace the label value. Continuing with the example above, you can remove the prefix of span_ only if the service label is my-service. By default the relabel rules concatenate the label values of the source_labels with the ; separator. The separator used can be changed by explicitly providing the separator paramter.
ingester:
config:
aggregator:
metric:
# relabelConfigs - Configure relabel rules for metric labels.
# Format follows Prometheus relabel_config syntax.
# Ref: https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config
relabelConfigs:
- ...
- regex: "my-service;span_(.*)"
replacement: ${1}
source_labels:
- service
- span_name
target_label: span_name
- ...Run the helm upgrade to apply the changes.
Add New Label Based on Multiple Labels
Continuing with the example above, you can create a new label service_span by combining value of the service and span_name labels. You can also utilize regex grouping to extract a part of the value. In the example below, the service_span label is created by combining the value of the service label and a portion of the span_name label.
ingester:
config:
aggregator:
metric:
# relabelConfigs - Configure relabel rules for metric labels.
# Format follows Prometheus relabel_config syntax.
# Ref: https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config
relabelConfigs:
- ...
- regex: "(.*);span_(.*)"
replacement: ${1}-${2}
source_labels:
- service
- span_name
target_label: service_span
- ...Run the helm upgrade to apply the changes.
Drop labels
Labels can be dropped from the data by setting the action to labeldrop and setting the regex to a pattern that matches the label name. In the example below, any label that starts with span_ will be dropped from the data.
ingester:
config:
aggregator:
metric:
# relabelConfigs - Configure relabel rules for metric labels.
# Format follows Prometheus relabel_config syntax.
# Ref: https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config
relabelConfigs:
- ...
- regex: "span_(.*)"
action: labeldrop
- ...Run the helm upgrade to apply the changes.
Keep labels and Drop Everything Else
You can retain only the labels in your data while removing all other labels by setting the action to labelkeep and specifying a regex pattern that matches the desired label names. In the following example, any label that does not begin with span_ will be excluded from the data.
ingester:
config:
aggregator:
metric:
# relabelConfigs - Configure relabel rules for metric labels.
# Format follows Prometheus relabel_config syntax.
# Ref: https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config
relabelConfigs:
- ...
- regex: "span_(.*)"
action: labelkeep
- ...Run the helm upgrade to apply the changes.
Drop or Keep Series with Matching Label Values
In business scenarios, some series have to be dropped based on existing label values. For example, to stop storing data for the label aws_account with value 123456, you must add aws_account_name label with AWS_account_name_for_123456 value in the relabel config with action drop. To keep certain series, use the keep action.
ingester:
config:
aggregator:
metric:
# relabelConfigs - Configure relabel rules for metric labels.
# Format follows Prometheus relabel_config syntax.
# Ref: https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config
relabelConfigs:
- ...
- regex: "123456"
action: drop
source_labels:
- aws_account
- ...Run the helm upgrade to apply the changes.
Drop or Keep Series with Matching Metric Name
In Metrics stream, name is a special label that contains the name of the metric. For example, to stop storing any metric with name that starts with aws_ec2, you must add name label with aws_ec2.* value in the relabel config with action drop. To only keep these metrics, use the keep action.
ingester:
config:
aggregator:
metric:
# relabelConfigs - Configure relabel rules for metric labels.
# Format follows Prometheus relabel_config syntax.
# Ref: https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config
relabelConfigs:
- ...
- regex: "aws_ec2.*"
action: drop
source_labels:
- __name__
- ...Run the helm upgrade to apply the changes.
