Retention Policies

Kloudfuse has default and additional custom policies for various data streams.

Stream-level Retention

By default, the Kloudfuse stack installs with stream-level retention policies for Metrics, Events, Logs, Traces, and so on. You can add these policies to the global.retentionPolicy section of the custom_values.yaml file.

Here are the default settings:

How to Set Default Retention Policies
global:
  retentionPolicy:
    logs:
      default:
        retentionTimeValue: 7
        retentionTimeUnit: DAYS
    metrics:
      default:
        retentionTimeValue: 30
        retentionTimeUnit: DAYS
    events:
      default:
        retentionTimeValue: 30
        retentionTimeUnit: DAYS
    traces:
      default:
        retentionTimeValue: 7
        retentionTimeUnit: DAYS
yaml

Record-level Retention

Kloudfuse also supports configuring retention at the level of the record for each stream. We do this for metrics, events, and traces.

Be aware that there are some performance implications when you enable record-level retention policies. We recommended that you limit your organization to no more than 3 additional retention policies.

To enable record-level retention, you must make several changes to the custom_values.yaml file.

Configure Additional Retention Policies

The global section of helm values contains a retentionPolicy section. It uses the name of the stream and a list of retention policies. Similar to the format for the stream-level policy, it also uses retentionTimeValue and retentionTimeUnit fields, and a required name field for each custom policy.

Always add new custom policies to the end of the section. Do not delete or reorder existing policies.

The following example sets retention policies for metrics and logs: default policies and 2 additional custom policies.

.How to Set Default and Custom Retention Policies
global:
  retentionPolicy:
    expireRoundUp: 10800000 # 3 hour. Required.
    metrics:
      default:
        retentionTimeValue: 5
        retentionTimeUnit: DAYS
      custom:
        - retentionTimeValue: 7
          retentionTimeUnit: DAYS
          name: prod
        - retentionTimeValue: 14
          retentionTimeUnit: DAYS
          name: staging
    logs:
      default:
        retentionTimeValue: 14
        retentionTimeUnit: DAYS
      custom:
        - retentionTimeValue: 30
          retentionTimeUnit: DAYS
          name: prod
        - retentionTimeValue: 90
          retentionTimeUnit: DAYS
          name: staging
    events: []
    traces: []
yaml

Configure Kafka Partitions

You must adjust the corresponding Kafka partitions for each stream, so they accommodate the new policies. The helm chart checks that the number of configured partitions is divisible by the sum of retention policies, both default and custom. In the previous Example, the Kafka metrics topic requires 3 partitions, to ensure that incoming records are sent to different Kafka partitions.

How to Configure Kafka Partitions
global:
  kafkaTopics:
    - name: kf_metrics_topic
      partitions: 3
      replicationFactor: 1
yaml

Configure Relabel Rules

Metrics, Traces, and Event Streams

Kloudfuse uses relabel rules to attach retention policies to each record. If a record does not match any of the retention policies specified in the global.retentionPolicy.custom section, it defaults to the stream-level retention policy. Specify the relabeling rules in the ingester.config section of the custom_values.yaml file. See Relabel Rules.

The relabeling rules syntax supports a retention action for retention policies; it has a retention_policy field that specifies the target retention policy of the rule. Similarly, Kloudfuse uses the source_labels separator regex fields to match the incoming record.

In the following example: * The prod retention policy configures metrics that have the env label with prod value. * The staging retention policy configures metrics that use a prefixed name test_app.

Kloudfuse evaluates relabel rules in order of appearance. If the record matches both rules, the last rule gets priority.

  1. How to Configure Metrics, Traces, and Events Streams

ingester:
  config:
    aggregator:
      metric:
        relabelConfigs:
          - source_labels:
            - env
            regex: "prod"
            action: retention
            retention_policy: prod
          - source_labels:
            - __name__
            regex: test_app(.*)
            action: retention
            retention_policy: staging

Logs Stream

Log streams have a different expression than metrics, traces, and events.

For example, to configure retention policy staging for logs from a source with sub-string -stag-, use the following relabel rule:

  1. How to Configure Logs Streams

logs-parser:
  kf_parsing_config:
    config: |-
      - relabel:
          args:
            - action: replace
            - sourceLabels: "#source"
            - targetLabel: "__kf_retention"
            - regex: "^.*-stag-.*$"
            - replacement: "staging"

To define and use relabel rules for logs streams, see Parsing logs