Configuring Log Archives

An archive is a rule that routes matching logs to your cloud storage bucket. Each archive has routing filters that decide which logs it captures, a priority that decides the order archives are evaluated in, a compression format, and a setting that controls whether its logs also stay searchable in Kloudfuse.

You define archives one of two ways:

  • The Archives UI — the recommended method, and the default since release 4.3.0.

  • The global.archive YAML block in deployments.yaml — used when the archive-writer service is disabled.

Both methods share the same storage connection. For a high-level picture of how archiving fits into the log pipeline, see Log Archive Architecture.

Storage connection vs. archive definitions

Log archiving has two configuration layers. Only the archive definitions move to the UI.

Layer What it sets Where it is configured

Storage connection

The cloud provider, bucket, region, and credentials Kloudfuse writes archives to.

Always in deployments.yaml (global.archive), at deployment time — with either method. See Storage connection.

Archive definitions

Which logs are archived, the archive names, routing filters, evaluation order, compression, and whether logs stay indexed.

The Archives UI (default), or the global.archive.rules block.

How routing works

Every archive has a priority, a whole number starting at 0. When a log is ingested, Kloudfuse checks the archives in priority order — 0 first — and writes the log to the first archive whose routing filters match. A log is written to at most one archive, so order matters: put more specific archives ahead of broad, catch-all ones.

Routing filters are evaluated after the log is normalized by the remap pipeline, so they match the normalized label and log-line values.

Storage connection

Kloudfuse supports log archive and hydration for AWS S3 and Google Cloud Storage (GCS). You must create the bucket yourself and grant Kloudfuse write access to it. Configure the connection in the global.archive block, regardless of which method you use to define archives:

global:
  archive:
    enabled: true
    prefix: "<Example_Cluster/Example_Folder>"    # Optional, can specify as ""
    useSecret: true                               # See "Storage access" below
    createSecret: true
    secretName: "<Example_Secret>"
    type: s3                                      # s3 or gcs
    s3:
      region: <Example_Region>                    # Such as us-west-2
      bucket: <Example_Bucket_Name>               # You must create the bucket yourself
      accessKey: <Example_Access_Key>
      secretKey: <Example_Secret_Key>
yaml

To turn off log archiving entirely, set global.archive.enabled: false.

Storage access

Grant Kloudfuse permission to write into the bucket with one of the following approaches:

  1. Helm creates the secret — createSecret: true, useSecret: true

    Helm creates the Kubernetes secret from the accessKey and secretKey values you provide, and Kloudfuse reads the credentials from that secret.

  2. You create the secret — createSecret: false, useSecret: true

    You create the Kubernetes secret yourself. Kloudfuse picks up the credentials from it automatically as environment variables.

  3. Node IAM role — createSecret: false, useSecret: false

    No secret or environment variables are needed. This assumes the node IAM role already has permission to access the bucket.

  4. Service account — createSecret: false, useSecret: false

    You create a service account with permission to access the bucket and set serviceAccountName to it. No environment variables are needed; the pod inherits the service account’s permissions.

Define archives in the Archives UI

When the archive-writer service is enabled — the default since 4.3.0 — define and manage archives from the Archives screen in the Admin UI. Changes take effect immediately, with no redeploy.

Open the Archives screen

Starting with Release 4.3.0, when RBAC is enabled you must have the Admin role to create, edit, start, stop, or delete archives. Users with other roles can view the Archives screen.

Select Admin in the top navigation bar, then select Archives under Data Management.

Create an archive

  1. Select Create archive.

  2. Fill in the Definition:

    • Name (required) — a unique identifier, using only letters, digits, and the _, ., and - characters. You cannot change the name after the archive is created.

    • Prefix — an optional sub-path within the storage bucket. Leave it blank to write to the default path.

    • Compression format — the object container the logs are written in. See Compression format.

    • Priority — the evaluation position. Lower is evaluated first; a new archive defaults to the next free number. See Change evaluation priority to change it later.

    • Index logs — leave selected to write matching logs to the archive and index them in Kloudfuse so they stay searchable (Index + archive). Clear it to write only to the archive (Archive only); those logs are then available only through hydration.

  3. Add Routing filters to decide which logs the archive captures. See Routing filters.

  4. Select Create archive.

The archive starts capturing matching logs immediately.

Edit an archive

Select an archive, then select Edit (from the row menu or the details panel) to reopen the form. Every field except Name can be changed. Select Save changes to apply, or Cancel to discard. Changing the Compression format applies to the next batch written; existing objects keep their original format.

Compression format

The archived payload is the same newline-delimited JSON (NDJSON) in every format — the choice only changes the object container and file extension.

Option File extension

Zstandard (default)

.log.zst

gzip

.gz

tar (uncompressed)

.tar

tar + gzip

.tar.gz

Routing filters

An archive captures a log only if the log matches every filter. An archive with no filters captures every log that reaches it — use one as the low-priority catch-all if you want a fallback destination.

Label filters match on a log label or tag. Each row has three parts:

  • Key — the label or tag to match. Choose one from the list or type your own.

  • Operator — = (equals), != (not equals), =~ (matches regex), !~ (does not match regex), contains, or not contains.

  • Value — the value or regular expression to compare against. Choose a suggested value or type your own.

Log line filters match on the raw log message instead of a label, using the same operators.

Select Add label filter or Add log line filter to add a row, or Remove to delete one.

The Archives list

The list shows every archive in priority order. Search by name or prefix, or filter by Status (In progress or Stopped). Select Refresh to reload sizes and ingest-rate charts.

Column What it shows

Priority

The evaluation order; 0 is evaluated first. See Change evaluation priority.

Archive

The archive name. Select a row to see its full details, including raw and compressed size and whether it came from YAML or the API.

Status

In progress — the archive is actively writing matching logs. Stopped — archiving is paused. See Start or stop archiving.

Location

The full path in cloud storage where the archive’s logs are written. Use the copy button to copy it.

Filters

The routing filters, shown as chips. An archive with no filters captures everything that reaches it.

Destination

Index + archive or Archive only, set by the Index logs option.

Compression

The object format the archive is written in. See Compression format.

Rate 1h, Rate 24h, Rate 7d

A sparkline of the archive’s ingest rate over the last hour, day, and week. A blank cell means there is no data for that window.

Created, Updated

When the archive was created and last changed.

To hydrate an archive’s logs back into Kloudfuse, select Hydrate from its row menu or details panel. See Hydrate From Archive.

Change evaluation priority

Because the first matching archive wins, list order matters.

  1. Clear any search text or status filter so the full list is shown.

  2. From an archive’s row menu, select Move up or Move down to shift it one position. Move up is hidden for the first row, Move down for the last.

  3. The Priority column shows each pending change as old → new and an Unsaved priority changes banner appears.

  4. Select Save order to commit, or Cancel to discard.

Start or stop archiving

While editing an archive, use the Stop or Start control next to its status:

  • Stop — pause archiving. The archive stops writing new logs; its configuration and previously written data are kept.

  • Start — resume a stopped archive.

The change takes effect immediately, independently of Save changes.

Delete an archive

Select Delete archive on the edit form and confirm. This removes the archive’s configuration and stops it from capturing new logs. Logs already written to cloud storage are not deleted — remove them with your cloud provider’s tools if you no longer need them.

Define archives in YAML

This method applies only when the archive-writer service is disabled (global.logs.remap.enabled: false). When it is enabled, define archives in the Archives UI instead. See Archive Writer and Ingest Integration.

Add a rules block inside global.archive to define which logs are written to which archive:

global:
  archive:
    # ... storage connection settings from above ...
    rules: |-
      - archive:                                  # First archive
          args:
            archiveName: a1
            doNotIndex: false                     # Archive and index
          conditions:
            - matcher: "#source"
              value: "s1"
              op: "=="
            - matcher: "@label"
              value: "l1"
              op: "=="
      - archive:                                  # Second archive
          args:
            archiveName: a2
            doNotIndex: false
          conditions:
            - matcher: "#source"
              value: "s1"
              op: "=="
yaml

Rules are evaluated in order, and every condition in a rule must match. In this example, a log line from source s1 is written to a1 if it also carries label l1; otherwise it falls through to a2. A log line from any other source matches neither rule and is not archived.

YAML-to-UI field reference

Use this table to translate an existing rules block into UI archives:

YAML Archives UI equivalent

args.archiveName

Name

global.archive.prefix

Prefix (set per archive in the UI)

args.doNotIndex: false

Index logs selected — Index + archive

args.doNotIndex: true

Index logs cleared — Archive only

conditions with matcher: "#source"

A Label filter on the source key

conditions with matcher: "@label"

A Label filter on the label name (drop the @)

op: "=="

The = operator (the UI also offers !=, =~, !~, contains, not contains)

Position in the rules list

Priority (lower is evaluated first)

(no equivalent)

Compression format — UI only

When the archive-writer service is enabled, archives you previously defined in YAML appear on the Archives screen marked Source: YAML, so you can list, inspect, and hydrate them there. Create and change archives in the UI from that point on, and treat the rules block as read-only history.

Archive rules reference

Each entry in the rules block has two keys: conditions and args.

conditions

conditions is a list of matchers that must all be satisfied (logical AND) for the rule to apply. If no conditions are specified, the rule matches every log line — use this as a catch-all last rule.

Each condition has three fields:

Field Required Description

matcher

Yes

The attribute to match against. Uses a sigil prefix to distinguish attribute types:

#source — matches the log source name (the originating agent or integration).
@label — matches a label extracted from the log by the Kloudfuse pipeline (for example, a Kubernetes namespace or pod name).

op

Yes

The comparison operator. Currently supported: == (exact equality).

value

Yes

The string value to compare against the matched attribute.
Example: match logs from a specific source and namespace
rules: |-
  - archive:
      conditions:
        - matcher: "#source"
          value: "kubernetes"
          op: "=="
        - matcher: "@label"
          value: "production"
          op: "=="
      args:
        archiveName: k8s-production
        doNotIndex: false
  - archive:
      conditions:
        - matcher: "#source"
          value: "kubernetes"
          op: "=="
      args:
        archiveName: k8s-all
        doNotIndex: false
yaml

In this example, Kubernetes logs from the production namespace match the first rule and are written to k8s-production. All other Kubernetes logs fall through to the second rule and are written to k8s-all. Non-Kubernetes logs do not match either rule and are not archived.

args

args controls how the matched log is archived.

Field Required Description

archiveName

Yes

A short identifier for the archive destination within the storage bucket. Used as a path segment when Kloudfuse writes the files and when you select an archive to hydrate. Must be unique across all rules.

doNotIndex

No (default: false)

When false, the log is both archived and indexed for live search. When true, the log is archived only and excluded from the live search index. Set to true to reduce index storage for logs you only need to access through hydration.

How archives are written

Kloudfuse writes matching logs to the bucket after ingestion, with their facets, labels, and tags applied. Within the bucket, objects are laid out by archive name, then by date (yyyymmdd), then by hour. This layout is the same whether the archive was defined in the UI or in YAML, so you can locate and open a record directly in cloud storage; each line in a decompressed file still carries all of its Kloudfuse attributes.

Hydration

To bring archived logs back into Kloudfuse for search and analysis, run a hydration job. Hydration re-runs the logs through metadata analysis and labeling, applying the current parsing grammar and rules, so older logs stay comparable with your live data.