Logs Archive and Hydration

You may have to save transactional information for compliance, legal, or other regulatory requirements. In addition to processing logs for observability and analytics, Kloudfuse introduced a supplementary mechanism for archiving pre-processed logs (with identified filters, facets, and so on) into longer-term storage, and a separate mechanism to hydrate these logs to examine them for the relevant data.

The benefits of this approach extend beyond basic regulatory compliance:

You store important historical data in a cost-effective compressed format in a location that you own and control.
When decompressed, the logs are human-readable and highly searchable because of the high level of indexing through labels and other data attributes.
You can configure the archival instructions in a manner that categorizes data consumption by internal cost center.

We currently support log archive and hydration for AWS S3 and Google Cloud Storage (GCS).

Archiving

Based on your own set of archival rules and configurations, you can add an archive section to the deployments.yaml file to specify the logs you need to write to your own archive storage.

global:
  archive:
    enabled: true
    prefix: "<Example_Cluster/Example_Folder>"    # Optional, can specify as ""
    useSecret: true                               # Security, 4 methods, see note
    createSecret: true
    secretName: "<Example_Secret>"
    type: s3                                      # AWS storage
    s3:
      region: <Example_Region>                    # Such as us-west-2
      bucket: <Example_Bucket_Name>               # You MUST create the bucket in you archival storage location
      accessKey: <Example_Access_Key>
      secretKey: <Example_Secret_Key>
    rules: |-
      - archive:                                  # Define first archive
          args:
            archiveName: a1
            doNotIndex: false                       # Both archive and index
          conditions:
            - matcher: "#source"
              value: "s1"
              op: "=="
            - matcher: "@label"
              value: "l1"
              op: "=="
      - archive:                                  # Define next archive
          args:
            archiveName: a2
            doNotIndex: false                       # Both archive and index
          conditions:
            - matcher: "#source"
              value: "s1"
              op: "=="

yaml

If you only plan to archive the logs, set the doNotIndex variable to true.
If you plan to both archive and review the logs, you must index them; set the doNotIndex variable to false.

The archive rules apply in order, and must match all conditions. In the preceding example, a log line from source “s1” gets mapped to archive a1 if it also contains label l1. If it does not contain label l1, it gets mapped to archive a2.

Archive rules reference

Each rule in the rules block is a YAML list entry with two keys: conditions and args.

conditions

conditions is a list of matchers that must all be satisfied (logical AND) for the rule to apply. If no conditions are specified, the rule matches every log line — use this as a catch-all last rule.

Each condition has three fields:

Field Required Description

Field	Required	Description
`matcher`	Yes	The attribute to match against. Uses a sigil prefix to distinguish attribute types: `#source` — matches the log source name (the originating agent or integration). `@label` — matches a label extracted from the log by the Kloudfuse pipeline (for example, a Kubernetes namespace or pod name).
`op`	Yes	The comparison operator. Currently supported: `==` (exact equality).
`value`	Yes	The string value to compare against the matched attribute.

matcher

Yes

The attribute to match against. Uses a sigil prefix to distinguish attribute types:

#source — matches the log source name (the originating agent or integration).
@label — matches a label extracted from the log by the Kloudfuse pipeline (for example, a Kubernetes namespace or pod name).

op

Yes

The comparison operator. Currently supported: == (exact equality).

value

Yes

The string value to compare against the matched attribute.

Example: match logs from a specific source and namespace

rules: |-
  - archive:
      conditions:
        - matcher: "#source"
          value: "kubernetes"
          op: "=="
        - matcher: "@label"
          value: "production"
          op: "=="
      args:
        archiveName: k8s-production
        doNotIndex: false
  - archive:
      conditions:
        - matcher: "#source"
          value: "kubernetes"
          op: "=="
        archiveName: k8s-all
        doNotIndex: false

yaml

In this example, Kubernetes logs from the production namespace match the first rule and are written to k8s-production. All other Kubernetes logs fall through to the second rule and are written to k8s-all. Non-Kubernetes logs do not match either rule and are not archived.

args

args controls how the matched log is archived.

Field Required Description

Field	Required	Description
`archiveName`	Yes	A short identifier for the archive destination within the storage bucket. Used as a path segment when Kloudfuse writes the files and when you select an archive to hydrate. Must be unique across all rules.
`doNotIndex`	No (default: `false`)	When `false`, the log is both archived and indexed for live search. When `true`, the log is archived only and excluded from the live search index. Set to `true` to reduce index storage for logs you only need to access through hydration.

archiveName

Yes

A short identifier for the archive destination within the storage bucket. Used as a path segment when Kloudfuse writes the files and when you select an archive to hydrate. Must be unique across all rules.

doNotIndex

No (default: false)

When false, the log is both archived and indexed for live search. When true, the log is archived only and excluded from the live search index. Set to true to reduce index storage for logs you only need to access through hydration.

Archive prerequisites

You must grant access for Kloudfuse to write archives into the specified storage. There are four security approaches:

createSecret = true, useSecret=true

The helm creates the k8s secret based on the provider’s accessKey and secretKey. The deployments.yaml file gets the value from secret.
createSecret=false, useSecret=true

The customer creates the k8s secret. The deployments.yaml file works automatically because it picks up the env var from secret.
createSecret=false, useSecret=false

This approach assumes that the customer already configured the node IAM role to have permission to access the S3 bucket, so there is no requirement to set env var.
Customer creates a service account with access permission to S3.

That service account maps to serviceAccountName. There is no requirement to set env var, and the pod inherits the permissions of the service account.

Archive writes

Kloudfuse writes the logs to the specified archive location after ingestion, applying facets, labels, and so on.

Hydration

Whenever you need to examine a record, you can access it directly in your archival storage because of our simple storage and compression rubric: by date (yyyymmdd format), and then by hour. When you decompress and open a log file, you can see all the facets, labels, and other tags attributed to the log by Kloudfuse.

Additionally, you can hydrate the archived logs into Kloudfuse. We run them through the metadata analysis, labeling, and so on. For older logs, this gives us the opportunity to apply the current (newer) set of grammar and rules, making them compatible (and comparable) with current logs.

To submit and manage hydration jobs programmatically, see Hydration API Reference.