Fluent Bit API for Ingesting Custom Logs

Kloudfuse can ingest logs from Fluent Bit API agent payloads. You would have to add rules and custom configurations in custom-values.yaml in Kloudfuse.

Basic Payload

A logs payload from Fluent Bit API looks similar to the following example:

Ingesting Fluent Bit API Logs Payload

 [
  {
    'k1':'foo',
    'k2': 'bar'
    'source': 'app1',
    'msg': json.dumps(parsed_payload[0]),
    'date': time.time()
  },
  {
    'k1':'baz',
    'k2': 'bat',
    'source': 'app2'',
    'msg': json.dumps(parsed_payload[1]), (1)
    'date': time.time() (2)
  }
]

1	The `'msg'` key contains the actual log line.
2	The `'date'` key contains the timestamp to attach to the log line.

Enrich Logs During Ingestion

You can configure Kloudfuse ingestion to extract and enrich logs. Consider the following logs-parser rule that instructs Kloudfuse to extract important attributes from the JSON code:

Configure Log Enrichment

 logs-parser
   parsing_config:
    - remap: (1)
        args:
          kf_source: (2)
            - "$.source"
          kf_additional_tags: $ (3)
            - "$.k1"
            - "$.k2"

1	The `remap` rule instructs Kloudfuse to implement the following details.
2	Extracts from the `source` field in the incoming JSON payload.
3	Extracts `k1` and `k2` as labels from the JSON payload.

Attach Labels from HTTP Headers

You can send custom labels as HTTP headers, along with the incoming payload. Consider the following ingester config:

ingester:
  config:
    labelsFromHeaders: (1)
      X-KFUSE-ENV: "env"
      X-KFUSE-CUSTOM: "custom_key_1"

1	Informs kloudfuse to look for `X-KFUSE-ENV` and `X-KFUSE-CUSTOM` in http headers of incoming logs.

When Kloudfuse finds the matching headers, it renames the labels inside them to the new name, and attaches the values that delivered by the payload. In this example, the header X-KFUSE-ENV': 'prod will be renamed to the label env with value prod, env:prod.