Fluentd

Kloudfuse accepts logs from Fluentd using the built-in HTTP output plugin. Point the plugin at the Kloudfuse ingester endpoint and Fluentd delivers log records directly — no additional plugins or protocol adapters required.

Overview

Fluentd is an open-source data collector that unifies log collection and routing across diverse sources. It uses a plugin-based architecture: input plugins collect data, filter plugins transform it, and output plugins deliver it to a destination.

Kloudfuse exposes an ingester endpoint compatible with the Fluentd HTTP output plugin:

https://<kloudfuse-hostname>/ingester/v1/fluentd

Kloudfuse supports both json and msgpack format types for the HTTP plugin.

Prerequisites

Configure Fluentd

HTTP Output Plugin

Add a <match> block to your Fluentd configuration to forward logs to Kloudfuse:

<match *>
  @type http

  endpoint https://<kloudfuse-hostname>/ingester/v1/fluentd
  open_timeout 2

  <format>
    @type json
  </format>

  <buffer>
    chunk_limit_size 1048576  # 1 MB
    flush_interval 10s
  </buffer>
</match>
xml

If ingestion authentication is enabled, add the API key as a request header:

<match *>
  @type http

  endpoint https://<kloudfuse-hostname>/ingester/v1/fluentd
  open_timeout 2

  <headers>
    Kf-Api-Key <token>
  </headers>

  <format>
    @type json
  </format>

  <buffer>
    chunk_limit_size 1048576  # 1 MB
    flush_interval 10s
  </buffer>
</match>
xml

Adjust the buffer settings (chunk_limit_size, flush_interval) to match your log volume and latency requirements. The endpoint URI path must remain /ingester/v1/fluentd.

Helm Installations

If Fluentd is installed via a Helm chart (for example, fluent/helm-charts), add the output configuration to your values.yaml:

configMapConfigs:
  - fluentd-prometheus-conf
output:
  type: http
  host: <kloudfuse-hostname>
  path: /ingester/v1/fluentd
  port: 443
  scheme: https
  selfSignedCert: false
yaml

Alternatively, supply the full <match> block as a raw config section in your values file under fileConfigs.

Kubernetes Metadata Enrichment

If your applications run in Kubernetes, use the kubernetes_metadata filter to enrich log records with pod, namespace, and container metadata before forwarding to Kloudfuse.

Install the plugin if not already present:

gem install fluent-plugin-kubernetes_metadata_filter

Add the filter before your <match> block:

<filter *>
  @type kubernetes_metadata
  @id filter_kube_metadata
  skip_labels false
  skip_container_metadata false
  skip_namespace_metadata false
  skip_master_url true
</filter>
xml

See fluent-plugin-kubernetes_metadata_filter for installation and all configuration options.

With this filter enabled, Kloudfuse automatically uses the container_name field from the Kubernetes metadata as the log source.

Label and Tag Extraction

Kloudfuse automatically promotes structured fields from log records into searchable facets and labels. The following sections describe how to customise which fields are used as the log source, message, and additional labels.

Log Source

By default, Kloudfuse uses container_name from the Kubernetes metadata filter as the log source. To use a different field as the source, add the following to the logs-parser section of your Kloudfuse custom-values.yaml:

kf_parsing_config:
  config: |-
    - remap:
        args:
          kf_source:
            - "$.<KEY_FOR_LOG_SOURCE>"  # must be a JSONPath expression
        conditions:
          - matcher: "__kf_agent"
            value: "fluentd"
            op: "=="
yaml

Replace <KEY_FOR_LOG_SOURCE> with the JSONPath key in your log payload that holds the source name.

Log Message

Kloudfuse looks for the log message text under the following keys by default: log, LOG, Log, message, msg, MSG, and Message.

To specify a different key, add the following to the logs-parser section of your Kloudfuse custom-values.yaml:

kf_parsing_config:
  config: |-
    - remap:
        args:
          kf_msg:
            - "$.<MSG_KEY_FROM_AGENT_CONFIG>"  # must be a JSONPath expression
        conditions:
          - matcher: "__kf_agent"
            value: "fluentd"
            op: "=="
yaml

Structured Key-Value Pairs

Fluentd parsers (for example, regexp, json, or csv) can extract key-value pairs from unstructured log lines. Kloudfuse adds all extracted key-value pairs as log facets by default, making them available for filtering in the UI.

To promote a set of keys to log labels and tags instead of facets, specify key prefixes in the logs-parser section of your Kloudfuse custom-values.yaml:

kf_parsing_config:
  config: |-
    - remap:
        args:
          kf_additional_tags:         (1)
            - "$.<PREFIX_KEY_FOR_AGENT_KV>"  # must be a JSONPath expression
        conditions:
          - matcher: "__kf_agent"
            value: "fluentd"
            op: "=="
yaml
1 kf_additional_tags is a list of key prefixes. Any top-level JSON key matching a prefix is stored as a log label/tag rather than a facet.
Keys already handled as log source, message, or Kubernetes metadata are automatically treated as metadata and excluded from facets, regardless of this setting.
When adding label selectors, ensure the prefix patterns do not conflict with existing filter definitions.

Verify Logs Are Arriving

After restarting Fluentd with the updated configuration, generate some log output in your application, then confirm logs are arriving in the Kloudfuse UI:

  1. Click the Logs tab to open the Logs search view.

  2. In the Filters panel on the left, locate the Source facet and select fluentd, or type fluentd in the search bar to filter by source.

  3. Set the time range to the last 15 minutes using the interval picker in the top-right corner.

  4. Confirm that log entries appear in the list.

  5. Click any log row to open the Detail view. The Facets section displays parsed fields including any structured key-value pairs extracted by Fluentd parsers. The Kubernetes Labels section shows enriched metadata such as namespace, pod name, and container name when the kubernetes_metadata filter is enabled.

  6. To narrow results by Kubernetes namespace, container, or log level, click the relevant value in the Kubernetes Labels or Facets section and choose Filter By.

Troubleshooting

No Logs Appearing in Kloudfuse

If source="fluentd" returns no results:

  1. Check the Fluentd logs for output plugin errors:

    # For a systemd-managed Fluentd:
journalctl -u fluentd --since "5 minutes ago" | grep -i "http\|error\|warn"

# For a Kubernetes DaemonSet:
kubectl logs -n <namespace> -l app=fluentd --tail=50

  2. Confirm the endpoint URL is correct — it must be exactly /ingester/v1/fluentd. A missing or incorrect path returns a 404.

  3. Verify the Kloudfuse hostname is reachable from the Fluentd host or pod:

    curl -sf https://<kloudfuse-hostname>/health/ && echo "reachable"

  4. Check that the <match> pattern covers your log tags. Using <match *> matches everything; more specific patterns may inadvertently exclude records.

401 Unauthorized Errors

If Fluentd logs show HTTP 401 responses from the ingester:

  • Ingestion authentication is enabled on your cluster.

  • Add a <headers> block with Kf-Api-Key <token> to your <match> configuration — see Configure HTTP Output.

  • Confirm the token is active in Admin > Settings > Auth key labels in the Kloudfuse UI.

Logs Are Delayed or Buffered

If logs appear in Kloudfuse with significant delay:

  • Decrease flush_interval in the <buffer> block to reduce the delivery window.

  • Decrease chunk_limit_size if individual chunks are large and taking time to fill.

  • Check Fluentd’s buffer queue depth in its metrics or logs — a growing queue indicates the ingester is not keeping up or is temporarily unavailable.

Missing Kubernetes Labels in Kloudfuse

If pod, namespace, or container fields are absent from log records:

  • Confirm the kubernetes_metadata filter is placed before the <match> block in your configuration.

  • Verify the fluent-plugin-kubernetes_metadata_filter gem is installed in your Fluentd image.

  • Ensure the Fluentd service account has RBAC permissions to read pod and namespace resources:

    rules:
  - apiGroups: [""]
    resources: ["pods", "namespaces"]
    verbs: ["get", "list", "watch"]
    yaml

External References