Fluentd integration

Fluentd is an open source data collector that unifies data collection and consumption for a better use and understanding of data.

See Fluentd Documentation.

Integrating Fluentd with Kloudfuse Stack

See Fluentd documentation on Installation.

To configure Fluentd to send data to Kloudfuse stack, you must modify the agent’s config, or values.yaml if you install using helm.

You must configure HTTP output plugin to forward data to Kloudfuse.

HTTP Plugin Integration

Add the configuration in Configure HTTP Plugin to Fluentd agent’s config for HTTP configuration.

You can change this example buffer configuration to suit your application needs. However, you cannot change the end-point URI path.

Kloudfuse supports json and msgpack types for Fluentd HTTP plugin.

If you configured external ingress with TLS for Kloudfuse, use https://<KFUSE_INGESTER_IP>:443/ingester/v1/fluentd as the endpoint, instead.

Configure HTTP Plugin
<match *> # Match everything
  @type http

  endpoint http://<KFUSE_INGESTER_IP>:80/ingester/v1/fluentd
  open_timeout 2

  <format>
    @type json
  </format>
  <buffer>
    chunk_limit_size 1048576 # 1MB
    flush_interval 10s
  </buffer>
</match>

Configuration

Kloudfuse UI enables you to filter log events based on log labels and tags. Find the label selectors and filter on the left navigation bar.

When adding recommended filters, ensure that they don’t conflict with existing filter definitions.

To have a seamless experience with Kloudfuse while using Fluentd agent, we recommend the following configuration and customizations:

Kubernetes Labels

Use the kubernetes filter to enrich the log events using Kubernetes metadata. See Fluentd download and documentation for fluent-plugin-kubernetes_metadata_filter.

If your applications deploy in a Kubernetes environment, be sure to enable this filter.

Here’s an example configuration:

Extract Kubernetes Labels
  <filter *>  # Match everything
    @type kubernetes_metadata
    @id filter_kube_metadata_catalog
    skip_labels false
    skip_container_metadata false
    skip_namespace_metadata false
    skip_master_url true
  </filter>

Log source

By default, Kloudfuse looks for the container_name in the message payload, as the log source. This information is present only if the agent operates with a kubernetes filter. To use a different key as the log source, include the code in Example Extract Log Source Label under the logs-parser section in the Kloudfuse custom-values.yaml file.

Extract Log Source Label
kf_parsing_config:
  config: |-
    - remap:
        args:
          kf_source:
            - "$.<KEY_FOR_LOG_SOURCE>" # must be JSONPath
        conditions:
          - matcher: "__kf_agent"
            value: "fluentd"
            op: "=="

Log message

The Fluent Bit agent includes the log event message key in the payload. Occasionally, the agent configuration overwrites it.

Kloudfuse looks in the payload for log message with the following key names, by default: log, LOG, Log, message, msg, MSG, and Message.

Customize the key that the Kloudfuse Platform gets to easily tag log event messages by including the code in Example Extract Log Message Label following section under the logs-parser section in the Kloudfuse custom-values.yaml file.

Extract Log Message Label
kf_parsing_config:
  config: |-
    - remap:
        args:
          kf_msg:
            - "$.<MSG_KEY_FROM_AGENT_CONFIG>" # must be JSONPath
        conditions:
          - matcher: "__kf_agent"
            value: "fluentd"
            op: "=="

Agent extracted key-value pairs

Fluentd supports parsers that extract key-value pairs from an unstructured log; see Fluentd documentation on Parser Plugins.

By default, Kloudfuse adds all key-value pairs to log facets, which can be filtered in the UI.

Kloudfuse cannot differentiate between these key-value pairs and metadata fields added by any filter other than kubernetes. However, you can customize the Kloudfuse stack by adding a list of prefix labels. To accomplish this, include the following section under the logs-parser section in your Kloudfuse custom-values.yaml file.

Any keys specified in log_source, message, or metadata fields from kubernetes filter are automatically treated as metadata, and not included as a log facet.
Extract Static Custom Labels as Labels and Tags
kf_parsing_config:
  config: |-
    - remap:
        args:
          kf_additional_tags: (1)
            - "$.<PREFIX_KEY_FOR_AGENT_KV>" # must be JSONPath
        conditions:
          - matcher: "__kf_agent"
            value: "fluentd"
            op: "=="
1 kf_additional_tags is a list of key prefixes. Any key that is a prefix matches at the top level json gets included as a log label/tag, and not included as a log facet.