Fluent Bit for Kinesis Firehose Datastream

You can configure the Fluent Bit agent to ingest logs from Kinesis Firehose datastream directly into Kloudfuse. This integration forwards logs to a different datastore/backend, not Kloudfuse. For example, you may want to forward logs to Elastic search.

Architecture

The following figure represents high-level architecture of the Kloudfuse-FluentBit integration.

These are the integration components:

Configure and define log streams in AWS Kinesis Firehose data stream, and connect it to an AWS CloudWatch subscription. See Configure AWS Kinesis Firehose.
The nginx load balancer (deployed as part of Kloudfuse stack) forwards the incoming data to the Ingester service.
The Ingester deploys as a Kubernetes deployment within the Kloudfuse stack. It unpacks the data and streams individual log events to a Fluent-bit service, also deployed as part of the Kloudfuse stack.
Fluent-bit deploys within the Kloudfuse stack, with an HTTP plugin (see Fluent Bit HTTP documentation) to accept incoming traffic locally within the Kubernetes cluster.
Fluent-bit subsequently forwards the incoming log events to the ElasticSearch cluster, using the ES plugin; see Fluent Bit documentation for Elasticsearch.

Configuration

Consider your specific usage scenario when configuring this integration. See these sections:

Enable Fluent Bit Deployment
Routing Rules
Add, Remove, and Modify Fields
Monitor

Enable Fluent Bit Deployment

The Kloudfuse stack supports integration with Fluent Bit agent. These steps illustrate the specific scenario of supporting Kloudfuse integration for Elasticsearch using Fluent Bit, to configure the ingested to forward traffic to Elasticsearch.

Add the following kfuse-fbit section to global values in your custom-values.yaml file.
```
  kfuse-fbit:
    enabled: true
```
yaml

Add the following code to the ingester configuration in your custom-values.yaml file.

ingester:
  config:
    logs:
      msgFormat: fluent-bit
    # Possible value for 'msgFormat' are 'kfuse', 'fluent-bit' or 'all'
    # The default value is 'kfuse'. 'all' will forward logs to both Kloudfuse
    # backend and Elasticsearch backend.

yaml

Enable Fluent Bit deployment.

fluent-bit:
  # Add any affinity rules, if you have affinity setup in your Kubernetes cluster.
  affinity: {}
  # Add any tolerations, if you have them enabled in your Kubernetes cluster.
  tolerations: []
  config:
    inputs: |-
      [INPUT]
          Name   http
          listen 0.0.0.0
          # default port is 9880. If you want to override it, uncomment line
          # below, update the value
          # port 9880
    outputs: |-
      [OUTPUT]
          Name es
          Match *
          Host <ES_HOST> (1)
          Index <ES_INDEX_NAME> (2)
          HTTP_User <ES_USER> (3)
          HTTP_Passwd <ES_PASSWD> (4)
          Retry_Limit False
          compress gzip
          Suppress_Type_Name On
  extraPorts:
  # ensure that this port matches the port defined in the input HTTP plugin.
  # If you're going with the default value, you don't need to change anything
  # below.
  - containerPort: 9880
    name: in-http
    port: 9880
    protocol: TCP
    targetPort: 9880
  imagePullSecrets:
  - name: kfuse-image-pull-credentials
  kind: Deployment

yaml

1	Specify Elasticsearch `host`.
2	Specify Elasticsearch `index name`.
3	Specify Elasticsearch `user`.
4	Specify Elasticsearch `password`.

Routing Rules

The Fluent Bit agent routes log events based on tag values; see Fluent Bit documentation on Tags.

To forward logs to different Elasticsearch hosts or indices, use the rewrite_tag filter; see Fluent Bit documentation on Rewrite Tag. Use as many rewrite_tag plugins as necessary for your specific situation.

After applying a new rule defined in the rewrite_tag plugin, Fluent bit re-ingests the log event with the new tag from the beginning of the pipeline.

Be careful not to write multiple rules that lead to an infinite loop and cause runtime issues.

Add, Remove, and Modify Fields

To add, remove, and modify fields, use the records_modifier filter, as documented by Fluent Bit in Record Modifier. Use as many records_modifier plugins as necessary for your specific situation.

Monitor

You can also use Fluent Bit to report prometheus-style metrics. To enable collection and scraping of these metrics and ingest them into the Kloudfuse stack, add the following configuration to your custom_values.yaml file.

Fluent-bit publishes metrics across various input, filter, and output plugins, including the number of records processed and dropped for each plugin. See Fluent Bit documentation on Monitoring.

This configuration uses common keys that are shared with the basic configuration. Make sure that you merge them properly.

config:
  inputs: |-
    [INPUT]
        Name fluentbit_metrics
        Tag internal_metrics
        scrape_interval 15
  outputs: |-
    [OUTPUT]
        Name prometheus_exporter
        match internal_metrics
        host 0.0.0.0
        port 8080

extraPorts:
  - containerPort: 8080
    name: out-metrics
    port: 8080
    protocol: TCP
    targetPort: 8080

podAnnotations:
  prometheus.io/path: /metrics
  prometheus.io/port: "8080"
  prometheus.io/scrape: "true"

yaml