Routing Logs to Additional Endpoints

Kloudfuse integrates with the Fluentbit agent to ingest logs from the Kinesis Firehose data stream. This enables you to forward logs to a different datastore or backend, instead of just Kloudfuse. For example, you could examine logs through ElasticSearch using the appropriate configuration on Fluentbit.

Architecture

Kloudfuse Integrates with Fluentbit Agent to stream data to an ElasticSearch Database

Configure and define log streams in AWS Kinesis Firehose data stream, and connect it them to an AWS CloudWatch subscription.

See AWS CloudWatch Logs Integration.
The Kloudfuse stack receives the log data stream, and process it:
1. The Nginx load balancer forwards the incoming data to the Kloudfuse Ingester service.
2. The Ingester, a Kubernetes instance within the Kloudfuse stack, unpacks the data and streams individual log events to the Fluentbit service.
3. Fluentbit, deployed with an HTTP plugin within the Kloudfuse stack, accepts the incoming traffic within the Kubernetes cluster.
  
  See Fluentbit documentation on HTTP.
4. Subsequently, Fluentbit forwards the log events to the ElasticSearch cluster using the ES Plugin.
  
  See Fluentbit documentation on Elasticsearch.

Configuration

Enable Fluentbit Deployment

The Kloudfuse stack independently supports integration with the Fluentbit agent. See Fluent Bit API for Ingesting Custom Logs.

Use the following configuration, specific to supporting Kloudfuse integration for ElasticSearch, to enable the ingester to forward traffic:

Add this code to the global values section in the custom-values.yaml file.
```
  kfuse-fbit:
    enabled: true
```
Add this code to the ingester configuration section in your custom-values.yaml file.
```
ingester:
  config:
    logs:
      msgFormat: fluent-bit (1)
```
1 msgFormat: Possible values are kfuse (default), `fluent-bit, or all (forwards logs to both Kloudfuse backend and ElasticSearch backend).

Add this code to enable Fluentbit deployment.

Be sure to replace the placeholders, <…>, with appropriate values.

fluent-bit:
  affinity: {} (1)
  tolerations: [] (2)
  config:
    inputs: |-
      [INPUT]
          Name   http
          listen 0.0.0.0
          # default port is 9880. If you want to override it, uncomment line
          # below, update the value
          # port 9880
    outputs: |-
      [OUTPUT]
          Name es
          Match *
          Host <ES_HOST>
          Index <ES_INDEX_NAME>
          HTTP_User <ES_USER>
          HTTP_Passwd <ES_PASSWD>
          Retry_Limit False
          compress gzip
          Suppress_Type_Name On
  extraPorts: (3)
  - containerPort: 9880
    name: in-http
    port: 9880
    protocol: TCP
    targetPort: 9880
  imagePullSecrets:
  - name: kfuse-image-pull-credentials
  kind: Deployment

1	`affinity`: Add any affinity rules if affinity is set up in your Kubernetes cluster.
2	`tolerations`: Add any tolerations that are enabled in your Kubernetes cluster.
3	`extraPorts`: Ensure that the port matches the port definition in the input HTTP plugin. If using the default value, don’t change anything.

Routing Rules

The Fluentbit agent routes log events based on tag values; see Fluentbit documentation on Tags.

To forward logs to a different ElasticSearch host or index, use the rewrite_tag filter; see Fluentbit documentation on the Rewrite Tag filter. You can add as many rewrite_tag plugins as necessary for your business scenario.

When you apply the rule defined by the rewrite_tag filter, the Fluentbit service re-ingests the log event from the beginning of the pipeline, using the new tag.

Manage Fields: Add, Remove, Modify

Fluentbit enables you to add and remove fields using the record_modifier filter. Similarly to using the rewrite_tag filter, you can add and define as many filters as you want.

See Fluentbit documentation on Record Modifier.

Monitor

Fluentbit can also report Prometheus-style metrics. To enable the collection and scraping of these metrics and ingest them into the Kloudfuse stack, add the following configuration to your custom_values.yaml file.

Fluentbit publishes metrics across various input, filter, and output plugins. This includes the number of records processed and dropped for each plugin.

This configuration contains some common keys that are shared with the basic configuration; ensure that you merge them correctly.

config:
  inputs: |-
    [INPUT]
        Name fluentbit_metrics
        Tag internal_metrics
        scrape_interval 15
  outputs: |-
    [OUTPUT]
        Name prometheus_exporter
        match internal_metrics
        host 0.0.0.0
        port 8080

extraPorts:
  - containerPort: 8080
    name: out-metrics
    port: 8080
    protocol: TCP
    targetPort: 8080

podAnnotations:
  prometheus.io/path: /metrics
  prometheus.io/port: "8080"
  prometheus.io/scrape: "true"

See Fluentbit documentation on Monitoring.