Filebeat

Kloudfuse accepts logs from Filebeat using the built-in Elasticsearch output plugin. Point the output at the Kloudfuse ingester endpoint and Filebeat delivers log records directly — no Elasticsearch or Logstash installation required.

Overview

Filebeat is a lightweight log shipper from the Elastic stack. It monitors log files and locations you specify, collects log events, and forwards them to a destination using output plugins.

Kloudfuse exposes an ingester endpoint compatible with the Filebeat Elasticsearch output:

https://<kloudfuse-hostname>/ingester/api/v1/filebeat

Redirecting the Elasticsearch output to this URL is the only configuration change required.

Prerequisites

Configure Filebeat

Elasticsearch Output

Set the output.elasticsearch hosts value in your filebeat.yml to the Kloudfuse ingester endpoint:

setup.ilm.enabled: false
setup.template.enabled: false

output.elasticsearch:
  hosts: ["https://<kloudfuse-hostname>:443/ingester/api/v1/filebeat"]
yaml

setup.ilm.enabled: false and setup.template.enabled: false are required — Kloudfuse does not implement the Elasticsearch index management APIs.

Include :443 explicitly in the hosts URL. Filebeat’s Elasticsearch output does not infer the port from the https:// scheme — it defaults to port 9200 if no port is specified, which will fail.

If ingestion authentication is enabled, add the API key as a header:

setup.ilm.enabled: false
setup.template.enabled: false

output.elasticsearch:
  hosts: ["https://<kloudfuse-hostname>:443/ingester/api/v1/filebeat"]
  headers:
    Kf-Api-Key: <token>
yaml

Helm Installation

Filebeat is commonly deployed as a Kubernetes DaemonSet using the elastic/filebeat Helm chart.

  1. Add the Elastic Helm repository:

    helm repo add elastic https://helm.elastic.co
helm repo update

  2. Create a custom-values.yaml file with the Filebeat configuration:

    daemonset:
  # Disable default Elasticsearch TLS cert and credential mounts — not needed for Kloudfuse
  secretMounts: []
  extraEnvs: []

  filebeatConfig:
    filebeat.yml: |
      filebeat.inputs:
        - type: container
          paths:
            - /var/log/containers/*.log
          processors:
            - add_kubernetes_metadata:
                host: ${NODE_NAME}
                matchers:
                  - logs_path:
                      logs_path: "/var/log/containers/"

      setup.ilm.enabled: false
      setup.template.enabled: false

      output.elasticsearch:
        hosts: ["https://<kloudfuse-hostname>:443/ingester/api/v1/filebeat"]
        headers:
          Kf-Api-Key: <token>   # omit if auth is not enabled
    yaml
    Copy across any existing filebeatConfig you already have — the filebeat.yml key is a full file replacement, so anything not included will be lost.
    secretMounts: [] and extraEnvs: [] are required when using the elastic/filebeat Helm chart. The chart’s defaults mount an Elasticsearch TLS certificate secret (elasticsearch-master-certs) and inject Elasticsearch credentials as environment variables (elasticsearch-master-credentials). Neither secret exists when pointing Filebeat at Kloudfuse, so the pods will fail to start without these overrides.

  3. Install or upgrade Filebeat:

    helm upgrade --install filebeat elastic/filebeat \
  -f custom-values.yaml \
  --namespace logging \
  --create-namespace

Use deployment instead of daemonset if Filebeat is configured to run as a Deployment rather than a DaemonSet.

Kubernetes Metadata Enrichment

The add_kubernetes_metadata processor enriches each log event with pod name, namespace, labels, and container name. Add it to any filebeat.inputs block collecting container logs:

filebeat.inputs:
  - type: container
    paths:
      - /var/log/containers/*.log
    processors:
      - add_kubernetes_metadata:
          host: ${NODE_NAME}
          matchers:
            - logs_path:
                logs_path: "/var/log/containers/"
yaml

Kloudfuse automatically uses the kubernetes.container.name field as the log source when it is present.

Ingest Logs from an AWS S3 Bucket

Filebeat’s aws-s3 input type reads log files stored in S3 and forwards them to Kloudfuse. Run Filebeat as a Deployment (not a DaemonSet) for S3 ingestion.

daemonset:
  enabled: false

deployment:
  enabled: true
  filebeatConfig:
    filebeat.yml: |
      filebeat.inputs:
        - type: aws-s3
          number_of_workers: 5
          bucket_arn: "arn:aws:s3:::<your-bucket-name>"
          access_key_id: "<access-key-id>"         (1)
          secret_access_key: "<secret-access-key>" (2)
          endpoint: amazonaws.com
          default_region: "${AWS_REGION:us-east-1}"
          fields:
            source: <log-source-label>             (3)

      setup.ilm.enabled: false
      setup.template.enabled: false

      output.elasticsearch:
        hosts: ["https://<kloudfuse-hostname>:443/ingester/api/v1/filebeat"]
        headers:
          Kf-Api-Key: <token>   # omit if auth is not enabled
yaml
1 Not required when using a node IAM role or IRSA.
2 Not required when using a node IAM role or IRSA.
3 Logs appear under this source name in Kloudfuse.

IAM Permissions for S3

When using a node IAM role or IRSA instead of static credentials, attach the following permissions to the role:

{
  "Effect": "Allow",
  "Action": [
    "s3:ListAllMyBuckets",
    "s3:GetBucketLocation",
    "s3:ListBucket",
    "s3:GetObject"
  ],
  "Resource": [
    "arn:aws:s3:::<your-bucket-name>",
    "arn:aws:s3:::<your-bucket-name>/*"
  ]
}
json

Verify Logs Are Arriving

After restarting Filebeat with the updated configuration, generate some log output, then confirm logs are arriving in the Kloudfuse UI:

  1. Click the Logs tab to open the Logs search view.

  2. In the Filters panel on the left, locate the Source facet and select filebeat, or type filebeat in the search bar to filter by source.

  3. Set the time range to the last 15 minutes using the interval picker in the top-right corner.

  4. Confirm that log entries appear in the list.

  5. Click any log row to open the Detail view. The Facets section displays parsed fields, and the Kubernetes Labels section shows enriched metadata such as namespace, pod name, and container name when the add_kubernetes_metadata processor is enabled.

  6. To narrow results by Kubernetes namespace or container, click the relevant value in the Kubernetes Labels section and choose Filter By.

Troubleshooting

No Logs Appearing in Kloudfuse

If source="filebeat" returns no results after restarting Filebeat:

  1. Check the Filebeat logs for output errors:

    # Systemd:
journalctl -u filebeat --since "5 minutes ago" | grep -iE "error|warn|output"

# Kubernetes:
kubectl logs -n logging -l app=filebeat --tail=50

  2. Confirm the output host is correct — the path must be exactly /ingester/api/v1/filebeat. A missing /ingester/ prefix returns a 404.

  3. Verify the Kloudfuse hostname is reachable from the Filebeat host or pod:

    curl -sf https://<kloudfuse-hostname>/health/ && echo "reachable"

  4. Confirm setup.ilm.enabled: false and setup.template.enabled: false are set. Without these, Filebeat attempts to configure Elasticsearch index management at startup, which fails and may prevent log delivery.

Filebeat Connecting to Port 9200 Instead of 443

If Filebeat logs show connection errors to <kloudfuse-hostname>:9200, the hosts URL is missing an explicit port. Filebeat’s Elasticsearch output does not infer the port from the https:// scheme — it defaults to 9200 when no port is given.

Change:

hosts: ["https://<kloudfuse-hostname>/ingester/api/v1/filebeat"]
yaml

To:

hosts: ["https://<kloudfuse-hostname>:443/ingester/api/v1/filebeat"]
yaml

Helm Pods Failing with "secret not found"

If pods deployed via the elastic/filebeat Helm chart fail with errors like:

secret "elasticsearch-master-certs" not found
secret "elasticsearch-master-credentials" not found

Add the following to your Helm values to disable the chart’s default Elasticsearch credential mounts:

daemonset:
  secretMounts: []
  extraEnvs: []
yaml

401 Unauthorized Errors

If Filebeat logs show HTTP 401 responses from the ingester:

  • Ingestion authentication is enabled on your cluster.

  • Add headers: Kf-Api-Key: <token> to the output.elasticsearch block — see Configure Output.

  • Confirm the token is active in Admin > Settings > Auth key labels in the Kloudfuse UI.

ILM or Template Errors at Startup

If Filebeat logs contain errors like error creating index template or ILM not supported:

  • Ensure setup.ilm.enabled: false and setup.template.enabled: false are present in your filebeat.yml.

  • In Helm deployments, confirm these settings are inside the filebeat.yml: block under filebeatConfig, not at the top-level values.yaml.

S3 Input Not Reading Files

If S3 logs are not appearing in Kloudfuse:

  • Confirm the Filebeat pod is running as a Deployment (not a DaemonSet) — see Configure S3 Input.

  • Verify the IAM role or credentials have s3:GetObject and s3:ListBucket on the target bucket — see Configure S3 IAM.

  • Check the bucket_arn value is correct and in the right region.

  • Increase number_of_workers if the bucket has high log volume.

External References