Ingest logs from AWS S3 bucket using Filebeat

Use the filebeat agent and configure it to ingest log files from an AWS S3 bucket into Kloudfuse.

Prerequisites

Complete the installation and configuration of the Filebeat agent; see Filebeat Configuration.

Configure Log Ingestion

daemonset:
  enabled: false
deployment:
  enabled: true
  filebeatConfig:
    filebeat.yml: | (1)
      filebeat.inputs:
      - type: aws-s3 (2)
        number_of_workers: 5
        bucket_arn: '<<TBA s3 bucket ARN>>'
        access_key_id: '<<TBA>>' (3)
        secret_access_key: '<<TBA>>' (4)
        endpoint: amazonaws.com
        default_region: "${AWS_REGION:<<TBA>>}"
        fields:
          source: <<TBA Define the source for import> # The logs will be visible with the same source.
      setup.ilm.enabled: false
      setup.template.enabled: false
      output.elasticsearch:
        hosts: ["https://<<TBA Kloudfuse Endpoint>>:443/ingester/api/v1/filebeat"]

yaml

1	Specify Filebeat agent configuration file.
2	Specify the AWS-S3 source bucket.
3	Not needed in case of Node IAM Role
4	Not needed in case of Node IAM Role

If you are ingesting from an S3 bucket with Node IAM role, you must add the following permissions to the configured policy:

				"s3:ListAllMyBuckets",
				"s3:GetBucketLocation",
				"s3:ListBucket",
				"s3:GetObject"

yaml

After it is configured, the Filebeat agent imports the logs in the specified Kloudfuse endpoint.