AWS CloudWatch Logs Integration

Kloudfuse ingests logs from AWS CloudWatch by subscribing CloudWatch log groups to a Kinesis Firehose delivery stream. Log records are delivered to Kloudfuse in real time as they are written to CloudWatch.

For an overview of how this fits into the broader AWS integration architecture, see AWS Integration Architecture.

Overview

Any AWS service that writes to CloudWatch Logs can be ingested — including Lambda functions, VPC Flow Logs, API Gateway access logs, ECS container logs, and custom application logs forwarded by the CloudWatch Agent.

Data flows as follows:

  1. AWS services write log records to CloudWatch Log Groups.

  2. A subscription filter on each log group forwards records to a Kinesis Firehose delivery stream.

  3. Firehose delivers the records to the Kloudfuse logs ingestion endpoint.

  4. Optionally, the Kloudfuse enrichment scraper attaches AWS resource metadata to the logs as labels.

Prerequisites

  • A Kinesis Firehose delivery stream named kloudfuse-logs pointed at your Kloudfuse logs endpoint. If you have not created this yet, complete Configure AWS Kinesis Firehose for Kloudfuse first.

  • The ARN of the KloudfuseFirehoseRole created in the Kinesis Firehose setup.

  • AWS CLI configured with permissions to manage CloudWatch Logs subscription filters and IAM.

Step 1: Subscribe Log Groups to Kinesis Firehose

For each CloudWatch log group you want to ingest, create a subscription filter pointing to the kloudfuse-logs Firehose stream.

Subscribe a Single Log Group

aws logs put-subscription-filter \
  --log-group-name "/aws/lambda/my-function" \
  --filter-name "kloudfuse" \
  --filter-pattern "" \
  --destination-arn "arn:aws:firehose:<region>:<account-id>:deliverystream/kloudfuse-logs" \
  --role-arn "arn:aws:iam::<account-id>:role/KloudfuseFirehoseRole"
bash

Set --filter-pattern "" to forward all log records. Use a non-empty pattern if you want to forward only matching records.

Subscribe Multiple Log Groups

To subscribe all log groups matching a prefix:

LOG_GROUPS=$(aws logs describe-log-groups \
  --log-group-name-prefix "/aws/lambda/" \
  --query 'logGroups[*].logGroupName' \
  --output text)

for GROUP in $LOG_GROUPS; do
  aws logs put-subscription-filter \
    --log-group-name "$GROUP" \
    --filter-name "kloudfuse" \
    --filter-pattern "" \
    --destination-arn "arn:aws:firehose:<region>:<account-id>:deliverystream/kloudfuse-logs" \
    --role-arn "arn:aws:iam::<account-id>:role/KloudfuseFirehoseRole"
  echo "Subscribed: $GROUP"
done
bash
Each log group can have at most two subscription filters. If a log group already has two filters, you must remove one before adding the Kloudfuse filter.

Step 2: Enable Log Enrichment (Optional)

Log enrichment adds AWS resource metadata — such as EC2 instance tags, ECS cluster names, and RDS instance identifiers — as labels to ingested log records. This allows you to filter logs in Kloudfuse by any tag applied to your AWS resources.

Configure IAM for Enrichment

Create an IAM policy that grants the enrichment scraper read access to AWS resource metadata:

cat > kloudfuse-enrichment-policy.json <<'EOF'
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ec2:DescribeInstances",
        "ec2:DescribeImages",
        "ec2:DescribeRegions",
        "ec2:DescribeSnapshots",
        "ec2:DescribeSecurityGroups",
        "ec2:DescribeVolumes",
        "ec2:DescribeVpcs",
        "ec2:DescribeSubnets",
        "ec2:DescribeTags",
        "ecs:ListClusters",
        "ecs:ListContainerInstances",
        "ecs:DescribeContainerInstances",
        "rds:DescribeDBInstances",
        "rds:ListTagsForResource",
        "lambda:ListFunctions",
        "lambda:ListTags",
        "logs:DescribeLogGroups",
        "logs:DescribeLogStreams",
        "logs:ListTagsLogGroup",
        "tag:GetResources",
        "tag:GetTagKeys",
        "tag:GetTagValues"
      ],
      "Resource": "*"
    }
  ]
}
EOF

aws iam create-policy \
  --policy-name KloudfuseEnrichmentPolicy \
  --policy-document file://kloudfuse-enrichment-policy.json
bash

Attach the policy to either an IAM user (for access key authentication) or an IAM role (for role-based authentication).

Configure Kloudfuse via Helm

See Configure Kloudfuse for AWS Enrichment for the full Helm configuration, including AWS credential options, enrichment values, and the Helm upgrade command.

Verify the Integration

Confirm Subscription Filters are Active

aws logs describe-subscription-filters \
  --log-group-name "/aws/lambda/my-function" \
  --query 'subscriptionFilters[*].{Filter:filterName,Destination:destinationArn,State:distribution}'
bash

Confirm Logs are being Sent

  1. In the AWS Console, open Amazon Kinesis and select your kloudfuse-logs delivery stream.

  2. Click the Monitoring tab and check the IncomingRecords and DeliveryToHTTPEndpoint.Success graphs. Both should show non-zero values within a few minutes of a log event being written.

  3. If DeliveryToHTTPEndpoint.Success is zero but IncomingRecords is non-zero, check DeliveryToHTTPEndpoint.DataFreshness for delivery errors and confirm the Kloudfuse ingester endpoint URL and API key are correct in the Firehose configuration.

  4. To confirm a specific log group is subscribed, run:

    aws logs describe-subscription-filters \
  --log-group-name "/aws/lambda/my-function" \
  --query 'subscriptionFilters[*].{Filter:filterName,Destination:destinationArn}'
    bash

    The destinationArn must match the kloudfuse-logs delivery stream ARN.

Confirm Logs are Arriving in Kloudfuse

  1. In the Kloudfuse UI, click the Logs tab and select Search from the drop-down.

  2. Set the time picker to the last 15 minutes.

  3. In the search bar, click Advanced Search and enter the following FuseQL query to confirm logs are flowing from CloudWatch:

    source="cloudwatch"
    fuseql

    Any result confirms that CloudWatch log records are reaching Kloudfuse.

  4. To filter by a specific log group, use the logGroup field:

    logGroup="/aws/lambda/my-function"
    fuseql

  5. To count log volume by log group across all CloudWatch sources:

    source="cloudwatch" | count by logGroup
    fuseql