AWS CloudWatch Logs Integration

Kloudfuse ingests logs from AWS CloudWatch by subscribing CloudWatch log groups to a Kinesis Firehose delivery stream. Log records are delivered to Kloudfuse in real time as they are written to CloudWatch.

For an overview of how this fits into the broader AWS integration architecture, see AWS Integration Architecture.

Overview

Any AWS service that writes to CloudWatch Logs can be ingested — including Lambda functions, VPC Flow Logs, API Gateway access logs, ECS container logs, and custom application logs forwarded by the CloudWatch Agent.

Data flows as follows:

  1. AWS services write log records to CloudWatch Log Groups.

  2. A subscription filter on each log group forwards records to a Kinesis Firehose delivery stream.

  3. Firehose delivers the records to the Kloudfuse logs ingestion endpoint.

  4. Optionally, the Kloudfuse enrichment scraper attaches AWS resource metadata to the logs as labels.

Prerequisites

Before you begin, ensure the following requirements are in place:

Requirement Details

Kinesis Firehose delivery stream

A Kinesis Firehose delivery stream named kloudfuse-logs pointed at your Kloudfuse logs endpoint. If you have not created this yet, complete AWS Kinesis Firehose Integration first.

IAM role ARN

The ARN of the KloudfuseFirehoseRole created in the Kinesis Firehose setup.

AWS CLI

AWS CLI configured with permissions to manage CloudWatch Logs subscription filters and IAM.

Step 1a: Enable Services to Emit Logs (Optional)

Several AWS services do not write to CloudWatch Logs by default and require per-resource opt-in before a log group exists to subscribe. Affected services include VPC Flow Logs, RDS, EKS control plane logs, CloudTrail, API Gateway access logs, and WAF.

See CloudWatch Enablement Reference — Logs Enablement for enablement instructions for each service.

Step 1: Subscribe Log Groups to Kinesis Firehose

For each CloudWatch log group you want to ingest, create a subscription filter pointing to the kloudfuse-logs Firehose stream.

Subscribe a Single Log Group

aws logs put-subscription-filter \
  --log-group-name "/aws/lambda/my-function" \
  --filter-name "kloudfuse" \
  --filter-pattern "" \
  --destination-arn "arn:aws:firehose:<region>:<account-id>:deliverystream/kloudfuse-logs" \
  --role-arn "arn:aws:iam::<account-id>:role/KloudfuseFirehoseRole"
bash

Set --filter-pattern "" to forward all log records. Use a non-empty pattern if you want to forward only matching records.

Subscribe Multiple Log Groups

To subscribe all log groups matching a prefix:

LOG_GROUPS=$(aws logs describe-log-groups \
  --log-group-name-prefix "/aws/lambda/" \
  --query 'logGroups[*].logGroupName' \
  --output text)

for GROUP in $LOG_GROUPS; do
  aws logs put-subscription-filter \
    --log-group-name "$GROUP" \
    --filter-name "kloudfuse" \
    --filter-pattern "" \
    --destination-arn "arn:aws:firehose:<region>:<account-id>:deliverystream/kloudfuse-logs" \
    --role-arn "arn:aws:iam::<account-id>:role/KloudfuseFirehoseRole"
  echo "Subscribed: $GROUP"
done
bash
Each log group can have at most two subscription filters. If a log group already has two filters, you must remove one before adding the Kloudfuse filter.

Step 2: Enable Logs Enrichment

Logs enrichment attaches AWS resource metadata — such as instance type, availability zone, environment tags, and custom resource tags — to incoming CloudWatch Logs. Without enrichment, Logs only carry the default CloudWatch dimensions (for example, InstanceId). With enrichment enabled, every metric also carries any tag you have applied to that resource in AWS.

Configure IAM for Enrichment

Create an IAM policy that grants the enrichment scraper read access to AWS resource metadata:

cat > kloudfuse-enrichment-policy.json <<'EOF'
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ec2:DescribeInstances",
        "ec2:DescribeImages",
        "ec2:DescribeRegions",
        "ec2:DescribeSnapshots",
        "ec2:DescribeSecurityGroups",
        "ec2:DescribeVolumes",
        "ec2:DescribeVpcs",
        "ec2:DescribeSubnets",
        "ec2:DescribeTags",
        "ecs:ListClusters",
        "ecs:ListContainerInstances",
        "ecs:DescribeContainerInstances",
        "rds:DescribeDBInstances",
        "rds:ListTagsForResource",
        "lambda:ListFunctions",
        "lambda:ListTags",
        "logs:DescribeLogGroups",
        "logs:DescribeLogStreams",
        "logs:ListTagsLogGroup",
        "tag:GetResources",
        "tag:GetTagKeys",
        "tag:GetTagValues"
      ],
      "Resource": "*"
    }
  ]
}
EOF

aws iam create-policy \
  --policy-name KloudfuseEnrichmentPolicy \
  --policy-document file://kloudfuse-enrichment-policy.json
bash

Attach the policy to either an IAM user (for access key authentication) or an IAM role (for role-based authentication).

Configure Kloudfuse via Helm

See Kloudfuse Enrichment for the full Helm configuration, including AWS credential options, enrichment values, and the Helm upgrade command.

Enrichment Reference

The enrichment scraper attaches metadata labels to metrics, by namespace. The table below shows a representative sample of common namespaces:

Namespace Labels added

AWS/EC2

instance_id, instance_type, availability_zone, region, image_id, vpc_id, plus all instance tags

AWS/RDS

db_instance_identifier, db_instance_class, engine, availability_zone, plus all RDS tags

AWS/ECS

cluster_name, container_instance_id, availability_zone, plus all ECS tags

AWS/Lambda

function_name, runtime, memory_size, plus all Lambda tags

AWS/DynamoDB

table_name, plus all DynamoDB tags

AWS/ElastiCache

cache_cluster_id, cache_node_type, engine, plus all ElastiCache tags
This is a sample only. For the complete list of enriched labels and IAM permissions for supported AWS services, see AWS Supported Services.

Step 3: Verify the Integration

Confirm Subscription Filters are Active

aws logs describe-subscription-filters \
  --log-group-name "/aws/lambda/my-function" \
  --query 'subscriptionFilters[*].{Filter:filterName,Destination:destinationArn,State:distribution}'
bash

Confirm Logs are being Sent

  1. In the AWS Console, open Amazon Kinesis and select your kloudfuse-logs delivery stream.

  2. Click the Monitoring tab and check the IncomingRecords and DeliveryToHTTPEndpoint.Success graphs. Both should show non-zero values within a few minutes of a log event being written.

  3. If DeliveryToHTTPEndpoint.Success is zero but IncomingRecords is non-zero, check DeliveryToHTTPEndpoint.DataFreshness for delivery errors and confirm the Kloudfuse ingester endpoint URL and API key are correct in the Firehose configuration.

  4. To confirm a specific log group is subscribed, run:

    aws logs describe-subscription-filters \
  --log-group-name "/aws/lambda/my-function" \
  --query 'subscriptionFilters[*].{Filter:filterName,Destination:destinationArn}'
    bash

    The destinationArn must match the kloudfuse-logs delivery stream ARN.

Confirm Logs are Arriving in Kloudfuse

  1. In the Kloudfuse UI, click the Logs tab and select Search from the drop-down.

  2. Set the time picker to the last 15 minutes.

  3. In the search bar, click Advanced Search and enter the following FuseQL query to confirm logs are flowing from CloudWatch:

    source="cloudwatch"
    fuseql

    Any result confirms that CloudWatch log records are reaching Kloudfuse.

  4. To filter by a specific log group, use the logGroup field:

    logGroup="/aws/lambda/my-function"
    fuseql

  5. To count log volume by log group across all CloudWatch sources:

    source="cloudwatch" | count by logGroup
    fuseql

References