AWS CloudWatch Metrics Integration

Kloudfuse ingests AWS CloudWatch metrics by connecting a CloudWatch Metrics Stream to a Kinesis Firehose delivery stream. Metrics are forwarded continuously to Kloudfuse as they are published by AWS services.

For an overview of how this fits into the broader AWS integration architecture, see AWS Integration Architecture.

Overview

A CloudWatch Metrics Stream pushes metrics from one or more AWS namespaces into a Kinesis Firehose delivery stream, which delivers them to the Kloudfuse metrics ingestion endpoint. Kloudfuse supports metrics from over 40 AWS service namespaces. See AWS Supported Services for the full list.

Prerequisites

  • A Kinesis Firehose delivery stream named kloudfuse-metrics pointed at your Kloudfuse metrics endpoint. If you have not created this yet, complete Configure AWS Kinesis Firehose for Kloudfuse first.

  • AWS CLI configured with permissions to create CloudWatch Metrics Streams.

Step 1: Create a CloudWatch Metrics Stream

Create a metrics stream that forwards all namespaces to the Firehose stream:

aws cloudwatch put-metric-stream \
  --name kloudfuse-metrics-stream \
  --firehose-arn "arn:aws:firehose:<region>:<account-id>:deliverystream/kloudfuse-metrics" \
  --role-arn "arn:aws:iam::<account-id>:role/KloudfuseFirehoseRole" \
  --output-format opentelemetry1.0
bash

To forward only specific namespaces, add --include-filters:

aws cloudwatch put-metric-stream \
  --name kloudfuse-metrics-stream \
  --firehose-arn "arn:aws:firehose:<region>:<account-id>:deliverystream/kloudfuse-metrics" \
  --role-arn "arn:aws:iam::<account-id>:role/KloudfuseFirehoseRole" \
  --output-format opentelemetry1.0 \
  --include-filters '[
    {"Namespace": "AWS/EC2"},
    {"Namespace": "AWS/RDS"},
    {"Namespace": "AWS/Lambda"},
    {"Namespace": "AWS/ApplicationELB"}
  ]'
bash

Step 2: Enable Additional Metrics (Optional)

Some AWS services require extra configuration to publish metrics.

AutoScaling Group Metrics

AutoScaling group metrics are not enabled by default. Enable them for each group you want to monitor:

aws autoscaling enable-metrics-collection \
  --auto-scaling-group-name my-asg \
  --granularity "1Minute"
bash

To enable for all AutoScaling groups in a region:

aws autoscaling describe-auto-scaling-groups \
  --query 'AutoScalingGroups[*].AutoScalingGroupName' \
  --output text | tr '\t' '\n' | while read ASG; do
    aws autoscaling enable-metrics-collection \
      --auto-scaling-group-name "$ASG" \
      --granularity "1Minute"
    echo "Enabled metrics for: $ASG"
  done
bash

S3 Request Metrics

S3 request metrics are charged separately and must be enabled per bucket:

aws s3api put-bucket-metrics-configuration \
  --bucket my-bucket \
  --id kloudfuse-metrics \
  --metrics-configuration '{"Id": "kloudfuse-metrics"}'
bash

Step 3: Enable Metrics Enrichment

Metrics enrichment attaches AWS resource metadata — such as instance type, availability zone, environment tags, and custom resource tags — to incoming CloudWatch metrics. Without enrichment, metrics only carry the default CloudWatch dimensions (for example, InstanceId). With enrichment enabled, every metric also carries any tag you have applied to that resource in AWS.

Configure IAM for Enrichment

Create an IAM policy for the enrichment scraper:

cat > kloudfuse-enrichment-policy.json <<'EOF'
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ec2:DescribeInstances",
        "ec2:DescribeImages",
        "ec2:DescribeRegions",
        "ec2:DescribeSnapshots",
        "ec2:DescribeSecurityGroups",
        "ec2:DescribeVolumes",
        "ec2:DescribeVpcs",
        "ec2:DescribeSubnets",
        "ec2:DescribeTags",
        "ecs:ListClusters",
        "ecs:ListContainerInstances",
        "ecs:DescribeContainerInstances",
        "elasticache:DescribeCacheClusters",
        "elasticache:ListTagsForResource",
        "rds:DescribeDBInstances",
        "rds:ListTagsForResource",
        "lambda:ListFunctions",
        "lambda:ListTags",
        "dynamodb:ListTables",
        "dynamodb:DescribeTable",
        "dynamodb:ListTagsOfResource",
        "logs:DescribeLogGroups",
        "logs:DescribeLogStreams",
        "logs:ListTagsLogGroup",
        "tag:GetResources",
        "tag:GetTagKeys",
        "tag:GetTagValues"
      ],
      "Resource": "*"
    }
  ]
}
EOF

aws iam create-policy \
  --policy-name KloudfuseEnrichmentPolicy \
  --policy-document file://kloudfuse-enrichment-policy.json
bash

Configure Kloudfuse via Helm

See Configure Kloudfuse for AWS Enrichment for the full Helm configuration, including AWS credential options, namespace and region filtering, and the Helm upgrade command.

Enrichment Reference

The enrichment scraper attaches metadata labels to metrics, by namespace. The table below shows a representative sample of common namespaces:

Namespace Labels added

AWS/EC2

instance_id, instance_type, availability_zone, region, image_id, vpc_id, plus all instance tags

AWS/RDS

db_instance_identifier, db_instance_class, engine, availability_zone, plus all RDS tags

AWS/ECS

cluster_name, container_instance_id, availability_zone, plus all ECS tags

AWS/Lambda

function_name, runtime, memory_size, plus all Lambda tags

AWS/DynamoDB

table_name, plus all DynamoDB tags

AWS/ElastiCache

cache_cluster_id, cache_node_type, engine, plus all ElastiCache tags
This is a sample only. For the complete list of enriched labels for every supported AWS service, see Namespace/Service Reference.

Limit Namespace and Region Scope

By default, the enrichment scraper fetches metadata from all supported namespaces in all regions. To reduce API call volume and cost, restrict the scope using the namespaces and regions lists in the Helm values shown in Configure Kloudfuse via Helm.

You can also limit which namespaces are included in the CloudWatch Metrics Stream itself (see Step 1) to reduce the volume of metrics forwarded to Kloudfuse.

Reduce Ingestion Costs

CloudWatch charges for metrics streams based on the number of metric updates forwarded. To reduce costs:

  • Use --include-filters when creating the metrics stream to forward only the namespaces you need.

  • Set namespaces in the Helm awsScraper config to match the namespaces in your stream.

  • Disable AutoScaling and S3 request metrics unless you specifically need them.

  • Use regions filtering to scrape only the regions where your workloads run.

Verify the Integration

Confirm the Metrics Stream is Running

aws cloudwatch get-metric-stream \
  --name kloudfuse-metrics-stream \
  --query '{State: State, OutputFormat: OutputFormat, CreationDate: CreationDate}'
bash

The State should be RUNNING.

  • Check the Firehose delivery stream monitoring for delivery errors (see Troubleshooting).

Confirm Metrics are Arriving in Kloudfuse

  1. In the Kloudfuse UI, click the Metrics tab and select Explorer from the drop-down.

  2. In the metric search field, type aws. to list all ingested CloudWatch metrics. Metrics follow the naming convention aws.<namespace>.<metric_name> in lowercase, for example:

    CloudWatch metric Kloudfuse metric name

    AWS/EC2 CPUUtilization

    aws.ec2.cpuutilization

    AWS/RDS DatabaseConnections

    aws.rds.databaseconnections

    AWS/Lambda Duration

    aws.lambda.duration

  3. Select a metric and set the time range to the last 15 minutes. A non-empty chart confirms end-to-end delivery.

  4. To verify enrichment labels are being applied (environment tags, instance types, etc.), click on a metric data point and confirm that resource tags such as environment or team appear as labels.