AWS CloudWatch Logs Integration

To fully set up ingestion of logs from AWS to Kloudfuse through CloudWatch, complete these tasks:

Configure AWS Kinesis Firehose
Create IAM Role
Setup CloudWatch logs subscriptions
Enable Enrichment of AWS Logs

Configure AWS Kinesis Firehose

Use different Firehose accounts for logs and metrics.

Create a new delivery stream in the account that emits the logs, in the Kinesis Firehose AWS console.

Specify the following attribute values:

Source: Direct PUT

Destination: HTTP Endpoint

Destination settings

Provide the external-facing endpoint of the Kloudfuse cluster as part of the following URL address format:

https://<external facing endpoint of Kfuse cluster>/ingester/kinesis/logs

Access token key: Provide if required

Content encoding

GZIP

Provide an existing S3 bucket, or create a new one for storing Kinesis records as a backup.

Backing up only failed data should be sufficient.
Change the name of the stream, as necessary.

Create IAM Role

Create an IAM role to allow CloudWatch logs to write to Kinesis Firehose.

Log into the account that emits the logs you plan for Kloudfuse to ingest.
In the IAM AWS Console, under Roles, click Create Role.

Select Custom Trust Policy and add the following code, replacing the existing region and AWS account information:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Service": "logs.<region>.amazonaws.com"
            },
            "Action": "sts:AssumeRole",
            "Condition": {
                "StringLike": {
                    "aws:SourceArn": "arn:aws:logs:<region>:<aws account number>:*"
                }
            }
        }
    ]
}

yaml

Click Next to navigate to the Add Permissions page.
Select Create Policy.

This will open a new window.

Select JSON and add the following code to allow all firehose in the same account. Adjust only if adding permissions to a specific firehose:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "firehose:*"
            ],
            "Resource": [
                "arn:aws:firehose:<region>:<aws account number>:*"
            ]
        }
    ]
}

yaml

Return to the Roles page, and select the permission policy you created.
Click Create Role.
Name the new role.

Setup CloudWatch logs subscriptions

In the account that emits the logs, in the IAM AWS Console, perform these steps:

Navigate to the Logs section on the left side of the console, and select Log Groups.
Select the Log group to send to the Kinesis Firehose.
Navigate through Actions → Subscription Filters → Create Kinesis Firehose subscription filter.
In the Kinesis Firehose delivery stream section, select the previously created Kinesis Firehose for Logs.
In the Grant Permission section, select the previously created role.
Provide a Subscription filter pattern, or leave this blank if you plan to stream everything.
Provide a Subscription filter name; this is required, and can be anything.
Click Start Streaming.

Enable Enrichment of AWS Logs

The logs sent by AWS CloudWatch to the Kinesis Firehose include minimal labels. Kloudfuse enables you to attach more labels and user-defined custom tags to the ingested logs, from within the AWS console, by scraping AWS.

To enable this enrichment of AWS logs, follow these steps:

Modify yaml in the global section of the custom-values.yaml file:
```
global:
  enrichmentEnabled:
    - aws
```
yaml

Create IAM scraper role in the AWS account where the services that emit the logs run.

Attach the following policy, for Kloudfuse to scrape the additional labels from AWS. See AWS documentation Define custom IAM permissions with customer managed policies

Create a scraper role with custom policies

			"Action": [
                "logs:DescribeLogGroups",
				"logs:ListTagsLogGroup",
				"acm:ListCertificates",
				"acm:ListTagsForCertificate",
				"apigateway:GET",
				"athena:ListWorkGroups",
				"athena:ListTagsForResource",
				"autoscaling:DescribeAutoScalingGroups",
				"cloudwatch:ListMetrics",
				"cloudwatch:GetMetricStatistics",
				"dynamodb:ListTables",
				"dynamodb:DescribeTable",
				"dynamodb:ListTagsOfResource",
				"ec2:DescribeInstances",
				"ec2:DescribeInstanceStatus",
				"ec2:DescribeSecurityGroups",
				"ec2:DescribeNatGateways",
				"ec2:DescribeVolumes",
				"ecs:ListClusters",
				"ecs:ListContainerInstances",
				"ecs:ListServices",
				"ecs:DescribeContainerInstances",
				"ecs:DescribeServices",
				"ecs:ListTagsForResource",
				"elasticache:DescribeCacheClusters",
				"elasticache:ListTagsForResource",
				"elasticfilesystem:DescribeFileSystems",
				"elasticfilesystem:DescribeBackupPolicy",
				"elasticloadbalancing:DescribeTags",
				"elasticloadbalancing:DescribeLoadBalancers",
				"es:ListDomainNames",
				"es:DescribeDomains",
				"es:ListTags",
				"events:ListRules",
				"events:ListTagsForResource",
				"events:ListEventBuses",
				"firehose:DescribeDeliveryStream",
				"firehose:ListDeliveryStreams",
				"firehose:ListTagsForDeliveryStream",
				"glue:ListJobs",
				"glue:GetTags",
				"kafka:ListTagsForResource",
				"kafka:ListClustersV2",
				"kinesis:ListStreams",
				"kinesis:ListTagsForStream",
				"kinesis:DescribeStream",
				"lambda:GetPolicy",
				"lambda:List*",
				"lambda:ListTags",
				"logs:DescribeLogGroups",
				"logs:ListTagsLogGroup"
				"mq:ListBrokers",
				"mq:DescribeBroker",
				"rds:DescribeDBInstances",
				"rds:ListTagsForResource",
				"rds:DescribeEvents",
				"redshift:DescribeClusters",
				"redshift:DescribeTags",
				"route53:ListHealthChecks",
				"route53:ListTagsForResource",
				"s3:ListAllMyBuckets",
				"s3:GetBucketTagging",
				"sns:ListTagsForResource",
				"sns:ListTopics",
				"sqs:ListQueues",
				"sqs:ListQueueTags",
				"wafv2:ListWebACLs",
				"wafv2:ListRuleGroups",
				"wafv2:ListTagsForResource"
			]

yaml

Modify the Trust Relationship for the policy of the scrape role ARN to add the node-group (Node IAM Role ARN), in which Kloudfuse is running on, as the Principal on the Account.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Statement1",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::ACCOUNT-NUMBER:role/eksctl-XXXXX-nodegroup-ng-XXXXXX-NodeInstanceRole-XXXXXXXXXX"
            },
            "Action": "sts:AssumeRole"
        }
    ]
}

yaml

Ensure that the permissions map to the node pool that hosts the EKS cluster for Kloudfuse.
Enable Kloudfuse to consume the new role; there are two approaches: through AWS credentials or through Role ARNs.
- AWS credentials
- Role ARNs
Add your AWS credentials as a secret, and use the secret in the ingester config.
Retrieve your aws credentials; see Configure tool authentication with AWS.

In the Kloudfuse namespace, create a kube secret name aws-access-key, with keys accessKey and secretKey.

kubectl create secret generic aws-access-key --from-literal=accessKey=<AWS_ACCESS_KEY_ID> --from-literal=secretKey=<AWS_SECRET_ACCESS_KEY>

Specify the secretName in the custom-values.yaml file.

ingester: config: awsScraper: secretName: aws-access-key

yaml

By default, Kloudfuse attempts to scrape from all regions and all AWS Namespaces. Customize this by adding the following configuration in the custom-values.yaml file:

ingester: config: awsScraper: secretName: aws-access-key namespaces: - AWS/Logs regions: - <add region>

yaml
Add Role ARNs in the ingester config: This option enables you to scrape multiple AWS accounts.
Add the scraper Role ARNs that you created with the new permissions to the awsRoleArns list to your custom-values.yaml file.

ingester: config: awsRoleArns: - role: <ADD ROLE ARN HERE>

yaml

By default, Kloudfuse attempts to scrape from all regions and all AWS Namespaces. Customize this by adding the following configuration in the custom-values.yaml file:

ingester: config: awsRoleArns: role: <ADD ROLE ARN HERE> namespaces: - AWS/Logs regions: - <add region>

yaml

To modify the node-group IAM role where Kloudfuse Platform runs, add the following permissions policy to the node-group (Node IAM Role ARN) to assume the role.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "sts:AssumeRole",
            "Resource": <REPLACE SCRAPER ROLE ARN HERE>
        }
    ]
}

yaml

Complete a helm upgrade to save the changes.

helm upgrade --create-namespace --install kfuse . -f <custom_values.yaml>