Deploy Kloudfuse on AWS with Terraform

Kloudfuse can be deployed on AWS using Terraform to provision the underlying infrastructure, followed by Helm to install the application. This page covers the Terraform configuration for three deployment scenarios and provides guidance on connecting the infrastructure to Kloudfuse data collection after deployment.

Overview

Terraform manages the AWS infrastructure layer:

  • VPC, subnets, and security groups

  • EKS cluster and node groups

  • ACM certificates and Route 53 DNS records

  • IAM roles and policies

  • S3 bucket and DynamoDB table for Terraform state

Helm manages the Kloudfuse application layer on top of the EKS cluster. After running Terraform, follow the data collection guides to configure CloudWatch metrics, logs, and other integrations.

Prerequisites

  • Terraform >= 1.3 installed

  • AWS CLI configured with administrator access

  • Helm >= 3.x installed

  • A registered domain name with a Route 53 hosted zone

  • An ACM certificate for your domain (or permission to create one)

Step 1: Configure Terraform Backend

Kloudfuse uses an S3 bucket and DynamoDB table to store Terraform state. Create these before running terraform init.

# Create the S3 state bucket
aws s3api create-bucket \
  --bucket kloudfuse-terraform-state \
  --region us-west-2 \
  --create-bucket-configuration LocationConstraint=us-west-2

# Enable versioning on the state bucket
aws s3api put-bucket-versioning \
  --bucket kloudfuse-terraform-state \
  --versioning-configuration Status=Enabled

# Create the DynamoDB lock table
aws dynamodb create-table \
  --table-name kloudfuse-terraform-locks \
  --attribute-definitions AttributeName=LockID,AttributeType=S \
  --key-schema AttributeName=LockID,KeyType=HASH \
  --billing-mode PAY_PER_REQUEST \
  --region us-west-2
bash

Create backend.tf in your Terraform workspace:

terraform {
  backend "s3" {
    bucket         = "kloudfuse-terraform-state"
    key            = "kloudfuse/terraform.tfstate"
    region         = "us-west-2"
    dynamodb_table = "kloudfuse-terraform-locks"
    encrypt        = true
  }
}
hcl

Step 2: Configure Terraform Variables

Create a terraform.tfvars file with your environment-specific values:

# AWS region
aws_region = "us-west-2"

# Availability zones to use
availability_zones = ["us-west-2a", "us-west-2b", "us-west-2c"]

# VPC CIDR block
vpc_cidr = "10.0.0.0/16"

# Private subnets (one per AZ)
private_subnets = ["10.0.1.0/24", "10.0.2.0/24", "10.0.3.0/24"]

# Public subnets (one per AZ)
public_subnets = ["10.0.101.0/24", "10.0.102.0/24", "10.0.103.0/24"]

# EKS cluster name
cluster_name = "kloudfuse"

# Kloudfuse domain (must match ACM certificate)
kloudfuse_domain = "kloudfuse.example.com"

# Route 53 hosted zone ID for the domain
hosted_zone_id = "Z1234567890ABC"

# ACM certificate ARN
acm_certificate_arn = "arn:aws:acm:us-west-2:123456789012:certificate/..."

# EKS node group instance type
node_instance_type = "m5.4xlarge"

# Number of nodes in the default node group
node_desired_capacity = 3
node_min_capacity     = 2
node_max_capacity     = 6
hcl

Step 3: Deploy Infrastructure

Scenario A: New Account (No Existing Infrastructure)

Use this scenario when you are provisioning a new VPC and EKS cluster from scratch.

terraform init
terraform plan -out=tfplan
terraform apply tfplan
bash

After Terraform completes, configure kubectl:

aws eks update-kubeconfig \
  --name kloudfuse \
  --region us-west-2
bash

Install Kloudfuse with Helm:

helm repo add kloudfuse https://kloudfuse.github.io/charts
helm repo update

helm upgrade --install kfuse kloudfuse/kfuse \
  --namespace kfuse \
  --create-namespace \
  -f custom-values.yaml
bash

Scenario B: Existing VPC (Non-EKS)

Use this scenario when a VPC already exists but no EKS cluster has been provisioned.

Set the following additional variables in terraform.tfvars:

# Use existing VPC instead of creating a new one
create_vpc = false
existing_vpc_id = "vpc-0abc1234"
existing_private_subnet_ids = ["subnet-0a1b2c3d", "subnet-0e4f5a6b"]
existing_public_subnet_ids  = ["subnet-0g7h8i9j", "subnet-0k1l2m3n"]
hcl

Run Terraform, then install Kloudfuse as in Scenario A.

Scenario C: Existing EKS Cluster

Use this scenario when you are deploying Kloudfuse into an existing EKS cluster. Terraform manages only the supporting resources (ACM, Route 53, IAM roles).

Set the following in terraform.tfvars:

create_vpc     = false
create_eks     = false
existing_vpc_id        = "vpc-0abc1234"
existing_cluster_name  = "my-existing-cluster"
hcl

Install with Helm

terraform init
terraform apply

# Configure kubectl for the existing cluster
aws eks update-kubeconfig --name my-existing-cluster --region us-west-2

# Install Kloudfuse
helm upgrade --install kfuse kloudfuse/kfuse \
  --namespace kfuse \
  --create-namespace \
  -f custom-values.yaml
bash

Install with kubectl and Helm (Air-gapped / Private Registry)

If your environment does not have outbound internet access from the EKS cluster:

# Pull the Helm chart
helm pull kloudfuse/kfuse --untar

# Push images to your private ECR (example)
aws ecr get-login-password --region us-west-2 | \
  docker login --username AWS --password-stdin \
  <account-id>.dkr.ecr.us-west-2.amazonaws.com

# Install from the local chart with private registry overrides
helm upgrade --install kfuse ./kfuse \
  --namespace kfuse \
  --create-namespace \
  -f custom-values.yaml \
  -f private-registry-values.yaml
bash

Step 4: Configure Data Collection

After the Kloudfuse cluster is running, configure AWS data collection:

  1. Set up Kinesis Firehose delivery streams: Configure AWS Kinesis Firehose for Kloudfuse

  2. Enable CloudWatch Metrics ingestion: AWS CloudWatch Metrics Integration

  3. Enable CloudWatch Logs ingestion: AWS CloudWatch Logs Integration

  4. Optionally enable CloudTrail, EventBridge, and Lambda integrations.

Verify the Deployment

Verify EKS Nodes

kubectl get nodes -o wide
bash

All nodes should show Ready status.

Verify Kloudfuse Pods

kubectl get pods -n kfuse
bash

All pods should be in Running or Completed state. Common pods include: ingester, querier, compactor, pinot, kafka, and grafana.

Verify Ingress and DNS

kubectl get svc -n kfuse | grep LoadBalancer
bash

Confirm the load balancer hostname matches the Route 53 record created by Terraform:

aws route53 list-resource-record-sets \
  --hosted-zone-id <hosted-zone-id> \
  --query "ResourceRecordSets[?Name=='kloudfuse.example.com.']"
bash