GCP Metrics Collection

Kloudfuse integrates with the GCP cloud service to collect metrics.

Complete these tasks to successfully collect metrics:

Create Credentials File
Create Kubernetes Secret
Configure Kloudfuse to Collect Metrics
- Enable Cloud Exporter
- Configure GCP Metrics Collection
  - Method 1: Using Kubernetes Secret (Recommended for Production)
  - Method 2: Using Direct secretKey (Quick Setup Only)
Configure GCP Metrics Enrichment

Create Credentials File

In the Google Cloud console, navigate to Menu > IAM & Admin > Service Accounts.
Select your service account, or create a new one.

Ensure that the service account has Monitoring Viewer permissions to collect GCP metrics.
Select Keys > Add key > Create new key.
Select JSON, then click Create.
Click Close.

Create Kubernetes Secret

The credentials file must be named credentials.json when creating the Kubernetes secret.

Create a Kubernetes secret from the credentials file:

kubectl create secret generic kfuse-sd-secret --from-file=credentials.json -n kfuse

Configure Kloudfuse to Collect Metrics

Enable Cloud Exporter

First, enable the cloud exporter service in the global configuration:

global:
  cloud-exporter:
    enabled: true
  gcpConfig:
    secretName: "kfuse-sd-secret"

yaml

Configure GCP Metrics Collection

There are two methods to provide GCP credentials for metrics collection:

Method 1: Using Kubernetes Secret (Recommended for Production)

This is the recommended approach for production environments as it keeps sensitive credentials secure.

Single Project Configuration

Update the helm values using the following configuration for a single project.

kfuse-cloud-exporter:
  prometheus-stackdriver-exporter:
    enabled: true
    stackdriver:
      httpTimeout: 30s
      maxRetries: 3
      projectId: "YOUR-PROJECT-ID" (1)
      metrics:
        typePrefixes: "compute.googleapis.com"

yaml

1	`projectId`: GCP project ID to monitor

Multi-Project Configuration

For monitoring multiple GCP projects, use one of the following approaches:

Recommended Approach (Clean Array Format)

kfuse-cloud-exporter:
  prometheus-stackdriver-exporter:
    enabled: true
    stackdriver:
      httpTimeout: 30s
      maxRetries: 3
      metrics:
        typePrefixes: "compute.googleapis.com"
    config:
      project_ids: (1)
        - project-1
        - project-2
        - project-3

yaml

1	`project_ids`: Array of GCP project IDs to monitor (recommended for multiple projects)

Alternative Approach (Backward Compatible)

kfuse-cloud-exporter:
  prometheus-stackdriver-exporter:
    enabled: true
    stackdriver:
      httpTimeout: 30s
      maxRetries: 3
      projectId: "project-1,project-2,project-3" (1)
      metrics:
        typePrefixes: "compute.googleapis.com"

yaml

1	`projectId`: Comma-separated list of GCP project IDs (backward compatible format)

Method 2: Using Direct secretKey (Quick Setup Only)

This method embeds credentials directly in the configuration file. Only use this for quick testing or proof-of-concept setups. Never use this approach in production environments.

Convert the JSON credentials file to base64:
```
base64 -w 0 credentials.json
```

Update the helm values with the base64-encoded credentials:

Single Project

kfuse-cloud-exporter:
  prometheus-stackdriver-exporter:
    enabled: true
    secretKey: "<BASE64_ENCODED_CREDENTIALS>" (1)
    stackdriver:
      httpTimeout: 30s
      maxRetries: 3
      projectId: "YOUR-PROJECT-ID" (2)
      metrics:
        typePrefixes: "compute.googleapis.com"

yaml

1 secretKey: Base64-encoded service account JSON credentials (not recommended for production)

projectId: GCP project ID to monitor

Multiple Projects

kfuse-cloud-exporter:
  prometheus-stackdriver-exporter:
    enabled: true
    secretKey: "<BASE64_ENCODED_CREDENTIALS>"
    stackdriver:
      httpTimeout: 30s
      maxRetries: 3
      metrics:
        typePrefixes: "compute.googleapis.com"
    config:
      project_ids: (1)
        - project-1
        - project-2
        - project-3

yaml

3 project_ids: Array of GCP project IDs to monitor

Update the corresponding projectId (or project_ids for multiple projects) of the GCP account and update the typePrefixes to collect the metrics from relevant services of GCP; see Google Cloud documentation on Google Cloud metrics.

Configure GCP Metrics Enrichment

Kloudfuse can enrich GCP metrics with custom labels and metadata from your GCP resources. This feature adds resource labels (like custom tags) from GCE instances to the corresponding metrics, providing better context for monitoring and alerting.

Prerequisites

GCP credentials with appropriate permissions for resource enrichment
The same Kubernetes secret (kfuse-sd-secret) created in the previous steps

Create IAM Role and Service Account

Create a service account in your GCP project with the required permissions for Kloudfuse to scrape additional labels from GCP resources.

# Create service account
gcloud iam service-accounts create kfuse-gcp-enrichment \
    --display-name="Kloudfuse GCP Enrichment" \
    --description="Service account for Kloudfuse GCP metrics enrichment"

# Assign Monitoring Viewer role for metrics collection
gcloud projects add-iam-policy-binding YOUR-PROJECT-ID \
    --member="serviceAccount:kfuse-gcp-enrichment@YOUR-PROJECT-ID.iam.gserviceaccount.com" \
    --role="roles/monitoring.viewer"

# Assign Compute Viewer role for GCE instance label enrichment
gcloud projects add-iam-policy-binding YOUR-PROJECT-ID \
    --member="serviceAccount:kfuse-gcp-enrichment@YOUR-PROJECT-ID.iam.gserviceaccount.com" \
    --role="roles/compute.viewer"

# Create JSON key file
gcloud iam service-accounts keys create kfuse-gcp-credentials.json \
    --iam-account=kfuse-gcp-enrichment@YOUR-PROJECT-ID.iam.gserviceaccount.com

Required IAM Role Permissions

The following table lists the IAM roles required for the service account:

IAM Role Purpose Required For

IAM Role	Purpose	Required For
`roles/monitoring.viewer`	Provides read-only access to GCP monitoring metrics	Metrics collection (required for all configurations)
`roles/compute.viewer`	Provides read-only access to GCE instance metadata and labels	Metrics enrichment (required only when enrichment is enabled)

roles/monitoring.viewer

Provides read-only access to GCP monitoring metrics

Metrics collection (required for all configurations)

roles/compute.viewer

Provides read-only access to GCE instance metadata and labels

Metrics enrichment (required only when enrichment is enabled)

Replace YOUR-PROJECT-ID with your actual GCP project ID in all commands above.

Enable GCP Metrics Enrichment

To enable automatic enrichment of GCP metrics with resource labels, add the following configuration to your helm values:

Global Configuration

Add the GCP configuration under the global section:

global:
  enrichmentEnabled: (1)
    - gcp
  gcpConfig:
    secretName: "kfuse-sd-secret" (2)

yaml

1	`enrichmentEnabled`: Enable enrichment for cloud providers (include "gcp" to enable GCP metrics enrichment)
2	`secretName`: The Kubernetes secret containing GCP credentials (same as used for metrics collection)

Ingester Configuration

Configure the ingester service to scrape and enrich GCP resources:

ingester:
  config:
    gcpScrapeIntervalMinutes: 30 (1)
    gcpProjectConfigs:
      - projectId: "YOUR-PROJECT-ID" (2)
        zones: (3)
          - "us-west1-b"
          - "us-central1-a"
          - "europe-west1-b"
        services: (4)
          - "GCE"
    gcpZones: (5)
      - "us-west1-b"
      - "us-central1-a"
      - "europe-west1-b"
    gcpServices: (6)
      - "GCE"

yaml

1	`gcpScrapeIntervalMinutes`: How often to scrape GCP resources for label enrichment (default: 30 minutes, minimum: 5 minutes)
2	`projectId`: GCP project ID to scrape resources from
3	`zones`: List of GCP zones/regions where your resources are deployed for this specific project
4	`services`: Services to enable enrichment for this specific project (currently supports "GCE" for Compute Engine)
5	`gcpZones`: Default list of zones used when zones are not specified for a project in gcpProjectConfigs
6	`gcpServices`: Default list of services used when services are not specified for a project in gcpProjectConfigs

Setting gcpScrapeIntervalMinutes to a lower value (e.g., 5 minutes) will reduce the time for new resources or label changes to appear in enriched metrics, but may increase API calls to GCP.

Multi-Project Configuration

To enable enrichment across multiple GCP projects, configure multiple project entries under gcpProjectConfigs:

ingester:
  config:
    gcpScrapeIntervalMinutes: 30
    gcpProjectConfigs:
      - projectId: "project-1" (1)
        zones: ["us-west1-b"] (2)
        services: ["GCE"] (3)
      - projectId: "project-2" (4)
        zones: ["europe-west1-b"] (5)
        services: ["GCE"] (6)

yaml

1	First GCP project ID to scrape resources from
2	Zones to scrape in the first project
3	Services to enable for the first project
4	Second GCP project ID to scrape resources from
5	Zones to scrape in the second project
6	Services to enable for the second project

The enrichment process runs periodically based on gcpScrapeIntervalMinutes (default: every 30 minutes) to update the resource labels cache. New resources or label changes may take up to this interval to appear in metrics.