GCP Log Integration

Kloudfuse ingests GCP logs by pulling messages from a Cloud Pub/Sub subscription. A Cloud Logging log sink routes log entries to a Pub/Sub topic, and Kloudfuse continuously pulls from the corresponding subscription.

For an overview of how this fits into the broader GCP integration architecture, see GCP Integration Architecture.

Overview

Any GCP service that writes to Cloud Logging can be ingested — including Compute Engine, GKE, Cloud SQL, App Engine, Cloud Functions, and custom application logs forwarded via the Cloud Logging API.

Data flows as follows:

  1. GCP resources write log entries to Cloud Logging.

  2. A Log Sink routes matching log entries to a Pub/Sub topic.

  3. Kloudfuse maintains a pull subscription on that topic and continuously retrieves batches of log messages.

  4. Logs appear in the Kloudfuse Log Analytics interface within 30–60 seconds.

Common use cases:

  • Security monitoring — Detect unusual API patterns, unauthorized access, or privilege escalation from Cloud Audit Logs.

  • Compliance reporting — Maintain a record of all configuration changes to GCP resources.

  • Incident investigation — Correlate GCP infrastructure changes with application errors using Kloudfuse’s unified log and metrics view.

Prerequisites

  • A GCP project with Cloud Logging and Pub/Sub APIs enabled.

  • The gcloud CLI installed and authenticated, or access to the Google Cloud Console.

  • A running Kloudfuse deployment on GKE.

  • The Kloudfuse ingestion namespace (default: kfuse).

Step 1: Create a Pub/Sub Topic and Subscription

Create the Topic

gcloud pubsub topics create kloudfuse-logs \
  --project=<project-id>
bash

Create the Pull Subscription

gcloud pubsub subscriptions create kloudfuse-gcp-subscription \
  --topic=kloudfuse-logs \
  --project=<project-id> \
  --message-retention-duration=1d \
  --expiration-period=never
bash

The subscription ID kloudfuse-gcp-subscription matches the default value in the Helm configuration. Use a different name if needed and update the subscriptionId field accordingly.

Step 2: Create a Log Sink

A log sink routes Cloud Logging entries to the Pub/Sub topic.

Route All Log Entries

To forward all log entries from the project:

gcloud logging sinks create kloudfuse-logs-sink \
  pubsub.googleapis.com/projects/<project-id>/topics/kloudfuse-logs \
  --project=<project-id>
bash

Route Filtered Log Entries

To forward only specific log entries (for example, Cloud Audit Logs):

gcloud logging sinks create kloudfuse-audit-sink \
  pubsub.googleapis.com/projects/<project-id>/topics/kloudfuse-logs \
  --project=<project-id> \
  --log-filter='logName=~"cloudaudit.googleapis.com"'
bash

Grant the Sink Writer Permission

After creating the sink, retrieve the sink’s service account and grant it publish rights on the topic:

SINK_SERVICE_ACCOUNT=$(gcloud logging sinks describe kloudfuse-logs-sink \
  --project=<project-id> \
  --format='value(writerIdentity)')

gcloud pubsub topics add-iam-policy-binding kloudfuse-logs \
  --project=<project-id> \
  --member="$SINK_SERVICE_ACCOUNT" \
  --role="roles/pubsub.publisher"
bash

Step 3: Configure a Service Account

Kloudfuse needs a service account with the roles/pubsub.subscriber role to pull messages from the subscription.

Create the Service Account

gcloud iam service-accounts create kloudfuse-gcp \
  --display-name="Kloudfuse GCP Integration" \
  --project=<project-id>
bash

Assign Roles

gcloud projects add-iam-policy-binding <project-id> \
  --member="serviceAccount:kloudfuse-gcp@<project-id>.iam.gserviceaccount.com" \
  --role="roles/pubsub.subscriber"
bash
If you also want to collect metrics, assign roles/monitoring.viewer and roles/compute.viewer to the same service account and reuse the same Kubernetes secret for both pipelines. See GCP Metrics — Service Account setup.

Create the JSON Key

gcloud iam service-accounts keys create kloudfuse-gcp-credentials.json \
  --iam-account=kloudfuse-gcp@<project-id>.iam.gserviceaccount.com
bash

Create the Kubernetes Secret

The credentials file must be named credentials.json inside the secret. 
kubectl create secret generic kfuse-gcp-credentials \
  --from-file=credentials.json=kloudfuse-gcp-credentials.json \
  --namespace <kloudfuse-namespace>
bash

Replace <kloudfuse-namespace> with your Kloudfuse installation namespace (kfuse by default).

Step 4: Configure Kloudfuse via Helm

Add the following to your custom-values.yaml:

global:
  enrichmentEnabled:
    - gcp
  gcpConfig:
    secretName: "kfuse-gcp-credentials" (1)
    projectId: "<project-id>" (2)
    pubsub:
      enabled: true
      subscriptions: (3)
        - projectId: "<project-id>" (4)
          subscriptionId: "kloudfuse-gcp-subscription" (5)
yaml
1 The Kubernetes secret containing the GCP service account credentials.
2 (Required) The default GCP project ID. The ingester will not start if this field is absent or empty.
3 A list of Pub/Sub subscriptions to consume logs from. Add one entry per subscription; see Consume Logs from Multiple Subscriptions.
4 (Required) The GCP project ID where this Pub/Sub subscription was created. Pub/Sub log ingestion fails to start for any subscription that is missing its projectId.
5 The Pub/Sub subscription ID created in Create the Pull Subscription.

Consume Logs from Multiple Subscriptions

A single secret can consume logs from more than one Pub/Sub subscription, including subscriptions that belong to different GCP projects.

Ensure the service account has the roles/pubsub.subscriber role in each project (or on each subscription) you list.

gcpConfig:
  pubsub:
    subscriptions:
      - projectId: "gcp-project-a-481726"
        subscriptionId: "kloudfuse-logs-sub-1"
      - projectId: "gcp-project-b-573204"
        subscriptionId: "kloudfuse-logs-sub-2"
yaml

Then apply the Helm upgrade:

helm upgrade --install kfuse oci://us-east1-docker.pkg.dev/mvp-demo-301906/kfuse-helm/kfuse \
  -n kfuse \
  --version <VERSION> \ (1)
  -f custom-values.yaml
1 Replace <VERSION> with a valid Kloudfuse release value; use the most recent one.

Verify the Integration

Confirm the Subscription is Receiving Messages

Check that messages are flowing into the Pub/Sub subscription:

gcloud pubsub subscriptions describe kloudfuse-gcp-subscription \
  --project=<project-id> \
  --format='table(name, messageRetentionDuration, expirationPolicy)'
bash

To pull a single test message and confirm the subscription is active:

gcloud pubsub subscriptions pull kloudfuse-gcp-subscription \
  --project=<project-id> \
  --limit=1 \
  --auto-ack
bash

A returned message confirms that Cloud Logging is routing entries to the topic.

Confirm Logs are Arriving in Kloudfuse

  1. In the Kloudfuse UI, click the Logs tab and select Search from the drop-down.

  2. Set the time picker to the last 15 minutes.

  3. In the search bar, click Advanced Search and enter the following FuseQL query to confirm GCP logs are flowing:

    source="gcp"
    fuseql

    Any result confirms that GCP log records are reaching Kloudfuse.

  4. To filter by GCP log name:

    logName="projects/<project-id>/logs/cloudaudit.googleapis.com%2Factivity"
    fuseql

  5. To count log volume by GCP resource type:

    source="gcp" | count by resource.type
    fuseql

Troubleshooting

No Messages in the Pub/Sub Subscription

The subscription is not receiving messages from Cloud Logging.

  1. Confirm the log sink was created and is active:

    gcloud logging sinks describe kloudfuse-logs-sink \
  --project=<project-id> \
  --format='table(name, destination, writerIdentity)'
    bash

  2. Confirm the sink’s writer identity has the roles/pubsub.publisher role on the topic (see Grant the Sink Writer Permission).

  3. Confirm logs are being written to Cloud Logging by checking the Logs Explorer in the Google Cloud Console.

  4. If using an inclusion filter on the sink, verify the filter matches logs that are actually being emitted.

Kloudfuse Pod Cannot Pull from Pub/Sub

If Kloudfuse logs show authentication or permission errors when pulling from Pub/Sub:

  1. Confirm the Kubernetes secret exists and contains credentials.json:

    kubectl get secret kfuse-gcp-credentials -n <kloudfuse-namespace> -o jsonpath='{.data}' | jq 'keys'
    bash

  2. Confirm the service account has the roles/pubsub.subscriber role:

    gcloud projects get-iam-policy <project-id> \
  --flatten="bindings[].members" \
  --filter="bindings.members:kloudfuse-gcp@" \
  --format="table(bindings.role)"
    bash

  3. Check the Kloudfuse ingester pod logs for specific error messages:

    kubectl logs -n <kloudfuse-namespace> -l app=ingester --tail=100 | grep -i "pubsub\|gcp\|error"
    bash

Logs Appear in Pub/Sub but Not in Kloudfuse

  1. Confirm pubsub.enabled: true and the correct subscriptionId are set in custom-values.yaml.

  2. Confirm enrichmentEnabled includes gcp and gcpConfig.secretName matches the Kubernetes secret name.

  3. Restart the ingester pod to pick up configuration changes:

    kubectl rollout restart deployment -n <kloudfuse-namespace> -l app=ingester
    bash

Pub/Sub Subscription Lag is Growing

The Kloudfuse puller is falling behind the rate of incoming log messages.

  1. Check the subscription’s undelivered message count in the Google Cloud Console under Pub/Sub > Subscriptions.

  2. If lag is consistently growing, increase the puller concurrency or the number of ingester replicas via Helm.

  3. Consider adding an exclusion filter to the log sink to reduce the volume of log entries forwarded (see Route Filtered Log Entries).

External References