Store Profiles in Google Cloud Storage
By default, the profiler server keeps profiles only on its local volume and deletes the oldest ones as it fills. Configure a Google Cloud Storage (GCS) bucket to keep profiles for a fixed retention period instead. The profiler server uploads a block to the bucket every hour and continues to query it from there after the local copy is gone. For details, see Storage tiers.
Prerequisites
-
Continuous Profiling is enabled. See Enable the profiler server.
-
A GCS bucket dedicated to profiling data, in the region where Kloudfuse runs.
-
A way for the profiler server to authenticate to the bucket, with the Storage Object Admin role (
roles/storage.objectAdmin) on it. The compactor needs delete access to merge blocks and enforce retention. Use one of:-
a GCP service account and its JSON key; see Configure with a service account key.
-
GKE Workload Identity Federation, with no key; see Configure with Workload Identity.
-
Configure with a service account key
-
Grant the service account access to the bucket:
gcloud storage buckets add-iam-policy-binding gs://<BUCKET_NAME> \ --member="serviceAccount:<SA_NAME>@<PROJECT_ID>.iam.gserviceaccount.com" \ --role="roles/storage.objectAdmin" -
Download a JSON key for the service account, and store it in a Kubernetes Secret in the
kfusenamespace:kubectl create secret generic kfuse-profiler-storage -n kfuse \ --from-file=GCS_SERVICE_ACCOUNT=<path/to/key.json> -
Add the following to your
custom-values.yamlfile:pyroscope: pyroscope: extraArgs: storage.gcs.service-account: $(GCS_SERVICE_ACCOUNT) (1) extraEnvFrom: - secretRef: name: kfuse-profiler-storage (2) structuredConfig: storage: backend: gcs gcs: bucket_name: <BUCKET_NAME> (3) limits: compactor_blocks_retention_period: 30d (4)yaml1 Passes the key to the profiler server as a command-line flag. Kubernetes replaces $(GCS_SERVICE_ACCOUNT)with the contents of the Secret when the pod starts. Keep it exactly as shown, with parentheses.2 Loads the key from the Secret you created as an environment variable. 3 The name of your bucket, without the gs://prefix.4 How long to keep profiles in the bucket. Blocks older than this are deleted. The default, 0s, keeps data forever.The key is passed as a flag, not as a
${GCS_SERVICE_ACCOUNT}placeholder understorage.gcs. The profiler server would expand that placeholder into raw JSON inside its YAML configuration and fail to start. -
Apply the change:
helm upgrade --install kfuse oci://us-east1-docker.pkg.dev/mvp-demo-301906/kfuse-helm/kfuse \ -n kfuse \ --version <VERSION> \ (1) -f custom-values.yamlbash1 Replace <VERSION>with a valid Kloudfuse release value. See Release Notes for the latest release.
Configure with Workload Identity
On GKE clusters with Workload Identity Federation enabled, the profiler server can authenticate without a key. When no service account key is configured, it uses Google Application Default Credentials.
By default, the profiler server runs as the default Kubernetes service account in the Kloudfuse namespace, which other Kloudfuse pods share. Granting the bucket role to that account would give every one of those pods access to your profiles, so this procedure runs the profiler server under its own service account, kfuse-profiler-server, and grants the role only to it.
-
Grant the profiler server’s Kubernetes service account access to the bucket:
gcloud storage buckets add-iam-policy-binding gs://<BUCKET_NAME> \ --role="roles/storage.objectAdmin" \ --member="principal://iam.googleapis.com/projects/<PROJECT_NUMBER>/locations/global/workloadIdentityPools/<PROJECT_ID>.svc.id.goog/subject/ns/kfuse/sa/kfuse-profiler-server" -
Add the following to your
custom-values.yamlfile:pyroscope: pyroscope: serviceAccount: (1) create: true name: kfuse-profiler-server structuredConfig: storage: backend: gcs gcs: bucket_name: <BUCKET_NAME> limits: compactor_blocks_retention_period: 30dyaml1 Creates the kfuse-profiler-serverservice account and runs the profiler server under it. The name must match thesa/part of the member in the previous step. -
Apply the change:
helm upgrade --install kfuse oci://us-east1-docker.pkg.dev/mvp-demo-301906/kfuse-helm/kfuse \ -n kfuse \ --version <VERSION> \ (1) -f custom-values.yamlbash1 Replace <VERSION>with a valid Kloudfuse release value. See Release Notes for the latest release. -
Confirm that the profiler server uses the new service account:
kubectl get pod -n kfuse kfuse-profiler-server-0 -o jsonpath='{.spec.serviceAccountName}'The command prints
kfuse-profiler-server.
Optional settings
To share one bucket with other data, set pyroscope.pyroscope.structuredConfig.storage.storage_prefix to a prefix of letters and digits only, such as profiles.
Verify
-
Check that the profiler server started without storage errors:
kubectl logs -n kfuse kfuse-profiler-server-0 | grep -i "bucket\|level=error"could not find default credentialsmeans no key was passed and Workload Identity is not set up.403errors point to the IAM binding. -
Within about an hour, the first block is uploaded. List the bucket to confirm:
gcloud storage ls --recursive gs://<BUCKET_NAME>/ | head
|
Earlier versions of this page placed the storage settings, including the full JSON key, in the |