Store Profiles in Amazon S3
By default, the profiler server keeps profiles only on its local volume and deletes the oldest ones as it fills. Configure an Amazon S3 bucket to keep profiles for a fixed retention period instead. The profiler server uploads a block to the bucket every hour and continues to query it from there after the local copy is gone. For details, see Storage tiers.
Prerequisites
-
Continuous Profiling is enabled. See Enable the profiler server.
-
An S3 bucket dedicated to profiling data, in the region where Kloudfuse runs.
-
An IAM user with an access key, and a policy that grants these actions on the bucket:
{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": ["s3:ListBucket"], "Resource": "arn:aws:s3:::<BUCKET_NAME>" }, { "Effect": "Allow", "Action": ["s3:GetObject", "s3:PutObject", "s3:DeleteObject"], "Resource": "arn:aws:s3:::<BUCKET_NAME>/*" } ] }jsonThe compactor needs
s3:DeleteObjectto merge blocks and enforce retention.
Configure S3 storage
-
Store the access key in a Kubernetes Secret in the
kfusenamespace:kubectl create secret generic kfuse-profiler-storage -n kfuse \ --from-literal=AWS_ACCESS_KEY_ID=<ACCESS_KEY_ID> \ --from-literal=AWS_SECRET_ACCESS_KEY=<SECRET_ACCESS_KEY> -
Add the following to your
custom-values.yamlfile:pyroscope: pyroscope: extraArgs: config.expand-env: true (1) extraEnvFrom: - secretRef: name: kfuse-profiler-storage (2) structuredConfig: storage: backend: s3 s3: bucket_name: <BUCKET_NAME> (3) region: <REGION> (4) endpoint: s3.<REGION>.amazonaws.com (5) access_key_id: ${AWS_ACCESS_KEY_ID} (6) secret_access_key: ${AWS_SECRET_ACCESS_KEY} limits: compactor_blocks_retention_period: 30d (7)yaml1 Lets the profiler server replace ${…}placeholders in its configuration with environment variables.2 Loads the keys from the Secret you created as environment variables. 3 The name of your bucket, without the s3://prefix.4 The AWS region of the bucket, for example us-west-2.5 The regional S3 endpoint, without a scheme, for example s3.us-west-2.amazonaws.com. See Amazon S3 endpoints.6 Placeholders filled from the Secret at startup. Keep them exactly as shown. 7 How long to keep profiles in the bucket. Blocks older than this are deleted. The default, 0s, keeps data forever. -
Apply the change:
helm upgrade --install kfuse oci://us-east1-docker.pkg.dev/mvp-demo-301906/kfuse-helm/kfuse \ -n kfuse \ --version <VERSION> \ (1) -f custom-values.yamlbash1 Replace <VERSION>with a valid Kloudfuse release value. See Release Notes for the latest release.
Optional settings
Add these under pyroscope.pyroscope.structuredConfig.storage.s3:
| Setting | Description |
|---|---|
|
Server-side encryption: |
|
Set to |
|
Set to |
To share one bucket with other data, set storage.storage_prefix to a prefix of letters and digits only, such as profiles.
Verify
-
Check that the profiler server started without storage errors:
kubectl logs -n kfuse kfuse-profiler-server-0 | grep -i "bucket\|level=error"Errors such as
Access DeniedorThe specified bucket does not existpoint to the IAM policy or the bucket name. -
Within about an hour, the first block is uploaded. List the bucket to confirm:
aws s3 ls s3://<BUCKET_NAME>/ --recursive | head
|
Earlier versions of this page placed the storage settings in the |