Store Profiles in Amazon S3

By default, the profiler server keeps profiles only on its local volume and deletes the oldest ones as it fills. Configure an Amazon S3 bucket to keep profiles for a fixed retention period instead. The profiler server uploads a block to the bucket every hour and continues to query it from there after the local copy is gone. For details, see Storage tiers.

Prerequisites

  • Continuous Profiling is enabled. See Enable the profiler server.

  • An S3 bucket dedicated to profiling data, in the region where Kloudfuse runs.

  • An IAM user with an access key, and a policy that grants these actions on the bucket:

    {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Effect": "Allow",
          "Action": ["s3:ListBucket"],
          "Resource": "arn:aws:s3:::<BUCKET_NAME>"
        },
        {
          "Effect": "Allow",
          "Action": ["s3:GetObject", "s3:PutObject", "s3:DeleteObject"],
          "Resource": "arn:aws:s3:::<BUCKET_NAME>/*"
        }
      ]
    }
    json

    The compactor needs s3:DeleteObject to merge blocks and enforce retention.

Configure S3 storage

  1. Store the access key in a Kubernetes Secret in the kfuse namespace:

    kubectl create secret generic kfuse-profiler-storage -n kfuse \
      --from-literal=AWS_ACCESS_KEY_ID=<ACCESS_KEY_ID> \
      --from-literal=AWS_SECRET_ACCESS_KEY=<SECRET_ACCESS_KEY>
  2. Add the following to your custom-values.yaml file:

    pyroscope:
      pyroscope:
        extraArgs:
          config.expand-env: true (1)
        extraEnvFrom:
          - secretRef:
              name: kfuse-profiler-storage (2)
        structuredConfig:
          storage:
            backend: s3
            s3:
              bucket_name: <BUCKET_NAME> (3)
              region: <REGION> (4)
              endpoint: s3.<REGION>.amazonaws.com (5)
              access_key_id: ${AWS_ACCESS_KEY_ID} (6)
              secret_access_key: ${AWS_SECRET_ACCESS_KEY}
          limits:
            compactor_blocks_retention_period: 30d (7)
    yaml
    1 Lets the profiler server replace ${…​} placeholders in its configuration with environment variables.
    2 Loads the keys from the Secret you created as environment variables.
    3 The name of your bucket, without the s3:// prefix.
    4 The AWS region of the bucket, for example us-west-2.
    5 The regional S3 endpoint, without a scheme, for example s3.us-west-2.amazonaws.com. See Amazon S3 endpoints.
    6 Placeholders filled from the Secret at startup. Keep them exactly as shown.
    7 How long to keep profiles in the bucket. Blocks older than this are deleted. The default, 0s, keeps data forever.
  3. Apply the change:

    helm upgrade --install kfuse oci://us-east1-docker.pkg.dev/mvp-demo-301906/kfuse-helm/kfuse \
      -n kfuse \
      --version <VERSION> \ (1)
      -f custom-values.yaml
    bash
    1 Replace <VERSION> with a valid Kloudfuse release value. See Release Notes for the latest release.

Optional settings

Add these under pyroscope.pyroscope.structuredConfig.storage.s3:

Setting Description

sse.type

Server-side encryption: SSE-S3 or SSE-KMS. For SSE-KMS, also set sse.kms_key_id.

force_path_style

Set to true to use path-style URLs. Required by some S3-compatible stores.

insecure

Set to true to connect over HTTP instead of HTTPS. Use only for local testing against an S3-compatible store such as MinIO. The default is false.

To share one bucket with other data, set storage.storage_prefix to a prefix of letters and digits only, such as profiles.

Verify

  1. Check that the profiler server started without storage errors:

    kubectl logs -n kfuse kfuse-profiler-server-0 | grep -i "bucket\|level=error"

    Errors such as Access Denied or The specified bucket does not exist point to the IAM policy or the bucket name.

  2. Within about an hour, the first block is uploaded. List the bucket to confirm:

    aws s3 ls s3://<BUCKET_NAME>/ --recursive | head

Earlier versions of this page placed the storage settings in the pyroscope.pyroscope.config string, with the keys in plain text. That still works, but structuredConfig takes precedence over it when both are set. Move to the Secret-based configuration above so that keys are not stored in custom-values.yaml.