Configure Continuous Profiling

Continuous Profiling is turned off by default. To use it, enable the profiler server in Kloudfuse, then run a collector in each cluster you want to profile. To learn how the pieces fit together, see Continuous Profiling Architecture.

Prerequisites

  • Access to the custom-values.yaml file for your Kloudfuse installation.

  • FIPS mode is off (global.fips.enabled: false). The profile ingest route is not available in FIPS deployments.

  • For each application you want to profile: either a Go service that exposes net/http/pprof endpoints, or a service instrumented with a Pyroscope language SDK.

Enable the profiler server

  1. Add the following to your custom-values.yaml file:

    global:
      kfuse-profiling:
        enabled: true
    yaml
  2. Optional: change the size of the local volume. The default of 50Gi keeps profiles until the volume fills, then deletes the oldest. Increase it to keep more history locally:

    pyroscope:
      pyroscope:
        persistence:
          size: 200Gi
    yaml

    To keep profiles longer than the volume can hold, also configure object storage. See Configure long-term retention.

  3. Apply the change:

    helm upgrade --install kfuse oci://us-east1-docker.pkg.dev/mvp-demo-301906/kfuse-helm/kfuse \
      -n kfuse \
      --version <VERSION> \ (1)
      -f custom-values.yaml
    bash
    1 Replace <VERSION> with a valid Kloudfuse release value. See Release Notes for the latest release.
  4. Confirm that the profiler server is running:

    kubectl get pods -n kfuse | grep kfuse-profiler-server

    The kfuse-profiler-server-0 pod should report 1/1 Running.

Annotate the pods to profile

Alloy profiles only the pods that opt in through annotations. Add these to the pod template of each workload, for example under spec.template.metadata.annotations in a Deployment:

annotations:
  pyroscope.io/scrape: "true" (1)
  pyroscope.io/application_name: checkout-service (2)
  pyroscope.io/port: "6060" (3)
yaml
1 Required. Only pods with this annotation set to "true" are scraped.
2 The service name shown in Kloudfuse. Becomes the service_name label.
3 The port that serves /debug/pprof/*. Omit it to use the pod’s container port.

For a Go service, importing net/http/pprof and starting an HTTP listener on that port is enough to expose the endpoints. See Set up Go profiling in pull mode.

Install Grafana Alloy

Install Alloy in each Kubernetes cluster you want to profile. It runs as a single pod that discovers the annotated pods in every namespace and forwards their profiles to Kloudfuse.

  1. Create a file named alloy-values.yaml with the following content:

    alloy:
      configMap:
        content: |-
          discovery.kubernetes "pods" {
            role = "pod"
          }
    
          discovery.relabel "pods" {
            targets = discovery.kubernetes.pods.targets
    
            rule { (1)
              action        = "drop"
              source_labels = ["__meta_kubernetes_pod_phase"]
              regex         = "Pending|Succeeded|Failed|Completed"
            }
            rule { (2)
              action        = "keep"
              source_labels = ["__meta_kubernetes_pod_annotation_pyroscope_io_scrape"]
              regex         = "true"
            }
            rule {
              action        = "replace"
              source_labels = ["__meta_kubernetes_pod_annotation_pyroscope_io_application_name"]
              target_label  = "service_name"
            }
            rule {
              action        = "replace"
              source_labels = ["__address__", "__meta_kubernetes_pod_annotation_pyroscope_io_port"]
              regex         = "(.+?)(?::\\d+)?;(\\d+)"
              replacement   = "$1:$2"
              target_label  = "__address__"
            }
            rule {
              action        = "replace"
              source_labels = ["__meta_kubernetes_pod_annotation_pyroscope_io_scheme"]
              regex         = "(https?)"
              target_label  = "__scheme__"
            }
            rule { (3)
              action        = "replace"
              source_labels = ["__meta_kubernetes_namespace"]
              target_label  = "kubernetes_namespace"
            }
            rule {
              action        = "replace"
              source_labels = ["__meta_kubernetes_pod_name"]
              target_label  = "kubernetes_pod_name"
            }
            rule {
              action = "labelmap"
              regex  = "__meta_kubernetes_pod_label_(.+)"
            }
          }
    
          pyroscope.scrape "pods" {
            targets    = discovery.relabel.pods.output
            forward_to = [pyroscope.write.kloudfuse.receiver]
    
            profiling_config { (4)
              profile.process_cpu { enabled = true }
              profile.memory      { enabled = true }
              profile.goroutine   { enabled = true }
              profile.block       { enabled = false }
              profile.mutex       { enabled = false }
              profile.fgprof      { enabled = false }
            }
          }
    
          pyroscope.write "kloudfuse" {
            endpoint {
              url = "https://<KLOUDFUSE_HOST>/profile" (5)
            }
          }
    
    controller: (6)
      type: deployment
      replicas: 1
    yaml
    1 Skips pods that are not running.
    2 Keeps only pods annotated with pyroscope.io/scrape: "true".
    3 Adds the namespace, pod name, and pod labels as profile labels, so you can filter and group by them in Kloudfuse.
    4 The profile types to collect. Block and mutex profiles are off by default because the application must enable them with runtime.SetBlockProfileRate and runtime.SetMutexProfileFraction. For the full list, see the pyroscope.scrape reference.
    5 Replace <KLOUDFUSE_HOST> with the host name of your Kloudfuse installation.
    6 Runs Alloy as one pod. The chart’s default is a DaemonSet, with one Alloy pod per node; because this configuration discovers pods across the whole cluster, every one of those Alloy pods would scrape every target, collecting each profile once per node.

    One Alloy pod handles most clusters. To spread scraping across several pods instead, set alloy.clustering.enabled: true and controller.replicas in alloy-values.yaml, and add clustering { enabled = true } inside pyroscope.scrape "pods". See the pyroscope.scrape clustering reference.

  2. Install Alloy with the Grafana Helm chart. Use any namespace; alloy is shown here:

    helm repo add grafana https://grafana.github.io/helm-charts
    helm repo update
    helm upgrade --install alloy grafana/alloy \
      --namespace alloy --create-namespace \
      -f alloy-values.yaml
  3. Confirm that Alloy is scraping your pods. Forward the Alloy UI port, then open http://localhost:12345 and check that the pyroscope.scrape.pods component is healthy and lists your targets:

    kubectl port-forward -n alloy svc/alloy 12345:12345

To push profiles from a language SDK instead of scraping them, point the SDK’s server address at https://<KLOUDFUSE_HOST>/profile. For Java, see the Pyroscope Java client.

View profiles

In the Kloudfuse UI, open the avatar menu, select Grafana, open Explore, and choose the KfuseProfileDatasource data source. In the legacy UI, open Grafana › Profiler instead. Choose a profile type, then select your service by its service_name. Profiles appear within a minute of the first scrape.

Profiles are accepted only if their timestamp is no more than 1 hour old. If Alloy is down for longer than that, the profiles from the gap are lost.

Configure long-term retention

By default, profiles live only on the profiler server’s local volume and are deleted, oldest first, as it fills. To keep profiles for a fixed period, such as 30 or 90 days, store them in a bucket you own: