Configure Continuous Profiling
Continuous Profiling is turned off by default. To use it, enable the profiler server in Kloudfuse, then run a collector in each cluster you want to profile. To learn how the pieces fit together, see Continuous Profiling Architecture.
Prerequisites
-
Access to the
custom-values.yamlfile for your Kloudfuse installation. -
FIPS mode is off (
global.fips.enabled: false). The profile ingest route is not available in FIPS deployments. -
For each application you want to profile: either a Go service that exposes
net/http/pprofendpoints, or a service instrumented with a Pyroscope language SDK.
Enable the profiler server
-
Add the following to your
custom-values.yamlfile:global: kfuse-profiling: enabled: trueyaml -
Optional: change the size of the local volume. The default of
50Gikeeps profiles until the volume fills, then deletes the oldest. Increase it to keep more history locally:pyroscope: pyroscope: persistence: size: 200GiyamlTo keep profiles longer than the volume can hold, also configure object storage. See Configure long-term retention.
-
Apply the change:
helm upgrade --install kfuse oci://us-east1-docker.pkg.dev/mvp-demo-301906/kfuse-helm/kfuse \ -n kfuse \ --version <VERSION> \ (1) -f custom-values.yamlbash1 Replace <VERSION>with a valid Kloudfuse release value. See Release Notes for the latest release. -
Confirm that the profiler server is running:
kubectl get pods -n kfuse | grep kfuse-profiler-serverThe
kfuse-profiler-server-0pod should report1/1 Running.
Annotate the pods to profile
Alloy profiles only the pods that opt in through annotations. Add these to the pod template of each workload, for example under spec.template.metadata.annotations in a Deployment:
annotations:
pyroscope.io/scrape: "true" (1)
pyroscope.io/application_name: checkout-service (2)
pyroscope.io/port: "6060" (3)
| 1 | Required. Only pods with this annotation set to "true" are scraped. |
| 2 | The service name shown in Kloudfuse. Becomes the service_name label. |
| 3 | The port that serves /debug/pprof/*. Omit it to use the pod’s container port. |
For a Go service, importing net/http/pprof and starting an HTTP listener on that port is enough to expose the endpoints. See Set up Go profiling in pull mode.
Install Grafana Alloy
Install Alloy in each Kubernetes cluster you want to profile. It runs as a single pod that discovers the annotated pods in every namespace and forwards their profiles to Kloudfuse.
-
Create a file named
alloy-values.yamlwith the following content:alloy: configMap: content: |- discovery.kubernetes "pods" { role = "pod" } discovery.relabel "pods" { targets = discovery.kubernetes.pods.targets rule { (1) action = "drop" source_labels = ["__meta_kubernetes_pod_phase"] regex = "Pending|Succeeded|Failed|Completed" } rule { (2) action = "keep" source_labels = ["__meta_kubernetes_pod_annotation_pyroscope_io_scrape"] regex = "true" } rule { action = "replace" source_labels = ["__meta_kubernetes_pod_annotation_pyroscope_io_application_name"] target_label = "service_name" } rule { action = "replace" source_labels = ["__address__", "__meta_kubernetes_pod_annotation_pyroscope_io_port"] regex = "(.+?)(?::\\d+)?;(\\d+)" replacement = "$1:$2" target_label = "__address__" } rule { action = "replace" source_labels = ["__meta_kubernetes_pod_annotation_pyroscope_io_scheme"] regex = "(https?)" target_label = "__scheme__" } rule { (3) action = "replace" source_labels = ["__meta_kubernetes_namespace"] target_label = "kubernetes_namespace" } rule { action = "replace" source_labels = ["__meta_kubernetes_pod_name"] target_label = "kubernetes_pod_name" } rule { action = "labelmap" regex = "__meta_kubernetes_pod_label_(.+)" } } pyroscope.scrape "pods" { targets = discovery.relabel.pods.output forward_to = [pyroscope.write.kloudfuse.receiver] profiling_config { (4) profile.process_cpu { enabled = true } profile.memory { enabled = true } profile.goroutine { enabled = true } profile.block { enabled = false } profile.mutex { enabled = false } profile.fgprof { enabled = false } } } pyroscope.write "kloudfuse" { endpoint { url = "https://<KLOUDFUSE_HOST>/profile" (5) } } controller: (6) type: deployment replicas: 1yaml1 Skips pods that are not running. 2 Keeps only pods annotated with pyroscope.io/scrape: "true".3 Adds the namespace, pod name, and pod labels as profile labels, so you can filter and group by them in Kloudfuse. 4 The profile types to collect. Block and mutex profiles are off by default because the application must enable them with runtime.SetBlockProfileRateandruntime.SetMutexProfileFraction. For the full list, see the pyroscope.scrape reference.5 Replace <KLOUDFUSE_HOST>with the host name of your Kloudfuse installation.6 Runs Alloy as one pod. The chart’s default is a DaemonSet, with one Alloy pod per node; because this configuration discovers pods across the whole cluster, every one of those Alloy pods would scrape every target, collecting each profile once per node. One Alloy pod handles most clusters. To spread scraping across several pods instead, set
alloy.clustering.enabled: trueandcontroller.replicasinalloy-values.yaml, and addclustering { enabled = true }insidepyroscope.scrape "pods". See the pyroscope.scrape clustering reference. -
Install Alloy with the Grafana Helm chart. Use any namespace;
alloyis shown here:helm repo add grafana https://grafana.github.io/helm-charts helm repo update helm upgrade --install alloy grafana/alloy \ --namespace alloy --create-namespace \ -f alloy-values.yaml -
Confirm that Alloy is scraping your pods. Forward the Alloy UI port, then open
http://localhost:12345and check that thepyroscope.scrape.podscomponent is healthy and lists your targets:kubectl port-forward -n alloy svc/alloy 12345:12345
To push profiles from a language SDK instead of scraping them, point the SDK’s server address at https://<KLOUDFUSE_HOST>/profile. For Java, see the Pyroscope Java client.
View profiles
In the Kloudfuse UI, open the avatar menu, select Grafana, open Explore, and choose the KfuseProfileDatasource data source. In the legacy UI, open Grafana › Profiler instead. Choose a profile type, then select your service by its service_name. Profiles appear within a minute of the first scrape.
Profiles are accepted only if their timestamp is no more than 1 hour old. If Alloy is down for longer than that, the profiles from the gap are lost.