Prometheus Pushgateway

Kloudfuse bundles the Prometheus Pushgateway to collect metrics from ephemeral and batch jobs that cannot be scraped directly by Prometheus. Jobs push their metrics to the Pushgateway over HTTP, and Kloudfuse scrapes the Pushgateway to ingest those metrics into the platform.

Overview

The Pushgateway pattern is intended for short-lived jobs such as cron tasks, CI/CD pipeline steps, and batch processing scripts that exit before a Prometheus scrape interval completes. By pushing metrics to the Pushgateway, these jobs can expose their results even after they have terminated.

When Pushgateway is enabled in Kloudfuse, the kfuse-pushgateway service is deployed inside your cluster and becomes available at the ingress path /metrics/.

Prerequisites

  • A running Kloudfuse installation with access to modify custom-values.yaml.

  • The cloud-exporter feature enabled in your Helm values (required for scraping the Pushgateway).

  • HTTP access to the Kloudfuse ingress from the hosts running your batch jobs.

Enable Pushgateway

  1. Add the following to your custom-values.yaml:

    global:
  pushgateway:
    enabled: true
    yaml

  2. Apply the change by upgrading your Kloudfuse Helm release:

    helm upgrade --install kfuse oci://us-east1-docker.pkg.dev/mvp-demo-301906/kfuse-helm/kfuse \
  -n kfuse \
  --version <VERSION> \ (1)
  -f custom-values.yaml
    1 Replace <VERSION> with a valid Kloudfuse release value; use the most recent one.

  3. Confirm the Pushgateway pod is running:

    kubectl get pods -n <kloudfuse-namespace> -l app.kubernetes.io/name=prometheus-pushgateway

    The pod should reach Running status within a minute.

Send Metrics to the Pushgateway

Use the Prometheus text exposition format to push metrics to the Pushgateway endpoint. Any HTTP client works — curl is shown in the examples below.

In the text format, every line must end with a Unix line-feed (\n). Lines ending with \r\n (Windows CRLF) or \r (CR only) cause a protocol error.

Metrics are grouped by a grouping key — a set of labels that together identify the job. The job label is required and must be the first element of the grouping key.

Send a Single Metric

Push a single metric into the group {job="batch-job"}:

echo "records_processed 1024" | \
  curl --data-binary @- https://<kloudfuse-hostname>/metrics/job/batch-job

Because no type information is provided, the metric is stored as type untyped.

Send Multiple Metrics with Metadata

Push multiple metrics with type and help annotations into the group {job="nightly-report",instance="host1"}:

cat <<EOF | curl --data-binary @- \
  https://<kloudfuse-hostname>/metrics/job/nightly-report/instance/host1
# TYPE records_processed counter
# HELP records_processed Total records processed in this run.
records_processed{status="success"} 9842
records_processed{status="error"} 17
# TYPE duration_seconds gauge
# HELP duration_seconds Job duration in seconds.
duration_seconds 142.3
EOF

Verify Metrics Are Ingested

After pushing, confirm metrics are arriving in the Kloudfuse UI:

  1. Click the Metrics tab, then select Explorer from the drop-down menu.

  2. Set the time range to the last 15 minutes using the interval picker in the top-right corner.

  3. In the metric selector, type the name of a metric you pushed (for example, records_processed).

  4. Select the metric from the suggestions list and confirm a chart appears with data points.

  5. Use the label filters to narrow results by job or instance — for example, filter by job="nightly-report" to isolate metrics from a specific push group.

The Kloudfuse ingester scrapes the Pushgateway on a regular interval (default 15 seconds). Allow up to 30 seconds after a successful push before checking for data in the UI.

Troubleshooting

Pushgateway Pod Is Not Running

If kubectl get pods shows no Pushgateway pod:

  • Confirm global.pushgateway.enabled: true is set in custom-values.yaml.

  • Re-run the Helm upgrade and check for errors in the output.

  • Run kubectl describe helmrelease kfuse -n <kloudfuse-namespace> to inspect Helm release status.

Push Returns a Non-2xx Response

If your curl push returns an HTTP error:

  • 400 Bad Request — the metric text is malformed. Verify that lines end with \n, not \r\n, and that metric names follow Prometheus naming rules (ASCII letters, digits, underscores, and colons only).

  • 403 Forbidden — ingestion authentication is enabled. Add the Kf-Api-Key header to your curl command: --header "Kf-Api-Key: <token>".

  • 404 Not Found — confirm the path is /metrics/job/<job-name> and that <kloudfuse-hostname> is correct.

Metrics Not Appearing in Kloudfuse

  • The Kloudfuse ingester scrapes the Pushgateway on a regular interval (default 15 s). Allow up to 30 seconds after a successful push before querying.

  • Confirm global.pushgateway.enabled: true is set and the Helm upgrade was applied successfully.

  • Check ingester logs for scrape errors: kubectl logs -n <kloudfuse-namespace> -l app=ingester --tail=50.

Old Metric Values Are Still Visible

The Pushgateway retains the last pushed value for a group until the group is explicitly deleted. If a job has not run recently, the Pushgateway still serves its last values.

Delete a stale group using the HTTP DELETE method:

curl -X DELETE https://<kloudfuse-hostname>/metrics/job/batch-job

External References