Prometheus Remote Write

Kloudfuse accepts metrics from any Prometheus-compatible source using the Prometheus remote write protocol. Point your existing Prometheus server, Prometheus Operator deployment, or Grafana Agent at the Kloudfuse remote write endpoint — no migration or schema changes required.

Overview

The Prometheus remote write protocol sends compressed, time-series metric batches over HTTP to a remote storage backend. Kloudfuse exposes this endpoint at:

https://<kloudfuse-hostname>/ingester/write

All Prometheus versions 2.x and later support remote write. Native histograms are supported from Prometheus 2.40 and Kloudfuse 3.4.3 onwards.

Prerequisites

  • The external hostname of your Kloudfuse cluster (for example, kloudfuse.example.com).

  • If ingestion authentication is enabled on your cluster, an API key — see Ingestion Authentication with API Key.

  • If your Prometheus server is in a different VPC or cluster than Kloudfuse, use the external IP or DNS name reachable from outside.

    If cross-VPC DNS does not resolve, use the Kloudfuse ingress IP directly. Find it with: kubectl get svc | grep kfuse-ingress-nginx-controller

Standalone Prometheus

prometheus.yml

Add a remote_write block to your prometheus.yml:

remote_write:
  - url: https://<kloudfuse-hostname>/ingester/write
    authorization:
      credentials: <token>            # omit if auth is not enabled
    queue_config:
      max_samples_per_send: 4000
    send_native_histograms: true      # Prometheus 2.40+ and Kloudfuse 3.4.3+

scrape_configs:
  - job_name: my-app
    scrape_native_histograms: true    # Prometheus 2.50+ — enables native histogram scraping per job
    static_configs:
      - targets: [localhost:9090]
yaml

Standalone Helm Chart

If Prometheus is installed via the prometheus Helm chart, add the following values and run helm upgrade:

serverFiles:
  prometheus.yml:
    remote_write:
      - url: https://<kloudfuse-hostname>/ingester/write
        authorization:
          credentials: <token>           # omit if auth is not enabled
        queue_config:
          max_samples_per_send: 4000
        send_native_histograms: true     # Prometheus 2.40+ and Kloudfuse 3.4.3+
yaml
The --enable-feature=native-histograms server flag is a no-op in Prometheus 2.50 and later. To enable native histogram scraping, set scrape_native_histograms: true on each scrape_configs job instead.

Prometheus Operator

kube-prometheus-stack Helm Chart

If Prometheus is managed via kube-prometheus-stack, add the following Helm values:

prometheus:
  prometheusSpec:
    remoteWrite:
      - url: https://<kloudfuse-hostname>/ingester/write
        authorization:
          credentials: <token>         # omit if auth is not enabled
        queueConfig:
          maxSamplesPerSend: 4000
        sendNativeHistograms: true     # Prometheus 2.40+ and Kloudfuse 3.4.3+
yaml

Then run helm upgrade.

Prometheus CRD

To configure remote write directly on the Prometheus custom resource:

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  name: prometheus
  namespace: monitoring
spec:
  remoteWrite:
    - url: https://<kloudfuse-hostname>/ingester/write
      authorization:
        credentials: <token>           # omit if auth is not enabled
      queueConfig:
        maxSamplesPerSend: 4000
      sendNativeHistograms: true       # Prometheus 2.40+ and Kloudfuse 3.4.3+
yaml

Apply with kubectl apply -f prometheus-cr.yaml.

Grafana Agent

The Grafana Agent is a lighter-weight alternative to a full Prometheus server. It can scrape all existing scrape_configs and write to Kloudfuse simultaneously.

Helm Chart Values

agent:
  configMap:
    create: true
    content: |
      prometheus.remote_write "kloudfuse" {
        endpoint {
          url = "https://<kloudfuse-hostname>/ingester/write"
          bearer_token = "<token>"       // omit if auth is not enabled
        }
        queue_config {
          max_samples_per_send = 4000
        }
      }

      // Copy your existing scrape targets here, e.g.:
      prometheus.scrape "default" {
        targets = prometheus.kubernetes.pods.targets
        forward_to = [prometheus.remote_write.kloudfuse.receiver]
      }
yaml

Legacy Grafana Agent Config Format

For Grafana Agent versions prior to 0.40 using the static configuration format:

prometheus:
  global:
    remote_write:
      - url: https://<kloudfuse-hostname>/ingester/write
        authorization:
          credentials: <token>
        queue_config:
          max_samples_per_send: 4000
  configs:
    - name: default
      scrape_configs:
        # paste your existing scrape_configs here
yaml

Native Histograms

Native histograms (also called exponential histograms) provide higher-resolution bucketing with lower cardinality than classic Prometheus histograms. Kloudfuse supports native histograms from version 3.4.3 onwards, and Prometheus supports sending them from version 2.40.

To enable, add to your Prometheus configuration:

  • enable-feature: native-histograms in extraArgs (or enableFeatures in CRD spec)

  • send_native_histograms: true on each remote_write entry

See Metric Native and Exponential Histograms for details on querying and visualizing native histograms in Kloudfuse.

Verify Metrics Are Arriving

After applying the configuration, confirm metrics are arriving in the Kloudfuse UI:

  1. Click the Metrics tab, then select Explorer from the drop-down menu.

  2. Set the time range to the last 15 minutes using the interval picker in the top-right corner.

  3. In the metric selector, type a well-known metric name from your environment (for example, node_cpu_seconds_total if scraping Kubernetes nodes, or up which is emitted by all Prometheus scrape jobs).

  4. Select the metric from the suggestions list and confirm a chart appears with data points.

  5. Use the label filters to narrow results by job or instance — for example, filter by job="node-exporter" to isolate metrics from a specific scrape target.

Troubleshooting

No Metrics in Kloudfuse

If queries return no data after configuring remote write:

  1. Check Prometheus logs for remote write errors:

    kubectl logs -n <namespace> -l app=prometheus --tail=50 | grep -i "remote"

  2. Confirm the remote write URL is reachable from the Prometheus pod:

    kubectl exec -n <namespace> <prometheus-pod> -- \
  wget -qO- https://<kloudfuse-hostname>/health/

  3. Verify the endpoint path is exactly /ingester/write. A missing /ingester/ prefix will result in 404 errors.

401 Unauthorized Errors

If Prometheus logs show remote write: server returned HTTP status 401 Unauthorized:

  • Authentication is enabled on your cluster — add authorization.credentials: <token> to the remote_write block.

  • Confirm the token is active in Admin > Settings > Auth key labels in the Kloudfuse UI.

Native Histograms Not Arriving

If send_native_histograms: true is set but no native histogram data appears in Kloudfuse:

  • In Prometheus 2.50+, the --enable-feature=native-histograms server flag is a no-op. Enable scraping per job with scrape_native_histograms: true in each scrape_configs entry instead:

    scrape_configs:
  - job_name: my-app
    scrape_native_histograms: true
    static_configs:
      - targets: [my-app:9090]
    yaml

  • Confirm the target application exposes native histograms — classic HISTOGRAM types in the exposition format are not native histograms.

  • Verify Kloudfuse version is 3.4.3 or later.

Metrics Are Delayed or Backlogged

If prometheus_remote_storage_pending_samples is growing:

  • Increase max_samples_per_send to reduce the number of round trips.

  • Increase capacity in queue_config to buffer more samples in memory.

  • Check Kloudfuse ingester availability — a temporary outage causes backpressure that Prometheus will drain automatically once connectivity is restored.

External References