VictoriaMetrics Agent

The VictoriaMetrics Agent (vmagent) is a lightweight metrics collection agent that scrapes Prometheus-compatible endpoints. It forwards the results to Kloudfuse using the Prometheus remote write protocol. It is a drop-in alternative to a Prometheus server for pure metric collection, with lower memory usage and built-in sharding support for large environments.

Overview

vmagent scrapes targets defined in a scrape_configs block — the same format as Prometheus — and writes the collected samples to one or more remote write endpoints. Kloudfuse accepts remote write at /ingester/write, making it compatible with any vmagent deployment without additional configuration.

When data reaches the Kloudfuse ingester it is stored in the time-series store and becomes immediately queryable via PromQL, dashboards, and alerts.

Prerequisites

  • A running Kloudfuse installation with its external hostname (for example, kloudfuse.example.com).

  • Helm 3 installed on your workstation.

  • If ingestion authentication is enabled on your cluster, an API key — see Ingestion Authentication with API Key.

Install with Helm

The official vmagent Helm chart is published to the GitHub Container Registry.

  1. Create a values.yaml file with the Kloudfuse remote write endpoint:

    remoteWrite:
  - url: https://<kloudfuse-hostname>/ingester/write
    yaml

    If ingestion authentication is enabled, add the Authorization header:

    remoteWrite:
  - url: https://<kloudfuse-hostname>/ingester/write
    headers:
      - "Authorization: Bearer <token>"
    yaml

  2. Install the agent — replace <namespace> with the Kubernetes namespace where you want to deploy it:

    helm upgrade --install vmagent \
  oci://ghcr.io/victoriametrics/helm-charts/victoria-metrics-agent \
  -f values.yaml \
  -n <namespace> \
  --create-namespace

  3. Confirm the pod is running:

    kubectl get pods -n <namespace> -l app=vmagent

Configure Scraping

vmagent uses the same scrape_configs format as Prometheus. Add targets to your values.yaml under config.scrape_configs:

config:
  scrape_configs:
    - job_name: my-app
      static_configs:
        - targets:
            - my-app-service.my-namespace.svc.cluster.local:8080

    - job_name: node-exporter
      kubernetes_sd_configs:
        - role: node
      relabel_configs:
        - source_labels: [__address__]
          regex: (.+):(.+)
          target_label: __address__
          replacement: ${1}:9100
yaml

For Kubernetes service discovery, grant vmagent the required RBAC permissions. The Helm chart creates a ClusterRole and ClusterRoleBinding by default when rbac.create: true (the default).

Multiple Remote Write Endpoints

vmagent supports writing to multiple endpoints simultaneously. To send metrics to Kloudfuse and retain a local Prometheus instance at the same time:

remoteWrite:
  - url: https://<kloudfuse-hostname>/ingester/write
    headers:
      - "Authorization: Bearer <token>"
  - url: http://prometheus.monitoring.svc.cluster.local:9090/api/v1/write
yaml

Verify Metrics Are Arriving

After installation, confirm metrics are arriving in the Kloudfuse UI:

  1. Click the Metrics tab, then select Explorer from the drop-down menu.

  2. Set the time range to the last 15 minutes using the interval picker in the top-right corner.

  3. In the metric selector, type the name of a metric scraped by vmagent (for example, node_cpu_seconds_total if scraping node-exporter, or vmagent_remotewrite_requests_total to check `vmagent’s own self-monitoring metrics).

  4. Select the metric from the suggestions list and confirm a chart appears with data points.

  5. Use the label filters to narrow results by job or instance — for example, filter by job="node-exporter" to isolate metrics from a specific scrape target.

Troubleshooting

No Metrics in Kloudfuse

If PromQL queries return no data:

  1. Check vmagent pod logs for remote write errors:

    kubectl logs -n <namespace> -l app=vmagent --tail=50

    Look for cannot send data or connection refused messages.

  2. Confirm the remote write URL is reachable from inside the cluster:

    kubectl exec -n <namespace> <vmagent-pod-name> -- \
  wget -qO- https://<kloudfuse-hostname>/health/

  3. Check that the ingester endpoint path is exactly /ingester/write — the /ingester/ prefix is required when routing through the Kloudfuse ingress.

400 or 401 Errors in Logs

If vmagent logs show HTTP 400 or 401 responses from the remote write endpoint:

  • Confirm ingestion authentication is enabled on your cluster — if it is, the Authorization: Bearer <token> header is required.

  • Verify the token is valid via Admin > Settings > Auth key labels in the Kloudfuse UI.

  • Check that the header is formatted exactly as Authorization: Bearer <token> with no extra whitespace.

High Cardinality or Memory Pressure

vmagent streams metrics without local storage by default. If you are scraping high-cardinality targets:

  • Enable persistent queue on disk to buffer remote write failures:

    remoteWrite:
  - url: https://<kloudfuse-hostname>/ingester/write
    persistentQueue:
      enabled: true
    yaml

  • Use remoteWrite.maxBlockSize and remoteWrite.queues to tune throughput.

External References