Datadog Agent for PostgreSQL

Kloudfuse collects metrics and logs from self-managed PostgreSQL deployments using the Kloudfuse observability agent. The agent discovers PostgreSQL pods automatically via Kubernetes pod annotations — no separate scraper deployment is required.

Overview

The Kloudfuse observability agent (kfuse-observability-agent) is a customised Datadog agent that runs as a DaemonSet — one pod on every Kubernetes node in your cluster. It uses Datadog Autodiscovery: it continuously watches pod annotations prefixed with ad.datadoghq.com/ to find services to monitor.

When the agent finds a PostgreSQL pod with the right annotations, it:

  1. Resolves %%host%% to the pod’s cluster IP and %%port%% to its port — automatically, from the pod’s own network identity.

  2. Opens a TCP connection to PostgreSQL using the credentials in the annotation.

  3. Runs the built-in postgres check, querying pg_stat_statements, pg_stat_bgwriter, pg_stat_replication, and pg_stat_database.

  4. Forwards the results to the Kloudfuse ingester as time-series metrics.

The flow for logs is similar: the agent tail-reads the PostgreSQL log output from the same pod and sends log lines to the Kloudfuse log ingestion endpoint.

Prerequisites

Enable pg_stat_statements

The postgres check depends on the pg_stat_statements extension for query-level metrics.

  1. Add it to shared_preload_libraries in postgresql.conf:

    shared_preload_libraries = 'pg_stat_statements'
    ini

    If you manage PostgreSQL with Helm (for example, Bitnami’s postgresql chart):

    postgresql:
  primary:
    postgresqlSharedPreloadLibraries: "pg_stat_statements"
    yaml

  2. Restart PostgreSQL, then create the extension in each database you want to monitor:

    CREATE EXTENSION IF NOT EXISTS pg_stat_statements;
    sql

Create a Monitoring User

The postgres check connects using a dedicated low-privilege user. Follow Datadog’s pre-flight steps to create this user and grant the required permissions.

At minimum, the user needs:

CREATE USER kfmon WITH PASSWORD '<password>' CONNECTION LIMIT 5;
GRANT pg_monitor TO kfmon;              -- PostgreSQL 10+
GRANT SELECT ON pg_stat_database TO kfmon;
GRANT SELECT ON pg_stat_statements TO kfmon;
sql

On PostgreSQL 14+, also grant access to the statements info view:

GRANT SELECT ON pg_stat_statements_info TO kfmon;
sql

pg_monitor (PostgreSQL 10+) grants read-only access to all pg_stat_* views. CONNECTION LIMIT 5 prevents the monitoring user from consuming connection slots needed by the application — the Datadog Agent postgres check uses a single persistent connection per instance.

For PostgreSQL 9.6 and earlier, replace the pg_monitor role with the explicit grants in the Datadog documentation.

Also ensure pg_hba.conf allows connections from the Kubernetes pod network CIDR.

Verify the Observability Agent Is Running

Confirm DaemonSet pods are running on the same nodes as your PostgreSQL pods:

kubectl get pods -n <kloudfuse-namespace> -l app=kfuse-observability-agent -o wide
kubectl get pods -n <postgresql-namespace> -l app.kubernetes.io/name=postgresql -o wide

Every node that runs a PostgreSQL pod must also have a healthy observability agent pod.

Configure Autodiscovery Annotations

The observability agent picks up the PostgreSQL check when your pod carries the following annotations. Add them to the pod template of your PostgreSQL Deployment, StatefulSet, or Helm chart values.

Metrics

podAnnotations:
  ad.datadoghq.com/postgresql.check_names: '["postgres"]'
  ad.datadoghq.com/postgresql.init_configs: '[{}]'
  ad.datadoghq.com/postgresql.instances: >-
    [{
      "host": "%%host%%",
      "port": 5432,
      "username": "kfmon",
      "password": "<password>",
      "tags": ["env:<environment>", "service:postgresql"]
    }]
yaml

%%host%% is an Autodiscovery template variable that the agent replaces with the pod’s cluster IP at runtime. The agent on the same node resolves the IP automatically — you do not specify it manually.

If you use the Bitnami postgresql chart, add the annotations under both primary.podAnnotations and readReplicas.podAnnotations:

postgresql:
  primary:
    podAnnotations:
      ad.datadoghq.com/postgresql.check_names: '["postgres"]'
      ad.datadoghq.com/postgresql.init_configs: '[{}]'
      ad.datadoghq.com/postgresql.instances: >-
        [{"host": "%%host%%", "port": 5432, "username": "kfmon", "password": "<password>", "tags": []}]
  readReplicas:
    podAnnotations:
      ad.datadoghq.com/postgresql.check_names: '["postgres"]'
      ad.datadoghq.com/postgresql.init_configs: '[{}]'
      ad.datadoghq.com/postgresql.instances: >-
        [{"host": "%%host%%", "port": 5432, "username": "kfmon", "password": "<password>", "tags": []}]
yaml

Logs

To collect PostgreSQL logs alongside metrics, add:

podAnnotations:
  ad.datadoghq.com/postgresql.logs: >-
    [{"source": "postgresql", "service": "postgresql"}]
yaml

Combine this with the metrics annotations in a single podAnnotations block.

Apply the Annotations

kubectl rollout restart statefulset/<postgresql-statefulset> -n <namespace>

The agent detects the new annotations within the next Autodiscovery refresh cycle (typically within 30 seconds of pod startup).

Verify Metrics Are Arriving

After restarting the agent, confirm metrics are arriving in the Kloudfuse UI:

  1. Click the Metrics tab, then select Explorer from the drop-down menu.

  2. Set the time range to the last 15 minutes using the interval picker in the top-right corner.

  3. In the metric selector, type postgresql_connections and select it from the suggestions list. Confirm a chart appears with data points.

  4. Use the label filters to narrow results by database or host — for example, filter by db="mydb" to isolate metrics for a specific database.

Key metrics collected by the postgres check:

Metric Description

postgresql_connections

Number of active client connections

postgresql_rows_fetched

Rows fetched per second by database

postgresql_rows_inserted / _updated / _deleted

DML activity rates

postgresql_commits / _rollbacks

Transaction commit and rollback rates

postgresql_deadlocks

Deadlock count per interval

postgresql_bgwriter_buffers_checkpoint

Buffers written by the checkpointer

postgresql_replication_delay

Replication lag in bytes (primary only)

postgresql_cache_hit

Buffer cache hit ratio

postgresql_statement_*

Per-query execution statistics from pg_stat_statements

To verify logs are arriving, click the Logs tab, filter by source postgresql in the Filters panel, and confirm log entries appear in the list.

Troubleshooting

No PostgreSQL Metrics in Kloudfuse

If postgresql_connections returns no data:

  1. Check that the observability agent pod is on the same node as the PostgreSQL pod:

    kubectl get pods -n <kloudfuse-namespace> -l app=kfuse-observability-agent -o wide
kubectl get pods -n <postgresql-namespace> -l app.kubernetes.io/name=postgresql -o wide

  2. Run the check manually from the agent pod:

    kubectl exec -n <kloudfuse-namespace> <agent-pod-name> -- agent check postgres

  3. Check agent logs for connection errors:

    kubectl logs -n <kloudfuse-namespace> <agent-pod-name> | grep -i "postgres"

Authentication Error in Agent Logs

If the agent logs show FATAL: password authentication failed for user "kfmon":

  • Confirm the password in the ad.datadoghq.com/postgresql.instances annotation matches the PostgreSQL user password.

  • Verify the user exists: SELECT usename FROM pg_user WHERE usename='kfmon';

  • Confirm pg_hba.conf allows connections from the pod network CIDR.

Statement Metrics Are Missing

  • Confirm pg_stat_statements is in shared_preload_libraries and PostgreSQL was restarted after the change.

  • Run SELECT * FROM pg_stat_statements LIMIT 1; as the monitoring user.

  • On PostgreSQL 14+, the monitoring user also needs SELECT on pg_stat_statements_info.

Replica Metrics Are Missing

postgresql.replication_delay is only populated on the primary. Use readReplicas.podAnnotations in the Bitnami chart, or annotate each replica pod separately.

External References