Kloudfuse Architecture

The Kloudfuse platform ingests all observability telemetry streams into a single unified datastore, enabling consistent storage and querying across metrics, logs, traces, events, and RUM data. The data plane runs within your cloud account (VPC); an optional managed control plane handles lifecycle and health monitoring.

Telemetry Streams

Kloudfuse consolidates five telemetry stream types into a purpose-built unified datastore.

Kloudfuse telemetry streams

Platform Architecture

The Kloudfuse data plane runs entirely within the customer VPC. An optional managed control plane, operated by Kloudfuse outside the VPC, monitors health and manages the lifecycle of the data plane.

Kloudfuse data plane architecture

Components

Sources (external — outside VPC)

  • RUM — Real User Monitoring data from browser and mobile clients, routed into the VPC via the API protocol endpoint.

  • Applications — Services sending distributed traces via the Traces endpoint.

  • Cloud Sources — Cloud-native event streams delivered via Pub/Sub.

  • Agents — Host and container agents forwarding logs, metrics, and events.

Protocol

  • API — HTTP gateway entry point for RUM traffic.

  • Traces — Endpoint for distributed trace ingestion (OpenTelemetry, Datadog Agent).

  • Pub/Sub — Subscription-based endpoint for cloud service event streams.

  • Logs/Metrics/Events — Endpoint for agent-based telemetry (Prometheus, FluentBit, FileBeat).

Ingest

  • Ingester — Receives all inbound telemetry from the protocol layer and fans out to Kafka, Redis, and PostgreSQL.

  • Redis — In-memory cache written directly by the Ingester to accelerate hot-path queries.

  • PostgreSQL — Relational store written by the Ingester for platform metadata and configuration.

Processors

  • Transformers — Normalize and enrich data on the Kafka bus (bidirectional).

  • Archive Service — Moves data to and from cold storage via Kafka (bidirectional).

  • Log Parser — Parses and structures raw log streams from Kafka (bidirectional).

Message Bus

  • Kafka — Central message bus connecting the Ingester to all processors and Pinot.

Storage

  • Pinot — Real-time OLAP index. Receives processed data from Kafka and serves the Query Service.

Query

  • Query Service — Queries Pinot across three signal types and returns results to the interface layer:

    • Metrics — time-series metric queries (PromQL)

    • Logs — structured log queries (LogQL)

    • Traces — distributed trace queries

Interface

  • Grafana — Dashboarding and visualization; queries via the Query Service.

  • Kloudfuse UI — Native platform UI; queries via the Query Service.

  • Alerts — Alerting engine; evaluates conditions via the Query Service.

Users (external — outside VPC)

  • Users interact with the platform through Grafana and Kloudfuse UI.

Cloud Storage (outside VPC)

  • DeepStore — Persistent storage backend. Receives indexed data from Pinot and is backed by:

    • S3 — Amazon Simple Storage Service

    • GCS — Google Cloud Storage

    • ABS — Azure Blob Storage