APM Python Instrumentation

Kloudfuse APM supports Python applications via the OpenTelemetry Python SDK. This guide covers both automatic instrumentation (zero-code, via the opentelemetry-instrument CLI wrapper) and manual SDK initialization for full control over span creation and export.

Prerequisites

OTLP Endpoints

There are three ways to route spans from a Python application to Kloudfuse, depending on how your cluster is deployed. Choose one and use it consistently across your OTEL_* environment variables.

The standard deployment runs kf-agent as a DaemonSet or sidecar in the cluster. Applications send spans to it over gRPC on port 4317; the agent batches and forwards them to the Kloudfuse backend.

OTEL_EXPORTER_OTLP_ENDPOINT=http://kf-agent:4317
OTEL_EXPORTER_OTLP_HEADERS=kf-api-key=<your-api-key>
bash

Use the gRPC exporter package:

pip install opentelemetry-exporter-otlp-proto-grpc
bash
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter

# Endpoint and headers are read from OTEL_EXPORTER_OTLP_ENDPOINT
# and OTEL_EXPORTER_OTLP_HEADERS automatically.
exporter = OTLPSpanExporter()
python

Authentication

Pass the Kloudfuse API key as the kf-api-key request header on the OTLP exporter.

Both the gRPC and HTTP exporters read OTEL_EXPORTER_OTLP_HEADERS automatically — no code changes are needed:

export OTEL_EXPORTER_OTLP_HEADERS="kf-api-key=<your-api-key>"
bash

The variable accepts a comma-separated list of name=value pairs:

export OTEL_EXPORTER_OTLP_HEADERS="kf-api-key=<your-api-key>,x-env=production"
bash

This also works with the opentelemetry-instrument CLI — just set the variable before running it:

OTEL_EXPORTER_OTLP_HEADERS="kf-api-key=<your-api-key>" \
opentelemetry-instrument --traces_exporter otlp python app.py
bash

In Kubernetes, pull the value from a Secret so it never appears in source:

env:
  - name: OTEL_EXPORTER_OTLP_HEADERS
    valueFrom:
      secretKeyRef:
        name: kloudfuse-api-key
        key: value
yaml

Create the Secret with the full header expression:

kubectl create secret generic kloudfuse-api-key \
  --from-literal=value="kf-api-key=<your-api-key>"
bash

Fork Safety (Gunicorn / uWSGI)

If you run your application under Gunicorn or uWSGI, the SDK must be initialized after the server forks worker processes. Initializing at module import time (before the fork) causes the BatchSpanProcessor background thread and exporter connection to be copied into each worker in a broken state.

See Python Fork Safety for the post_fork hook pattern and full explanation.

Automatic Instrumentation

For web frameworks (Flask, Django, FastAPI), the opentelemetry-instrument CLI wrapper monkey-patches the framework at import time, automatically creating SpanKind = SERVER spans for every HTTP request without any code changes.

Flask

pip install \
  opentelemetry-sdk \
  opentelemetry-exporter-otlp-proto-http \
  opentelemetry-instrumentation-flask
bash
OTEL_SERVICE_NAME=my-flask-service \
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://<KFUSE_CLUSTER_DNS>/ingester/otlp/traces \
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf \
OTEL_EXPORTER_OTLP_HEADERS="kf-api-key=<your-api-key>" \
opentelemetry-instrument \
  --traces_exporter otlp \
  --metrics_exporter none \
  --logs_exporter none \
  python app.py
bash

The Flask instrumentation sets SpanKind = SERVER and populates standard HTTP attributes (http.method, http.target, http.status_code, http.scheme, net.host.name) for every request. No changes to app.py are required.

Django

pip install \
  opentelemetry-sdk \
  opentelemetry-exporter-otlp-proto-http \
  opentelemetry-instrumentation-django
bash
OTEL_SERVICE_NAME=my-django-service \
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://<KFUSE_CLUSTER_DNS>/ingester/otlp/traces \
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf \
OTEL_EXPORTER_OTLP_HEADERS="kf-api-key=<your-api-key>" \
opentelemetry-instrument python manage.py runserver
bash

Setting Resource Attributes

Pass resource attributes via OTEL_RESOURCE_ATTRIBUTES — they are merged into every span exported by opentelemetry-instrument:

OTEL_RESOURCE_ATTRIBUTES=deployment.environment.name=production,service.namespace=my-team
bash

Manual Instrumentation

Step 1: Install the SDK

Install the core SDK and the HTTP exporter. The HTTP exporter works for Options 2 and 3 above and is recommended for Kubernetes deployments where only port 443 is guaranteed to be open.

pip install \
  opentelemetry-sdk \
  opentelemetry-exporter-otlp-proto-http
bash

For auto-instrumentation of web frameworks (Flask, Django, FastAPI), also install:

pip install opentelemetry-instrumentation-flask   # or -django, -fastapi, etc.
bash

To auto-detect and install instrumentation for all libraries already installed in your environment:

pip install opentelemetry-distro
opentelemetry-bootstrap --action=install
bash

Step 2: Configure the SDK

Initialize the TracerProvider with a BatchSpanProcessor and an OTLPSpanExporter. All connection settings — endpoint, headers, protocol — are read from OTEL_* environment variables automatically. Only the resource attributes (service name, environment, namespace) need to be set in code.

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource

# Resource attributes identify this service in Kloudfuse APM.
resource = Resource.create({
    "service.name": "my-python-service",
    "deployment.environment.name": "production",
    "service.namespace": "my-team",
})

# OTLPSpanExporter() with no arguments reads OTEL_EXPORTER_OTLP_TRACES_ENDPOINT (or
# OTEL_EXPORTER_OTLP_ENDPOINT), OTEL_EXPORTER_OTLP_HEADERS, and
# OTEL_EXPORTER_OTLP_PROTOCOL from the environment automatically.
provider = TracerProvider(resource=resource)
provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter()))
trace.set_tracer_provider(provider)
python

Call this initialization once at application startup, before creating any tracers or spans.

Step 3: Create Spans

Use start_as_current_span as a context manager to create spans. Always set kind explicitly on the entry-point span of your service — SpanKind.SERVER for inbound request handlers, SpanKind.CLIENT for outbound calls. Kloudfuse derives service-level throughput and latency metrics from SERVER spans.

from opentelemetry import trace
from opentelemetry.trace import SpanKind

tracer = trace.get_tracer("my-python-service")

# Root SERVER span — represents an inbound request handled by this service.
# Required for the service to appear in the Service Map with request metrics.
with tracer.start_as_current_span("handle-request", kind=SpanKind.SERVER) as server_span:
    server_span.set_attribute("http.method", "GET")
    server_span.set_attribute("http.route", "/orders")

    # Child CLIENT span — outbound call (DB, downstream service, cache, etc.).
    # Inherits the parent trace ID automatically via context manager nesting.
    with tracer.start_as_current_span("db-query", kind=SpanKind.CLIENT) as client_span:
        client_span.set_attribute("db.system", "postgresql")
        client_span.set_attribute("db.name", "orders")
        # ... execute the query ...
python

To record an exception and mark a span as failed:

from opentelemetry.trace import StatusCode

with tracer.start_as_current_span("process-order", kind=SpanKind.SERVER) as span:
    try:
        result = process(order_id)
    except Exception as e:
        span.record_exception(e)
        span.set_status(StatusCode.ERROR, str(e))
        raise
python
Call both record_exception() and set_status(StatusCode.ERROR, …​) together. They do not imply each other — omitting set_status leaves the span marked OK and it will not appear in error rate calculations.

Kubernetes Integration

The following pod spec shows a complete setup for direct ingester export (Option 3). The application code is written inline and run directly with python — no opentelemetry-instrument wrapper is needed when the SDK is initialized in code.

env:
  - name: OTEL_SERVICE_NAME
    value: "my-python-service"
  - name: OTEL_EXPORTER_OTLP_TRACES_ENDPOINT
    value: "https://<KFUSE_CLUSTER_DNS>/ingester/otlp/traces"
  - name: OTEL_EXPORTER_OTLP_PROTOCOL
    value: "http/protobuf"
  - name: OTEL_EXPORTER_OTLP_HEADERS
    valueFrom:
      secretKeyRef:
        name: kloudfuse-api-key
        key: value
  - name: OTEL_RESOURCE_ATTRIBUTES
    value: "deployment.environment.name=production,service.namespace=my-team"
yaml

Create the Secret with the full header expression:

kubectl create secret generic kloudfuse-api-key \
  --from-literal=value="kf-api-key=<your-api-key>"
bash

For kf-agent deployments (Option 1), replace the endpoint env vars with:

env:
  - name: OTEL_SERVICE_NAME
    value: "my-python-service"
  - name: OTEL_EXPORTER_OTLP_ENDPOINT
    value: "http://kf-agent:4317"
  - name: OTEL_EXPORTER_OTLP_HEADERS
    valueFrom:
      secretKeyRef:
        name: kloudfuse-api-key
        key: value
yaml

Verify Traces

  1. Open Kloudfuse UI → APM → Trace Explorer

  2. Filter by service.name = my-python-service

  3. Click a trace to inspect the span tree, attributes, and timing breakdown

References

Sample Manifest

An example Kubernetes manifest and instructions can be found at APM Demo - Python

Specification and Concepts

Python SDK and Packages

Framework and Server Hooks