Otel Collector Trace Collection

kf-agent is a Kubernetes-native collector that receives spans from OTel SDK-instrumented applications over OTLP and forwards them to the Kloudfuse ingester. Applications send spans to kf-agent over the cluster-internal network; no direct internet access from application pods is required.

Overview

kf-agent runs as a Kubernetes Deployment and exposes two OTLP endpoints:

Port Protocol Endpoint

4317

OTLP/gRPC

http://kf-agent.<namespace>:4317

4318

OTLP/HTTP (Protobuf)

http://kf-agent.<namespace>:4318

Replace <namespace> with the namespace where kf-agent is deployed (typically kloudfuse).

All spans received by kf-agent are forwarded to the Kloudfuse ingester over HTTPS. kf-agent buffers and compresses spans in transit, reducing the number of outbound connections from your cluster.

For agent installation and Helm configuration, see OTel Kubernetes Integration. This page focuses on configuring applications to send spans to a running agent.

Prerequisites

Before you begin, ensure the following requirements are in place:

Requirement Details

kf-agent

kf-agent deployed and running. Run kubectl get pods -n kloudfuse -l app=kf-agent to confirm.

OpenTelemetry SDK

Applications instrumented with an OpenTelemetry SDK. See the language guides under APM Instrumentation for SDK setup.

Configure Applications to Send Traces

Environment Variables

Configure every application pod with the following environment variables:

Variable Value

OTEL_EXPORTER_OTLP_ENDPOINT

http://kf-agent.<namespace>:4317 (gRPC) or http://kf-agent.<namespace>:4318 (HTTP)

OTEL_EXPORTER_OTLP_PROTOCOL

grpc (default) or http/protobuf

OTEL_SERVICE_NAME

A stable logical name for your service (e.g., payments-api)

OTEL_RESOURCE_ATTRIBUTES

service.namespace=<team>,deployment.environment.name=<env>

OTEL_SERVICE_NAME determines how the service appears in APM views and the service map. Use a stable, consistent name — avoid pod names or hostnames.

Kubernetes Pod Spec

env:
  - name: OTEL_EXPORTER_OTLP_ENDPOINT
    value: "http://kf-agent.kloudfuse:4317"
  - name: OTEL_SERVICE_NAME
    value: payments-api
  - name: OTEL_RESOURCE_ATTRIBUTES
    value: "service.namespace=payments,deployment.environment.name=production"
yaml

If your application namespace is isolated by a NetworkPolicy, you may need an egress rule to allow traffic to kf-agent — see NetworkPolicy.

gRPC vs HTTP

Most OTel SDKs default to gRPC on port 4317. Use HTTP on port 4318 if your language SDK or runtime does not support gRPC, or if your network blocks gRPC:

env:
  - name: OTEL_EXPORTER_OTLP_ENDPOINT
    value: "http://kf-agent.kloudfuse:4318"
  - name: OTEL_EXPORTER_OTLP_PROTOCOL
    value: "http/protobuf"
yaml

The SDK appends /v1/traces automatically when using the HTTP exporter — do not include it in the endpoint value.

Language-Specific Notes

The environment variables above work for Java, Python, and Go without additional code changes when the SDK is configured to read from the environment.

Java (auto-instrumentation agent) — the Java agent reads all OTEL_* variables automatically. Attach the agent with:

env:
  - name: JAVA_TOOL_OPTIONS
    value: "-javaagent:/agent/opentelemetry-javaagent.jar"
  - name: OTEL_EXPORTER_OTLP_ENDPOINT
    value: "http://kf-agent.kloudfuse:4317"
  - name: OTEL_SERVICE_NAME
    value: payments-api
yaml

Python (auto-instrumentation) — wrap the application command with opentelemetry-instrument:

command: ["opentelemetry-instrument", "python", "app.py"]
yaml

Then set OTEL_EXPORTER_OTLP_ENDPOINT and OTEL_SERVICE_NAME as environment variables on the container.

Go (manual SDK) — initialize the SDK from environment variables using autoconf:

import "go.opentelemetry.io/contrib/exporters/autoexport"
import "go.opentelemetry.io/otel/sdk/trace"

exp, _ := autoexport.NewSpanExporter(ctx)
tp := sdktrace.NewTracerProvider(sdktrace.WithBatcher(exp))
otel.SetTracerProvider(tp)
go

autoexport reads OTEL_EXPORTER_OTLP_ENDPOINT and OTEL_EXPORTER_OTLP_PROTOCOL automatically. Alternatively, configure the exporter explicitly — see APM Go Instrumentation.

NetworkPolicy

If your cluster enforces NetworkPolicy, add an egress rule to each application namespace that allows outbound traffic to kf-agent on ports 4317 and 4318:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-kf-agent-traces
  namespace: <app-namespace>
spec:
  podSelector: {}
  policyTypes:
    - Egress
  egress:
    - ports:
        - protocol: TCP
          port: 4317
        - protocol: TCP
          port: 4318
yaml

If kf-agent is in a different namespace, also add the namespaceSelector:

    - ports:
        - protocol: TCP
          port: 4317
        - protocol: TCP
          port: 4318
      to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kloudfuse
yaml

Verify Traces Are Arriving

  1. Confirm kf-agent is running and listening:

    kubectl get pods -n kloudfuse -l app=kf-agent
kubectl exec -it <app-pod> -- nc -zv kf-agent.kloudfuse 4317

  2. Generate traffic through your application — make a request to an instrumented endpoint.

  3. In the Kloudfuse UI, click APM in the top navigation.

  4. Select Services from the drop-down and set the time range to the last 15 minutes.

  5. Your service name (from OTEL_SERVICE_NAME) should appear. Click it, then select the Traces tab to see individual trace records.

Troubleshooting

Connection Refused to kf-agent

If nc -zv kf-agent.kloudfuse 4317 fails:

  1. Confirm kf-agent pods are running:

    kubectl get pods -n kloudfuse -l app=kf-agent

  2. Confirm the kf-agent Service exists and targets the correct port:

    kubectl get svc -n kloudfuse kf-agent

  3. Check whether a NetworkPolicy in the application namespace blocks egress on port 4317. Add the egress rule from NetworkPolicy if needed.

  4. Verify the DNS name resolves from inside the application pod:

    kubectl exec -it <app-pod> -- nslookup kf-agent.kloudfuse

Spans Not Appearing in Kloudfuse

If kf-agent is reachable but traces do not appear:

  1. Enable SDK debug logging to see export attempts and errors:

    SDK Debug flag

    Java agent

    -Dotel.javaagent.debug=true

    Python

    OTEL_LOG_LEVEL=debug

    Go

    Log any errors returned from tp.Shutdown(ctx) and from the exporter initialization

  2. Check kf-agent logs for export errors:

    kubectl logs -n kloudfuse -l app=kf-agent | grep -i "error\|warn\|failed"

  3. Confirm OTEL_SERVICE_NAME is set — without it the SDK defaults to unknown_service:<binary>.

  4. Confirm the endpoint does not include a path suffix — http://kf-agent.kloudfuse:4317 is correct; http://kf-agent.kloudfuse:4317/v1/traces is not (gRPC uses no path).

Wrong Protocol (gRPC vs HTTP)

Symptoms: HTTP 415 Unsupported Media Type or HTTP 405 Method Not Allowed in SDK logs.

  • Port 4317 accepts gRPC only.

  • Port 4318 accepts HTTP/Protobuf only.

Match the SDK exporter type to the port:

# gRPC (default for most SDKs)
OTEL_EXPORTER_OTLP_ENDPOINT=http://kf-agent.kloudfuse:4317

# HTTP
OTEL_EXPORTER_OTLP_ENDPOINT=http://kf-agent.kloudfuse:4318
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
bash

Partial or Broken Traces (Orphan Spans)

Orphan spans (spans with no parent in the UI) indicate that context propagation between services is broken.

  • Confirm all services in the call chain are using W3C TraceContext propagation (OTEL_PROPAGATORS=tracecontext,baggage).

  • Confirm outbound HTTP or gRPC calls pass the propagated headers — this is handled automatically by instrumentation libraries (otelhttp, otelgrpc) but requires explicit header injection for manual HTTP clients.

See Partial or Broken Traces for language-specific fixes.

Spans Dropped Under Load

Symptoms: traces appear complete at low throughput but are missing spans under high load; SDK logs show BatchSpanProcessor queue is full.

Increase the queue and batch sizes:

OTEL_BSP_MAX_QUEUE_SIZE=8192
OTEL_BSP_MAX_EXPORT_BATCH_SIZE=1024
OTEL_EXPORTER_OTLP_COMPRESSION=gzip
bash

If drops persist after tuning the BSP, add tail-based sampling at the kf-agent level using the Tail Sampling Processor in the agent configuration. Contact Kloudfuse support to enable this in the managed kf-agent configuration.

Authentication Errors (401 / 403)

If kf-agent logs show 401 Unauthorized or 403 Forbidden on the outbound ingester request:

  1. Confirm the API key in the kf-agent Helm values matches an active key in Kloudfuse under Administration → API Keys.

  2. The API key is set at the agent level, not the application level — application pods do not need to pass API credentials.

  3. If you recently rotated the key, update the kf-agent Helm values and restart the agent pods.

External References