APM Go Instrumentation

Kloudfuse APM supports Go-based distributed systems via the OpenTelemetry Go SDK. This guide walks through setting up instrumentation and exporting spans to the Kloudfuse backend.

Prerequisites

OTLP Endpoints

There are three ways to route spans from a Go application to Kloudfuse, depending on how your cluster is deployed. Choose one and use it consistently across your OTEL_* environment variables.

The standard deployment runs kf-agent as a DaemonSet or sidecar in the cluster. Applications send spans to it over gRPC on port 4317; the agent batches and forwards them to the Kloudfuse backend.

OTEL_EXPORTER_OTLP_ENDPOINT=http://kf-agent:4317
OTEL_EXPORTER_OTLP_HEADERS=kf-api-key=<your-api-key>
bash

Use the otlptracegrpc exporter package:

go get go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc
go
import "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc"

// All config is read from OTEL_EXPORTER_OTLP_ENDPOINT and OTEL_EXPORTER_OTLP_HEADERS.
exporter, err := otlptracegrpc.New(ctx)
go

Authentication

Pass the Kloudfuse API key as a request header on the OTLP exporter.

Both the gRPC and HTTP exporters read OTEL_EXPORTER_OTLP_HEADERS automatically — no code changes are needed:

export OTEL_EXPORTER_OTLP_HEADERS="kf-api-key=<your-api-key>"
bash

The variable accepts a comma-separated list of name=value pairs, so multiple headers can be passed:

export OTEL_EXPORTER_OTLP_HEADERS="kf-api-key=<your-api-key>,x-env=production"
bash

In Kubernetes, pull the value from a Secret so it never appears in source:

env:
  - name: OTEL_EXPORTER_OTLP_HEADERS
    valueFrom:
      secretKeyRef:
        name: kloudfuse-api-key
        key: value
yaml

HTTP Servers

For HTTP servers, use otelhttp.NewHandler from the contrib package. It automatically sets SpanKind = SERVER, extracts incoming W3C traceparent headers, and populates standard HTTP semantic convention attributes (http.method, http.status_code, http.target, etc.):

go get go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp
bash
import "go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"

http.Handle("/api/orders", otelhttp.NewHandler(ordersHandler, "GET /api/orders"))
http.ListenAndServe(":8080", nil)
go

For outbound HTTP calls, wrap the transport so traceparent headers are injected automatically:

client := &http.Client{Transport: otelhttp.NewTransport(http.DefaultTransport)}
req, _ := http.NewRequestWithContext(ctx, "GET", url, nil)
client.Do(req)
go
otelhttp v0.53.x emits deprecated HTTP attribute names (http.method, http.status_code). Upgrade to v0.56+ together with otel core v1.31+ to get the stable names (http.request.method, http.response.status_code). See Go: Semantic Convention Limitations in otelhttp v0.53.x for the full attribute mapping and upgrade steps.

Manual Instrumentation

Step 1: Import OpenTelemetry Modules

Add the core OpenTelemetry packages to your Go module. This example uses the HTTP exporter (Option 2 or 3 above); swap otlptracehttp for otlptracegrpc if using gRPC.

go get go.opentelemetry.io/otel@v1.28.0 \
  go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp@v1.28.0 \
  go.opentelemetry.io/otel/sdk@v1.28.0
bash

Your go.mod should look like:

require (
  go.opentelemetry.io/otel v1.28.0
  go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.28.0
  go.opentelemetry.io/otel/sdk v1.28.0
)

Step 2: Configure the Tracer Provider

Set up the exporter and tracer provider in main(). All OTLP connection settings — endpoint, headers, protocol, compression — are read from OTEL_* environment variables automatically. Only the resource attributes (service name, environment, namespace) need to be set in code.

package main

import (
	"context"
	"log"

	"go.opentelemetry.io/otel"
	"go.opentelemetry.io/otel/attribute"
	"go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp"
	"go.opentelemetry.io/otel/sdk/resource"
	sdktrace "go.opentelemetry.io/otel/sdk/trace"
)

func initTracer(ctx context.Context) (*sdktrace.TracerProvider, error) {
	// Endpoint, headers, and protocol are read from OTEL_EXPORTER_OTLP_* env vars.
	exporter, err := otlptracehttp.New(ctx)
	if err != nil {
		return nil, err
	}

	res := resource.NewWithAttributes(
		"https://opentelemetry.io/schemas/1.21.0",
		attribute.String("service.name", "my-go-service"),
		attribute.String("deployment.environment.name", "production"),
		attribute.String("service.namespace", "my-team"),
	)

	tp := sdktrace.NewTracerProvider(
		sdktrace.WithBatcher(exporter),
		sdktrace.WithResource(res),
	)

	otel.SetTracerProvider(tp)
	return tp, nil
}

func main() {
	ctx := context.Background()
	tp, err := initTracer(ctx)
	if err != nil {
		log.Fatalf("failed to init tracer: %v", err)
	}
	defer tp.Shutdown(ctx)

	// ... start your application
}
go

Call tp.Shutdown(ctx) on exit to flush any buffered spans before the process terminates.

Step 3: Create Spans

Use the global tracer to create spans. Always set SpanKind explicitly on the entry-point span of your service — SpanKind.Server for inbound HTTP/gRPC handlers, SpanKind.Client for outbound calls. Kloudfuse derives service-level throughput and latency metrics from SERVER spans.

import (
	"go.opentelemetry.io/otel"
	"go.opentelemetry.io/otel/trace"
)

tracer := otel.Tracer("my-go-service")

// Root SERVER span — represents an inbound request handled by this service.
ctx, serverSpan := tracer.Start(ctx, "handle-request",
	trace.WithSpanKind(trace.SpanKindServer),
)
defer serverSpan.End()

// Child CLIENT span — represents an outbound call (DB query, downstream service, etc.).
// Inherits the parent's trace ID automatically because ctx carries the parent.
_, clientSpan := tracer.Start(ctx, "db-query",
	trace.WithSpanKind(trace.SpanKindClient),
)
defer clientSpan.End()
go

To add attributes to a span:

import "go.opentelemetry.io/otel/attribute"

span.SetAttributes(
	attribute.String("db.system", "postgresql"),
	attribute.String("db.name", "orders"),
)
go

Verify Traces

  1. Open Kloudfuse UI → APM → Trace Explorer

  2. Filter by service.name = my-go-service

  3. Click a trace to inspect the span tree, attributes, and timing breakdown

References

Sample Manifest

An example Kubernetes manifest and instructions can be found at APM Demo - Go

Specification and Concepts

Go SDK and Packages

Exporter Configuration