APM Java Instrumentation

Kloudfuse APM supports zero-code automatic instrumentation of Java applications using the OpenTelemetry Java agent. Attaching the agent with a single JVM flag instruments 1,000+ libraries automatically—including Spring, JDBC, gRPC, and Kafka—without modifying any source code. Traces flow into Kloudfuse where you can analyze latency, errors, and service dependencies.

Diagram showing the OpenTelemetry Java agent attaching to a JVM process at startup — it instruments supported libraries via bytecode transformation and exports spans through the BatchSpanProcessor and OTLP Exporter to kf-agent which forwards to the Kloudfuse platform

How It Works

The OpenTelemetry Java agent is a standard Java agent JAR attached to the JVM at startup via the -javaagent: flag. It requires no source code changes and no build system integration.

At JVM startup the agent:

  1. Registers a ClassFileTransformer using the java.lang.instrument API

  2. Intercepts library classes as they are loaded and rewrites their bytecode

  3. Inserts span creation, attribute collection, and context propagation calls at instrumentation points (HTTP entry, SQL execution, message dispatch, etc.)

  4. Wires all generated spans through a BatchSpanProcessor to the configured OTLP exporter

Because instrumentation is injected at class-load time, your application code is unaware of it. The spans it produces are identical to what you would write manually with the OpenTelemetry SDK.

Prerequisites

  • Java 8 or later (any JVM: HotSpot, OpenJ9, GraalVM)

  • Access to set JVM startup flags (command line, environment variable, or Kubernetes pod spec)

  • Your Kloudfuse API key — see Ingestion Authentication with API Key

OTLP Endpoints

There are three ways to route spans from a Java application to Kloudfuse, depending on how your cluster is deployed. Choose one and use it consistently across your OTEL_* environment variables or -Dotel.* system properties.

The standard deployment runs kf-agent as a DaemonSet or sidecar in the cluster. Applications send spans to it over gRPC on port 4317; the agent batches and forwards them to the Kloudfuse backend.

OTEL_EXPORTER_OTLP_ENDPOINT=http://kf-agent:4317
OTEL_EXPORTER_OTLP_HEADERS=kf-api-key=<your-api-key>
bash

Or as JVM system properties:

-Dotel.exporter.otlp.endpoint=http://kf-agent:4317
bash

This is the default protocol used by the Java Agent. No additional protocol flag is needed.

Authentication

Pass the Kloudfuse API key as the kf-api-key request header on the OTLP exporter.

Supported Libraries

The agent instruments 1,000+ libraries across the most common Java ecosystem. Libraries are grouped below by category.

HTTP Servers

Library Versions

Spring MVC / Spring Boot

Spring 3.1+, Spring Boot 1.5 – 3.x

Servlet API (Tomcat, Jetty, Undertow, Glassfish)

Servlet 2.3+

Netty HTTP server

4.0+

Quarkus (Vert.x-based)

1.0+

Micronaut HTTP server

2.0+

JAX-RS (Jersey, RESTEasy)

1.0+

Akka HTTP

10.0+

HTTP Clients and RPC

Library Versions

Apache HttpClient (4.x and 5.x)

4.0+, 5.0+

OkHttp

3.0+

Java HttpClient (JDK 11+)

JDK 11+

gRPC client and server

1.6+

HttpURLConnection

all

Spring WebClient / WebFlux

5.0+

Databases and Caches

Library Versions

JDBC (all drivers: PostgreSQL, MySQL, Oracle, MSSQL, H2)

any JDBC driver

Hibernate ORM

3.3+

Spring Data JPA / Spring Data JDBC

1.8+

MongoDB driver

3.1+

Redis (Jedis and Lettuce)

1.4+, 4.0+

Elasticsearch client

6.0+

Cassandra driver

3.0+

Couchbase client

2.0+

Messaging

Library Versions

Apache Kafka (producer and consumer)

0.11+

RabbitMQ AMQP client

2.7+

AWS SDK v2 (SQS, SNS, S3, DynamoDB)

2.2+

AWS SDK v1

1.11+

Spring Kafka

2.7+

Spring AMQP / Spring Rabbit

1.0+

JMS (ActiveMQ, IBM MQ)

1.1+

Logging

The agent injects the active trace ID and span ID into log records automatically, enabling log-trace correlation in Kloudfuse.

Library Versions

Log4j 2 (MDC injection)

2.7+

Logback (MDC injection)

1.0+

Log4j 1.x

1.0+

JUL (java.util.logging)

any

After the agent injects trace context, each log line will contain trace_id and span_id fields that Kloudfuse uses to link logs directly to the corresponding trace.

Agent Configuration

Key Configuration Properties

All properties below can be set as JVM system properties (-Dotel.property.name=value) or as environment variables (OTEL_PROPERTY_NAME=value).

Property Description Default

otel.service.name

Service name — the most important attribute. Must be set.

unknown_service:<executable>

otel.exporter.otlp.endpoint

Base OTLP endpoint. The SDK appends /v1/traces for HTTP exporters. Use for kf-agent on port 4317 (gRPC) or 4318 (HTTP).

http://localhost:4317

otel.exporter.otlp.traces.endpoint

Signal-specific traces endpoint. Used verbatim — the SDK does NOT append /v1/traces. Required when sending directly to the Kloudfuse ingester at a non-standard path.

otel.exporter.otlp.protocol

Exporter wire protocol: grpc or http/protobuf. Default is gRPC. Set to http/protobuf when using port 4318 or the direct ingester path.

grpc

otel.exporter.otlp.compression

Set to gzip to reduce network bandwidth.

none

otel.traces.sampler

Sampling strategy. Use parentbased_traceidratio in production.

parentbased_always_on

otel.traces.sampler.arg

Sample rate for traceidratio and parentbased_traceidratio (0.0–1.0).

1.0

otel.propagators

Context propagation formats. Default covers W3C TraceContext and Baggage.

tracecontext,baggage

otel.resource.attributes

Comma-separated key=value resource attributes (namespace, version, env).

otel.javaagent.configuration-file

Path to a .properties file containing all OTel configuration.

otel.javaagent.logging

Agent log routing: simple (stderr), none, or application (routes to SLF4J).

simple

otel.javaagent.debug

Set to true to enable verbose agent debug logging.

false

otel.bsp.schedule.delay

How often the BatchSpanProcessor exports, in milliseconds.

5000

otel.bsp.max.queue.size

Maximum span queue size before spans are dropped.

2048

otel.bsp.max.export.batch.size

Maximum spans per export batch.

512

otel.resource.providers.aws.enabled

Enable AWS resource detection (EC2, ECS, EKS, Lambda).

false

otel.resource.providers.gcp.enabled

Enable GCP resource detection (GCE, GKE, Cloud Run).

false

otel.resource.providers.azure.enabled

Enable Azure resource detection (VMs, AKS).

false

Selective Instrumentation

Disable Specific Libraries

To silence a noisy or irrelevant instrumentation, disable it by name without affecting everything else:

# Suppress Log4j appender instrumentation (MDC injection)
-Dotel.instrumentation.log4j-appender.enabled=false

# Suppress JDBC instrumentation
-Dotel.instrumentation.jdbc.enabled=false

# Suppress Spring Scheduling (cron jobs creating spans)
-Dotel.instrumentation.spring-scheduling.enabled=false
bash

Start from Zero

To instrument only specific libraries and nothing else:

# Disable everything
-Dotel.instrumentation.common.default-enabled=false

# Re-enable only what you need
-Dotel.instrumentation.opentelemetry-api.enabled=true
-Dotel.instrumentation.spring-webmvc.enabled=true
-Dotel.instrumentation.jdbc.enabled=true
-Dotel.instrumentation.logback-appender.enabled=true
bash

Automatic Instrumentation

Step 1: Download the Agent

Download the latest OpenTelemetry Java agent JAR:

curl -L -o opentelemetry-javaagent.jar \
  https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/latest/download/opentelemetry-javaagent.jar
bash

Place the JAR where your JVM can read it at startup — for container deployments, bake it into the image or mount it as a volume (see Kubernetes Integration).

Step 2: Attach the Agent

Add the -javaagent: flag when launching your application. Set the service name and OTLP endpoint at minimum. See OTLP Endpoints above to choose the right endpoint for your deployment.

Using kf-agent (gRPC, port 4317):

java \
  -javaagent:/path/to/opentelemetry-javaagent.jar \
  -Dotel.service.name=my-java-service \
  -Dotel.exporter.otlp.endpoint=http://kf-agent:4317 \
  -Dotel.resource.attributes=service.namespace=payments,service.version=1.4.2,deployment.environment.name=production \
  -jar myapp.jar
bash

Direct to Kloudfuse Ingester (HTTPS, no kf-agent):

java \
  -javaagent:/path/to/opentelemetry-javaagent.jar \
  -Dotel.service.name=my-java-service \
  -Dotel.exporter.otlp.traces.endpoint=https://<KFUSE_CLUSTER_DNS>/ingester/otlp/traces \
  -Dotel.exporter.otlp.protocol=http/protobuf \
  -Dotel.exporter.otlp.compression=gzip \
  -Dotel.resource.attributes=service.namespace=payments,deployment.environment.name=production \
  -jar myapp.jar
bash

For larger deployments, use a properties file instead of a long list of -D flags:

java \
  -javaagent:/path/to/opentelemetry-javaagent.jar \
  -Dotel.javaagent.configuration-file=/etc/otel/otel-config.properties \
  -jar myapp.jar
bash
# /etc/otel/otel-config.properties
otel.service.name=my-java-service
# Use otel.exporter.otlp.endpoint for kf-agent, or
# otel.exporter.otlp.traces.endpoint for direct ingester (see OTLP Endpoint Options above)
otel.exporter.otlp.endpoint=http://kf-agent:4317
otel.exporter.otlp.compression=gzip
otel.resource.attributes=service.namespace=payments,service.version=1.4.2,deployment.environment.name=production
otel.traces.sampler=parentbased_traceidratio
otel.traces.sampler.arg=0.05
otel.propagators=tracecontext,baggage
otel.javaagent.logging=application
properties

All OTEL_* environment variables are equivalent to their -Dotel.* system property counterparts:

export OTEL_SERVICE_NAME=my-java-service
export OTEL_EXPORTER_OTLP_ENDPOINT=http://kf-agent:4317
export OTEL_EXPORTER_OTLP_COMPRESSION=gzip
export OTEL_TRACES_SAMPLER=parentbased_traceidratio
export OTEL_TRACES_SAMPLER_ARG=0.05

java -javaagent:/path/to/opentelemetry-javaagent.jar -jar myapp.jar
bash

Manual Instrumentation

The Java Agent covers most library boundaries automatically. Use manual instrumentation for business logic, background jobs, or code paths that auto-instrumentation does not reach.

Span Annotations (Recommended, Java Agent only)

When the Java Agent is attached, the @WithSpan annotation creates a span around a method automatically. No SDK imports are needed in your business logic:

import io.opentelemetry.instrumentation.annotations.WithSpan;
import io.opentelemetry.instrumentation.annotations.SpanAttribute;

@Service
public class PaymentService {

    @WithSpan("process-payment")
    public PaymentResult processPayment(
        @SpanAttribute("payment.method") String method,
        @SpanAttribute("payment.amount")  double amount
    ) {
        // A span named "process-payment" is created automatically.
        // Method parameters become span attributes.
        return charge(method, amount);
    }
}
java

Add the annotations dependency to your build (no agent version pinning needed — the agent provides the implementation at runtime):

<dependency>
  <groupId>io.opentelemetry.instrumentation</groupId>
  <artifactId>opentelemetry-instrumentation-annotations</artifactId>
  <version>2.8.0</version>
</dependency>
xml

Programmatic Spans with the Java Agent

For more control — custom attributes, conditional error recording, or nested operations — use the SDK Tracer API alongside the Java Agent. The agent registers the TracerProvider with GlobalOpenTelemetry during JVM startup, so GlobalOpenTelemetry.getTracer() is available in main() without any additional setup.

This approach works reliably when an auto-instrumented framework (Spring Boot, Servlet, gRPC) has already initialized before the first manual API call. For standalone batch jobs or background loops where no framework precedes your main() code, see SDK-Only Instrumentation (No Java Agent) below.

import io.opentelemetry.api.GlobalOpenTelemetry;
import io.opentelemetry.api.trace.Span;
import io.opentelemetry.api.trace.SpanKind;
import io.opentelemetry.api.trace.StatusCode;
import io.opentelemetry.api.trace.Tracer;
import io.opentelemetry.context.Scope;

Tracer tracer = GlobalOpenTelemetry.getTracer("com.example.payments");

public InvoiceResult processInvoice(String invoiceId) {
    Span span = tracer.spanBuilder("process-invoice")
        .setSpanKind(SpanKind.SERVER)   // required for service map + request metrics
        .setAttribute("invoice.id", invoiceId)
        .setAttribute("invoice.currency", "USD")
        .startSpan();

    try (Scope scope = span.makeCurrent()) {
        InvoiceResult result = doProcessing(invoiceId);
        span.setAttribute("invoice.amount", result.getAmount());
        return result;
    } catch (Exception e) {
        span.recordException(e);
        span.setStatus(StatusCode.ERROR, e.getMessage());
        throw e;
    } finally {
        span.end();
    }
}
java
Call both span.recordException(e) and span.setStatus(StatusCode.ERROR, …​) together. They do not imply each other — omitting setStatus means the span is still marked OK and won’t appear in error rate calculations.

Always set SpanKind.SERVER on the outermost span of a service handling inbound requests. Kloudfuse derives throughput, latency, and error rate metrics from SERVER spans — spans with SpanKind.INTERNAL (the default) are stored but excluded from service-level aggregations.

SDK-Only Instrumentation (No Java Agent)

For standalone applications — batch jobs, background loops, Kubernetes pods with no HTTP server — the Java Agent is not required. Initialize the SDK directly using AutoConfiguredOpenTelemetrySdk, which reads all OTEL_* environment variables automatically (endpoint, headers, protocol, service name, etc.).

Add these two Maven dependencies (use the BOM or pin all io.opentelemetry artifacts to the same version):

<dependencies>
  <!-- Reads OTEL_* env vars and sets up the SDK as the global TracerProvider -->
  <dependency>
    <groupId>io.opentelemetry</groupId>
    <artifactId>opentelemetry-sdk-extension-autoconfigure</artifactId>
    <version>1.40.0</version>
  </dependency>
  <!-- OTLP HTTP exporter — use with OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf -->
  <dependency>
    <groupId>io.opentelemetry</groupId>
    <artifactId>opentelemetry-exporter-otlp</artifactId>
    <version>1.40.0</version>
  </dependency>
</dependencies>
xml

Call AutoConfiguredOpenTelemetrySdk.builder().setResultAsGlobal().build() at the start of main() before creating any tracers:

import io.opentelemetry.api.GlobalOpenTelemetry;
import io.opentelemetry.api.trace.Span;
import io.opentelemetry.api.trace.SpanKind;
import io.opentelemetry.api.trace.Tracer;
import io.opentelemetry.context.Scope;
import io.opentelemetry.sdk.autoconfigure.AutoConfiguredOpenTelemetrySdk;

public class MyService {
    public static void main(String[] args) throws Exception {
        // Initialize SDK from OTEL_* env vars and register as global.
        // Reads: OTEL_SERVICE_NAME, OTEL_EXPORTER_OTLP_TRACES_ENDPOINT,
        //        OTEL_EXPORTER_OTLP_HEADERS, OTEL_EXPORTER_OTLP_PROTOCOL, etc.
        AutoConfiguredOpenTelemetrySdk.builder()
                .setResultAsGlobal()
                .build();

        Tracer tracer = GlobalOpenTelemetry.getTracer("my-service");

        // Root SERVER span — visible in service map and request metrics
        Span serverSpan = tracer.spanBuilder("handle-request")
                .setSpanKind(SpanKind.SERVER)
                .startSpan();
        try (Scope serverScope = serverSpan.makeCurrent()) {

            // Child CLIENT span — outbound call, shares the same trace ID
            Span clientSpan = tracer.spanBuilder("db-query")
                    .setSpanKind(SpanKind.CLIENT)
                    .startSpan();
            try (Scope clientScope = clientSpan.makeCurrent()) {
                // ... execute the query ...
            } finally {
                clientSpan.end();
            }

        } finally {
            serverSpan.end();
        }
    }
}
java

The OTEL_* environment variables are identical to those consumed by the Java Agent — no changes to Kubernetes pod specs or Secrets are required when switching between the two approaches.

Java Agent 2.x has a known limitation with manual API calls made at the very start of main() before any auto-instrumented framework has loaded. If you see java.lang.NoClassDefFoundError: io/opentelemetry/api/GlobalOpenTelemetry when using the Java Agent with a standalone application, use the SDK-only approach above instead. See Java Agent 2.x: ClassNotFoundException with Manual API Usage for a full explanation.

Kubernetes Integration

Option 1: Init Container with kf-agent (Recommended)

Use an init container to download the agent JAR into a shared volume. This keeps the agent version managed separately from your application image. JAVA_TOOL_OPTIONS is read by the JVM automatically before any other flags — it is the most reliable way to inject -javaagent: in containers where you do not control the entrypoint.

initContainers:
  - name: otel-agent-init
    image: busybox
    command:
      - sh
      - -c
      - |
        wget -q -O /agent/opentelemetry-javaagent.jar \
          https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/latest/download/opentelemetry-javaagent.jar
    volumeMounts:
      - name: otel-agent
        mountPath: /agent

containers:
  - name: java-app
    image: myapp:latest
    env:
      - name: JAVA_TOOL_OPTIONS
        value: "-javaagent:/agent/opentelemetry-javaagent.jar"
      - name: OTEL_SERVICE_NAME
        value: my-java-service
      - name: OTEL_EXPORTER_OTLP_ENDPOINT
        value: http://kf-agent:4317
      - name: OTEL_EXPORTER_OTLP_COMPRESSION
        value: gzip
      - name: OTEL_TRACES_SAMPLER
        value: parentbased_traceidratio
      - name: OTEL_TRACES_SAMPLER_ARG
        value: "0.05"
      - name: OTEL_RESOURCE_ATTRIBUTES
        value: service.namespace=payments,deployment.environment.name=production
      - name: OTEL_EXPORTER_OTLP_HEADERS
        valueFrom:
          secretKeyRef:
            name: kloudfuse-api-key
            key: value
    volumeMounts:
      - name: otel-agent
        mountPath: /agent

volumes:
  - name: otel-agent
    emptyDir: {}
yaml

Option 2: Direct to Kloudfuse Ingester (No kf-agent)

When kf-agent is not deployed, use the direct HTTPS ingester path with OTEL_EXPORTER_OTLP_TRACES_ENDPOINT (signal-specific). The agent is still used here for auto-instrumentation — only the destination changes.

initContainers:
  - name: otel-agent-init
    image: busybox
    command:
      - sh
      - -c
      - |
        wget -q -O /agent/opentelemetry-javaagent.jar \
          https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/latest/download/opentelemetry-javaagent.jar
    volumeMounts:
      - name: otel-agent
        mountPath: /agent

containers:
  - name: java-app
    image: myapp:latest
    env:
      - name: JAVA_TOOL_OPTIONS
        value: "-javaagent:/agent/opentelemetry-javaagent.jar"
      - name: OTEL_SERVICE_NAME
        value: my-java-service
      # Signal-specific endpoint — used verbatim, SDK does not append /v1/traces
      - name: OTEL_EXPORTER_OTLP_TRACES_ENDPOINT
        value: "https://<KFUSE_CLUSTER_DNS>/ingester/otlp/traces"
      - name: OTEL_EXPORTER_OTLP_PROTOCOL
        value: http/protobuf
      - name: OTEL_EXPORTER_OTLP_COMPRESSION
        value: gzip
      - name: OTEL_RESOURCE_ATTRIBUTES
        value: service.namespace=payments,deployment.environment.name=production
      - name: OTEL_EXPORTER_OTLP_HEADERS
        valueFrom:
          secretKeyRef:
            name: kloudfuse-api-key
            key: value
    volumeMounts:
      - name: otel-agent
        mountPath: /agent

volumes:
  - name: otel-agent
    emptyDir: {}
yaml

Option 3: Bake into the Image

Alternatively, include the JAR in your Docker image:

FROM eclipse-temurin:21-jre

COPY opentelemetry-javaagent.jar /otel/opentelemetry-javaagent.jar
COPY myapp.jar /app/myapp.jar

ENV JAVA_TOOL_OPTIONS="-javaagent:/otel/opentelemetry-javaagent.jar"
ENV OTEL_SERVICE_NAME=my-java-service
ENV OTEL_EXPORTER_OTLP_ENDPOINT=http://kf-agent:4317

ENTRYPOINT ["java", "-jar", "/app/myapp.jar"]
dockerfile

See Kubernetes Setup → for deploying and configuring kf-agent in your cluster.

Verify Traces

  1. Open Kloudfuse UI → APM → Trace Explorer

  2. In the search bar, filter by service.name = my-java-service

  3. Click a trace to inspect the span tree, attributes, and timing breakdown

To confirm auto-instrumentation is active, enable agent debug logging temporarily:

-Dotel.javaagent.debug=true
bash

Look for log lines like [opentelemetry-javaagent] Instrumented class: org.springframework.web.servlet.DispatcherServlet to confirm specific libraries are being instrumented.

For production-ready guidance on service naming, sampling strategy, cardinality, error handling, and Java-specific configuration, see APM Best Practices →.

References

Sample Manifest

An example Kubernetes manifest and instructions can be found at APM Demo - Java

Specification and Concepts

Java Agent

Java SDK (Manual and SDK-Only Instrumentation)