Quick Guide for OTel Agent on Kubernetes for Events

Use the following sample helm-values.yaml file to configure the OpenTelemetry Collector to collect events in Kloudfuse. Ensure that you correctly set the protocol as either https or http, based on TLS being enabled or disabled.

Due to limitations in the k8s_events receiver, the OpenTelemetry Collector must be run in Deployment mode rather than DaemonSet mode when collecting Kubernetes events. Running in DaemonSet mode can result in duplicate events being sent. This is a known upstream issue in the OpenTelemetry Collector. See GitHub issue #42266 for more details.

Add the k8s_events receiver into your configuration, so that OTel receives kubernetes events, and enable the pipeline to handle the receiver events, process them, and export them to a specific log point.

This is a separate logs pipeline. Kloudfuse does not use the existing logs pipeline to process events.

Learn how to Configure the Events Pipeline.
Define the exporter to send these processed Kubernetes events to the specified cluster.

The changes in the configuration look something like this. Ensure that you correctly set the protocol as either https or http, based on TLS being enabled or disabled.

exporters:
  <exporter_name>:
    logs_endpoint: https://<REPLACE KFUSE ADDRESS>/ingester/otlp/k8s_events (1)

receivers:
  k8s_events:

service:
  pipelines:
    logs/k8s_events:
      exporters:
      - <exporter_name>
      processors:
      - <processors>
      receivers:
      - k8s_events

yaml

1	Because OTEL wraps the events into logs, you must set a logs endpoint for the events stream. For example, use `ingester/otlp/k8s_events`. Kloudfuse can handle the data as events.

To add permissions, alter the clusterrole and add events to the resource.

rules:
  - apiGroups:
    - ""
    resources:
    - events
    - ...

yaml

To distinguish events from specific agents, add a processor and a key/value pair to distinguish OTel from other agents.

...
command:
...
  processors:
    batch:
      ...
    resource:
      attributes:
      - key: kf_events_agent
        value: "otlp"
        action: upsert
...

yaml

Restart your OTel collector, so it can begin sending events to your cluster.

Configure the Events Pipeline

You must add a new logs pipeline just for the Kubernetes events. The complete configuration would have 1 pipleline for logs, and another for logs/k8s_events.

The configuration may appear similar to this:

service:
  pipelines:
    logs/k8s_events:
      exporters:
      - <exporter_name>
      processors:
      ...
      - resource/k8s_events
      receivers:
      - k8s_events

processors:
  resource/k8s_events:
    attributes:
    - key: kf_events_agent
      value: "otlp"
      action: upsert

bash