Datadog Agent for Kubernetes

See the following documentation:

Collect high cardinality tags

The Datadog Agent is installed with cardinality set to orchestrator level, allowing granular (both pod and container-level) tagging of metrics.

For more information, see Datadog documentation on Assigning Tags.

The default setting in the Datadog Agent is set to low, tagging only at host level.

Install Command

If you haven’t before, add datadog helm repo:

helm repo add datadog https://helm.datadoghq.com
helm repo update

Install the datadog agent with updated values (cmd below assumes the file is called dd-values-kfuse.yaml. Replace that argument with the filename that’s relevant to your scenario).

# Create separate namespace for the agent to be installed (if required)
kubectl create namespace kfuse-agent
helm upgrade --install kfuse-agent -f dd-values-kfuse.yaml datadog/datadog -n kfuse-agent --version 3.6.7

Using Existing Datadog Agent Install

If you have an existing datadog agent, then you can use the same agent to send your data streams to the Kloudfuse platform.

Streaming to Kloudfuse in addition to Datadog

Datadog Agent supports dual shipping. To send the data to Kloudfuse in addition to the Datadog, add the following in the datadog agent helm values.

Change for each stream, as necessary.

Under the agents.customAgentConfig key, specify:

additional_endpoints:
  "http://<kfuse-ingress-external-ip>/ingester": (1)
   - <api_key>
use_v2_api:
  series: true
logs_config:
  ...
  logs_no_ssl: true
  use_v2_api: false
  additional_endpoints:
    - api_key: <api_key>
      Host: <kfuse-ingress-external-ip>
      Port: 80
      use_compression: true
apm_config:
  ...
  additional_endpoints:
    "https://kfuse-ingress-external-ip>":
       - <api_key>
process_config:
  ...
  additional_endpoints:
    "https://kfuse-ingress-external-ip>":
       - <api_key>
metadata_providers:
  - name: host
    interval: 300

Under the clusterAgent.datadog_cluster_yaml key, specify:

additional_endpoints:
  "http://<kfuse-ingress-external-ip>/ingester":
   - <api_key>
use_v2_api:
  series: true
process_config:
  ...
  additional_endpoints:
    "https://kfuse-ingress-external-ip>":
       - <api_key>
orchestrator_explorer:
  ...
  additional_endpoints:
    "https://kfuse-ingress-external-ip>":
       - <api_key>

1	Use `http://` instead of `https://` when referencing the Kloudfuse ingester.

Adding Custom Tags

You can add custom tags to the agent, ensuring that all metrics that you agent collects has the correct tag.

To add custom tags, follow these steps:

Update dd-values-kfuse.yaml file to include the following code to enable custom tags.

Each tag must be in the form key:value, a string separated by a column (:).
```
datadog:
  tags:
    - custom_tag_name:custom_tag_value
```
The Kloudfuse helm custom-values.yaml file must list the custom tags.
```
ingester:
  config:
    hostTagIncludes:
    ...
    - custom_tag_name (1)
    ...
```
1 custom_tag_name: Add all custom tag names; otherwise, the system may overwrite existing configurations.
Alternatively, if you don’t plan to maintain a long list of possible custom tags, set allowAllHostTags to true. This effectively adds all extracted custom tags to the allow list.
```
ingester:
  config:
    allowAllHostTags: true (1)
```
1 allowAllHostTags: Adds all custom tag names to the allow list; otherwise, the system may overwrite existing configurations.

Enabling Pods to be detected by Prometheus Autodiscovery

In addition to enabling prometheusScrape in the Datadog values.yaml, the pods must have the following annotations.

If you deploy application pods through helm, the helm values support a podAnnotations section.

prometheus.io/path: <specify prometheus endpoint path, e.g., /metrics>
prometheus.io/port: <SPECIFY prometheus endpoint here, e.g., "9090">
prometheus.io/scrape: "true"

Metadata for Metrics collected using the OpenMetrics check

If you use the preceding configuration, then the openmetrics check is enabled in the agent.

For more details, see Datadog documentation on Kubernetes Prometheus and OpenMetrics metrics collection.

The Kloudfuse agent installed using the provided values file uses a custom check, kf_openmetrics, does not collect metadata by default. You can:

Configure kf_openmetrics to collect metadata by annotating the required sources.
Enable the Autodiscovery agent for these pods.

See the Datadog documentation section Kubernetes and Integrations.
Run the custom check.

Kubernetes environment

To enable metrics metadata collection from a Kubernetes pod, as discussed in Datadog documentation section Kubernetes and Integrations: Configuration, add the following code.

You can add this to the helm and each deployment.

apiVersion: v1
kind: Pod
# (...)
metadata:
  name: '<POD_NAME>'
annotations:
    ad.datadoghq.com/<CONTAINER_IDENTIFIER>.check_names: '["kf_openmetrics"]'
    ad.datadoghq.com/<CONTAINER_IDENTIFIER>.init_configs: '[{}]'
    ad.datadoghq.com/<CONTAINER_IDENTIFIER>.instances: '[ { "openmetrics_endpoint": "<http://%%host%%:%%port%%/metrics>"}
      ]'
    # (...)
spec:
  containers:
    - name: '<CONTAINER_IDENTIFIER>'
# (...)

Advance monitoring using Knight

Enable kubernetes_state_metric

The Kloudfuse Advanced Analytics depend on kubernetes state metrics (KSM) check; by default, it is not enabled in the agent.

Ensure that the agent continues to capture these metrics through KSM by adding or updating the dd-agent values file.

datadog:
  kubeStateMetricsEnabled: true
  kubeStateMetricsCore:
    enabled: true
    ignoreLegacyKSMCheck: false

Enable Knight-based monitoring in Kloudfuse

Add the knightEnabled logic to the custom-values.yaml file.
```
ingester:
  ...
  config:
  knightEnabled: true
```
Upgrade the cluster.