Kubernetes Autoscaling (HPA) with Custom Metrics

You must have the following to scale components based on custom metrics:

To collect metrics from applications and store them in the Kloudfuse database.
To extend the Kubernetes Custom Metrics API with metrics supplied by a collector, the k8s-prometheus-adapter. The adapter is an implementation of the custom metrics API that supports arbitrary metrics.

Consider a case where Kloudfuse ingests an application-specific metric, some_metric. You plan to modify it, and use its value for autoscale the kubernetes pods. Follow these steps:

Install Prometheus adapter with proper configuration to emit a custom_metric based on a query for some_metric.

Create prometheus-values.yaml file for the Prometheus Adapter, configuring the proper endpoint, authentication, and custom metric rules.

metricsRelistInterval: 5m

prometheus:
  # Replace with your endpoint for Kloudfuse
  url: <<https://your.hostname.kloudfuse>>
  port: 443
  path: ""

# Rules to generate `some_custom_metric` as an external metric
rules:
  default: false
  external:
  - seriesQuery: '{__name__=~"some_metric"}'
    resources:
      template: <<.Resource>>
    name:
      as: "some_custom_metric"
    metricsQuery: avg (some_metric)

# Base64 Encoded Auth string in the format <<username:password>>
extraArguments:
  - --prometheus-header=Authorization=Basic YWRtaW46cGFzc3dvcmQ=

Deploy Prometheus adapter on your cluster.

helm install --name prometheus-adapter >prometheus-community/prometheus-adapter -f prometheus-values.yaml

Verify that the adapter emits custom-metric correctly, using the Kubernetes External Metrics API.

See Prometheus Adapter code and documentation for additional configuration information.

kubectl get --raw /apis/external.metrics.k8s.io/v1beta1 | jq .

#Should show some output like this
{
  "kind": "APIResourceList",
  "apiVersion": "v1",
  "groupVersion": "external.metrics.k8s.io/v1beta1",
  "resources": [
    {
      "name": "some_custom_metric",
      "singularName": "",
      "namespaced": true,
      "kind": "ExternalMetricValueList",
      "verbs": [
        "get"
      ]
    }
  ]
}

After custom-metric becomes available to use within the Kubernetes Metrics API, configure HPA to use the metric.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: <<some_name>>
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: <<some_name>>
  minReplicas: 1
  maxReplicas: 10
  metrics:
  - type: External
    external:
      metric:
        name: some_custom_metric
      target:
        type: AverageValue
        averageValue: 100

Explore how to configure HPA with Custom metrics in Kubernetes documentation for Autoscaling on multiple metrics and custom metrics.