Configure Stream Isolation on Pinot Tenants

When installing Kloudfuse, you may choose to isolate resources at the level of internal Pinot DB. You can designate specific deployments to service assigned streams: metrics, logs, traces, RUM, and so on. You must then mark the node with a specific label, such as kf_stream=metrics, and so on; see Tag Nodes.

Supported Streams

These are the supported streams:

pinot-metrics: Enable through global.metrics.isolated.

Node affinity: kf_stream=metrics

pinot-logs: Enable through global.logs.isolated.

Node affinity: kf_stream=logs

pinot-traces: Enable through global.traces.isolated.

Node affinity: kf_stream=traces

pinot-events: Enable through `global.events.isolated'.

Node affinity: kf_stream=events

pinot-rum: Enable through global.rum.isolated.

Node affinity: kf_stream=rum

Customize Helm Values

Consider the code in the Kloudfuse custom-values.yaml file, Configure Stream Isolation, and ensure that you implement all the requirements to correctly isolate telemetry streams:

Set global.metrics.isolated to true.
Add a new pinot-metrics section.
Set pinot-metrics.fullnameOverride to pinot-metrics; the pinot-* section matches the fullnameOverride.

For pinot-logs it must be pinot-logs, for pinot-traces it must be pinot-traces, and so on.
Set up the rest of the file like a default pinot deployment. The only required sections are pinot.broker, pinot.server.realtime, and pinot.server.offline
When enabled, nodes that service metrics must be marked with the label kf_stream=metrics.
Do not specify zookeeper or controller sections; they are specified in the default pinot section.
The pinot-metrics pods have a node affinity to the label kf_stream=metrics. Before deployment, you must tag the nodes you want to dedicate with kf_stream=metrics. Leave the default nodes untagged. The default pinot deployment has a node affinity that requires the absence of the kf_stream label.
Work with Kloudfuse Customer Support to get the correct jvmOpts values for your cluster.

Ensure that the sum of the replicaCount values accross all pinot-tenants sections equals the number of nodes.

Configure Stream Isolation

pinot-metrics: (1)
   fullnameOverride: pinot-metrics (2)
   peerDownload:
     enabled: true

   deepStore: (3)
     enabled: false

   broker: (4)
     jvmOpts: "[CONSULT WITH KLOUDFUSE SUPPORT]" (6)
     replicaCount: 1
     podAntiAffinityPreset: "hard"
     nodeSelector: {}
     affinity: {}
     tolerations: []

   server: (5)
     probes:
       livenessEnabled: false
       readinessEnabled: false
     realtime:
       replicaCount: 1
       jvmOpts: "[CONSULT WITH KLOUDFUSE SUPPORT]" (6)
       persistence:
         size: 100G
         storageClass: "kfuse-ssd"

     offline:
       enabled: true
       replicaCount: 1
       jvmOpts: "[CONSULT WITH KLOUDFUSE SUPPORT]" (6)
       persistence:
         size: 256G
         storageClass: "kfuse-ssd-offline"
     podAntiAffinityPreset: "hard"
     nodeSelector: {}
     affinity: {}
     tolerations: []

1	`pinot-metrics`: Configuration for Pinot metrics tenant requires `global.metrics.isolated` to be `true.
2	`fullnameOverride`: Must match the name of the deployment, here `pinot-metrics`.
3	When `deepStore` is enabled, configure additional parameters depending on where you host `deepStore`: GCP, AWS, or Azure object store.
4	Specify the `broker` section.
5	Specify the `server` section.
6	`jvmOpts`: Consult Kloudfuse Customer Support to get the correct values for your cluster.

Tag Nodes

We do not recommend that you manually tag nodes because if the cloud provider recycles the node, the configuration is lost.

Instead, create separate node group or pool for each stream, and attach the required Kubernetes label to identify the stream. This ensures that any node created in the node group has the correct label.

For example, this is how to configure node groups or node pools in [aws], Azure, and GCP.

Azure
AWS
GCP

Create a node pool.
Under the Optional setting tab, scroll to the Labels section.
Specify a key kf_stream with value metrics.

Create a Node Group.
In Step 1, Configure node group, scroll to the Kubernetes labels section.
Specify a key kf_stream with value metrics.

Add a Node pool.
In the Metadata section, scroll to the Labels section.
Specify a key kf_stream with value metrics.

List Available Nodes

To get a list of available nodes, issue this command:

kubectl get nodes

To filter the list of available nodes by node group or namespace, issue this command:

kubectl get nodes -l

Verify

During the deployment process, the set-tags job dedicates/assigns the pinot-metrics pods only to service metrics.

After finishing the deployment, port forward the pinot-controller:

kubectl port-forward pinot-controller-0 9000:9000
Go to localhost:9000

Click on Tenants.
Find realtime_metrics and verify that:
- The server count is the same as the realtime replicas you specified in your deployment.
- The broker count is the same as the broker replicas you specified in your deployment.
Find offline_metrics and verify that:
- The server count is the same as the offline replicas you specified in your deployment.
- The broker count is the same as the broker replicas you specified in your deployment.

Scale the Stream-Isolated Cluster

Follow these steps to ensure that your stream-isolated cluster scales to the workload:

Add the desired number of nodes to your existing cluster.
Tag these nodes with the stream you are scaling. See the list of Supported Streams, and how to Tag Nodes.
In your configuration.yaml file, increase replicaCount for all services by the number of additional nodes. See the code listing Configure Stream Isolation.

Do not increase the replicaCount for pinot and the kafka.zookeeper sections.
In the pinot section, increase the replicaCount of the controller.
In the pinot-STREAM section, increase the replicaCount values of server.offline, server.realtime, and broker.
Proportionally increase the number of topic partitions in Kafka.
Increase the Kafka disk space using the following formula:

\(diskSpace = numberOfPartitions * replicationFactor * 10GB\)
Upgrade the helm using the new configuration.
Re-assign Kafka partitions to new brokers; see Increase Kafka partition replication factor.
Re-balance Pinot segments:
1. Run the following command:
  kubectl port-forward pinot-controller-0 9000:9000
2. Navigate to the configuration UI in your Web browser:
  
  http://localhost:9000/#/tenants/table/kf_logs_REALTIME
3. Select the Rebalance Servers option at the top of the page.
4. In the Rebalance Server dialog:
  - Turn on the Downtime option.
  - Turn on the Best Efforts option.
  - Click Save.

The rebalancing process may take a long time, depending on the volume of data that must be moved.