Pinot Object Storage

Apache Pinot Deep Store is a permanent, persistent storage layer for segment files, acting as a backup and recovery mechanism rather than a real-time query path. It stores compressed segments, allowing servers to recover data, handle node failures, and facilitate cluster scaling by downloading segments from cloud storage (S3, ADLS, GCS) or HDFS.

Ensure that the deep store is set up in the same region that hosts your Kubernetes cluster.

Add the following configurations in the custom_values.yaml file for use with the Helm installation of Kloudfuse.

GCP
AWS
Azure

Kloudfuse supports two options for object store configuration in GCP:

Use a Service Account Key
Use Google Cloud Workload Identity

Use a Service Account Key

Prerequisites

Download the service account key from the GCP console.
Create a Kubernetes secret with the GCP credentials to allow access to the GCS bucket. Name the key file secretKey:
```
kubectl create secret generic cloud-storage-secret --from-file=./secretKey -n kfuse
```

Helm values

Add the following values in the custom_values.yaml file. Replace the GCS details as required.

GCP Configuration with Service Account Key

global:
  cloudStorage:
    type: gcs
    useSecret: true
    secretName: cloud-storage-secret
    gcs:
      bucket: "REPLACE BUCKET HERE"

pinot:
  deepStore: (1)
    enabled: true
    prefix: "kfuse/controller/data" (2)

yaml

1	`deepStore`: Enable/disable storage of Pinot segments in deep store.
2	`prefix`: Folder prefix in the specified bucket for deep storage.

Use Google Cloud Workload Identity

Prerequisites

Follow the steps in Authenticate to Google Cloud APIs from GKE workloads to create and associate a service account with the GKE cluster.
Use the following values:
ROLE_NAME
roles/storage.admin

Alternatively, create a custom role with the following permissions:
storage.buckets.get

storage.objects.create

storage.objects.delete

storage.objects.get

storage.objects.getIamPolicy

storage.objects.list

storage.objects.update
NAMESPACE

kfuse; create this namespace if you have not done so yet.

Alternatively, use the namespace of the Kloudfuse deployment.

KSA_NAME

default

Helm values

Add the following values in the custom_values.yaml file. Replace the GCS details as required.

GCP Configuration with Google Cloud Workload Identity

global:
  cloudStorage:
    type: gcs
    useSecret: false
    gcs:
      bucket: "REPLACE BUCKET HERE"

pinot:
  deepStore: (1)
    enabled: true
    prefix: "kfuse/controller/data" (2)

yaml

1	`deepStore`: Enable/disable storage of Pinot segments in deep store.
2	`prefix`: Folder prefix in the specified bucket for deep storage.

Pinot must have an IAM policy with read and write permissions to the S3 bucket for deep storage. Kloudfuse supports the following options:

Use an IAM User Secret Access Key
Attach the IAM policy to the NodeInstanceRole
Use a Kubernetes ServiceAccount with an IAM Role

Use an IAM User Secret Access Key

Prerequisites

Refer to the AWS documentation to Create an IAM user in your AWS account.
Ensure that the user has an IAM policy with read and write access to the S3 bucket for deep storage.
After creating the IAM user, generate access key credentials and note the access key and secret key values.

Create a Kubernetes secret with the AWS credentials to allow access to the S3 bucket:

kubectl create secret generic cloud-storage-secret --from-literal=accessKey='<accessKey>' --from-literal=secretKey='<secretKey>' -n kfuse

Helm values

Add the following values to the custom_values.yaml file:

AWS Configuration with IAM User Secret Access Key

global:
  cloudStorage:
    type: s3
    useSecret: true
    secretName: cloud-storage-secret
    s3:
      region: "REPLACE BUCKET REGION HERE"
      bucket: "REPLACE BUCKET HERE"
pinot:
  deepStore:  (1)
    enabled: true
    prefix: "kfuse/controller/data" (2)

yaml

1	`deepStore`: Enable/disable storing of Pinot segments in deep store.
2	`prefix`: Folder prefix in the specified bucket for deep storage.

Attach the IAM Policy to the NodeInstanceRole

Attach the IAM policy to the NodeInstanceRole of the node that runs the Kloudfuse stack. On the EKS console, navigate to the corresponding EKS cluster’s node group detail page and locate the NodeInstanceRole under the Node IAM role ARN.
Set both createSecret and useSecret to false.
Add the following values in the custom_values.yaml file. Replace the S3 details with your values:
AWS Configuration with IAM Policy Attached to the NodeInstanceRole
```
global:
  cloudStorage:
    type: s3
    useSecret: false
    s3:
      region: "REPLACE BUCKET REGION HERE"
      bucket: "REPLACE BUCKET HERE"
pinot:
  deepStore:  (1)
    enabled: true
    prefix: "kfuse/controller/data" (2)
```
yaml
1 deepStore: Enable/disable storing of Pinot segments in deep store.

2 prefix: Folder prefix in the specified bucket for deep storage.

Use a Kubernetes ServiceAccount with an IAM Role

Ensure that the Kubernetes cluster has a ServiceAccount associated with an IAM role that has read and write permissions to S3.

For instructions, see the AWS documentation on Assign IAM roles to Kubernetes service accounts.
Ensure that Pinot is configured to use the ServiceAccount and that deepStore is configured in the custom_values.yaml file.

Set both useSecret and createSecret to false:
AWS Configuration with a Kubernetes ServiceAccount that Assumes an IAM Role
```
global:
  cloudStorage:
    type: s3
    useSecret: false
    s3:
      region: "REPLACE BUCKET REGION HERE"
      bucket: "REPLACE BUCKET HERE"
pinot:
  serviceAccountName: <REPLACE SERVICE ACCOUNT NAME HERE>
  deepStore:  (1)
    enabled: true
    prefix: "kfuse/controller/data" (2)
```
yaml
1 deepStore: Enable/disable storing of Pinot segments in deep store.

2 prefix: Folder prefix in the specified bucket for deep storage.

Prerequisites

The storage account must be enabled with Azure Data Lake Storage Gen 2.

When creating a storage account, select the Enable hierarchical namespace option in the Advanced section:
Find the connection string by navigating to Access Keys from the left pane of the storage account, in the Security + networking section:
The container value refers to the container name. Create a container by navigating to Containers from the left pane of the storage account, under the Data storage section.
Create a Kubernetes secret with the Azure credentials to allow access to the Azure Data Lake Storage container:
```
kubectl create secret generic cloud-storage-secret --from-literal=connectionString=<connectionString> -n kfuse
```

Helm values

Add the following values in the custom_values.yaml file. Replace the Azure Data Lake details with your values.

Azure Configuration for Azure Data Lake Storage Gen 2 with Access Key

global:
  cloudStorage:
    type: azure
    useSecret: true
    secretName: cloud-storage-secret
    azure:
      container: "REPLACE CONTAINER NAME HERE"

pinot:
  deepStore: (1)
    enabled: true
    prefix: "kfuse/controller/data" (2)

yaml

1	`deepStore`: Enable/disable storing of Pinot segments in deep store.
2	`prefix`: Folder prefix in the specified bucket for deep storage.