Backup and Restore Kloudfuse

These steps describe how to back up and restore your Kloudfuse cluster. Follow the instructions in Backup and Restore carefully to ensure the successful operation of the Kloudfuse platform.

Backup

Configuration Database Backup

Upgrade to 3.4.0 or higher release as Kfuse has a separate az-service which takes periodic backup of databases (configdb, orchestratordb, and pinot) and stores in the cloud storage. Starting in 3.4.2 or higher release, az-service is now enabled by default. Depending on the cloud provider we can configure az-service in kfuse yaml to do periodic backup.

# az-service - Configuration for AZ Service
az-service:
  enabled: true
  # replica count for az-service should stay at 1.
  # # Override for nodeSelector, affinity and tolerations available

  config:
    zone: "<zone-id>" # unique identifier so that if the bucket is shared, it doesn't get written in the same folder. We can keep it same as orgId as it is unique enough. Defaults to <orgId>-<Release.Name>-<Release.Namespace> if not specified.

    # cloud storage can be configured here or through the global.cloudStorage section.
    # The cloud storage configuration in az-service will take precedence and fallback to global cloudStorage configuration.
    cloudStorage: {}
      # type of cloud storage: s3, gcs, azure
      # type: s3
      # useSecret: false
      #
      # secretName is the kubernetes secret that contains the credentials to access the specified cloud storage.
      # For s3:
      # The secret contains accessKey and secretKey of the IAM credentials.
      # kubectl create secret generic cloud-storage-secret --from-literal=accessKey=<accessKey> --from-literal=secretKey='<secretKey>'
      # For GCS:
      # The secret contains the encoded content of the json credential file. Note that the file needs to be called secretKey.
      # kubectl create secret generic cloud-storage-secret --from-file=./secretKey
      # For Azure:
      # The secret contains the connectionString for the container storage.
      # kubectl create secret generic cloud-storage-secret --from-literal=connectionString=<connectionString>
      #
      # secretName: cloud-storage-secret
      # s3:
      #   region: <specify region>
      #   bucket: <specify bucket>
      # gcs:
      #   bucket: <specify bucket>
      # azure:
      #   container: <specify container>

Restore

To restore data from az-service

To restore data from the backup, run the below command:

curl  https://<KFUSE ENDPOINT>/az-service/activate?zone=<zone id from the backup>&latestTime=<optional timestamp>

Parameters: * zone: The zone id from the backed up data (required). IMPORTANT: If the backed up data is using a different zone than your current cluster configuration, ensure that you use the zone id from the backup, not the current cluster’s zone. * latestTime: Optional parameter that allows you to specify the latest time (as a timestamp) of configdb and orchestratordb dump to restore. If not specified, the most recent backup will be used.

The above will pause alerts, pause pinot consumption, then rehydrate pinot segments (any missing), resume pinot consumption, restore config and orchestrator db Afterwards there is a separate command to resume the alerts.

curl  https://<KFUSE ENDPOINT>/az-service/resume

Create New Kloudfuse Cluster

Follow the Installation instructions to create a new Kloudfuse cluster.

The new cluster must be configured with the same cloud storage configuration (S3, GCS, or Azure) as the one you had previously backed up, including the same bucket/container and credentials. However, the zone configuration can be different from the original cluster.