Upgrade validation

Verify that alerts are updated

When you successfully execute pre- and post-upgrade tasks, the backend does not get queries with 1-second step.

Verify this by looking at the query-service logs. Issue a grep command for step:[1]; there shouldn’t be any.

Verify deepstore settings

During an upgrade, it is extremely important that you retain deepstore settings. They must not be removed; the deepstore configuration must remain identical to the state before the upgrade.

To verify, run the helm diff plugin to check the differences. You can build this plugin from the source, and then install it. See the helm-diff documentation for instructions.

  1. Install the helm diff plugin:

    helm plugin install https://github.com/databus23/helm-diff

  2. Run docker login before running the diff command.

    To run the correct command for login, see how to Login for Kloudfuse helm registry.

  3. Generate the diff with the new version:

    helm diff upgrade kfuse oci://us-east1-docker.pkg.dev/mvp-demo-301906/kfuse-helm/kfuse --version <VERSION.NUM.BER> --values values.yaml > /tmp/kfuse_upgrade_diff (1)
    1 version: Use the version number of the most current Kloudfuse release; the pattern looks like 3.0.0, 3.1.3, and so on. See Version documentation.

  4. If the deepstore settings changed, you will find changes in pinot-controller.conf and pinot-server.conf files. Search or grep for them in the /tmp/kfuse_upgrade_diff file.

    The following sample output demonstrates that deepstore settings may change from enabled to not enabled during the upgrade. Verify your deepstore settings in your custom-values.yaml file.

    pinot-controller.conf:

+     controller.data.dir=/var/pinot/controller/data
-     controller.realtime.segment.deepStoreUploadRetryEnabled=true
-     controller.realtime.segment.validation.frequencyPeriod=5m
-     controller.data.dir=gs://pinot-gcp/dev-gcp/vinayk/controller/data
-     controller.local.temp.dir=/var/pinot/controller/data
-     pinot.controller.instance.enable.split.commit=true
-     pinot.controller.storage.factory.class.gs=org.apache.pinot.plugin.filesystem.GcsPinotFS
-     pinot.controller.storage.factory.gs.projectId=mvp-demo-301906
-     pinot.controller.storage.factory.gs.gcpKey=/etc/pinot/server/secretKey
-     pinot.controller.segment.fetcher.protocols=file,http,gs
-     pinot.controller.segment.fetcher.gs.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher

pinot-server.conf:

+     # enable peer download; refer: https://docs.pinot.apache.org/operators/operating-pinot/decoupling-controller-from-the-data-path
+     pinot.server.storage.factory.class.peer=org.apache.pinot.spi.filesystem.LocalPinotFS
+     pinot.server.instance.segment.store.uri=peer:///var/pinot/server/data
-     pinot.server.instance.segment.store.uri=gs://pinot-gcp/dev-gcp/vinayk/controller/data
-     pinot.server.storage.factory.class.gs=org.apache.pinot.plugin.filesystem.GcsPinotFS
-     pinot.server.storage.factory.gs.projectId=mvp-demo-301906
-
-     pinot.server.storage.factory.gs.gcpKey=/etc/pinot/server/secretKey
-     pinot.server.segment.fetcher.protocols=file,http,gs
-     pinot.server.segment.fetcher.gs.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher