Catalog Service

The Kloudfuse platform has a catalog service that provides following features for ease of use and onboarding:

It presents ready-to-use artifacts (dashboards or alerts) for several commonly used software components (Postgres, Redis, Mongo, Kafka, K8s, Apache Pinot, etc. etc.) towards the “works out of the box” experience. These artifacts can be deployed directly into your environment through the catalog service.
The catalog service is also aimed at providing simpler migration of artifacts from other source platforms to kloudfuse platform. See Migrating artifacts from other platforms.

The artifacts that you install through the catalog are “staged” for the user to customize, as needed. Ideally, we do not consider initially-staged artifacts to be in their final form, or at their final destination. It is at the user’s discretion how to manage their artifacts and their lifecycle as code, because each organization has their own business requirements. We encourage you to move imported dashboards and alerts to other folders where they make sense.

If artifacts with a matching name already exist in a destination folder, the catalog does not overwrite them. If you plan to update your installed artifacts, delete or rename the older version. We also provide a sample config.yaml file that you can use to control how to merge alert artifacts.

Explore/Install through the Kloudfuse artifacts catalog

The UI for consuming (listing/exploring/installing) these artifacts is coming in an upcoming version. However, the users can use these easy-to-follow instructions to do the same by using a CLI. The CLI is available as part of the catalog-service which runs as a service within the kloudfuse installed stack. The steps are as follows:

Logging into the catalog service

# make sure you are connected to the kfuse k8s cluster and in the kfuse namespace
kubectx <your-kfuse-cluster>
kubens kfuse
kubectl exec -it catalog-service-<...> -- bash

Usage

# Make sure to understand the usage.
# cd /catalog_service
python3 catalog.py -h
usage: catalog.py [-h] [--artifact_type {dashboards,alerts,contactpoints}] [--explore] [--list [LIST]] [--detailed] [--install] [--download_installed] [--grafana_address GRAFANA_ADDRESS] [--grafana_username GRAFANA_USERNAME]
                  [--grafana_password GRAFANA_PASSWORD] [-c CONFIG] [--no_upload]

catalog service

options:
  -h, --help            show this help message and exit
  --artifact_type {dashboards,alerts,contactpoints}
                        which artifact type you want to work with
  --explore             explore the catalog, making selections, and finally installing chosen artifacts to grafana
  --list [LIST]         list all the components for which artifacts are available. if list of comma separated components are provided, then those are used
  --detailed            show detailed list (only honored with --list option)
  --install             directly installs artifacts of specified type as per --list option without exploring/prompting
  --download_installed  download already installed artifacts from provided grafana instance or from the grafana in the cluster context
  --grafana_address GRAFANA_ADDRESS
                        grafana server address. optional, default is http://kfuse-grafana:80)
  --grafana_username GRAFANA_USERNAME
                        grafana username. optional, if not provided, it is taken from environment
  --grafana_password GRAFANA_PASSWORD
                        grafana user password. taken from the environment
  -c CONFIG, --config CONFIG
                        Config file containing settings for install if selectivity is needed
  --no_upload           just process artifacts but do not upload

List artifacts (optionally, of a given type)

# To list all artifacts in the catalog
python3 catalog.py
{'kafka': {'dashboards': {'count': 1}},
 'kubernetes': {'alerts': {'count': 1}, 'dashboards': {'count': 5}},
 'sock-shop': {'dashboards': {'count': 1}}}

 # To list only alerts in the catalog
python3 catalog.py --artifact_type=alerts
{'kafka': {}, 'kubernetes': {'alerts': {'count': 1}}, 'sock-shop': {}}

Explore artifacts (optionally, of a given type)

# catalog lets you explore artifacts and make "selections" along the way to what
# you want to install, and installing it after one confirms the selection.
# Provding --artifact_type as an argument confines the exploration to a given artifact
# type.
# Follow the help and prompt to make selections. Following example uses
# "dashboards" artifact type and makes "kafka" and "kubernetes" selection
# for components and then "Select All" for installing all dashboards for these
# components.

python3 catalog.py --explore --artifact_type=dashboards
? Select component:

Note: use up or down arrow to navigate
    : use <space> to select or unselect
    : multiple choices can be selected
    : press <enter> to go to next step

❯ Select All
  kafka
  kubernetes
  sock-shop

['kafka', 'kubernetes']
2023-02-24T01:40:11.377805Z listing artifacts for          component=['kafka', 'kubernetes']
? Select artifacts:

Note: use up or down arrow to navigate
    : use <space> to select or unselect
    : multiple choices can be selected
    : press <enter> to go to next step

❯ Select All
  kafka: dashboards: kafka.json
  kubernetes: dashboards: dd-Kubernetes-O_grafana.json
  kubernetes: dashboards: dd-KubernetesAPIServerO_grafana.json
  kubernetes: dashboards: dd-KubernetesClusterOverviewDa_grafana.json
  kubernetes: dashboards: dd-KubernetesJobsandCronJobsO_grafana.json
  kubernetes: dashboards: dd-KubernetesNodesO_grafana.json

   ['Select All']

Following artifacts will be downloaded and installed:

 selection=['kafka: dashboards: kafka.json',
 'kubernetes: dashboards: dd-Kubernetes-O_grafana.json',
 'kubernetes: dashboards: dd-KubernetesAPIServerO_grafana.json',
 'kubernetes: dashboards: dd-KubernetesClusterOverviewDa_grafana.json',
 'kubernetes: dashboards: dd-KubernetesJobsandCronJobsO_grafana.json',
 'kubernetes: dashboards: dd-KubernetesNodesO_grafana.json',
 'sock-shop: dashboards: sock-shop-perf.json']
?

Proceed? (Y/n)

Install kubernetes artifacts

# Install all kubernetes dashboard artifacts.
# NOTE THAT --install DOES NOT ASK FOR ANY CONFIRMATION. Use --explore for that.

python3 catalog.py --install --artifact_type=dashboards --list kubernetes

Install all artifacts

# Install all available artifacts (optionally, of a given type. if artifact_type
# is not supplied then all artifact types will be installed).
# NOTE THAT --install DOES NOT ASK FOR ANY CONFIRMATION. Use --explore for that.

python3 catalog.py --install --artifact_type=dashboards

Install out-of-the-box Kloudfuse artifacts

python3 catalog.py --explore --artifact_type=alerts --use_dir auto_installed_artifacts

Migrating artifacts from other platforms

Migrating from Grafana

Download from existing

Connect to Kloudfuse cluster and log into the catalog service pod.

# make sure you are connected to the kfuse k8s cluster and in the kfuse namespace
kubectx <your-kfuse-cluster>
kubens kfuse
kubectl exec -it catalog-service-<...> -- bash

Download existing artifacts (alerts in the following example) from source grafana instance using the catalog script (check usage with -h for more options), appropriately specifying arguments (be sure that the catalog service pod can reach the source grafana address):

# Review usage with -h
#
cd /catalog_service
python3 catalog.py --download_installed --artifact_type=alerts --grafana_address=SOURCE_GRAFANA_ADDRESS --grafana_username="SOURCE_GRAFANA_USERNAME" --grafana_password="SOURCE_GRAFANA_PASSWD"

Upload to Kloudfuse

The download copies the artifacts into the /tmp/<artifact_type>_downloads directory. Use the following command to explore and install these artifacts into the Kloudfuse-embedded Grafana.

If uploading a directory, your alerts must be ina directory structure with the form /tmp/alerts_downloads/assests/<any_folder_name>/alerts/. Use the following command to see all alerts and to select from them.

python3 catalog.py --explore --artifact_type=alerts --use_dir=/tmp/alerts_downloads --grafana_address=https://<target-cluster-addr> --grafana_username="SOURCE_GRAFANA_USERNAME" --grafana_password="SOURCE_GRAFANA_PASSWD"

If you cannot execute these commands from pod because the <target-cluster-addr> is not reachable, use the following instructions to extract the catalog tool to a location where you can reach the <target-cluster-addr>:

# Requires python 3.10

mkdir /tmp/catalog_service/
cd /tmp/

# Get the catalog-service pod id for the following command.
kubectl cp catalog-service-<...>:/catalog_service ./catalog_service/
pip3 install -r ./catalog_service/requirements.txt

# migrate alerts (use -h for help)
python3 ./catalog_service/catalog.py --download_installed --grafana_address https://<source-kfs-cluster>.kloudfuse.io --artifact_type alerts
python3 ./catalog_service/catalog.py --grafana_address https://<target-kfs-cluster>.kloudfuse.io --artifact_type alerts --explore --use_dir /tmp/alerts_downloads

# migrate dashboards (use -h for help)

python3 ./catalog_service/catalog.py --download_installed --grafana_address https://<source-kfs-cluster>.kloudfuse.io --artifact_type dashboards
python3 ./catalog_service/catalog.py --grafana_address https://<target-kfs-cluster>.kloudfuse.io --artifact_type dashboards --explore --use_dir /tmp/dashboards_downloads

Manually uploading artifacts to Kloudfuse

Download from existing

Ensure that you download the alerts/dashboards in json format.

You can then proceed and upload them to Kloudfuse, and make them available in the catalog-service pod. You can connect to catalog service pod with Usage instructions.

Upload to Kloudfuse

Create a directory structure in your <download_directory>:

Alerts

mkdir -p <download_directory>/assets/upload/alerts

Dashboards

mkdir -p <download_directory>/assets/upload/dashboards
Copy the alerts and dashboards to their respective directories. Use the following command to explore and install these artifacts into the Kloudfuse-embedded Grafana.