POC Troubleshooting

Every command install.sh runs, and its output, is in install.log in the bundle folder. After a fix, run ./install.sh again; every step is safe to repeat. If you set another namespace=, use it in place of kfuse in the kubectl -n kfuse commands below. The README.md in the bundle folder lists the errors for your bundle.

Tools, license, and input

Symptom Fix

Missing required tools: …​

Install the tools, then run ./install.sh again.

The license check fails with unauthorized or not found

license.json is wrong or expired. Contact Kloudfuse.

…​ is for organization …​, but Kloudfuse here was installed for …​

The organization can’t change after installation. Use that organization’s license.json.

A value is rejected with invisible characters

Backspace or the arrow keys left invisible characters in it. Type it again without editing.

…​ has expired or expires within 7 days

Renew your TLS certificate, replace the files, then run ./install.sh again.

Sign-in and permissions

Symptom Fix

Resources created in the wrong account, or permission errors right after the static IP was created

You were signed in to another account. install.sh shows the account and asks before it creates anything: answer No and sign in to the right one. If something was already created, run ./uninstall.sh in that account first.

A permission check reports missing permissions or roles

Use an identity with the permissions in POC Permissions, then run ./install.sh again. Nothing has been created. If you are sure you have them (the check can’t evaluate every organization policy), add skip_permission_check = true to terraform.tfvars; a permission that really is missing then fails part-way through the install.

AWS: ExpiredToken, or The security token included in the request is expired

Your AWS sign-in expired. Run aws sso login, then ./install.sh again.

AWS: AccessDenied on iam:SimulatePrincipalPolicy

The permission check itself needs that action. Add it to your identity, use an admin role, or add skip_permission_check = true to terraform.tfvars.

AWS: kubectl or helm fails with Unauthorized or You must be logged in to the server

install.sh runs as a different AWS identity than the one that created the cluster. Sign in as that identity.

Google Cloud: could not find default credentials

Run gcloud auth application-default login, then ./install.sh again.

Google Cloud: cloud-platform scope is required but not consented

Sign in again and check Select all on the Google sign-in page.

Google Cloud: gke-gcloud-auth-plugin not found

Run gcloud components install gke-gcloud-auth-plugin.

Azure: …​ can create role assignments …​ but a condition on that role assignment stops it assigning: …​

Ask a subscription Owner to add the roles it names to the condition, or to grant User Access Administrator without a condition.

kubectl times out

A proxy or firewall on your side blocks the cluster’s API endpoint.

New cluster

Symptom Fix

Apply fails with a quota error

Raise the vCPU quota for the region. See POC Permissions.

AWS: AllocateAddress fails with AddressLimitExceeded

The region has no Elastic IPs left; the install needs two. Check with aws ec2 describe-addresses --region <region>, then pick another region, or raise the EC2-VPC Elastic IPs quota (L-0263D0A3). Nothing has been created.

AWS: r7i.4xlarge isn’t offered in …​

The region’s first availability zone doesn’t offer it. Pick another region.

Google Cloud: n4-highmem-16 isn’t available in zone …​

Pick another zone in the region that offers N4. All four us-central1 zones do.

Azure: No VM size for the …​ node is available in …​

Neither the v5 nor the v7 size is offered in that zone, or both are blocked for your subscription. Pick another region or zone, or ask Azure support to enable a size.

…​ can’t change after the first apply

Put the old value back, or uninstall and install again.

A Terraform folder creates its own cluster

A cloud bundle always creates a new cluster. To install on a cluster you already have, run bash install.sh --existing-cluster in an empty folder.

This folder holds an install on a cluster you already have

An earlier bundle installed on an existing cluster from this folder. Remove it with ./uninstall.sh, or move install.env to the helm bundle’s folder and run ./install.sh there.

kubectl shows another cluster’s pods, or the kfuse namespace isn’t found

kubectl is on another context. Run export KUBECONFIG=$PWD/kubeconfig in the bundle folder.

Existing cluster

Symptom Fix

Could not reach the cluster of context …​

Sign in to your cloud and refresh your kubeconfig, then run ./install.sh again.

Your kubeconfig has no context …​

kubectl config get-contexts lists the contexts you have. Add the cluster’s context (see Sign in to your cloud), or type one from the list.

…​ isn’t a cluster admin

Use a kubeconfig context with cluster-admin on the cluster.

No node has the label kf_role=kfuse

Label and taint a node for Kloudfuse, then run ./install.sh again.

<node>: is running other pods

The table under the message lists them. Move them off the node with kubectl drain <node> --ignore-daemonsets, then kubectl uncordon <node>, or take the kf_role label off that node.

The cluster of context …​ runs on gcp (its nodes' provider IDs), not aws

--cloud, or cloud= in install.env, names another cloud than the cluster’s. Leave --cloud out, since the cloud is read from the nodes, fix install.env, or choose the right context.

Kloudfuse was installed from this folder in the context …​, but you are currently in the context …​

This folder’s install is on another cluster. Run kubectl config use-context <original context> to carry on with it. To install on the other cluster instead, run ./uninstall.sh here first, or use another folder.

The storage class kloudfuse-disk exists with other settings

A storage class can’t be changed once it exists, and install.sh never deletes one. If nothing else uses it, run kubectl delete storageclass kloudfuse-disk (existing volumes keep their disks), then ./install.sh again.

Kloudfuse and HTTPS

Symptom Fix

Pods in ImagePullBackOff

license.json isn’t the current key. Replace it, then run ./install.sh again.

helm upgrade fails with context deadline exceeded

Something wasn’t ready within 30 minutes. kubectl -n kfuse get pods shows which; fix it, then run ./install.sh again.

Kloudfuse didn’t answer at https://…​; within 15 minutes

Kloudfuse is installed, but its address doesn’t answer yet. kubectl -n kfuse get svc envoy must show an EXTERNAL-IP, and kubectl -n kfuse get certificate must show READY True. Fix what is missing, then run ./install.sh again. ingress_wait_minutes= in install.env sets the wait; 0 skips it.

envoy service stays <pending>

kubectl -n kfuse describe svc envoy shows the error. On an existing AWS cluster, check the subnet and the Elastic IP allocation. On Azure, check the IP’s name and resource group, and the cluster identity’s Network Contributor role; a new role assignment can take a few minutes to take effect.

Disks (PVCs) stuck Pending

The disk driver isn’t running. On AWS, check kubectl -n kube-system get pods -l app.kubernetes.io/name=aws-ebs-csi-driver. On Azure, a message about PremiumV2_LRS means the region or zone doesn’t support Premium SSD v2: uninstall and pick another region or zone.

Let’s Encrypt’s check failed: asking it again (1 of 4)

Expected on a first install: Let’s Encrypt checked before the load balancer was ready. The script asks again by itself, up to 4 times, 2 minutes apart.

Certificate READY False

The DNS name doesn’t resolve to the static IP yet, or port 80 is blocked from the internet. kubectl -n kfuse describe challenge shows why. After you fix it, run kubectl -n kfuse delete certificate letsencrypt-ingress-tls to ask Let’s Encrypt again.

The wait for https://<domain>; says no certificate yet

cert-manager isn’t creating the certificate. Check that it runs, with Gateway API support: kubectl get pods -A -l app.kubernetes.io/name=cert-manager.

cert-manager runs on this cluster …​ without Gateway API support

A cert-manager that was already on the cluster can’t issue the certificate. Turn on its Gateway API support (config.enableGatewayAPI=true), or run ./install.sh again and answer No to Use 'Let’s Encrypt' for TLS to use your own certificate.

Browser certificate warning

With Let’s Encrypt, the certificate isn’t issued yet. With your own certificate, it doesn’t cover the DNS name, or the certificate file is missing the intermediate certificates.

After installation

Symptom Fix

Lost the admin password

Reset the admin password.

Alert emails don’t arrive

The SMTP host, port, or password is wrong, or the port is 25. If they arrive in spam, the from address isn’t set up for your mail server (SPF, DKIM).

custom-values.yaml differs from what install.sh writes

It was edited, or the settings changed. Answer Yes to replace it (the old one is backed up), or No to stop with nothing changed. Put lasting changes in the settings or in extended-values.yaml.

This install was last changed by bundle …​, but this is bundle …​

A newer bundle changed this install, and an older one can’t safely run on it. Run ./install.sh or ./uninstall.sh from the newer bundle’s folder.

Azure: Pinot restarts with an access error on the storage account

The storage account key was rotated. Run ./install.sh again, then delete the Pinot pods so they restart. On a new cluster, Terraform gives install.sh the new connection string. On an existing cluster, a re-run reuses the old one from custom-values.yaml, so set the new one: KFUSE_AZURE_STORAGE_CONNECTION_STRING='<connection string>' ./install.sh.

uninstall.sh says No Kloudfuse install here

Run ./uninstall.sh in the folder you started the install in.

terraform destroy stops with Kloudfuse is still installed

Kloudfuse’s load balancer still uses the static IP. Run ./uninstall.sh, which removes Kloudfuse first.

AWS: terraform destroy hangs on the VPC or subnets

A load balancer is left over. Delete it in the AWS console (tag kfuse-poc=true) and run ./uninstall.sh again.