Configure Envoy Ingress
Overview
Envoy Gateway implements the Kubernetes Gateway API and replaces ingress-nginx as the ingress controller for Kloudfuse. This guide covers new cluster installation, zero-downtime migration from ingress-nginx, rollback procedures, and post-uninstall cleanup.
Prerequisites
Hardware Requirements
-
Kloudfuse version 4.0.0 or later — Envoy Gateway is not compatible with earlier versions
-
Kubernetes cluster version 1.27 or later
-
kubectlconfigured with cluster access -
Helm 3.x installed
Install Envoy CRDs
Install the CRDs before running helm install or helm upgrade. The version must match the Envoy Gateway version used by the Kloudfuse chart:
| Kloudfuse Version | Envoy Gateway CRD Version |
|---|---|
4.0.0 |
v1.3.0 |
4.0.1+ |
v1.7.1 |
helm show crds oci://docker.io/envoyproxy/gateway-helm \
--version <VERSION> \
| kubectl apply --server-side --force-conflicts -f -
This command is idempotent — safe to run on clusters that already have the CRDs installed. The --server-side flag avoids annotation size limits on large CRDs.
|
Install or Upgrade cert-manager
cert-manager v1.14+ with Gateway API support is required for automatic TLS certificate issuance.
| cert-manager is cluster-wide but runs in a single namespace. If cert-manager is already installed on your cluster, upgrade the existing installation in its current namespace rather than installing a second instance. |
helm repo add jetstack https://charts.jetstack.io
helm repo update
helm upgrade --install cert-manager jetstack/cert-manager \
--namespace <cert-manager-namespace> \ (1)
--version v1.17.1 \
--set crds.enabled=true \
--set config.enableGatewayAPI=true
| 1 | Use the namespace where cert-manager is already installed (e.g., cert-manager), not the Kloudfuse namespace. Run kubectl get pods -A | grep cert-manager to find it. |
The config.enableGatewayAPI=true flag is required. Without it, cert-manager only watches Ingress resources and will not create certificates for Gateway resources.
|
Add the following TLS section to your custom_values.yaml to enable automatic certificate issuance:
tls:
enabled: true
host: <your-hostname>
email: <your-email>
clusterIssuer: <namespace>-letsencrypt-prod
createClusterIssuer: true
New Install
For new clusters where ingress-nginx has never been installed, enable envoy-gateway and disable ingress-nginx:
-
AWS (EKS)
-
GCP (GKE)
envoy-gateway:
enabled: true
installGatewayRoutes: true
envoyService:
annotations:
service.beta.kubernetes.io/aws-load-balancer-type: nlb
service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: 'true'
service.beta.kubernetes.io/aws-load-balancer-eip-allocations: <YOUR_EIP_ALLOC_IDS>
patch:
externalTrafficPolicy: Local
external:
enabled: true
internal:
enabled: true
annotations:
service.beta.kubernetes.io/aws-load-balancer-internal: "true"
ingress-nginx:
enabled: false
installIngressRules: false
envoy-gateway:
enabled: true
installGatewayRoutes: true
envoyService:
patch:
loadBalancerIP: "<YOUR_STATIC_IP>"
external:
enabled: true
internal:
enabled: true
annotations:
networking.gke.io/load-balancer-type: "Internal"
cloud.google.com/load-balancer-type: "Internal"
ingress-nginx:
enabled: false
installIngressRules: false
TLS settings do not change. The existing tls.enabled, tls.host, and tls.clusterIssuer work with envoy-gateway. To disable the HTTP listener on port 80 (HTTPS only), set envoy-gateway.enableHttp to false in your values file.
|
envoy-gateway:
enableHttp: false
Separate Internal and External Hosts
To route ingest traffic to an internal load balancer and query traffic to an external load balancer, configure the tls.internalIngest and tls.externalQuery fields:
tls:
host: <PRIMARY DNS HOST>
internalIngest:
hosts:
- <INTERNAL INGEST DNS HOST>
secretName: "<TLS secret for ingest hosts, or empty for ACM>"
hostOnly: false
externalQuery:
hosts:
- <EXTERNAL QUERY DNS HOST>
secretName: "<TLS secret for query hosts, or empty for ACM>"
hostOnly: false
| Field | Description |
|---|---|
|
DNS hostnames for internal ingestion routes (Gateway listeners and ingest |
|
Kubernetes TLS secret for the ingest hosts. Leave empty when using ACM termination. |
|
If |
|
DNS hostnames for external query/browser routes (Gateway listeners and query |
|
Kubernetes TLS secret for the query hosts. Leave empty when using ACM termination. |
|
If |
These fields are used by Envoy Gateway templates only. If migrating from ingress-nginx, these replace the flat fields tls.ingestInternalHosts, tls.ingestInternalHostOnly, tls.ingestInternalSecretName, tls.queryHosts, tls.queryHostOnly, and tls.queryHostSecretName.
|
Upgrade from Nginx to Envoy
This procedure migrates an existing Kloudfuse installation from ingress-nginx to envoy-gateway without recreating load balancers or changing IPs or DNS. This migration will result in zero-downtime.
The Nginx Load Balancer Service is preserved throughout the migration. Instead of deleting it and creating a new Envoy LB, the Nginx service’s selector is repointed to Envoy proxy pods. The same NLB/LB keeps the same IPs and DNS.
| Step | Change Values | Traffic Served By | What Happens |
|---|---|---|---|
D0 (current) |
None |
Nginx |
Starting state |
Step 1 (prepare) |
Enable Envoy + |
Nginx (unchanged) |
Envoy starts alongside nginx. Nginx LB gets |
Step 2 (switch) |
Set |
Envoy |
Nginx LB selector switches to Envoy pods. TargetPorts change to 10080/10443. Traffic flows through Envoy via the same LB. |
Step 3 (cleanup) |
Set |
Envoy |
Remove Nginx Controller. Nginx LB Service is kept (resource-policy annotation). |
Step 1: Enable Envoy
-
Add the
envoy-gatewaysection and the migration flag to yourcustom_values.yaml:global: envoyMigration: enabled: true external: false internal: false envoy-gateway: enabled: true installGatewayRoutes: true envoyService: external: enabled: true # Enable if you have an internal LB (1) # internal: # enabled: trueyaml1 Enable if you have an internal LB -
If your existing nginx configuration uses separate internal/external hosts, migrate the flat TLS fields to the new nested format. Add the
tls.internalIngestandtls.externalQueryfields alongside your existing flat fields (which nginx still reads until it is disabled):Nginx (existing) Envoy (add these) tls.ingestInternalHosts: [host]tls.internalIngest.hosts: [host]tls.ingestInternalSecretName: secrettls.internalIngest.secretName: secrettls.ingestInternalHostOnly: truetls.internalIngest.hostOnly: truetls.queryHosts: [host]tls.externalQuery.hosts: [host]tls.queryHostSecretName: secrettls.externalQuery.secretName: secrettls.queryHostOnly: truetls.externalQuery.hostOnly: trueFor example, if your current config has:
tls: host: app.example.com ingestInternalHosts: - ingest.example.com ingestInternalHostOnly: falseyamlAdd the envoy equivalent:
tls: host: app.example.com ingestInternalHosts: # keep for nginx during migration - ingest.example.com ingestInternalHostOnly: false # keep for nginx during migration internalIngest: # new for envoy hosts: - ingest.example.com hostOnly: falseyamlKeep both sets of fields during migration (Steps 1-2). After nginx is disabled in Step 3, the flat fields are no longer needed and can be removed. -
Run
helm upgrade:helm upgrade --install kfuse oci://us-east1-docker.pkg.dev/mvp-demo-301906/kfuse-helm/kfuse \ -n kfuse \ --version <VERSION> \ (1) -f custom-values.yaml1 Replace <VERSION>with a valid Kloudfuse release value. See Release Notes for the latest release.
After completion, verify:
# Envoy pods should be running
kubectl get pods -n <namespace> | grep envoy
# Gateway should be Programmed: False
kubectl get gateway -n <namespace>
# Nginx should still be serving (302 redirect to login)
curl -sk https://your-hostname/ -o /dev/null -w "%{http_code}"
# Nginx LB should have keep annotation
kubectl get svc kfuse-ingress-nginx-controller -n <namespace> \
-o jsonpath='{.metadata.annotations.helm\.sh/resource-policy}'
Step 2: Switch Traffic to Envoy
-
Change
global.envoyMigration.externaltotruein yourcustom_values.yaml. If you have an internal LB, also setinternal: true:global: envoyMigration: enabled: true external: true # Set to true if you have an internal Nginx LB to switch (1) internal: falseyaml1 If you have an internal Nginx LB to switch, change to true -
Run
helm upgrade:helm upgrade --install kfuse oci://us-east1-docker.pkg.dev/mvp-demo-301906/kfuse-helm/kfuse \ -n kfuse \ --version <VERSION> \ (1) -f custom-values.yaml1 Replace <VERSION>with a valid Kloudfuse release value. See Release Notes for the latest release.
After completion, verify:
# Selector should point to envoy, with targetPorts: 10080 10443
export NAMESPACE="<namespace>"
kubectl get svc kfuse-ingress-nginx-controller -n $NAMESPACE -o jsonpath='selector: {.spec.selector}{"\n"}targetPorts: {.spec.ports[*].targetPort}{"\n"}'
# Should return 401 (Envoy auth) instead of 302 (Nginx redirect)
curl -sk https://your-hostname/ -o /dev/null -w "%{http_code}"
| AWS NLB health checks may take 15-30 seconds to converge after the targetPort change. This is transient. |
Step 3: Disable Nginx
-
Set
ingress-nginx.enabledtofalsein yourcustom_values.yaml:ingress-nginx: enabled: false installIngressRules: falseyaml -
Run
helm upgrade:helm upgrade --install kfuse oci://us-east1-docker.pkg.dev/mvp-demo-301906/kfuse-helm/kfuse \ -n kfuse \ --version <VERSION> \ (1) -f custom-values.yaml1 Replace <VERSION>with a valid Kloudfuse release value. See Release Notes for the latest release.
The Nginx controller pod is removed. The Nginx LB Service is not deleted because of the resource-policy: keep annotation — it continues routing traffic to Envoy pods.
Step 4: Verification
export NAMESPACE="<namespace>"
# Nginx controller pod should be gone; Envoy pods should be running
kubectl get pods -n $NAMESPACE | grep -E "nginx|envoy"
# Nginx LB service should still exist; selector and targetPorts should point to Envoy
kubectl get svc kfuse-ingress-nginx-controller -n $NAMESPACE \
-o jsonpath='selector: {.spec.selector}{"\n"}targetPorts: {.spec.ports[*].targetPort}{"\n"}'
# Gateway should show Programmed: False
kubectl get gateway -n $NAMESPACE
# All HTTPRoutes should show Accepted and Resolved
kubectl get httproute -n $NAMESPACE
# HTTPS should return 401 (Envoy auth challenge) — not 302 (Nginx redirect)
curl -sk https://your-hostname/ -o /dev/null -w "%{http_code}\n"
Rollback to Nginx
To revert from Envoy back to Nginx:
-
Only if the original Nginx LB Service still exists (i.e. you ran the in-place migration in Upgrade from Nginx to Envoy, which annotated the Nginx LB with
helm.sh/resource-policy: keepso it survived through Step 3), remove those annotations so Helm can manage the Service again:export NAMESPACE="<namespace>" kubectl annotate svc kfuse-ingress-nginx-controller \ helm.sh/resource-policy- -n $NAMESPACE kubectl annotate svc kfuse-ingress-nginx-controller-internal \ helm.sh/resource-policy- -n $NAMESPACE # if internal LB existsbashCheck before running:
kubectl get svc -n <namespace> kfuse-ingress-nginx-controller -o jsonpath='{.metadata.annotations.helm\.sh/resource-policy}{"\n"}'. If the Service does not exist (you migrated by deleting Nginx outright, or did a fresh Envoy install), skip this step — Helm will create a brand-new Nginx Service from yourcustom_values.yamlin step 2. -
Update the
custom_values.yamlto disable Envoy and re-enable Nginx with the full controller config that was in place before the Envoy migration. Helm will not infer Nginx’s controller settings (LB IP, service annotations, headers) from theenvoy-gatewayblock — every field that was set oningress-nginx.controllerbefore the migration must be present here, otherwise the Nginx LB will come up with chart defaults.Do not delete the existing envoy-gateway:block fromcustom_values.yaml. Setenvoy-gateway.enabled: falseandenvoy-gateway.installGatewayRoutes: false— those two flags are enough to fully disable Envoy (the controller, the EnvoyProxy data plane pods, and theGateway+HTTPRouteobjects all go away). Removing the rest of the block during rollback risks two concrete failure modes:-
If the
envoy-gatewayblock is deleted butenvoyMigration/installGatewayRoutesare not consistent, the Nginx or Envoy pods can land inCrashLoopBackOffbecause the rendered Service or controller config is incomplete. -
If the
ingress-nginx.controller.service.loadBalancerIP/ EIPs / annotations below do not exactly match the LB the cluster had before the migration, the cloud provider allocates a fresh load balancer with a new IP, breaking DNS and causing a downtime window until the A records are updated.Keeping the full
envoy-gateway:block around also means a future re-upgrade to Envoy is just a matter of flipping both flags back totrueand re-running the Upgrade from Nginx to Envoy flow — no need to re-derive the LB IPs, service annotations, or traffic policies you previously tuned.-
AWS (EKS)
-
GCP (GKE)
-
Azure (AKS)
global: envoyMigration: enabled: false external: false internal: false envoy-gateway: enabled: false installGatewayRoutes: false ingress-nginx: enabled: true installIngressRules: true controller: ingressClassResource: enabled: true name: kfuse-ingress ingressClass: kfuse-ingress watchIngressWithoutClass: false service: annotations: service.beta.kubernetes.io/aws-load-balancer-type: nlb service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: 'true' service.beta.kubernetes.io/aws-load-balancer-eip-allocations: <YOUR_EIP_ALLOC_IDS> (1) externalTrafficPolicy: Local external: enabled: true internal: (2) enabled: true annotations: service.beta.kubernetes.io/aws-load-balancer-internal: "true" service.beta.kubernetes.io/aws-load-balancer-type: nlbyaml1 Replace with the comma-separated Elastic IP allocation IDs that were attached to the Nginx NLB before the migration. Without these, AWS provisions a fresh NLB with new public IPs and the existing DNS records break. 2 Omit the internal:block if your cluster only had an external NLB before the migration. If you did have an internal NLB, the annotations here must match what it had originally — otherwise AWS provisions a fresh internal NLB with a new IP and the internal DNS records break.global: envoyMigration: enabled: false external: false internal: false envoy-gateway: enabled: false installGatewayRoutes: false ingress-nginx: enabled: true installIngressRules: true controller: ingressClassResource: enabled: true name: kfuse-ingress ingressClass: kfuse-ingress watchIngressWithoutClass: false addHeaders: Content-Security-Policy: "<YOUR_CSP>" (1) service: loadBalancerIP: <YOUR_STATIC_IP> (2) external: enabled: true externalTrafficPolicy: Local internal: (3) enabled: true annotations: networking.gke.io/load-balancer-type: "Internal" cloud.google.com/load-balancer-type: "Internal"yaml1 Restore the Content-Security-Policy header value you had on Nginx, if any. Envoy serves this via its own addHeadersmechanism — switching back to Nginx loses that header unless you put it back here.2 The reserved external static IP. Must match the IP attached to the Nginx LB before the migration so the existing DNS A record continues to resolve. 3 Omit the internal:block if your cluster only has an external LB.global: envoyMigration: enabled: false external: false internal: false envoy-gateway: enabled: false installGatewayRoutes: false ingress-nginx: enabled: true installIngressRules: true controller: ingressClassResource: enabled: true name: kfuse-ingress ingressClass: kfuse-ingress watchIngressWithoutClass: false service: loadBalancerIP: <YOUR_STATIC_IP> (1) externalTrafficPolicy: Local external: enabled: true internal: (2) enabled: true annotations: service.beta.kubernetes.io/azure-load-balancer-internal: "true"yaml1 The reserved Azure public IP. Must match the IP attached to the Nginx LB before the migration. 2 Omit the internal:block if your cluster only has an external LB. -
-
-
Run
helm upgrade:helm upgrade --install kfuse oci://us-east1-docker.pkg.dev/mvp-demo-301906/kfuse-helm/kfuse \ -n kfuse \ --version <VERSION> \ (1) -f custom-values.yaml1 Replace <VERSION>with a valid Kloudfuse release value. See Release Notes for the latest release. -
Clean up orphaned Envoy resources:
export NAMESPACE="<namespace>" # Delete controller-managed proxy Deployments and Services kubectl delete deploy -l app.kubernetes.io/managed-by=envoy-gateway -n $NAMESPACE kubectl delete svc -l app.kubernetes.io/managed-by=envoy-gateway -n $NAMESPACE # Remove GatewayClass finalizers and delete them for gc in $(kubectl get gatewayclass -o name | grep "$NAMESPACE"); do kubectl patch "$gc" --type=merge -p '{"metadata":{"finalizers":[]}}' kubectl delete "$gc" donebash -
Verify the rollback:
export NAMESPACE="<namespace>" # Nginx controller pod should be running; no Envoy proxy pods should remain kubectl get pods -n $NAMESPACE | grep -E "nginx|envoy" # Nginx LB selector should point back to Nginx pods; targetPorts should be 80/443 kubectl get svc kfuse-ingress-nginx-controller -n $NAMESPACE \ -o jsonpath='selector: {.spec.selector}{"\n"}targetPorts: {.spec.ports[*].targetPort}{"\n"}' # HTTPS should return 302 (Nginx login redirect) — not 401 curl -sk https://your-hostname/ -o /dev/null -w "%{http_code}\n" # Confirm no orphaned Envoy resources remain kubectl get all -l app.kubernetes.io/managed-by=envoy-gateway -n $NAMESPACE kubectl get gatewayclass | grep $NAMESPACEbash
Re-upgrading to Envoy after a rollback
If you kept the envoy-gateway: block in custom_values.yaml during the rollback (as recommended in Step 2 above), re-upgrading to Envoy is the same as a first-time upgrade — no need to re-derive LB IPs, service annotations, or traffic policies:
-
Re-enable the migration flags in
custom_values.yaml:global: envoyMigration: enabled: true external: false # Step 1 — flip to true at Step 2 internal: false envoy-gateway: enabled: true installGatewayRoutes: true # ... (preserved from before the rollback)yaml -
Follow the Upgrade from Nginx to Envoy flow from Step 1 onward.
If you deleted the envoy-gateway: block during the rollback, restore it from version control (or from the per-cloud examples in New Install) before re-upgrading.
Uninstall
When running helm delete (or helm uninstall), the envoy-gateway controller is removed but the resources it created at runtime are not deleted by Helm. These orphaned resources must be cleaned up manually.
The Envoy Deployments, Services, and ConfigMaps created by the envoy-gateway controller (not by Helm templates). Since Helm didn’t create them, it doesn’t track or delete them.
Cleanup Steps
After helm delete kfuse -n <namespace>:
# 1. Delete controller-managed proxy Deployments and Services
export NAMESPACE="<namespace>"
kubectl delete deploy -l app.kubernetes.io/managed-by=envoy-gateway -n $NAMESPACE
kubectl delete svc -l app.kubernetes.io/managed-by=envoy-gateway -n $NAMESPACE
kubectl delete configmap -l app.kubernetes.io/managed-by=envoy-gateway -n $NAMESPACE
# 2. Remove GatewayClass finalizers and delete them (cluster-scoped)
for gc in $(kubectl get gatewayclass -o name | grep "$NAMESPACE"); do
kubectl patch "$gc" --type=merge -p '{"metadata":{"finalizers":[]}}'
kubectl delete "$gc"
done
# 3. Verify no orphaned resources remain
kubectl get all -l app.kubernetes.io/managed-by=envoy-gateway -n $NAMESPACE
kubectl get gatewayclass | grep $NAMESPACE
| Shared clusters — Do NOT delete Gateway API or Envoy Gateway CRDs on clusters where other namespaces also use envoy-gateway. CRDs are cluster-scoped; deleting them removes all Gateway, HTTPRoute, and SecurityPolicy resources across all namespaces. Only delete the GatewayClass resources for your own namespace (step 2 above). |
If a CRD gets stuck in a terminating state, remove the finalizer and reinstall:
kubectl patch crd <crd-name> \
-p '{"metadata":{"finalizers":[]}}' --type=merge
Then re-run the CRD install command from Prerequisites.
If using the Nginx LB service repoint (migration), the Nginx LB service will also survive helm delete due to the helm.sh/resource-policy: keep annotation. Delete it manually if no longer needed: kubectl delete svc kfuse-ingress-nginx-controller -n <namespace>
|
Troubleshooting
Kubernetes Gateway Programmed
If you are doing an upgrade, the envoy-gateway will says Programmed is False. you’re reusing the Nginx LB instead of creating a new one — so the Gateway object never gets an address assigned to it directly. This is expected and harmless.
Failed to install CRDS
Error Message:
Error: failed to install CRD crds/gatewayapi-crds.yaml: 10 errors occurred:
* customresourcedefinitions.apiextensions.k8s.io "gatewayclasses.gateway.networking.k8s.io" is forbidden: ValidatingAdmissionPolicy 'safe-upgrades.gateway.networking.k8s.io' with binding 'safe-upgrades.gateway.networking.k8s.io' denied request: Installing CRDs with version before v1.5.0 is prohibited by default. Uninstall ValidatingAdmissionPolicy safe-upgrades.gateway.networking.k8s.io to install older versions.
There are two potential issues here,
The cluster already has Gateway API CRDs at v1.5.0+ and the safe-upgrades ValidatingAdmissionPolicy is blocking a downgrade. The tag you tried to install is resolving to an older version.
An older version than 1.5.0 is being accidently installed and this is there to prevent that from being a problem.
If you deleted a version of kfuse from the cluster, but you did not execute the Cleanup Steps
It will find the existing envoy-gateway resources and prevent it from being updated.
AWS NLB Hairpin
On AWS, cert-manager’s HTTP-01 self-check may fail during initial certificate issuance because pods cannot reach the external NLB from inside the cluster. This only affects the first-time cert setup — not ongoing operations. Not applicable to GCP.
If the certificate takes too long to issue, add a hostAliases entry to cert-manager that maps your domain to the envoy ClusterIP inside the cluster:
export NAMESPACE="kfuse"
export HOSTNAME="observe.example.com"
ENVOY_IP=$(kubectl get svc envoy -n $NAMESPACE \
-o jsonpath='{.spec.clusterIP}')
helm upgrade cert-manager jetstack/cert-manager \
--namespace $NAMESPACE \
--set "hostAliases[0].ip=$ENVOY_IP" \
--set "hostAliases[0].hostnames[0]=$HOSTNAME"
| If the envoy service is recreated (new ClusterIP), update the hostAlias with the new IP. |
Shared Cluster CRD Deletion
Do NOT delete CRDs on clusters where other namespaces also use envoy-gateway. CRDs are cluster-scoped — deleting them removes all Gateway, HTTPRoute, and SecurityPolicy resources across all namespaces.
Only delete the GatewayClass resources for your own namespace.
If a CRD gets stuck in terminating state:
kubectl patch crd <crd-name> \
-p '{"metadata":{"finalizers":[]}}' --type=merge
Then re-run the CRD install command from Prerequisites.
Multi-Tenant Clusters
When multiple Kloudfuse instances share the same cluster, each namespace must scope its envoy-gateway controller:
envoy-gateway:
config:
envoyGateway:
provider:
kubernetes:
watch:
type: Namespaces
namespaces:
- <your-namespace>
The chart automatically uses a namespace-specific controllerName to prevent cross-namespace interference.