Configure Envoy Ingress

Overview

Envoy Gateway implements the Kubernetes Gateway API and replaces ingress-nginx as the ingress controller for Kloudfuse. This guide covers new cluster installation, zero-downtime migration from ingress-nginx, rollback procedures, and post-uninstall cleanup.

Prerequisites

Hardware Requirements

  • Kloudfuse version 4.0.0 or later — Envoy Gateway is not compatible with earlier versions

  • Kubernetes cluster version 1.27 or later

  • kubectl configured with cluster access

  • Helm 3.x installed

Install Envoy CRDs

Install the CRDs before running helm install or helm upgrade. The version must match the Envoy Gateway version used by the Kloudfuse chart:

Kloudfuse Version Envoy Gateway CRD Version

4.0.0

v1.3.0

4.0.1+

v1.7.1

helm show crds oci://docker.io/envoyproxy/gateway-helm \
  --version <VERSION> \
  | kubectl apply --server-side --force-conflicts -f -
bash
This command is idempotent — safe to run on clusters that already have the CRDs installed. The --server-side flag avoids annotation size limits on large CRDs.

Install or Upgrade cert-manager

cert-manager v1.14+ with Gateway API support is required for automatic TLS certificate issuance.

cert-manager is cluster-wide but runs in a single namespace. If cert-manager is already installed on your cluster, upgrade the existing installation in its current namespace rather than installing a second instance.
helm repo add jetstack https://charts.jetstack.io
helm repo update
helm upgrade --install cert-manager jetstack/cert-manager \
  --namespace <cert-manager-namespace> \ (1)
  --version v1.17.1 \
  --set crds.enabled=true \
  --set config.enableGatewayAPI=true
bash
1 Use the namespace where cert-manager is already installed (e.g., cert-manager), not the Kloudfuse namespace. Run kubectl get pods -A | grep cert-manager to find it.
The config.enableGatewayAPI=true flag is required. Without it, cert-manager only watches Ingress resources and will not create certificates for Gateway resources.

Add the following TLS section to your custom_values.yaml to enable automatic certificate issuance:

tls:
  enabled: true
  host: <your-hostname>
  email: <your-email>
  clusterIssuer: <namespace>-letsencrypt-prod
  createClusterIssuer: true
yaml

New Install

For new clusters where ingress-nginx has never been installed, enable envoy-gateway and disable ingress-nginx:

envoy-gateway:
  enabled: true
  installGatewayRoutes: true
  envoyService:
    annotations:
      service.beta.kubernetes.io/aws-load-balancer-type: nlb
      service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: 'true'
      service.beta.kubernetes.io/aws-load-balancer-eip-allocations: <YOUR_EIP_ALLOC_IDS>
    patch:
      externalTrafficPolicy: Local
    external:
      enabled: true
    internal:
      enabled: true
      annotations:
        service.beta.kubernetes.io/aws-load-balancer-internal: "true"

ingress-nginx:
  enabled: false
  installIngressRules: false
yaml
TLS settings do not change. The existing tls.enabled, tls.host, and tls.clusterIssuer work with envoy-gateway. To disable the HTTP listener on port 80 (HTTPS only), set envoy-gateway.enableHttp to false in your values file.
envoy-gateway:
  enableHttp: false
yaml

Separate Internal and External Hosts

To route ingest traffic to an internal load balancer and query traffic to an external load balancer, configure the tls.internalIngest and tls.externalQuery fields:

tls:
  host: <PRIMARY DNS HOST>
  internalIngest:
    hosts:
      - <INTERNAL INGEST DNS HOST>
    secretName: "<TLS secret for ingest hosts, or empty for ACM>"
    hostOnly: false
  externalQuery:
    hosts:
      - <EXTERNAL QUERY DNS HOST>
    secretName: "<TLS secret for query hosts, or empty for ACM>"
    hostOnly: false
yaml
Field Description

tls.internalIngest.hosts

DNS hostnames for internal ingestion routes (Gateway listeners and ingest HTTPRoutes).

tls.internalIngest.secretName

Kubernetes TLS secret for the ingest hosts. Leave empty when using ACM termination.

tls.internalIngest.hostOnly

If true, ingest routes only match internalIngest.hosts and exclude tls.host.

tls.externalQuery.hosts

DNS hostnames for external query/browser routes (Gateway listeners and query HTTPRoutes).

tls.externalQuery.secretName

Kubernetes TLS secret for the query hosts. Leave empty when using ACM termination.

tls.externalQuery.hostOnly

If true, query routes only match externalQuery.hosts and exclude tls.host.

These fields are used by Envoy Gateway templates only. If migrating from ingress-nginx, these replace the flat fields tls.ingestInternalHosts, tls.ingestInternalHostOnly, tls.ingestInternalSecretName, tls.queryHosts, tls.queryHostOnly, and tls.queryHostSecretName.

Upgrade from Nginx to Envoy

This procedure migrates an existing Kloudfuse installation from ingress-nginx to envoy-gateway without recreating load balancers or changing IPs or DNS. This migration will result in zero-downtime.

The Nginx Load Balancer Service is preserved throughout the migration. Instead of deleting it and creating a new Envoy LB, the Nginx service’s selector is repointed to Envoy proxy pods. The same NLB/LB keeps the same IPs and DNS.

Step Change Values Traffic Served By What Happens

D0 (current)

None

Nginx

Starting state

Step 1 (prepare)

Enable Envoy + envoyMigration.enabled

Nginx (unchanged)

Envoy starts alongside nginx. Nginx LB gets resource-policy: keep annotation. Envoy creates ClusterIP services (no new LB).

Step 2 (switch)

Set envoyMigration.external/internal: true

Envoy

Nginx LB selector switches to Envoy pods. TargetPorts change to 10080/10443. Traffic flows through Envoy via the same LB.

Step 3 (cleanup)

Set ingress-nginx.enabled: false

Envoy

Remove Nginx Controller. Nginx LB Service is kept (resource-policy annotation).

Step 1: Enable Envoy

  1. Add the envoy-gateway section and the migration flag to your custom_values.yaml:

    global:
      envoyMigration:
        enabled: true
        external: false
        internal: false
    
    envoy-gateway:
      enabled: true
      installGatewayRoutes: true
      envoyService:
        external:
          enabled: true
        # Enable if you have an internal LB (1)
        # internal:
        #   enabled: true
    yaml
    1 Enable if you have an internal LB
  2. If your existing nginx configuration uses separate internal/external hosts, migrate the flat TLS fields to the new nested format. Add the tls.internalIngest and tls.externalQuery fields alongside your existing flat fields (which nginx still reads until it is disabled):

    Nginx (existing) Envoy (add these)

    tls.ingestInternalHosts: [host]

    tls.internalIngest.hosts: [host]

    tls.ingestInternalSecretName: secret

    tls.internalIngest.secretName: secret

    tls.ingestInternalHostOnly: true

    tls.internalIngest.hostOnly: true

    tls.queryHosts: [host]

    tls.externalQuery.hosts: [host]

    tls.queryHostSecretName: secret

    tls.externalQuery.secretName: secret

    tls.queryHostOnly: true

    tls.externalQuery.hostOnly: true

    For example, if your current config has:

    tls:
      host: app.example.com
      ingestInternalHosts:
        - ingest.example.com
      ingestInternalHostOnly: false
    yaml

    Add the envoy equivalent:

    tls:
      host: app.example.com
      ingestInternalHosts:          # keep for nginx during migration
        - ingest.example.com
      ingestInternalHostOnly: false # keep for nginx during migration
      internalIngest:               # new for envoy
        hosts:
          - ingest.example.com
        hostOnly: false
    yaml
    Keep both sets of fields during migration (Steps 1-2). After nginx is disabled in Step 3, the flat fields are no longer needed and can be removed.
  3. Run helm upgrade:

    helm upgrade --install kfuse oci://us-east1-docker.pkg.dev/mvp-demo-301906/kfuse-helm/kfuse \
      -n kfuse \
      --version <VERSION> \ (1)
      -f custom-values.yaml
    1 Replace <VERSION> with a valid Kloudfuse release value. See Release Notes for the latest release.

After completion, verify:

# Envoy pods should be running
kubectl get pods -n <namespace> | grep envoy

# Gateway should be Programmed: False
kubectl get gateway -n <namespace>

# Nginx should still be serving (302 redirect to login)
curl -sk https://your-hostname/ -o /dev/null -w "%{http_code}"

# Nginx LB should have keep annotation
kubectl get svc kfuse-ingress-nginx-controller -n <namespace> \
  -o jsonpath='{.metadata.annotations.helm\.sh/resource-policy}'
bash

Step 2: Switch Traffic to Envoy

  1. Change global.envoyMigration.external to true in your custom_values.yaml. If you have an internal LB, also set internal: true:

    global:
      envoyMigration:
        enabled: true
        external: true
        # Set to true if you have an internal Nginx LB to switch (1)
        internal: false
    yaml
    1 If you have an internal Nginx LB to switch, change to true
  2. Run helm upgrade:

    helm upgrade --install kfuse oci://us-east1-docker.pkg.dev/mvp-demo-301906/kfuse-helm/kfuse \
      -n kfuse \
      --version <VERSION> \ (1)
      -f custom-values.yaml
    1 Replace <VERSION> with a valid Kloudfuse release value. See Release Notes for the latest release.

After completion, verify:

# Selector should point to envoy, with targetPorts: 10080 10443
export NAMESPACE="<namespace>"
 kubectl get svc kfuse-ingress-nginx-controller -n $NAMESPACE -o jsonpath='selector: {.spec.selector}{"\n"}targetPorts: {.spec.ports[*].targetPort}{"\n"}'

# Should return 401 (Envoy auth) instead of 302 (Nginx redirect)
curl -sk https://your-hostname/ -o /dev/null -w "%{http_code}"
bash
AWS NLB health checks may take 15-30 seconds to converge after the targetPort change. This is transient.

Step 3: Disable Nginx

  1. Set ingress-nginx.enabled to false in your custom_values.yaml:

    ingress-nginx:
      enabled: false
      installIngressRules: false
    yaml
  2. Run helm upgrade:

    helm upgrade --install kfuse oci://us-east1-docker.pkg.dev/mvp-demo-301906/kfuse-helm/kfuse \
      -n kfuse \
      --version <VERSION> \ (1)
      -f custom-values.yaml
    1 Replace <VERSION> with a valid Kloudfuse release value. See Release Notes for the latest release.

The Nginx controller pod is removed. The Nginx LB Service is not deleted because of the resource-policy: keep annotation — it continues routing traffic to Envoy pods.

Step 4: Verification

export NAMESPACE="<namespace>"

# Nginx controller pod should be gone; Envoy pods should be running
kubectl get pods -n $NAMESPACE | grep -E "nginx|envoy"

# Nginx LB service should still exist; selector and targetPorts should point to Envoy
kubectl get svc kfuse-ingress-nginx-controller -n $NAMESPACE \
  -o jsonpath='selector: {.spec.selector}{"\n"}targetPorts: {.spec.ports[*].targetPort}{"\n"}'

# Gateway should show Programmed: False
kubectl get gateway -n $NAMESPACE

# All HTTPRoutes should show Accepted and Resolved
kubectl get httproute -n $NAMESPACE

# HTTPS should return 401 (Envoy auth challenge) — not 302 (Nginx redirect)
curl -sk https://your-hostname/ -o /dev/null -w "%{http_code}\n"
bash

Rollback to Nginx

To revert from Envoy back to Nginx:

  1. Only if the original Nginx LB Service still exists (i.e. you ran the in-place migration in Upgrade from Nginx to Envoy, which annotated the Nginx LB with helm.sh/resource-policy: keep so it survived through Step 3), remove those annotations so Helm can manage the Service again:

    export NAMESPACE="<namespace>"
    kubectl annotate svc kfuse-ingress-nginx-controller \
      helm.sh/resource-policy- -n $NAMESPACE
    kubectl annotate svc kfuse-ingress-nginx-controller-internal \
      helm.sh/resource-policy- -n $NAMESPACE  # if internal LB exists
    bash

    Check before running: kubectl get svc -n <namespace> kfuse-ingress-nginx-controller -o jsonpath='{.metadata.annotations.helm\.sh/resource-policy}{"\n"}'. If the Service does not exist (you migrated by deleting Nginx outright, or did a fresh Envoy install), skip this step — Helm will create a brand-new Nginx Service from your custom_values.yaml in step 2.

  2. Update the custom_values.yaml to disable Envoy and re-enable Nginx with the full controller config that was in place before the Envoy migration. Helm will not infer Nginx’s controller settings (LB IP, service annotations, headers) from the envoy-gateway block — every field that was set on ingress-nginx.controller before the migration must be present here, otherwise the Nginx LB will come up with chart defaults.

    Do not delete the existing envoy-gateway: block from custom_values.yaml. Set envoy-gateway.enabled: false and envoy-gateway.installGatewayRoutes: false — those two flags are enough to fully disable Envoy (the controller, the EnvoyProxy data plane pods, and the Gateway + HTTPRoute objects all go away). Removing the rest of the block during rollback risks two concrete failure modes:
    • If the envoy-gateway block is deleted but envoyMigration / installGatewayRoutes are not consistent, the Nginx or Envoy pods can land in CrashLoopBackOff because the rendered Service or controller config is incomplete.

    • If the ingress-nginx.controller.service.loadBalancerIP / EIPs / annotations below do not exactly match the LB the cluster had before the migration, the cloud provider allocates a fresh load balancer with a new IP, breaking DNS and causing a downtime window until the A records are updated.

      Keeping the full envoy-gateway: block around also means a future re-upgrade to Envoy is just a matter of flipping both flags back to true and re-running the Upgrade from Nginx to Envoy flow — no need to re-derive the LB IPs, service annotations, or traffic policies you previously tuned.

      global:
        envoyMigration:
          enabled: false
          external: false
          internal: false
      
      envoy-gateway:
        enabled: false
        installGatewayRoutes: false
      
      ingress-nginx:
        enabled: true
        installIngressRules: true
        controller:
          ingressClassResource:
            enabled: true
            name: kfuse-ingress
          ingressClass: kfuse-ingress
          watchIngressWithoutClass: false
          service:
            annotations:
              service.beta.kubernetes.io/aws-load-balancer-type: nlb
              service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: 'true'
              service.beta.kubernetes.io/aws-load-balancer-eip-allocations: <YOUR_EIP_ALLOC_IDS>  (1)
            externalTrafficPolicy: Local
            external:
              enabled: true
            internal:                          (2)
              enabled: true
              annotations:
                service.beta.kubernetes.io/aws-load-balancer-internal: "true"
                service.beta.kubernetes.io/aws-load-balancer-type: nlb
      yaml
      1 Replace with the comma-separated Elastic IP allocation IDs that were attached to the Nginx NLB before the migration. Without these, AWS provisions a fresh NLB with new public IPs and the existing DNS records break.
      2 Omit the internal: block if your cluster only had an external NLB before the migration. If you did have an internal NLB, the annotations here must match what it had originally — otherwise AWS provisions a fresh internal NLB with a new IP and the internal DNS records break.
  3. Run helm upgrade:

    helm upgrade --install kfuse oci://us-east1-docker.pkg.dev/mvp-demo-301906/kfuse-helm/kfuse \
      -n kfuse \
      --version <VERSION> \ (1)
      -f custom-values.yaml
    1 Replace <VERSION> with a valid Kloudfuse release value. See Release Notes for the latest release.
  4. Clean up orphaned Envoy resources:

    export NAMESPACE="<namespace>"
    
    # Delete controller-managed proxy Deployments and Services
    kubectl delete deploy -l app.kubernetes.io/managed-by=envoy-gateway -n $NAMESPACE
    kubectl delete svc -l app.kubernetes.io/managed-by=envoy-gateway -n $NAMESPACE
    
    # Remove GatewayClass finalizers and delete them
    for gc in $(kubectl get gatewayclass -o name | grep "$NAMESPACE"); do
      kubectl patch "$gc" --type=merge -p '{"metadata":{"finalizers":[]}}'
      kubectl delete "$gc"
    done
    bash
  5. Verify the rollback:

    export NAMESPACE="<namespace>"
    
    # Nginx controller pod should be running; no Envoy proxy pods should remain
    kubectl get pods -n $NAMESPACE | grep -E "nginx|envoy"
    
    # Nginx LB selector should point back to Nginx pods; targetPorts should be 80/443
    kubectl get svc kfuse-ingress-nginx-controller -n $NAMESPACE \
      -o jsonpath='selector: {.spec.selector}{"\n"}targetPorts: {.spec.ports[*].targetPort}{"\n"}'
    
    # HTTPS should return 302 (Nginx login redirect) — not 401
    curl -sk https://your-hostname/ -o /dev/null -w "%{http_code}\n"
    
    # Confirm no orphaned Envoy resources remain
    kubectl get all -l app.kubernetes.io/managed-by=envoy-gateway -n $NAMESPACE
    kubectl get gatewayclass | grep $NAMESPACE
    bash

Re-upgrading to Envoy after a rollback

If you kept the envoy-gateway: block in custom_values.yaml during the rollback (as recommended in Step 2 above), re-upgrading to Envoy is the same as a first-time upgrade — no need to re-derive LB IPs, service annotations, or traffic policies:

  1. Re-enable the migration flags in custom_values.yaml:

    global:
      envoyMigration:
        enabled: true
        external: false   # Step 1 — flip to true at Step 2
        internal: false
    
    envoy-gateway:
      enabled: true
      installGatewayRoutes: true
      # ... (preserved from before the rollback)
    yaml
  2. Follow the Upgrade from Nginx to Envoy flow from Step 1 onward.

If you deleted the envoy-gateway: block during the rollback, restore it from version control (or from the per-cloud examples in New Install) before re-upgrading.

Uninstall

When running helm delete (or helm uninstall), the envoy-gateway controller is removed but the resources it created at runtime are not deleted by Helm. These orphaned resources must be cleaned up manually.

The Envoy Deployments, Services, and ConfigMaps created by the envoy-gateway controller (not by Helm templates). Since Helm didn’t create them, it doesn’t track or delete them.

Cleanup Steps

After helm delete kfuse -n <namespace>:

# 1. Delete controller-managed proxy Deployments and Services
export NAMESPACE="<namespace>"
kubectl delete deploy -l app.kubernetes.io/managed-by=envoy-gateway -n $NAMESPACE
kubectl delete svc -l app.kubernetes.io/managed-by=envoy-gateway -n $NAMESPACE
kubectl delete configmap -l app.kubernetes.io/managed-by=envoy-gateway -n $NAMESPACE

# 2. Remove GatewayClass finalizers and delete them (cluster-scoped)
for gc in $(kubectl get gatewayclass -o name | grep "$NAMESPACE"); do
  kubectl patch "$gc" --type=merge -p '{"metadata":{"finalizers":[]}}'
  kubectl delete "$gc"
done

# 3. Verify no orphaned resources remain
kubectl get all -l app.kubernetes.io/managed-by=envoy-gateway -n $NAMESPACE
kubectl get gatewayclass | grep $NAMESPACE
bash
Shared clusters — Do NOT delete Gateway API or Envoy Gateway CRDs on clusters where other namespaces also use envoy-gateway. CRDs are cluster-scoped; deleting them removes all Gateway, HTTPRoute, and SecurityPolicy resources across all namespaces. Only delete the GatewayClass resources for your own namespace (step 2 above).

If a CRD gets stuck in a terminating state, remove the finalizer and reinstall:

kubectl patch crd <crd-name> \
  -p '{"metadata":{"finalizers":[]}}' --type=merge
bash

Then re-run the CRD install command from Prerequisites.

If using the Nginx LB service repoint (migration), the Nginx LB service will also survive helm delete due to the helm.sh/resource-policy: keep annotation. Delete it manually if no longer needed: kubectl delete svc kfuse-ingress-nginx-controller -n <namespace>

Troubleshooting

Kubernetes Gateway Programmed

If you are doing an upgrade, the envoy-gateway will says Programmed is False. you’re reusing the Nginx LB instead of creating a new one — so the Gateway object never gets an address assigned to it directly. This is expected and harmless.

Failed to install CRDS

Error Message:

Error: failed to install CRD crds/gatewayapi-crds.yaml: 10 errors occurred:
        * customresourcedefinitions.apiextensions.k8s.io "gatewayclasses.gateway.networking.k8s.io" is forbidden: ValidatingAdmissionPolicy 'safe-upgrades.gateway.networking.k8s.io' with binding 'safe-upgrades.gateway.networking.k8s.io' denied request: Installing CRDs with version before v1.5.0 is prohibited by default. Uninstall ValidatingAdmissionPolicy safe-upgrades.gateway.networking.k8s.io to install older versions.

There are two potential issues here,

The cluster already has Gateway API CRDs at v1.5.0+ and the safe-upgrades ValidatingAdmissionPolicy is blocking a downgrade. The tag you tried to install is resolving to an older version.

An older version than 1.5.0 is being accidently installed and this is there to prevent that from being a problem.

If you deleted a version of kfuse from the cluster, but you did not execute the Cleanup Steps

It will find the existing envoy-gateway resources and prevent it from being updated.

AWS NLB Hairpin

On AWS, cert-manager’s HTTP-01 self-check may fail during initial certificate issuance because pods cannot reach the external NLB from inside the cluster. This only affects the first-time cert setup — not ongoing operations. Not applicable to GCP.

If the certificate takes too long to issue, add a hostAliases entry to cert-manager that maps your domain to the envoy ClusterIP inside the cluster:

export NAMESPACE="kfuse"
export HOSTNAME="observe.example.com"

ENVOY_IP=$(kubectl get svc envoy -n $NAMESPACE \
  -o jsonpath='{.spec.clusterIP}')

helm upgrade cert-manager jetstack/cert-manager \
  --namespace $NAMESPACE \
  --set "hostAliases[0].ip=$ENVOY_IP" \
  --set "hostAliases[0].hostnames[0]=$HOSTNAME"
bash
If the envoy service is recreated (new ClusterIP), update the hostAlias with the new IP.

Shared Cluster CRD Deletion

Do NOT delete CRDs on clusters where other namespaces also use envoy-gateway. CRDs are cluster-scoped — deleting them removes all Gateway, HTTPRoute, and SecurityPolicy resources across all namespaces.

Only delete the GatewayClass resources for your own namespace.

If a CRD gets stuck in terminating state:

kubectl patch crd <crd-name> \
  -p '{"metadata":{"finalizers":[]}}' --type=merge
bash

Then re-run the CRD install command from Prerequisites.

Multi-Tenant Clusters

When multiple Kloudfuse instances share the same cluster, each namespace must scope its envoy-gateway controller:

envoy-gateway:
  config:
    envoyGateway:
      provider:
        kubernetes:
          watch:
            type: Namespaces
            namespaces:
              - <your-namespace>
yaml

The chart automatically uses a namespace-specific controllerName to prevent cross-namespace interference.