#Rolling back the migration

Every phase before Mesh v1 is decommissioned can be undone. This document lists what to do per phase, and what recovery looks like after the point of no return.

#When to roll back

Roll back when one of these is true and you cannot explain it within your change window:

• The probes report failures that do not stop by themselves.
• Calls between workloads fail with TLS errors.
• A verification step of the current phase does not pass.

#Roll back by phase

The observability part of Phase 6 has no rollback path of its own. The Mesh v1 tracing components were deleted in Phase 5, so returning to them means the recovery procedure below; everything up to that point — the new collector, the Telemetry provider, the Instrumentation resources, the Kiali tracing configuration — is reconfigured forward rather than rolled back.

A mesh that spans several clusters rolls back the same way, once per cluster, with the additions described under Multi-cluster meshes.

#Phase 1

Deleting the Kiali resource removes what the operator created for it: the deployment, the service, the ingress, the service account and the configuration.

The Kiali Operator, the kiali and kiali-monitoring-basic-auth secrets in istio-system and the cpaas.io/project label on that namespace stay behind. Nothing depends on them and they cost one small deployment. Redoing Phase 1 reuses them: the two kubectl create secret commands then report already exists, which is expected and does not have to be corrected.

In a multi-cluster mesh, the Kiali server also keeps one kiali-remote-cluster-secret-<cluster> secret per other cluster, and that one must not be reused. Deleting the Kiali resource of a remote cluster deletes the service account the secret authenticates with, and redoing Phase 1 creates a new one with a new token. The Kiali server then does not start at all: it reports failed to get server groups: the server has asked for the client to provide credentials and shows nothing, not even the cluster it runs on. Generate the remote cluster secrets again, see Create a long-lived service account token and Create a remote cluster secret. To remove Kiali from a multi-cluster mesh rather than redo it, follow Cleaning up Kiali, which unregisters the remote clusters before the server is deleted.

#Phase 2

kubectl delete istio default
kubectl delete istiocni default

The Mesh v1 control plane is untouched by this, and no workload restarts. Keep Kiali if you want to go on watching the mesh.

A multi-network mesh has one more resource to remove per cluster. Deploy the Mesh v2 east-west gateways added an istio-eastwestgateway-v2 deployment beside the Mesh v1 one, and deleting the control plane leaves it running. Both generations are selected by the same service, so the Mesh v2 pod stays a ready endpoint of a mesh that has no Mesh v2 control plane any more, serving the configuration it received last: cross-cluster traffic that reaches it is sent to the pod addresses it knew at that moment, and every connection to an address that has been replaced since fails. Nothing reports it — a mesh with locality failover answers from the local cluster instead, with 200 — so remove it on every cluster:

kubectl -n istio-system delete deploy istio-eastwestgateway-v2
kubectl -n istio-system delete sa istio-eastwestgateway-v2
kubectl -n istio-system get endpoints istio-eastwestgateway

The endpoints of the service are the Mesh v1 gateway pods again.

If you redo Phase 2 later instead of staying on Mesh v1, the gateway does not have to be removed and recreated: the new control plane is published under the same name, the pod reconnects to it on its own and its endpoints are correct again.

#Phase 3

kubectl label ns "${MESH_NAMESPACE}" istio.io/rev="${V1_REVISION}" --overwrite
kubectl -n "${MESH_NAMESPACE}" rollout restart "deploy/${WORKLOAD}"
kubectl -n "${MESH_NAMESPACE}" rollout status "deploy/${WORKLOAD}" --timeout=300s

The pod settings written into the workload template during the migration are also valid for Mesh v1 — they are the values Mesh v1 itself used — so nothing else has to be reverted.

A namespace label applies to the whole namespace. Rolling back one workload also means that every other workload of that namespace returns to Mesh v1 the next time it restarts.

#Phase 4

kubectl -n "${GATEWAY_NAMESPACE}" scale deploy "${GATEWAY_NAME}" --replicas="${GATEWAY_REPLICAS}"
kubectl -n "${GATEWAY_NAMESPACE}" rollout status "deploy/${GATEWAY_NAME}" --timeout=300s
kubectl -n "${GATEWAY_NAMESPACE}" scale deploy "${GATEWAY_NAME}-v2" --replicas=0

Wait for the Mesh v1 gateway before you scale the Mesh v2 gateway down. Both deployments sit behind the same service, and scaling both in one go leaves that service without a ready endpoint for as long as the Mesh v1 pod needs to start — about nine seconds in a test cluster, which a probe that runs every few seconds does not see and a client does.

GATEWAY_REPLICAS is the replica count the Mesh v1 gateway had, recorded when the Mesh v2 gateway was scaled up, see Migrating gateways. The Mesh v1 deployment has been at zero since then, so the value cannot be read back from the cluster; if the shell is gone, take it from ${BACKUP_DIR}/gateway-replicas.txt.

#Phase 6

Restore the certificate authority with the function from Preparing for the migration, which applies the secret in place:

restore_from_backup "${BACKUP_DIR}/cacerts.yaml"

Do not delete the secret and create it again. istiod would briefly run without a configured certificate authority, which is what Hand over the certificate authority warns about. istiod reports the certificate it picked up about a minute later, no pod is restarted, and workload certificates are reissued from the restored intermediate certificate within their normal lifetime. Both certificates are signed by the same root, so a pod that still holds one issued by the other keeps being trusted.

The restored certificate is the one Mesh v1 issued, with the lifetime Mesh v1 gave it — a couple of months, where the certificate Hand over the certificate authority replaced it with lasts ten years. Nothing renews it: that was a Mesh v1 controller, see Check the lifetime of the intermediate certificate. Read its expiry date back and decide what happens before it is reached.

Deleting the default revision tag is safe only while no namespace uses it:

kubectl get ns -l istio-injection=enabled

Expected output: none. In a multi-cluster mesh, run this against every cluster and name each one with --context: No resources found answered for the cluster your kubeconfig happens to have selected reads exactly like the expected output for the cluster you meant. A namespace that Adopt the default revision tag moved from istio.io/rev to istio-injection=enabled is claimed by no webhook once the tag is gone. Its pods keep running and nothing reports an error; the next restart silently produces pods without a sidecar. Label such a namespace back first:

kubectl label ns "${MESH_NAMESPACE}" istio.io/rev="${V2_REVISION}" istio-injection- --overwrite
kubectl delete istiorevisiontag default

#Phase 5 and later

Mesh v1 is gone. Recovery means installing it again. Restore every resource with the function from Preparing for the migration: a backup holds the object as the cluster had it, and applying it unchanged recreates owner references to resources that no longer exist, which makes the garbage collector delete what you just restored.

1. Install the Service Mesh Essentials plugin on the global cluster from Marketplace → Cluster Plugins — but only if Phase 5 uninstalled it. It is left installed when the platform still serves another Mesh v1 mesh, and global-asm-controller and the Mesh v1 webhooks then never went away either, which makes this whole step a no-op. Check which of the two cases you are in:

   kubectl --context "${GLOBAL_CONTEXT}" get moduleinfoes.cluster.alauda.io -l cpaas.io/module-name=asm-global
   kubectl --context "${GLOBAL_CONTEXT}" -n cpaas-system get deploy global-asm-controller

   A ModuleInfo and a running controller mean the plugin is still installed and you continue with step 2. Otherwise install it: Phase 5 uninstalls the plugin without touching its catalog entry, so it is normally still listed in the marketplace. If it is not, apply ${BACKUP_DIR}/global-asm-moduleplugin.yaml first: that resource is the catalog entry, and installing is then recreating ${BACKUP_DIR}/global-asm-moduleinfo.yaml.

2. Restore the ServiceMesh and ServiceMeshGroup resources of this mesh, and wait for the Mesh v1 control plane to be running.

   ${BACKUP_DIR}/global-servicemeshes.yaml and ${BACKUP_DIR}/global-servicemeshgroups.yaml hold every mesh of the platform, not only yours. Applying them whole writes the resources of the other meshes as well, from a snapshot taken before your migration began, and silently reverts whatever has changed in them since. Select your own mesh with the variables Phase 5 deleted it by:

   kubectl config use-context "${GLOBAL_CONTEXT}"
   kubectl create -f "${BACKUP_DIR}/global-servicemeshes.yaml" --dry-run=client -o json \
     | jq --arg names "${CLUSTER_NAMES}" 'select(.metadata.name as $n | ($names | split(" ")) | index($n))' \
     > "${BACKUP_DIR}/mesh-servicemeshes.json"
   kubectl create -f "${BACKUP_DIR}/global-servicemeshgroups.yaml" --dry-run=client -o json \
     | jq --arg group "${MESH_GROUP}" 'select(.metadata.name == $group)' \
     > "${BACKUP_DIR}/mesh-servicemeshgroups.json"
   restore_from_backup "${BACKUP_DIR}/mesh-servicemeshes.json"
   restore_from_backup "${BACKUP_DIR}/mesh-servicemeshgroups.json"

   restore_from_backup applies to the selected context and these two resources live on the global cluster, which is why the block selects it. Select the mesh cluster again before you continue with step 3.

   global-asm-controller recreates the plugin records of the mesh clusters from these resources — the asm and istio ModuleInfo resources that Phase 5 cleared, which are not part of the backup — and installs the Mesh v1 operators recorded in them. If an operator does not come back this way, restore its subscription from ${BACKUP_DIR}/olm.yaml, at the version recorded there. Do not apply that file as a whole: it holds every subscription and every cluster service version of the cluster, including the ones that have nothing to do with the mesh. Select the Mesh v1 packages, the same four that Phase 5 removed:

   kubectl create -f "${BACKUP_DIR}/olm.yaml" --dry-run=client -o json \
     | jq 'select(.kind == "Subscription"
           and (.spec.name | test("^(asm-operator|flagger-operator|jaeger-operator|opentelemetry-operator)$")))' \
     > "${BACKUP_DIR}/v1-operator-subscriptions.json"
   restore_from_backup "${BACKUP_DIR}/v1-operator-subscriptions.json"

   A subscription with installPlanApproval: Manual waits for its install plan to be approved before anything is installed:

   kubectl -n istio-system get installplans.operators.coreos.com -o json \
     | jq -r '.items[] | select(.spec.approved | not) | .metadata.name' \
     | while read -r plan; do
         kubectl -n istio-system patch installplans.operators.coreos.com "${plan}" \
           --type=merge -p '{"spec":{"approved":true}}'
       done

3. Restore the platform resources — microservices, gateways, policies — from the ${BACKUP_DIR}/asm-*.yaml files. Leave out what Mesh v1 generates for itself; it is recreated by its controllers.

   The gateway services stand in the way of the GatewayDeploy resources. Phase 5 kept them because deleting them interrupts traffic, and the Mesh v1 webhook refuses to create a gateway whose service exists already: admission webhook "vgatewaydeploys.kb.io" denied the request: Service "<gateway>" already exists. The backup holds every gateway of the cluster in one file, so delete the service of each of them first — restoring the file while one service is left creates that gateway and is rejected for the rest:

   kubectl create -f "${BACKUP_DIR}/asm-gatewaydeploys.asm.alauda.io.yaml" --dry-run=client -o json \
     | jq -r '"\(.metadata.namespace) \(.metadata.name)"' \
     | while read -r ns name; do
         kubectl -n "${ns}" delete svc "${name}" --ignore-not-found
       done
   restore_from_backup "${BACKUP_DIR}/asm-gatewaydeploys.asm.alauda.io.yaml"

   Mesh v1 recreates each service together with its deployment. Read them back:

   kubectl create -f "${BACKUP_DIR}/asm-gatewaydeploys.asm.alauda.io.yaml" --dry-run=client -o json \
     | jq -r '"\(.metadata.namespace) \(.metadata.name)"' \
     | while read -r ns name; do
         kubectl -n "${ns}" get svc "${name}" -o wide
       done

   The node ports are part of the GatewayDeploy, so the addresses the gateway is reached at do not change. The cluster IP does: this is a new service. Ingress is refused from the moment the service is deleted until Mesh v1 has recreated it — a second and a half when the two commands follow each other, and as long as you take if you stop in between — and the node ports belong to nobody during that window, so another service can claim one. Compare each recreated service with its ${BACKUP_DIR}/gateway-svc-<namespace>-<gateway>.yaml before you continue.

4. Restore the namespace labels from ${BACKUP_DIR}/namespaces.yaml and restart the workloads.

5. Remove the Mesh v2 resources. Check first that the Mesh v1 gateways are ready: the two generations share one service, and deleting the Mesh v2 gateway while the Mesh v1 one is not running interrupts traffic.

   What to remove	Created in
   Istio, IstioCNI and the istio-cni namespace	Phase 2
   In a multi-network mesh, per cluster: the istio-eastwestgateway-v2 deployment and its service account	Phase 2
   The PodMonitor and the ServiceMonitor created for Mesh v2	Phase 1 and Phase 5
   Kiali and its secrets; in a multi-cluster mesh in the order of Cleaning up Kiali	Phase 1
   The access-logging Telemetry of each migrated namespace	Phase 3
   Per gateway: the <gateway>-v2 deployment, its service account, and the <gateway>-v2-sds role and role binding; the <gateway>-access-logging Telemetry	Phase 4
   The asm-default Telemetry in istio-system	Phase 5
   The IstioRevisionTag named default	Phase 6
   The Mesh v2 operator, if Mesh v2 is not to be reinstalled later	Phase 2

   Work down the table in this order. The operator is last because Istio, IstioCNI and the IstioRevisionTag carry a finalizer that only the operator clears: deleting the operator before them leaves them in Terminating for good, with a kubectl delete that never returns, and the object can then only be freed by patching its finalizers away by hand.

   The reinstalled Mesh v1 brings its own monitors and its own mesh-wide Telemetry back. Keeping both sets of monitors scrapes every proxy twice. Of two mesh-wide Telemetry resources only one is ever in effect — the oldest one in istio-system that has no selector — so asm-default keeps winning over the resource Mesh v1 recreates, and keeps the mesh pointed at providers that no longer exist.

   A Mesh v2 gateway left behind is the hardest of these to notice. Its pod is still selected by the restored gateway service, and it goes on serving the configuration it received last, answering 200 long after its control plane is gone. It stops following any change from that moment on. Step 7 is what reveals it. The east-west gateway of a multi-network mesh is the one most easily forgotten, and the one that fails most quietly, see Phase 2.

6. Restore cacerts from ${BACKUP_DIR}/cacerts.yaml if Phase 6 already replaced it, as described under Phase 6 above.

7. Verify with the probes, and check which control plane injected each pod:

   kubectl get pod -A -o json | jq -r '[.items[]
     | (.metadata.annotations["sidecar.istio.io/status"] // "") | fromjson? | .revision // empty]
     | group_by(.) | map({revision: .[0], pods: length})'

   Only the Mesh v1 revision may be left. A pod that still reports the Mesh v2 revision is a workload that was not restarted in step 4, or one of the Mesh v2 gateways of step 5 — including the east-west gateway of a multi-network mesh.

8. Remove the asm.cpaas.io/user-managed annotation from the resources generated for external services. Phase 4 set it to stop the Mesh v1 controllers from managing them; while it is set, the restored Mesh v1 does not manage them either, and a change made to an external service in the web console is never applied to the mesh.

   kubectl get gateways.networking.istio.io,virtualservices.networking.istio.io,destinationrules.networking.istio.io \
     -A -o json | jq -r '.items[]
       | select(.metadata.annotations["asm.cpaas.io/user-managed"] == "true")
       | "\(.kind) \(.metadata.namespace) \(.metadata.name)"' \
     | while read -r kind ns name; do
         kubectl -n "${ns}" annotate "$(echo "${kind}" | tr '[:upper:]' '[:lower:]').networking.istio.io" "${name}" \
           asm.cpaas.io/user-managed-
       done

Traces and metrics collected by Mesh v2 stay in their own indices and are not restored by this procedure.

Because this path is long, and because it depends on backups that were taken before the migration, rehearse the whole migration in a test environment first, as Migration overview requires.

#Multi-cluster meshes

Every step above acts on one cluster and is repeated on each cluster of the mesh. The exceptions are the global steps of the recovery procedure — step 1 and step 2 — which cover the whole mesh at once, exactly as they do in Phase 5.

The clusters may be rolled back one at a time, and the mesh may stay in between: a mesh whose first cluster is back on Mesh v1 while the second still serves Mesh v2 keeps its cross-cluster call distribution, its remote endpoints and its mutual TLS. Both generations of a cluster use the cacerts of that cluster, and all intermediate certificates are signed by the same root.

Name every cluster explicitly with --context, as Address one cluster at a time requires. The precondition of Phase 6 is where leaving it out is not noticed: No resources found answered for the wrong cluster reads like the expected output, and deleting the revision tag on that basis costs the sidecars of a namespace on the cluster you meant.

Three things are added to the procedure:

• The Mesh v2 east-west gateway of a multi-network mesh, see Phase 2. It is removed per cluster, when Phase 2 is rolled back and in step 5 of the recovery procedure alike.

• The remote cluster secrets of Kiali, see Phase 1. They survive the deletion of the Kiali resources and must be generated again before Phase 1 is redone.

• The remote secrets of the mesh. They are what each control plane reads the other clusters with, and both generations discover them by label. Deleting the Mesh v1 control plane does not delete them, so they are normally still in place after a decommissioning — confirm it rather than assume it:

  kubectl -n istio-system get secret -l istio/multiCluster=true

  One secret per other cluster. Restore a missing one with restore_from_backup "${BACKUP_DIR}/remote-secrets.yaml", and check that the restored Mesh v1 control plane reads the other clusters again:

  istioctl remote-clusters --revision "${V1_REVISION}"

  Each cluster lists the other one with status synced.