#Migrating a multi-cluster mesh

A mesh that spans several clusters follows the same phases as a single-cluster mesh. This section describes what is added per phase, and the order in which the clusters are migrated.

Read Migrating a single-cluster mesh first: every step there applies to each cluster of the mesh, and the pages in this section only describe the differences.

#How the clusters are migrated

• The global cluster orchestrates the mesh. Its steps — freezing the Mesh v1 controllers, deleting the mesh resources, uninstalling the plugin — are run once and cover every cluster.
• Every other command of Migrating a single-cluster mesh is run once per cluster, against that cluster.
• Two points in the order are fixed. Every cluster has its Mesh v2 control plane — and, in a multi-network mesh, its Mesh v2 east-west gateway — before the first workload of any cluster moves. And every cluster serves its workloads and gateways from Mesh v2 before Mesh v1 is decommissioned on any of them. Between those two points the clusters are migrated one at a time, so that a workload that has moved always finds a control plane and a gateway of its own generation on the other side.
• The certificate authority is shared: each cluster has its own intermediate certificate signed by one root. Mesh v2 reuses the cacerts secret of each cluster, so workload certificates stay mutually trusted across clusters during the whole migration.
• The remote secrets that let each control plane read the other clusters are reused as they are. Mesh v2 discovers them by label, exactly as Mesh v1 did.
• The custom configuration of the mesh is collected per cluster and expressed the Mesh v2 way per cluster, see Collect the custom Istio configuration. Clusters of one mesh rarely have identical control plane settings.

#The cross-cluster probe

The checks on this page and on the two pages below all call one service of the mesh from one workload, and they name it with the same variables. Set them on a service that has pods in both clusters, and keep the values for the whole migration. Replace <namespace-they-run-in>, <deployment-that-makes-the-call>, <its-application-container>, <service-it-calls> and <port-of-that-service> with your own values:

export MESH_NAMESPACE=<namespace-they-run-in>
export CLIENT=<deployment-that-makes-the-call>
export APP_CONTAINER=<its-application-container>
export TARGET_APP=<service-it-calls>
export TARGET_SERVICE="${TARGET_APP}.${MESH_NAMESPACE}.svc.cluster.local"
export APP_PORT=<port-of-that-service>

The service is named twice on purpose. TARGET_APP is its short name, which is what its EndpointSlice is labelled with; TARGET_SERVICE is its fully qualified name, which is what the Envoy cluster of the service is named after. A proxy-config endpoint --cluster "outbound|${APP_PORT}||${TARGET_SERVICE}" given the short name prints its header and no rows — the same output as a service whose cross-cluster endpoints are gone. A request is addressed by either form.

APP_PORT is the port of the service, not the port its pods listen on.

#Determine the network topology

The migration steps differ depending on whether the clusters share one network or use one network each. Determine this before you start. Replace <context-of-the-first-cluster> and <context-of-the-second-cluster> with your own values.

export CTX_CLUSTER1=<context-of-the-first-cluster>
export CTX_CLUSTER2=<context-of-the-second-cluster>
export V1_REVISION=1-22

for ctx in "${CTX_CLUSTER1}" "${CTX_CLUSTER2}"; do
  echo "== ${ctx}"
  kubectl --context "${ctx}" -n istio-system get istiooperators.install.istio.io -l '!asm.cpaas.io/component' -o json \
    | jq -r '.items[0] | "   meshID=\(.spec.values.global.meshID // "") network=\(.spec.values.global.network // "") clusterName=\(.spec.values.global.multiCluster.clusterName // "")"'
  echo "   east-west gateways:"
  kubectl --context "${ctx}" -n istio-system get svc -l topology.istio.io/network --no-headers 2>/dev/null \
    | awk '{print "     " $1, $2, $4}'
done

Example output

== cluster-1
   meshID=mesh-1 network=mesh-1-a1b2c clusterName=cluster-1
   east-west gateways:
     istio-eastwestgateway LoadBalancer 10.0.0.10
== cluster-2
   meshID=mesh-1 network=mesh-1-d3e4f clusterName=cluster-2
   east-west gateways:
     istio-eastwestgateway LoadBalancer 10.0.0.20

If pods of one cluster can reach pods of the other directly, the mesh is single network even when the network values differ by accident. Check before you decide:

PEER=$(kubectl --context "${CTX_CLUSTER2}" -n "${MESH_NAMESPACE}" get endpointslices \
  -l "kubernetes.io/service-name=${TARGET_APP}" \
  -o jsonpath='{.items[0].endpoints[0].addresses[0]}:{.items[0].ports[0].port}')
kubectl --context "${CTX_CLUSTER1}" -n "${MESH_NAMESPACE}" exec "deploy/${CLIENT}" -c "${APP_CONTAINER}" -- \
  curl -s -o /dev/null -w '%{http_code}\n' --max-time 5 "http://${PEER}/"

The address and the port come from the endpoints of TARGET_APP in the other cluster: the address of a pod and the port that pod listens on. This is the one check that uses the pod port rather than APP_PORT — a request to a pod address on the service port is answered by the sidecar with 503 in either topology and tells you nothing.

Run the same command against a pod of the local cluster as a control. It must answer 200. If it does not, the check is measuring something else — the wrong port, a workload that serves no HTTP, a policy that rejects the call — and its cross-cluster result means nothing.

#Address one cluster at a time

CTX_CLUSTER1 and CTX_CLUSTER2 name the two clusters of the mesh and keep those values for the whole migration. They appear in the commands that compare the clusters or act on both of them.

Every other command acts on one cluster, and names it ${CTX}. Set it to the cluster you are working on, and set it again each time you move to the next one:

export CTX="${CTX_CLUSTER1}"
kubectl config get-contexts "${CTX}"

A command without --context runs against whatever context your kubeconfig has selected, which is rarely the cluster you mean during a multi-cluster migration. Every command in this section passes it explicitly.

The commands of Migrating a single-cluster mesh do not: they act on the cluster your kubeconfig has selected. Select ${CTX} before you run one of those pages for a cluster, and again when you move to the next one:

kubectl config use-context "${CTX}"
kubectl config current-context

The selected context is stored in the kubeconfig file, so every shell that reads that file shares it — a traffic probe or a check that keeps running while you work on the other cluster follows the switch and reports on the wrong cluster. Where steps of both clusters run at the same time, give each cluster its own copy of the kubeconfig, kubectl config view --flatten --raw > "${HOME}/.kube/${CTX}" with the context of that cluster selected in it, and point KUBECONFIG at the copy of the cluster the shell works on.