#Migrating a multi-network mesh

Each cluster of a multi-network mesh has its own network and its own east-west gateway, and cross-cluster traffic passes through those gateways. This document lists what is added to each phase of Migrating a single-cluster mesh; everything not mentioned here is identical.

Confirm the topology first, see Determine the network topology.

Most blocks below act on one cluster and address it as ${CTX}. Set it before the first of them, and again every time you move to the next cluster, see Address one cluster at a time:

export CTX="${CTX_CLUSTER1}"

#What is added per phase

#Phase 0: record and probe every cluster

Run Preparing for the migration against each cluster, with --context set to that cluster, and keep one variable set per cluster. The mesh identity differs per cluster and must be recorded per cluster:

for ctx in "${CTX_CLUSTER1}" "${CTX_CLUSTER2}"; do
  kubectl --context "${ctx}" -n istio-system get istiooperators.install.istio.io -l '!asm.cpaas.io/component' -o json \
    | jq -r --arg ctx "${ctx}" '.items[0] | "\($ctx) meshID=\(.spec.values.global.meshID) network=\(.spec.values.global.network) clusterName=\(.spec.values.global.multiCluster.clusterName)"'
  kubectl --context "${ctx}" -n istio-system get secret -l istio/multiCluster=true --no-headers | awk -v c="${ctx}" '{print c, "remote secret:", $1}'
done

meshID is the same on every cluster; network and clusterName differ. Mesh v2 must reproduce all three per cluster.

Extend the traffic probes of Set up traffic probes to cover the cross-cluster path: give each cluster an in-mesh probe that calls a service backed by workloads in both clusters, the one named in The cross-cluster probe. Both directions are probed, because a migration step on one cluster changes what the other one reaches.

Record what that path resolves to before anything changes, with the endpoint check of Phase 3 and Phase 4 run against each cluster: the endpoints of the other cluster, and rq_success on them, are the baseline the later phases are compared against. Keep the records with your migration notes rather than in BACKUP_DIR, which is a different directory on every cluster.

Compare against what you recorded, not against an even split: the traffic policies of the mesh decide the distribution, and a weighted locality distribution, locality failover and a mesh with neither all look different. The endpoints are the part that always tells you something — they are there whether traffic uses them or not, so a mesh where locality keeps every call inside its own cluster still shows whether the cross-cluster path is programmed.

#Phase 1: Kiali for a multi-cluster mesh

Kiali reads the metrics of every cluster from one store and needs read access to each cluster. Follow Installing Kiali in a multi-cluster mesh instead of the single-cluster instructions: the Kiali server runs on one cluster of the mesh, and every other cluster gets a Kiali resource that produces the remote cluster resources only.

Its prerequisites are what a migrated mesh usually has to add:

• The clusters report to the same monitoring store. kubectl --context "${CTX}" get feature monitoring -o jsonpath='{.spec.accessInfo.database.address}' returns the same address on every cluster of the mesh.
• The Kiali Operator is installed on every cluster, including the ones that run no Kiali server.
• The PodMonitor exists on every cluster, restricted as described in Create the metrics resources; the ServiceMonitor follows in Phase 5.
• The mesh-wide Telemetry resource carries the mesh_id tag override on every cluster that runs a control plane. Kiali filters the metrics of the mesh by that label, so a proxy that reports mesh_id="unknown" is invisible to it. A migrating mesh has nothing to add here: the Mesh v1 resource of each cluster carries the tag and configures the proxies of both generations, and Phase 5 replaces it per cluster with an asm-default that carries it too. Creating asm-default now instead would change nothing, see Create the metrics resources.

Kiali reports what it found in its log: one Discovered cluster line per cluster, each with Accessible=true.

#Phase 2: one control plane per cluster

Create the IstioCNI and Istio resources on each cluster as described in Installing the Mesh v2 control plane, with the values recorded for that cluster:

spec:
  values:
    global:
      meshID: ${MESH_ID}
      network: ${NETWORK}
      multiCluster:
        clusterName: ${MESH_CLUSTER_NAME}

Nothing else changes: each control plane loads the cacerts secret of its own cluster, and both intermediates are signed by the same root, so workloads keep trusting each other across clusters and across generations.

The remote secrets are reused. Verify that each Mesh v2 control plane sees the other cluster:

istioctl --context "${CTX_CLUSTER1}" remote-clusters --revision "${V2_REVISION}"
istioctl --context "${CTX_CLUSTER2}" remote-clusters --revision "${V2_REVISION}"

Each control plane lists the other cluster with status synced.

#Deploy the Mesh v2 east-west gateways

Cross-cluster traffic reaches a workload through the east-west gateway of its cluster. A Mesh v2 proxy sends that traffic to the gateway address the control plane gives it, so every cluster needs a Mesh v2 east-west gateway before workloads are migrated.

The Mesh v1 east-west gateway keeps running in parallel: both generations terminate the same SNI-based passthrough, and each proxy uses the gateway its own control plane advertises.

kubectl --context "${CTX}" -n istio-system get svc istio-eastwestgateway -o json \
  | jq -c '{labels: .metadata.labels, selector: .spec.selector, type: .spec.type, ports: [.spec.ports[] | {name, port}]}'
kubectl --context "${CTX}" -n istio-system get gateways.networking.istio.io istio-eastwestgateway -o json \
  | jq -c '{selector: .spec.selector, servers: .spec.servers}'

cat <<EOF | kubectl --context "${CTX}" apply -f -
apiVersion: v1
kind: ServiceAccount
metadata:
  name: istio-eastwestgateway-v2
  namespace: istio-system
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: istio-eastwestgateway-v2
  namespace: istio-system
spec:
  replicas: 1
  selector:
    matchLabels:
      gateway.migration: istio-eastwestgateway-v2
  template:
    metadata:
      annotations:
        inject.istio.io/templates: gateway
      labels:
        gateway.migration: istio-eastwestgateway-v2
        sidecar.istio.io/inject: "true"
        istio.io/rev: ${V2_REVISION}
        app: istio-eastwestgateway
        istio: eastwestgateway
        topology.istio.io/network: ${NETWORK}
    spec:
      serviceAccountName: istio-eastwestgateway-v2
      containers:
        - name: istio-proxy
          image: auto
          env:
            - name: ISTIO_META_REQUESTED_NETWORK_VIEW
              value: ${NETWORK}
          securityContext:
            capabilities:
              drop:
                - ALL
            allowPrivilegeEscalation: false
            privileged: false
            readOnlyRootFilesystem: true
            runAsNonRoot: true
            seccompProfile:
              type: RuntimeDefault
EOF
kubectl --context "${CTX}" -n istio-system rollout status deploy/istio-eastwestgateway-v2 --timeout=300s

POD=$(kubectl --context "${CTX}" -n istio-system get pod -l gateway.migration=istio-eastwestgateway-v2 -o jsonpath='{.items[0].metadata.name}')
istioctl --context "${CTX}" proxy-config listener "${POD}" -n istio-system --port 15443 | tail -n +2 | wc -l
kubectl --context "${CTX}" -n istio-system get endpoints istio-eastwestgateway -o jsonpath='{.subsets[*].addresses[*].ip}{"\n"}'
kubectl --context "${CTX}" -n istio-system exec "${POD}" -c istio-proxy -- \
  pilot-agent request GET clusters | grep "${TARGET_SERVICE}" | grep cx_total

#Phase 3 and Phase 4: migrate cluster by cluster

Migrate the workloads and gateways of one cluster as described for a single-cluster mesh, then move to the next cluster. The probes report whether the cross-cluster path keeps serving traffic. After each namespace, check that it is still programmed, on both clusters — a migration step on one cluster changes what the other one reaches:

istioctl --context "${CTX}" proxy-config endpoint "deploy/${CLIENT}.${MESH_NAMESPACE}" \
  --cluster "outbound|${APP_PORT}||${TARGET_SERVICE}" -o json \
  | jq -r '.[0].hostStatuses[] | "\(.address.socketAddress.address):\(.address.socketAddress.portValue) \(.healthStatus.edsHealthStatus) rq_success=\(.stats[]? | select(.name=="rq_success") | .value // 0)"'

The endpoint of the other cluster is the address of its east-west gateway on port 15443, and it is healthy. Its rq_success keeps growing while the probes run, in the proportion recorded in Phase 0; a counter that stops while the local one grows means the calls no longer leave the cluster. An endpoint that is a raw pod address of the other cluster means the network value of that control plane is wrong. No output at all beyond the header means TARGET_SERVICE is not the fully qualified name of the service, see The cross-cluster probe.

#Phase 5: decommission cluster by cluster

The steps on the global cluster are run once: they delete the ServiceMesh resource of every cluster of the mesh and the ServiceMeshGroup that ties them together. CLUSTER_NAMES and MESH_GROUP, set in Preparing for the migration, keep that deletion inside this mesh. The plugin is only uninstalled if no other Mesh v1 mesh is left on the platform.

The steps on each mesh cluster are the same as for a single-cluster mesh, with one addition. The east-west gateway service belongs to the Mesh v1 control plane resource — not through an owner reference but through operator labels — and is deleted with it. Detach it first, on every cluster:

kubectl --context "${CTX}" -n istio-system label svc istio-eastwestgateway \
  install.operator.istio.io/owning-resource- \
  install.operator.istio.io/owning-resource-namespace- \
  operator.istio.io/managed-
kubectl --context "${CTX}" -n istio-system get svc istio-eastwestgateway -o jsonpath='{.metadata.labels}{"\n"}'

The remaining labels must still contain app, istio and topology.istio.io/network: they are what the Mesh v2 gateway pods are selected by. Deleting the Mesh v1 control plane resource then removes the Mesh v1 gateway pods and leaves the service, its address and the Mesh v2 pods behind it in place.

#Phase 6: finalize every cluster

• Certificates. Repeat Hand over the certificate authority per cluster, signing each intermediate certificate with the same root and using the cluster name in its subject. Cross-cluster mutual TLS keeps working while the clusters hold certificates from different intermediates, because the root is unchanged.
• Revision tag and grace period. Per cluster, as for a single-cluster mesh.
• Tracing. Use one index family for the whole mesh: run the installation on every cluster with the same index prefix, so that a trace which crosses clusters is assembled from one place. The steps that act on Elasticsearch — the lifecycle policy, the index initialization, and attaching the existing Mesh v1 indices — are run once for the mesh, not once per cluster. Each cluster then keeps its own collector and its own Jaeger instance, and every instance serves the traces of the whole mesh.

#Keep trace reporting inside each cluster

Each cluster runs its own collector, and both are published under the same service name, otel-collector.jaeger-system.svc.cluster.local. Left alone, the control planes merge them into one set of endpoints and every proxy load-balances its span exports over both. The collector has no sidecar and therefore no network identity, so its endpoint is not rewritten to the east-west gateway address: a proxy receives the raw pod address of the remote collector, which belongs to the other network and is unreachable. Both endpoints are reported HEALTHY — nothing checks them — so every export that goes to the remote one is lost without a trace of its own. How much of a trace disappears depends on how the exports are spread: traces appear broken, and some are missing entirely.

Repoint the mesh at the new collector declares the collector service cluster-local, which drops the endpoints of the other cluster. Run it on every cluster that runs a control plane, then check that one endpoint is left:

istioctl --context "${CTX}" proxy-config endpoint "deploy/${WORKLOAD}.${MESH_NAMESPACE}" \
  --cluster "outbound|4317||otel-collector.jaeger-system.svc.cluster.local"

One endpoint, in the local cluster. Traces that cross clusters then contain the spans of both sides.

The setting travels through the control plane — the Operator re-renders the mesh configuration, istiod reloads it and pushes it to every proxy — so a check that runs in the same breath as the patch still lists both collectors. Give it a few seconds and run it again.

The entry affects only that host. Cross-cluster load balancing of application services, and the east-west gateway path they use, are unchanged.

#Verification

• istioctl --context "${CTX}" remote-clusters --revision "${V2_REVISION}" reports synced on every cluster.
• The cross-cluster endpoint of every cluster is the east-west gateway of the other one, healthy, and its rq_success grows in the proportion recorded in Phase 0.
• Traffic policies that depend on the topology — locality load balancing and outlier detection, for example — behave as they did before the migration. They are Istio resources and are not changed by the migration.
• Kiali shows the workloads of every cluster, and traces that cross clusters contain the spans of both sides. A span carries the cluster it was produced in as istio.cluster_id, which is what a trace of a cross-cluster call is read by.
• The east-west gateway service on each cluster keeps its address, and its endpoints are the Mesh v2 gateway pods.

#After the migration

The traffic graph of Kiali is the view of the whole migrated mesh, and the checks above only cover the service the probes call. Open Traffic Graph, select the namespaces of the mesh and enable Display > Cluster Boxes: the workloads are grouped per cluster, and every call that crosses clusters is an edge between the boxes. Confirm that the edges are drawn in both directions and that no service lost the cross-cluster half of its traffic, see Verification of Installing Kiali in a multi-cluster mesh.

A graph that draws no edge between the clusters while the endpoint checks pass is usually Kiali, not the mesh: it reads the metrics of both clusters from one store, filtered by mesh_id, and a cluster whose metrics are missing or labelled mesh_id="unknown" is invisible to it, see Phase 1: Kiali for a multi-cluster mesh.