Preparing for the migration

Phase 0 of the migration. Nothing changes in the mesh: you check that the cluster is ready, record how Mesh v1 is configured, back up every resource the later phases touch, and start the probes that tell you whether application traffic keeps flowing.

Complete every section before you install the Mesh v2 control plane.

Prerequisites

Alauda Container Platform 4.2 or 4.3, with Kubernetes 1.32 or later on every node that runs mesh workloads. Istio 1.30 requires Kubernetes 1.32.
Mesh v1 runs Istio 1.22. Record the Mesh v1 configuration below checks this.
The Alauda Service Mesh v2 Operator, the Alauda Service Mesh Console (previously Alauda Build of Kiali) Operator and the Alauda Container Platform Application Observability (previously Alauda Build of OpenTelemetry v2) Operator are available in the cluster catalog. Do not install them yet.
If applications use OpenTelemetry Java auto-instrumentation, an OpenTelemetry Java agent 2.x image in a registry the cluster can pull from. Mesh v1 shipped its own agent image and Mesh v2 does not, see Prepare the Java agent image.
The ACP Multi-Network (previously ACP Networking for Multus) plugin is installed. IstioCNI writes its configuration to the Multus configuration directory.
cluster-admin on the cluster that runs the mesh and on the global cluster, in one kubeconfig that has a context for each of them. A mesh that spans several clusters needs a context per cluster, see Address one cluster at a time.

The following tools on the machine you run the migration from:

Tool	Version	Purpose
`kubectl`	Matching the cluster	All steps
`istioctl`	1.30.x, see Istioctl tool	Checking Mesh v2 proxies
`jq`	1.6 or later	Reading and rewriting resources
`openssl`	1.1.1 or later	Certificate authority handover in Phase 6

A maintenance window agreed with the application owners. Application traffic keeps flowing, but workloads and gateways are restarted, and tracing is interrupted between Phase 5 and the end of Phase 6.

istioctl and Mesh v1

istioctl 1.30 talks to the Mesh v2 control plane. Do not point it at the Mesh v1 control plane, which runs Istio 1.22: the versions are too far apart. Checks against Mesh v1 in this section read the sidecar.istio.io/status annotation of the pods instead.

Set the variables

Every page in this section uses these variables. Set them once per shell. Replace <global-cluster-context> and <cluster-name> with your own values.

umask 077
export GLOBAL_CONTEXT=<global-cluster-context>
export CLUSTER_NAME=<cluster-name>
export BACKUP_DIR=${HOME}/mesh-v1-migration-backup-${CLUSTER_NAME}
mkdir -p "${BACKUP_DIR}"
chmod 700 "${BACKUP_DIR}"

export MESH_GROUP=$(kubectl --context "${GLOBAL_CONTEXT}" -n cpaas-system get servicemeshs.asm.alauda.io \
  "${CLUSTER_NAME}" -o jsonpath='{.metadata.labels.asm\.cpaas\.io/meshgroup}')
export CLUSTER_NAMES=$(kubectl --context "${GLOBAL_CONTEXT}" -n cpaas-system get servicemeshgroups.asm.alauda.io \
  "${MESH_GROUP}" -o jsonpath='{.spec.clusters[*]}')
echo "mesh ${MESH_GROUP} spans: ${CLUSTER_NAMES}"

CLUSTER_NAME is the name of the cluster in the platform. It is also the name of the ServiceMesh resource that represents this cluster on the global cluster. MESH_GROUP is the mesh this cluster belongs to, and CLUSTER_NAMES every cluster of that mesh — one name for a single-cluster mesh. Phase 5 uses them to delete only the resources of this mesh: a platform can host several.

The backup written below holds the private key of the root certificate authority of the mesh and the private key of every gateway certificate. umask 077 and the directory mode keep them readable only by you. Keep the umask for the rest of the migration shell.

In a multi-cluster mesh, set CLUSTER_NAME and BACKUP_DIR again for each cluster. Everything recorded below differs per cluster.

Record the Mesh v1 configuration

Mesh v2 must be created with the same mesh identity and the same mesh-wide settings as Mesh v1. Record them now: the resources that hold them are deleted in Phase 5.

Find the control plane

The Mesh v1 gateways are IstioOperator resources too. The control plane is the one without the gateway label.

export V1_CONTROL_PLANE_IOP=$(kubectl -n istio-system get istiooperators.install.istio.io \
  -l '!asm.cpaas.io/component' -o jsonpath='{.items[0].metadata.name}')
export V1_REVISION=$(kubectl -n istio-system get istiooperators.install.istio.io "${V1_CONTROL_PLANE_IOP}" \
  -o jsonpath='{.spec.revision}')
echo "${V1_CONTROL_PLANE_IOP} ${V1_REVISION}"

Example output

istio-122 1-22

The revision must be 1-22. If it is lower, update Mesh v1 to Istio 1.22 before you migrate.

Record the mesh identity and the injection policy

IOP_JSON=$(kubectl -n istio-system get istiooperators.install.istio.io "${V1_CONTROL_PLANE_IOP}" -o json)
export MESH_ID=$(jq -r '.spec.values.global.meshID // ""' <<<"${IOP_JSON}")
export NETWORK=$(jq -r '.spec.values.global.network // ""' <<<"${IOP_JSON}")
export MESH_CLUSTER_NAME=$(jq -r '.spec.values.global.multiCluster.clusterName // ""' <<<"${IOP_JSON}")
export V1_AUTO_INJECT=$(jq -r '.spec.values.global.proxy.autoInject // "enabled"' <<<"${IOP_JSON}")
export TRUST_DOMAIN=$(kubectl -n istio-system get configmap "istio-${V1_REVISION}" \
  -o jsonpath='{.data.mesh}' | awk '/^trustDomain:/{print $2}')
echo "meshID=${MESH_ID} network=${NETWORK} clusterName=${MESH_CLUSTER_NAME} trustDomain=${TRUST_DOMAIN} autoInject=${V1_AUTO_INJECT}"

Example output

meshID=mesh-1 network=mesh-1 clusterName=cluster-1 trustDomain=cluster.local autoInject=disabled

network is set even for a single-cluster mesh, and every injected pod carries a matching topology.istio.io/network label. Mesh v2 must use the same value, otherwise the two control planes disagree about which network the endpoints belong to and traffic between Mesh v1 and Mesh v2 pods fails.
autoInject is the injection policy of the mesh. With disabled, only pods that carry sidecar.istio.io/inject: "true" are injected, which is how Mesh v1 keeps uninjected workloads in a mesh namespace uninjected. Mesh v2 must use the same policy.

Record the mesh-wide settings

kubectl -n istio-system get istiooperators.install.istio.io "${V1_CONTROL_PLANE_IOP}" -o json \
  | jq '{meshConfig: (.spec.meshConfig // {}),
         proxy: ((.spec.values.global.proxy // {}) | {includeIPRanges, excludeIPRanges, resources}),
         traceSampling: .spec.values.pilot.traceSampling}' \
  | tee "${BACKUP_DIR}/v1-mesh-settings.json"
export V1_TRACE_SAMPLING=$(jq -r '.traceSampling // 1' "${BACKUP_DIR}/v1-mesh-settings.json")
echo "traceSampling=${V1_TRACE_SAMPLING}"

These settings change how every proxy in the mesh behaves — DNS capture for ServiceEntry hosts, the number of trusted proxies at the gateways, connection draining, the trace sampling rate. Phase 2 copies them into the Mesh v2 Istio resource.

traceSampling is the percentage of requests the mesh traces, 100 in a default Mesh v1 installation. Istio itself defaults to 1, so every phase that configures tracing uses ${V1_TRACE_SAMPLING} rather than a fixed value.

List the mesh namespaces and the injected workloads

for ns in $(kubectl get ns -l "istio.io/rev=${V1_REVISION}" -o jsonpath='{.items[*].metadata.name}'); do
  kubectl -n "${ns}" get pod -o json | jq -r --arg ns "${ns}" '
    ([.items[] | select(.metadata.annotations["sidecar.istio.io/status"] != null)] | length) as $injected
    | "\($ns) injected=\($injected) pods=\(.items | length)"'
done | tee "${BACKUP_DIR}/v1-mesh-namespaces.txt"

Example output

app-all      injected=6 pods=6
app-partial  injected=4 pods=6
java-apps    injected=0 pods=4

injected equals pods: every workload in the namespace is in the mesh.
injected is lower but not zero: a mixed namespace. Migrate these workloads one at a time, see Migrating workloads.
injected is zero: the namespace carries the revision label but has no mesh workloads, typically a namespace that only uses OpenTelemetry auto-instrumentation. Do not switch its label; Phase 5 removes it.

${BACKUP_DIR}/v1-mesh-namespaces.txt holds these lines. Phase 3 takes one namespace at a time from it, and the injected count decides how: a namespace is migrated as a whole, workload by workload, or not at all. Keep the file: it is the list as it was before anything changed, while in the cluster the revision label of a namespace changes the moment it is migrated, and the label selector above then no longer returns it.

Gateways run in their own namespace, which may not carry the revision label. Migrating gateways lists them.

Count every pod that Mesh v1 injected, whatever namespace it runs in. This is the baseline that Phase 2 compares against:

kubectl get pod -A -o json | jq -r --arg rev "${V1_REVISION}" '[.items[]
  | select((.metadata.annotations["sidecar.istio.io/status"] // "") | test("\"revision\":\"" + $rev + "\""))] | length' \
  | tee "${BACKUP_DIR}/v1-injected-pod-count.txt"

The number is higher than the sum of the namespace listing above whenever gateways run in a namespace without the revision label, which is the usual layout: two gateways add two pods.

Record the tracing backend

The Mesh v1 Jaeger resource is deleted in Phase 5. Record the index prefix and the retention of its Elasticsearch indices now; Phase 6 needs both to keep the existing traces queryable.

kubectl -n istio-system get jaegers.jaegertracing.io -o json | jq -r '.items[]
  | "name=\(.metadata.name) prefix=\(.spec.storage.options["es.index-prefix"] // "") days=\(.spec.storage.esIndexCleaner.numberOfDays // "") url=\(.spec.storage.options["es.server-urls"] // "") credentials=\(.spec.storage.options["es.asm.cnamespace"] // "istio-system")/\(.spec.storage.options["es.asm.cname"] // "")"' \
  | tee "${BACKUP_DIR}/v1-jaeger.txt"

Example output

name=jaeger-prod prefix=asm-mesh-mesh-1 days=7 url=https://elasticsearch.example.com/es_proxy credentials=istio-system/system-elasticsearch-basic-auth

The Jaeger resource does not hold the Elasticsearch user and password: it names a secret that does. Phase 6 needs those credentials to deploy the new backend against the same Elasticsearch cluster, and the secret lives in istio-system next to the Mesh v1 components that Phase 5 removes. Back it up:

sed -n 's|.*credentials=\([^/ ]*\)/\([^ ]*\).*|\1 \2|p' "${BACKUP_DIR}/v1-jaeger.txt" \
  | while read -r ns name; do
      [ -z "${name}" ] && continue
      kubectl -n "${ns}" get secret "${name}" -o yaml > "${BACKUP_DIR}/v1-jaeger-es-credentials.yaml"
      echo "${ns}/${name} keys: $(kubectl -n "${ns}" get secret "${name}" -o jsonpath='{.data}' | jq -r 'keys | join(",")')"
    done

The keys are ES_USERNAME and ES_PASSWORD, not username and password.

Record the instrumented workloads

Record how applications are instrumented today, so that the telemetry of your applications does not change when Mesh v1 is gone. Phase 6 restores it from this file, one workload at a time:

kubectl get instrumentations.opentelemetry.io -A -o yaml > "${BACKUP_DIR}/instrumentations.yaml"
kubectl get pod -A -o json | jq -r '.items[]
  | select(.metadata.annotations["instrumentation.opentelemetry.io/inject-java"] != null)
  | . as $p
  | ($p.metadata.labels["asm.cpaas.io/msname"] // "") as $ms
  | ([$p.metadata.ownerReferences[]? | select(.kind == "ReplicaSet") | .name] | first // $p.metadata.name) as $owner
  | ($owner | sub("-" + ($p.metadata.labels["pod-template-hash"] // "-") + "$"; "")) as $workload
  | (($p.metadata.annotations["instrumentation.opentelemetry.io/java-container-names"] // "")
      | split(",")[0]) as $annotated
  | ([$p.spec.containers[] | select(any((.env // [])[]; .name == "OTEL_SERVICE_NAME")) | .name] | first // "") as $withenv
  | (if $annotated != "" then $annotated else $withenv end) as $container
  | ([$p.spec.containers[] | select(.name == $container) | (.env // [])[]
      | select(.name == "OTEL_SERVICE_NAME") | .value] | first // "") as $raw
  | ($raw
      | gsub("\\$\\(ASM_MS_NAME\\)"; $ms)
      | gsub("\\$\\(ASM_POD_NAMESPACE\\)"; $p.metadata.namespace)) as $name
  | [$p.metadata.namespace, $workload, $container, $ms, $name]
  | map(if . == "" then "-" else . end) | join(" ")' \
  | sort -u | tee "${BACKUP_DIR}/v1-instrumented-pods.txt"

Example output

app-all frontend frontend frontend-ms frontend-ms.app-all

Five columns: the namespace, the workload, its application container, the Mesh v1 microservice name and the service name the application reports. Phase 6 sets the variables of a workload from its line, and by then none of these values can be read from the cluster any more.

The service name of a Mesh v1 application is built from variables that a Mesh v1 webhook adds to the pod, so the environment of the pod shows $(ASM_MS_NAME).$(ASM_POD_NAMESPACE) rather than a value. The command resolves it to the name the application actually reports, <microservice>.<namespace>, takes the microservice from the asm.cpaas.io/msname label and the application container from the instrumentation.opentelemetry.io/java-container-names annotation, both written by the same webhook. One line per workload, not per pod. A column that reads - could not be determined from the pod: fill it in by hand before Phase 6.

Record the variables the Mesh v1 webhooks add to pods

The same webhooks add the identity of the microservice to the application containers of every pod they handle. The variables are not in the workload template, so they disappear with the webhooks in Phase 5:

kubectl get pod -A -o json | jq -r '.items[]
  | select(.metadata.annotations["sidecar.istio.io/status"] != null
           or .metadata.annotations["instrumentation.opentelemetry.io/inject-java"] != null)
  | . as $p | .spec.containers[] | select(.name != "istio-proxy")
  | [(.env // [])[] | select(.name | startswith("ASM_")) | .name] as $asm
  | select($asm | length > 0)
  | "\($p.metadata.namespace) \($p.metadata.name) \(.name) \($asm | join(","))"' \
  | tee "${BACKUP_DIR}/v1-injected-env.txt"

Example output

app-all  frontend-7d9c6b8f2-abcde  frontend  ASM_MS_NAME,ASM_CLUSTER_NAME,ASM_CONTAINER_NAME,ASM_POD_NAMESPACE

OTEL_SERVICE_NAME is built from ASM_MS_NAME and ASM_POD_NAMESPACE, and Migrating the observability stack restores it as a literal value. Ask the application owners whether the applications themselves read any of these variables. An application that does keeps running after the migration but reads an empty value — a name such as frontend-ms.app-all becomes null.null — and nothing reports an error.

Save the variables

The resources these values come from are deleted during Phase 5, while later steps of the same phase still use them. Write them down now:

cat > "${BACKUP_DIR}/vars.env" <<EOF
export GLOBAL_CONTEXT='${GLOBAL_CONTEXT}' CLUSTER_NAME='${CLUSTER_NAME}' BACKUP_DIR='${BACKUP_DIR}'
export MESH_GROUP='${MESH_GROUP}' CLUSTER_NAMES='${CLUSTER_NAMES}'
export V1_CONTROL_PLANE_IOP='${V1_CONTROL_PLANE_IOP}' V1_REVISION='${V1_REVISION}'
export MESH_ID='${MESH_ID}' NETWORK='${NETWORK}' MESH_CLUSTER_NAME='${MESH_CLUSTER_NAME}'
export TRUST_DOMAIN='${TRUST_DOMAIN}' V1_AUTO_INJECT='${V1_AUTO_INJECT}' V1_TRACE_SAMPLING='${V1_TRACE_SAMPLING}'
EOF
cat "${BACKUP_DIR}/vars.env"

Load the file in every new shell

In a new shell, run source "${BACKUP_DIR}/vars.env" rather than deriving the values again. From the middle of Phase 5 on, the ServiceMesh resources and the Mesh v1 control plane are gone and the commands above return empty values. Nothing fails then: a kubectl delete with an empty name deletes an object that does not exist, a label selector with an empty value matches no namespace, and the cleanup steps silently skip the work they were supposed to do.

Check compatibility

Audit `ServiceEntry` resources

Istio 1.30 rejects a ServiceEntry without ports and one with more than 256 hosts.

kubectl get serviceentries.networking.istio.io -A -o json | jq -r '.items[]
  | select((.spec.ports == null) or ((.spec.ports | length) == 0) or ((.spec.hosts | length) > 256))
  | "\(.metadata.namespace)/\(.metadata.name)"'

Expected output: none. Fix every resource that is listed before you continue.

Audit `EnvoyFilter` resources

An EnvoyFilter that pins a proxy version stops applying once the workload runs on Mesh v2, because the data plane is then Istio 1.30.

kubectl get envoyfilters.networking.istio.io -A -o json | jq -r '.items[]
  | select([.. | objects | .proxyVersion? // empty] | length > 0)
  | "\(.metadata.namespace)/\(.metadata.name)"'

Resources generated by Mesh v1 features that Mesh v2 does not provide — API metrics and allow lists among them — appear here. They disappear with Mesh v1 in Phase 5. For every other resource, check whether the filter is still needed and rebuild it for Istio 1.30.

Find features that Mesh v2 does not provide

for kind in canarydeliveries globalratelimiters apiattributes isolatepods; do
  printf '%s: ' "${kind}"
  kubectl get "${kind}.asm.alauda.io" -A --no-headers 2>/dev/null | wc -l
done

Every resource counted here stops working when Mesh v1 is removed. Finish running canary releases, release isolated pods, and agree with the application owners on what replaces global rate limiting and API metrics before you start Phase 2.

Check the lifetime of the intermediate certificate

Mesh v1 renews the intermediate certificate of the mesh automatically, and stops doing so when its controllers are removed in Phase 5. Phase 6 replaces the certificate with a long-lived one.

kubectl -n istio-system get secret cacerts -o jsonpath='{.data.ca-cert\.pem}' \
  | base64 -d | openssl x509 -noout -subject -enddate

The expiry date must be later than the end of the planned migration, including Phase 6.

Collect the custom Istio configuration

Mesh v1 lets you change the mesh through the web console and through platform resources, and some of those settings end up in places that Mesh v2 does not read: the IstioOperator resources, the Mesh v1 ServiceMesh resource, and the injection templates of the Mesh v1 control plane.

kubectl -n istio-system get istiooperators.install.istio.io "${V1_CONTROL_PLANE_IOP}" -o yaml \
  > "${BACKUP_DIR}/v1-control-plane-iop.yaml"
kubectl --context "${GLOBAL_CONTEXT}" -n cpaas-system get servicemeshs.asm.alauda.io "${CLUSTER_NAME}" -o yaml \
  > "${BACKUP_DIR}/v1-servicemesh.yaml"

Go through both files and decide, for every setting that differs from the defaults, how it is expressed in Mesh v2:

Setting in Mesh v1	Where it goes in Mesh v2
`spec.meshConfig` of the control plane resource	`spec.values.meshConfig` of the `Istio` resource, see Installing the Mesh v2 control plane
`spec.values.global.proxy` — resources, traffic capture, injection policy	`spec.values.global.proxy` of the `Istio` resource
Per-microservice sidecar settings	Annotations in the workload template, see Migrating workloads
`spec.components` — replica counts, resources, scheduling of the control plane and the gateways	`spec.values.pilot` for the control plane, and your own gateway deployments
Injection templates defined by Mesh v1	Not available. Reproduce what your workloads need with pod annotations
Istio resources — `Gateway`, `VirtualService`, `DestinationRule`, `Telemetry` and the rest	Unchanged, Mesh v2 reads them as they are

Anything you do not carry over silently reverts to the Istio default when the workload restarts on Mesh v2.

Review the Istio behavior changes

The data plane moves from Istio 1.22 to Istio 1.30. Review the upgrade notes of Alauda Service Mesh v2.2 and the upstream release announcements between 1.23 and 1.30 for changes that affect your traffic policies.

Check cluster capacity

Between Phase 2 and Phase 5 both control planes run at the same time, and every workload that is being migrated runs a second pod during its rolling update. A pod is scheduled only on a node whose unrequested capacity covers the resource requests of the pod — how busy the node is does not matter. Check what is requested, not what is used:

kubectl describe node | grep -E '^Name:|^  (cpu|memory) +[0-9]'

For each node, the first value is what the pods on it request, followed by its share of what the node can allocate. The second value is the limits.

The migration changes the requests as follows. The values are the defaults measured on Alauda Container Platform 4.3, per replica; the Mesh v1 values vary with how Mesh v1 was installed:

Phase	Change	Namespace	CPU	Memory
1	Kiali Operator added	`kiali-operator`	+10m	+64Mi
1	Kiali added	`istio-system`	+100m	+64Mi
2	Alauda Service Mesh v2 Operator added	`sail-operator`	+10m	+64Mi
2	Mesh v2 `istiod` added	`istio-system`	+500m	+2Gi
2	`istio-cni-node` added, on every node	`istio-cni`	+100m per node	+100Mi per node
3	Second pod of a workload during its rolling update, sidecar included	Namespace of the workload	Requests of the pod	Requests of the pod
4	Mesh v2 gateway, until the Mesh v1 gateway is scaled to zero	Namespace of the gateway	+100m	+128Mi
5	Mesh v1 control plane, operators and controller removed	`istio-system`, `cpaas-system`	About −1050m	About −2.6Gi
5	Mesh v1 tracing removed: OpenTelemetry v1 Operator and collectors, Jaeger 1.x Operator, collector and query	`istio-system`, `cpaas-system`	About −700m	About −2.3Gi
6	Alauda Build of OpenTelemetry v2 Operator added	`opentelemetry-operator2`	+100m	+64Mi
6	OpenTelemetry collector and Jaeger v2 added	`jaeger-system`	+250m	+576Mi

The requests peak between Phase 2 and Phase 5: about 620m of CPU and 2.2Gi of memory on top of Mesh v1, plus 100m and 100Mi on every node. The Mesh v2 istiod alone asks for 2Gi — four times the Mesh v1 istiod — and stays Pending when no node has that much memory left unrequested. If your nodes are that full, set spec.values.pilot.resources of the Istio resource in Phase 2.

Actual usage grows far less. A three-node cluster running a small mesh — three namespaces, seven injected workloads, two gateways — used about 400m of CPU and 800Mi of memory more while both control planes were running than with Mesh v1 alone.

Back up the mesh

Phase 5 deletes the Mesh v1 resources. Some of the resources you depend on are owned by them and are garbage collected with them — typically the gateway services and the per-microservice and per-gateway telemetry resources. Record which resources Mesh v1 owns below produces the list that holds for your cluster; do not assume a resource is owned, or that it is not, before you have read that list. The gateway certificates are usually not owned, but they are stored in istio-system next to the Mesh v1 components, so they are backed up here as well.

The backup is what you restore from if a phase has to be rolled back.

Back up the Istio and Mesh v1 resources

kubectl get ns -o yaml > "${BACKUP_DIR}/namespaces.yaml"
for crd in $(kubectl get crd -o name | sed 's|.*/||' | grep -E '\.istio\.io$'); do
  kubectl get "${crd}" -A -o yaml > "${BACKUP_DIR}/istio-${crd}.yaml"
done
for crd in $(kubectl get crd -o name | sed 's|.*/||' | grep -E '\.asm\.alauda\.io$'); do
  kubectl get "${crd}" -A -o yaml > "${BACKUP_DIR}/asm-${crd}.yaml"
done
kubectl get asms.operator.alauda.io,flaggers.operator.alauda.io -A -o yaml > "${BACKUP_DIR}/asm-operator-crs.yaml"
kubectl get opentelemetrycollectors.opentelemetry.io,instrumentations.opentelemetry.io -A -o yaml > "${BACKUP_DIR}/otel.yaml"
kubectl get jaegers.jaegertracing.io -A -o yaml > "${BACKUP_DIR}/jaeger.yaml"
kubectl get subscriptions.operators.coreos.com,clusterserviceversions.operators.coreos.com -A -o yaml > "${BACKUP_DIR}/olm.yaml"
kubectl -n istio-system get secret cacerts -o yaml > "${BACKUP_DIR}/cacerts.yaml"
kubectl -n istio-system get secret -l istio/multiCluster=true -o yaml > "${BACKUP_DIR}/remote-secrets.yaml"

The loop over the custom resource definitions covers every Istio API group, including telemetry.istio.io. Listing resource types by hand misses the telemetry resources that Phase 5 garbage collects.

Back up the gateway services and certificates

kubectl get gatewaydeploys.asm.alauda.io -A -o json | jq -r '.items[] | "\(.metadata.namespace) \(.metadata.name)"' \
  | while read -r ns name; do
      kubectl -n "${ns}" get svc "${name}" -o yaml > "${BACKUP_DIR}/gateway-svc-${ns}-${name}.yaml"
    done
kubectl get gateways.networking.istio.io -A -o json \
  | jq -r '.items[].spec.servers[]? | .tls.credentialName // empty' | sort -u \
  | while read -r secret; do
      kubectl get secret -A --field-selector "metadata.name=${secret}" -o yaml > "${BACKUP_DIR}/gateway-tls-${secret}.yaml"
    done

Gateway certificates cannot be recreated

The private key of a gateway certificate exists only in its secret. If the secret is lost, the certificate has to be reissued. Keep this backup until the migration is finished and verified.

Record which resources Mesh v1 owns

for kind in $(kubectl get crd -o name | sed 's|.*/||' | grep -E '\.istio\.io$') secrets services; do
  kubectl get "${kind}" -A -o json 2>/dev/null | jq -r --arg k "${kind}" '.items[]
    | select(any(.metadata.ownerReferences[]?; .apiVersion | test("asm\\.alauda\\.io")))
    | "\($k) \(.metadata.namespace) \(.metadata.name) \(.metadata.ownerReferences[0].kind)/\(.metadata.ownerReferences[0].name)"'
done | tee "${BACKUP_DIR}/asm-owned-resources.txt"
while read -r kind ns name owner; do
  kubectl -n "${ns}" get "${kind}" "${name}" -o yaml > "${BACKUP_DIR}/owned-${kind}-${ns}-${name}.yaml"
done < "${BACKUP_DIR}/asm-owned-resources.txt"

Example output

envoyfilters.networking.istio.io  app-all  frontend-1.22  ApiAttribute/frontend
telemetries.telemetry.istio.io    app-all  frontend       MicroService/frontend
telemetries.telemetry.istio.io    gw       ingress-1      GatewayDeploy/ingress-1
services                          gw       ingress-1      GatewayDeploy/ingress-1

Every resource in this list is deleted by Kubernetes when its owner is deleted in Phase 5. Decommissioning Mesh v1 removes the owner reference from the resources you keep — and only from those: most of what is listed here is Mesh v1 configuration that Phase 3 and Phase 4 replace with equivalents of their own.

Resources that Mesh v1 generated but that are owned by something else — a DestinationRule owned by the ServiceEntry of an external service, for example — are not in this list and are not deleted in Phase 5. List them too, so that you know what stays behind without an owner to maintain it:

for crd in $(kubectl get crd -o name | sed 's|.*/||' | grep -E '\.istio\.io$'); do
  kubectl get "${crd}" -A -l 'asm.cpaas.io/creator=asm-controller' -o json 2>/dev/null | jq -r --arg k "${crd}" '.items[]
    | "\($k) \(.metadata.namespace) \(.metadata.name) \(.metadata.ownerReferences[0].kind // "-")/\(.metadata.ownerReferences[0].name // "-")"'
done | tee "${BACKUP_DIR}/asm-generated-resources.txt"

Back up the `global` cluster resources

kubectl --context "${GLOBAL_CONTEXT}" -n cpaas-system get servicemeshs.asm.alauda.io -o yaml > "${BACKUP_DIR}/global-servicemeshes.yaml"
kubectl --context "${GLOBAL_CONTEXT}" -n cpaas-system get servicemeshgroups.asm.alauda.io -o yaml > "${BACKUP_DIR}/global-servicemeshgroups.yaml"
kubectl --context "${GLOBAL_CONTEXT}" -n cpaas-system get secret asm-istio-rootca -o yaml > "${BACKUP_DIR}/global-asm-istio-rootca.yaml"
kubectl --context "${GLOBAL_CONTEXT}" get moduleplugins.cluster.alauda.io asm-global -o yaml \
  > "${BACKUP_DIR}/global-asm-moduleplugin.yaml"
kubectl --context "${GLOBAL_CONTEXT}" get moduleinfoes.cluster.alauda.io -l cpaas.io/module-name=asm-global -o yaml \
  > "${BACKUP_DIR}/global-asm-moduleinfo.yaml"
kubectl --context "${GLOBAL_CONTEXT}" get \
  mutatingwebhookconfiguration/servicemesh-v1alpha1-mutation \
  mutatingwebhookconfiguration/servicemeshgroup-v1alpha1-mutation \
  validatingwebhookconfiguration/global-asm-controller-validating-webhook-configuration \
  -o yaml > "${BACKUP_DIR}/global-webhooks.yaml"

The webhook configurations are needed again only if the global cluster keeps serving another Mesh v1 mesh, see Decommissioning Mesh v1.

asm-global is the Alauda Service Mesh Essentials cluster plugin. ModulePlugin is its entry in the plugin catalog of the platform and ModuleInfo the record that it is installed on the global cluster. Rolling back the migration reinstalls the plugin from these two files.

The root certificate authority is in this backup

asm-istio-rootca contains the private key of the root certificate authority of the mesh. Phase 6 signs a new intermediate certificate with it. Store the backup where you keep private keys, not on the cluster.

Restore a resource from the backup

Every file above is a resource as the cluster held it, including the owner references that Mesh v1 had set. Applying such a file after the owner is gone recreates the resource and the garbage collector deletes it again within seconds, because it references an owner that no longer exists. Restore through this function instead, which strips the owner references and the fields that belonged to the deleted object:

restore_from_backup() {
  kubectl create -f "$1" --dry-run=client -o json \
    | jq 'def clean: del(.metadata.ownerReferences, .metadata.uid, .metadata.resourceVersion,
                         .metadata.creationTimestamp, .metadata.generation, .metadata.managedFields,
                         .metadata.annotations["kubectl.kubernetes.io/last-applied-configuration"], .status);
          if .kind == "List" then .items |= map(clean) else clean end' \
    | kubectl apply -f -
}

It takes a file that holds one resource and a file that holds a list. If the API server rejects a restored Service because its cluster IP has been reassigned in the meantime, remove spec.clusterIP and spec.clusterIPs as well and apply again.

Freeze changes

For the duration of the migration:

Do not change the mesh from the web console: no new microservices, no new gateways, no policy changes.
Do not approve updates for the Mesh v1 operators.
Tell the application owners that workloads are restarted phase by phase.

Set up traffic probes

Every phase is verified against continuous traffic. Run probes that cover each path the mesh serves, and keep their output with timestamps:

a call between two workloads inside the mesh,
one request per ingress gateway route, including every TLS host,
one request per egress route,
for a multi-cluster mesh, one call per cluster pair.

A probe loop can be as simple as this:

while true; do
  ts=$(date -u +%FT%TZ)
  for target in "${PROBE_TARGETS[@]}"; do
    code=$(curl -sk -o /dev/null -w '%{http_code}' --max-time 5 "${target}")
    case "${code}" in
      2*|3*) echo "${ts} ${target} ${code} OK" ;;
      *)     echo "${ts} ${target} ${code:-000} FAIL" ;;
    esac
  done
  sleep 1
done

Set PROBE_TARGETS to your own URLs. Replace <ingress-gateway-host>, <ingress-gateway-tls-host> and <path>:

export PROBE_TARGETS=(
  "http://<ingress-gateway-host>/<path>"
  "https://<ingress-gateway-tls-host>/<path>"
)

The ingress routes are reachable from your machine. The other two paths — a call between two workloads and an egress route — are not: they only exist inside the mesh and have to be probed from a pod. Use a workload you migrate last, so that its own restart does not show up as an outage. Replace <namespace>, <workload>, <application-container>, <service>, <port> and <path> with your own values:

export PROBE_NAMESPACE=<namespace> PROBE_WORKLOAD=<workload> PROBE_CONTAINER=<application-container>
export IN_MESH_TARGET=http://<service>.<namespace>:<port>/<path>
while true; do
  ts=$(date -u +%FT%TZ)
  out=$(kubectl -n "${PROBE_NAMESPACE}" exec "deploy/${PROBE_WORKLOAD}" -c "${PROBE_CONTAINER}" -- \
    curl -s -o /dev/null -w '%{http_code}' --max-time 5 "${IN_MESH_TARGET}" 2>&1)
  code=${out%%[!0-9]*}
  case "${code}" in
    2??|3??)         echo "${ts} ${IN_MESH_TARGET} ${code} OK" ;;
    [0-9][0-9][0-9]) echo "${ts} ${IN_MESH_TARGET} ${code} FAIL" ;;
    *)               echo "${ts} ${IN_MESH_TARGET} PROBE-ERROR ${out}" ;;
  esac
  sleep 1
done

Where to run the probes from

Run the probes from outside the mesh, or from the namespace you migrate last. A probe that runs inside a workload being restarted reports its own restart as an outage.

HTTPS routes must be probed as HTTPS. A gateway that cannot load its certificate keeps answering its HTTP routes and only resets the TLS handshakes on its HTTPS port, so an HTTP-only probe does not detect the failure.

An egress probe must be checked against the gateway, not only against its status code. With the default outboundTrafficPolicy: ALLOW_ANY, a request to a host that resolves outside the cluster leaves through the passthrough cluster when the egress route is broken, and still answers 200. Confirm once, before the migration and again after each phase that touches the gateways, that the request really traverses the egress gateway — the access log of the gateway pod names the route, and istioctl proxy-config shows which cluster the proxy picked. Forcing the hostname to the address of its ServiceEntry with curl --resolve <host>:<port>:<address> keeps the probe on the intended path.

Verification

echo "${MESH_ID} ${NETWORK} ${MESH_CLUSTER_NAME} ${TRUST_DOMAIN} ${V1_AUTO_INJECT}" prints the mesh identity and the injection policy.
cat "${BACKUP_DIR}/vars.env" shows every variable the later phases need, with no empty value.
cat "${BACKUP_DIR}/v1-mesh-namespaces.txt" lists every namespace of the mesh with its injected pod count.
ls "${BACKUP_DIR}" lists the backup of the Istio resources, the Mesh v1 resources, the gateway services and certificates, the cacerts secret and the global cluster resources.
cat "${BACKUP_DIR}/v1-jaeger.txt" shows the index prefix, the retention and the credentials secret of the tracing backend, and ${BACKUP_DIR}/v1-jaeger-es-credentials.yaml holds that secret.
The probes report OK for every target, with no PROBE-ERROR line.
The compatibility audits return no unexpected resources.

Continue with Migrating a single-cluster mesh, or with Migrating a multi-cluster mesh if the mesh spans several clusters.

#Preparing for the migration

#TOC

#Prerequisites

#Set the variables

#Record the Mesh v1 configuration

#Find the control plane

#Record the mesh identity and the injection policy

#Record the mesh-wide settings

#List the mesh namespaces and the injected workloads

#Record the tracing backend

#Record the instrumented workloads

#Record the variables the Mesh v1 webhooks add to pods

#Save the variables

#Check compatibility

#Audit ServiceEntry resources

#Audit EnvoyFilter resources

#Find features that Mesh v2 does not provide

#Check the lifetime of the intermediate certificate

#Collect the custom Istio configuration

#Review the Istio behavior changes

#Check cluster capacity

#Back up the mesh

#Back up the Istio and Mesh v1 resources

#Back up the gateway services and certificates

#Record which resources Mesh v1 owns

#Back up the global cluster resources

#Restore a resource from the backup

#Freeze changes

#Set up traffic probes

#Verification

Preparing for the migration

TOC

Prerequisites

Set the variables

Record the Mesh v1 configuration

Find the control plane

Record the mesh identity and the injection policy

Record the mesh-wide settings

List the mesh namespaces and the injected workloads

Record the tracing backend

Record the instrumented workloads

Record the variables the Mesh v1 webhooks add to pods

Save the variables

Check compatibility

Audit `ServiceEntry` resources

Audit `EnvoyFilter` resources

Find features that Mesh v2 does not provide

Check the lifetime of the intermediate certificate

Collect the custom Istio configuration

Review the Istio behavior changes

Check cluster capacity

Back up the mesh

Back up the Istio and Mesh v1 resources

Back up the gateway services and certificates

Record which resources Mesh v1 owns

Back up the `global` cluster resources

Restore a resource from the backup

Freeze changes

Set up traffic probes

Verification