Installing the Mesh v2 control plane

Phase 1 installs the observability components that let you watch the mesh during the migration. Phase 2 creates the Mesh v2 control plane next to Mesh v1. No workload changes in either phase: the mesh keeps running on Mesh v1 until Migrating workloads.

Both phases are reversible — deleting the Istio and IstioCNI resources removes everything Phase 2 created.

Before you start, complete Preparing for the migration and keep the shell variables it sets.

Phase 1: Prepare observability

Kiali reads the metrics of the mesh, so it shows the Mesh v1 traffic before the migration starts and the Mesh v2 traffic afterwards. Installing it first gives you a view that spans the whole migration.

Install Kiali

Follow Alauda Build of Kiali to install the Kiali Operator and create the Kiali resource, and Configuring Monitoring with Kiali to connect it to the monitoring store of the platform.

Install it without a tracing configuration. The Mesh v1 Jaeger instance is removed in Phase 5 and Jaeger v2 is installed in Phase 6, which is where Migrating the observability stack connects Kiali to it. Until then Kiali shows the metrics of the mesh, and traces stay in the Mesh v1 Jaeger UI.

If you want to read traces in Kiali during the migration, point it at the Mesh v1 Jaeger instance as described in Integrating distributed tracing platform with Alauda Build of Kiali, and expect that configuration to stop working in Phase 5.

Create the metrics resources

Until Phase 5, Mesh v1 keeps collecting what it collects today. Its sidecar-monitor pod monitor scrapes every proxy that runs as a regular container — the Mesh v1 proxies, and from Phase 4 on the Mesh v2 gateways — and its istio-monitor service monitor scrapes every istiod in istio-system, the Mesh v2 one included. It skips only native sidecars, the init containers with restartPolicy: Always that Mesh v2 injects into workloads.

Create the PodMonitor resource of Metrics and Service Mesh once, in istio-system: its namespaceSelector covers every namespace, and a copy in another namespace would scrape every proxy again. Then restrict it to native sidecars:

kubectl -n istio-system get podmonitor istio-proxies-monitor -o json \
  | jq '.spec.podMetricsEndpoints[0].relabelings |=
          [{action: "keep", sourceLabels: ["__meta_kubernetes_pod_container_init"], regex: "true"}]
          + map(select(.sourceLabels != ["__meta_kubernetes_pod_container_init"]))' \
  | kubectl replace -f -

A proxy that both monitors scrape reports every request twice, and Kiali shows twice the traffic it carries. For the same reason, do not create the ServiceMonitor resource yet. Decommissioning Mesh v1 lifts the restriction and creates it once the Mesh v1 monitors are gone.

Do not create the asm-default telemetry resource yet

Mesh v1 already owns a mesh-wide Telemetry resource in istio-system, and both control planes read it. Only one mesh-wide resource is ever in effect — the oldest one in istio-system that has no selector — so a second one created now would be ignored without any error, and the Mesh v1 one would keep configuring both generations. Decommissioning Mesh v1 creates asm-default in its place in Phase 5.

Nothing is lost by waiting. The Mesh v1 resource carries the mesh_id tag override that Kiali filters the metrics of a mesh by, and applies it to the proxies of both generations, so a workload stays visible in Kiali from before the migration until asm-default takes over.

Verification

Open Kiali and confirm that the traffic graph shows the current Mesh v1 traffic.

Phase 2: Install the Mesh v2 control plane

Install the Alauda Service Mesh v2 Operator

Follow Installing Alauda Service Mesh to install the Operator. Stop before creating the IstioCNI and Istio resources: this document creates them with the values of your Mesh v1 mesh.

Relax the anti-affinity rule of the Mesh v1 control plane

The Mesh v1 istiod pod has a required anti-affinity rule against other istiod pods. On a cluster where the Mesh v2 istiod would have to be scheduled on a node that already runs the Mesh v1 istiod — a single-node cluster, or a cluster where only one node is available for it — the Mesh v2 pod stays Pending forever.

Check how many nodes can take the control plane:

kubectl get node -o json | jq -r '[.items[] | select(.spec.unschedulable != true)] | length'
kubectl -n istio-system get pod -l app=istiod -o wide --no-headers

If more than one node can take the control plane, skip this step. Relax the rule when the cluster has a single node available for it, or when the Mesh v2 istiod pod created in the next steps stays Pending with an event about pod anti-affinity:

kubectl -n istio-system patch istiooperators.install.istio.io "${V1_CONTROL_PLANE_IOP}" --type=json -p='[
  {"op":"replace","path":"/spec/components/pilot/k8s/affinity","value":{"podAntiAffinity":{"preferredDuringSchedulingIgnoredDuringExecution":[{"weight":100,"podAffinityTerm":{"labelSelector":{"matchExpressions":[{"key":"istio","operator":"In","values":["istiod"]}]},"topologyKey":"kubernetes.io/hostname"}}]}}},
  {"op":"add","path":"/spec/components/pilot/k8s/strategy","value":{"rollingUpdate":{"maxSurge":0,"maxUnavailable":1}}}
]'

The patch must replace the affinity, not merge into it: a merge keeps the original required rule.
maxSurge: 0 makes the Mesh v1 istiod roll one pod at a time, which is what a cluster without spare capacity can do. The proxies keep their configuration while istiod restarts, so the data plane is not affected.

Create the `IstioCNI` resource

Mesh v2 captures traffic with a CNI plugin instead of the istio-init container that Mesh v1 adds to each pod. The plugin skips pods that already carry istio-init, so Mesh v1 pods are not affected.

kubectl create namespace istio-cni --dry-run=client -o yaml | kubectl apply -f -
cat <<EOF | kubectl apply -f -
apiVersion: sailoperator.io/v1
kind: IstioCNI
metadata:
  name: default
spec:
  version: v1.30.4
  namespace: istio-cni
  values:
    cni:
      cniConfDir: /etc/cni/multus/net.d
      excludeNamespaces:
        - istio-cni
        - kube-system
EOF
kubectl wait --for condition=Ready istiocni/default --timeout=3m

Build the mesh configuration of Mesh v2

Mesh v2 must run with the mesh-wide settings of Mesh v1. The following command reads them from the Mesh v1 control plane and drops the two fields that belong to the Mesh v1 deployment itself.

export V2_MESH_CONFIG=$(kubectl -n istio-system get istiooperators.install.istio.io "${V1_CONTROL_PLANE_IOP}" -o json \
  | jq -c '(.spec.meshConfig // {})
      | del(.defaultConfig.discoveryAddress, .rootNamespace)')
jq . <<<"${V2_MESH_CONFIG}"

Example output

{
  "defaultConfig": {
    "gatewayTopology": { "numTrustedProxies": 1 },
    "holdApplicationUntilProxyStarts": true,
    "proxyMetadata": {
      "EXIT_ON_ZERO_ACTIVE_CONNECTIONS": "true",
      "ISTIO_META_DNS_AUTO_ALLOCATE": "true",
      "ISTIO_META_DNS_CAPTURE": "true"
    }
  },
  "defaultHttpRetryPolicy": { "attempts": 2 },
  "extensionProviders": [
    { "name": "zipkin", "zipkin": { "port": 9411, "service": "zipkin.istio-system.svc.cluster.local" } }
  ]
}

proxyMetadata carries settings that change how every proxy behaves. ISTIO_META_DNS_CAPTURE, for example, is what makes a ServiceEntry host resolvable inside the mesh. A Mesh v2 proxy without it cannot reach hosts that only exist as ServiceEntry resources.
The tracing provider is kept as it is, so that proxies of both generations report to the Mesh v1 tracing backend during the migration. Phase 6 repoints it.

The trust domain of the mesh is part of the identity of every workload certificate. Mesh v1 keeps it in the mesh configuration, where the command above copies it, but a mesh that was configured elsewhere would fall back to cluster.local and break every authorization policy whose principals name the trust domain the mesh used. Make it explicit:

if [ "$(jq -r '.trustDomain // "cluster.local"' <<<"${V2_MESH_CONFIG}")" != "${TRUST_DOMAIN}" ]; then
  export V2_MESH_CONFIG=$(jq -c --arg td "${TRUST_DOMAIN}" '.trustDomain = $td' <<<"${V2_MESH_CONFIG}")
fi
jq -r '.trustDomain // "cluster.local"' <<<"${V2_MESH_CONFIG}"

The output is the ${TRUST_DOMAIN} recorded in Phase 0.

Create the `Istio` resource

cat <<EOF | kubectl apply -f -
apiVersion: sailoperator.io/v1
kind: Istio
metadata:
  name: default
spec:
  version: v1.30.4
  namespace: istio-system
  updateStrategy:
    type: RevisionBased
    inactiveRevisionDeletionGracePeriodSeconds: 2592000
  values:
    global:
      meshID: ${MESH_ID}
      network: ${NETWORK}
      multiCluster:
        clusterName: ${MESH_CLUSTER_NAME}
      proxy:
        autoInject: ${V1_AUTO_INJECT}
        seccompProfile:
          type: RuntimeDefault
    pilot:
      traceSampling: ${V1_TRACE_SAMPLING}
    meshConfig: ${V2_MESH_CONFIG}
EOF
kubectl wait --for condition=Ready istio/default --timeout=5m

meshID, network and multiCluster.clusterName must match Mesh v1 exactly. network decides whether a proxy treats an endpoint as local or as reachable through a gateway only. If the two control planes disagree, calls between Mesh v1 and Mesh v2 workloads are routed to a gateway that does not exist.
autoInject keeps the injection policy of Mesh v1. With disabled, a pod is injected only when it carries sidecar.istio.io/inject: "true", so workloads that are not in the mesh today stay out of it.
RevisionBased keeps Mesh v1 and Mesh v2 apart: the Mesh v2 webhooks carry their own revision suffix and never claim the default slot that Mesh v1 owns.
inactiveRevisionDeletionGracePeriodSeconds keeps a revision that no longer has workloads for 30 days instead of 30 seconds, so a revision is not removed while the migration is still running. Finalizing the migration restores the default.
seccompProfile is required when any mesh namespace enforces the Restricted Pod Security Admission profile. See Pod Security Admission.
traceSampling is the sampling percentage recorded from Mesh v1. Without it the Mesh v2 control plane uses the Istio default of 1 percent, and a workload traced at 100 percent under Mesh v1 is traced at 1 percent from the moment it is migrated until Phase 5 replaces the mesh-wide Telemetry resource.

Copy any other Mesh v1 proxy setting that differs from the Istio defaults — includeIPRanges, excludeIPRanges and the default sidecar resources are in ${BACKUP_DIR}/v1-mesh-settings.json — into spec.values.global.proxy. The resource is created without them, so apply them now:

kubectl patch istio default --type=merge -p "$(jq -c '
  {spec: {values: {global: {proxy:
    ((if .proxy.resources then {resources: .proxy.resources} else {} end)
     + (if .proxy.includeIPRanges then {includeIPRanges: .proxy.includeIPRanges} else {} end)
     + (if .proxy.excludeIPRanges then {excludeIPRanges: .proxy.excludeIPRanges} else {} end))}}}}' \
  "${BACKUP_DIR}/v1-mesh-settings.json")"
kubectl get istio default -o jsonpath='{.spec.values.global.proxy}{"\n"}'
kubectl wait --for condition=Ready istio/default --timeout=5m

The patch only writes the three fields that Phase 0 recorded, and leaves autoInject and seccompProfile as the resource above set them. Mesh v1 typically caps the sidecar at 500m and 512Mi where Istio defaults to 2000m and 1024Mi, so skipping this step multiplies what every migrated pod requests.

Settings that are not proxy settings — replica counts, resources and scheduling of the control plane in spec.components of the Mesh v1 resource — belong under spec.values.pilot. Review ${BACKUP_DIR}/v1-control-plane-iop.yaml and carry over the ones you need.

Do not create an IstioRevisionTag named default

Mesh v1 owns the default injection webhook. An IstioRevisionTag named default collides with it and changes which control plane injects pods that have no revision label. Finalizing the migration creates it after Mesh v1 is gone.

Prepare the Istio reader service account

Mesh v1 created the istio-reader-service-account service account in istio-system without Helm metadata. The IstioRevisionTag that Phase 6 creates claims that account through Helm and fails while the metadata is missing. Adding it now costs nothing and has no effect on Mesh v1.

kubectl -n istio-system label sa istio-reader-service-account app.kubernetes.io/managed-by=Helm --overwrite
kubectl -n istio-system annotate sa istio-reader-service-account \
  meta.helm.sh/release-name=default-base meta.helm.sh/release-namespace=sail-operator --overwrite

Get the Mesh v2 revision

With the RevisionBased strategy the control plane runs under a revision name derived from the Istio version. Every later phase refers to it.

export V2_REVISION=$(kubectl get istio default -o jsonpath='{.status.activeRevisionName}')
echo "export V2_REVISION='${V2_REVISION}'" >> "${BACKUP_DIR}/vars.env"
echo "${V2_REVISION}"

Example output

default-v1-30-4

See Identifying the revision name for the naming rules.

Verification

kubectl get istio,istiocni,istiorevision
kubectl -n istio-system get deploy | grep istiod
kubectl -n istio-system logs "deploy/istiod-${V2_REVISION}" | grep -i "root cert" | head -3

The Istio and IstioCNI resources report Healthy, and the IstioRevision for ${V2_REVISION} is Healthy.
Both istiod deployments are running: the Mesh v1 one and istiod-${V2_REVISION}.
The Mesh v2 istiod log contains Use root cert from etc/cacerts, which means it loaded the same certificate authority as Mesh v1. Workload certificates issued by either control plane are then trusted by both.
The Mesh v1 proxies are unchanged, and the probes report no failures.

Count the pods that are still served by Mesh v1:

kubectl get pod -A -o json | jq -r --arg rev "${V1_REVISION}" '[.items[]
  | select((.metadata.annotations["sidecar.istio.io/status"] // "") | test("\"revision\":\"" + $rev + "\""))] | length'

The count matches ${BACKUP_DIR}/v1-injected-pod-count.txt, the baseline recorded in Phase 0: creating the Mesh v2 control plane does not restart or re-inject anything. It is higher than the sum of the per-namespace listing of that phase, which only covers namespaces carrying the revision label — the gateway pods are counted here and not there.

Never delete the Mesh v1 service mesh from the platform UI

After you create the Mesh v2 Istio resource, never delete the Mesh v1 service mesh from the platform UI, and never delete a ServiceMesh resource and leave the cleanup to the Mesh v1 controllers. The Mesh v1 deletion flow deletes the whole istio-system namespace, including the Mesh v2 control plane that runs in it. Remove Mesh v1 only by following Decommissioning Mesh v1.

Continue with Migrating workloads.

#Installing the Mesh v2 control plane

#TOC

#Phase 1: Prepare observability

#Install Kiali

#Create the metrics resources

#Verification

#Phase 2: Install the Mesh v2 control plane

#Install the Alauda Service Mesh v2 Operator

#Relax the anti-affinity rule of the Mesh v1 control plane

#Create the IstioCNI resource

#Build the mesh configuration of Mesh v2

#Create the Istio resource

#Prepare the Istio reader service account

#Get the Mesh v2 revision

#Verification

Installing the Mesh v2 control plane

TOC

Phase 1: Prepare observability

Install Kiali

Create the metrics resources

Verification

Phase 2: Install the Mesh v2 control plane

Install the Alauda Service Mesh v2 Operator

Relax the anti-affinity rule of the Mesh v1 control plane

Create the `IstioCNI` resource

Build the mesh configuration of Mesh v2

Create the `Istio` resource

Prepare the Istio reader service account

Get the Mesh v2 revision

Verification