Finalizing the migration

Phase 6 of the migration. Mesh v1 is gone, and the mesh runs on Mesh v2. Three things are left: adopting the default revision tag, taking over the certificate authority that Mesh v1 used to renew, and moving the observability stack.

Adopt the default revision tag

While Mesh v1 existed, the default injection webhook belonged to it and Mesh v2 could not claim it. An IstioRevisionTag named default now lets namespaces use istio-injection=enabled instead of a revision-specific label, and keeps that label valid across future Mesh v2 updates.

cat <<EOF | kubectl apply -f -
apiVersion: sailoperator.io/v1
kind: IstioRevisionTag
metadata:
  name: default
spec:
  targetRef:
    kind: Istio
    name: default
EOF
kubectl get istiorevisiontags

If the resource reports a Helm ownership error, the istio-reader-service-account service account left behind by Mesh v1 still lacks the Helm metadata, see Installing the Mesh v2 control plane.

Namespaces keep working with the revision label they have. To move a namespace to the tag instead:

kubectl label ns "${MESH_NAMESPACE}" istio-injection=enabled istio.io/rev- --overwrite
NOTE

This changes which control plane a namespace follows, not whether its pods are injected. With the injection policy inherited from Mesh v1 (autoInject: disabled), a pod is still injected only when it carries sidecar.istio.io/inject: "true". See Sidecar injection with IstioRevisionTag.

Hand over the certificate authority

The mesh keeps the root certificate authority it has always used. What changes is the intermediate certificate in the cacerts secret: Mesh v1 issued it with a short lifetime and renewed it automatically, and the component that did that is gone. Replace it with a long-lived certificate signed by the same root, so that every workload certificate stays valid and no proxy has to be restarted.

Export the root certificate authority

umask 077
export CA_DIR=${CA_DIR:-$(mktemp -d "${HOME}/ca-handover-XXXXXX")}
mkdir -p "${CA_DIR}"
chmod 700 "${CA_DIR}"
kubectl --context "${GLOBAL_CONTEXT}" -n cpaas-system get secret asm-istio-rootca \
  -o jsonpath='{.data.tls\.crt}' | base64 -d > "${CA_DIR}/root-cert.pem"
kubectl --context "${GLOBAL_CONTEXT}" -n cpaas-system get secret asm-istio-rootca \
  -o jsonpath='{.data.tls\.key}' | base64 -d > "${CA_DIR}/root-key.pem"
openssl x509 -in "${CA_DIR}/root-cert.pem" -noout -subject -enddate

If the secret no longer exists on the global cluster, take both files from ${BACKUP_DIR}/global-asm-istio-rootca.yaml, which holds the same two keys.

Issue the new intermediate certificate

export CA_DAYS=${CA_DAYS:-3650}
openssl genrsa -out "${CA_DIR}/ca-key.pem" 4096
openssl req -new -key "${CA_DIR}/ca-key.pem" -subj "/O=Istio/CN=Intermediate CA/L=${CLUSTER_NAME}" \
  -out "${CA_DIR}/ca.csr"
cat > "${CA_DIR}/ca-ext.cnf" <<EOF
[ v3_ca ]
basicConstraints = critical, CA:TRUE, pathlen:0
keyUsage = critical, digitalSignature, keyEncipherment, keyCertSign
subjectKeyIdentifier = hash
EOF
openssl x509 -req -in "${CA_DIR}/ca.csr" -CA "${CA_DIR}/root-cert.pem" -CAkey "${CA_DIR}/root-key.pem" \
  -CAcreateserial -days "${CA_DAYS}" -extensions v3_ca -extfile "${CA_DIR}/ca-ext.cnf" -out "${CA_DIR}/ca-cert.pem"
cat "${CA_DIR}/ca-cert.pem" "${CA_DIR}/root-cert.pem" > "${CA_DIR}/cert-chain.pem"
openssl x509 -in "${CA_DIR}/ca-cert.pem" -noout -subject -issuer -enddate

CA_DAYS is the lifetime of the intermediate certificate in days, 3650 unless you set it before running the block. Choose it according to your certificate policy, keep it within the lifetime of the root printed in the previous step, and plan the rotation before it expires. A mesh that spans several clusters gets one intermediate certificate per cluster, all signed by the same root.

Replace the cacerts secret

kubectl -n istio-system create secret generic cacerts \
  --from-file="${CA_DIR}/ca-cert.pem" --from-file="${CA_DIR}/ca-key.pem" \
  --from-file="${CA_DIR}/root-cert.pem" --from-file="${CA_DIR}/cert-chain.pem" \
  --dry-run=client -o yaml | kubectl replace -f -
Replace the secret, do not delete and recreate it

istiod watches this secret. If it is deleted first, the control plane briefly runs without a configured certificate authority and can issue workload certificates from a self-signed one, which no other proxy trusts. The command above swaps the contents in a single API call.

Remove the private keys from your workstation

Copy the root key to where you keep private keys first — it is needed again when the intermediate certificate is rotated — then remove the working directory. It holds the private key of the root and of the new intermediate certificate, and both are now in the cluster.

shred -u "${CA_DIR}"/*key.pem 2>/dev/null || rm -f "${CA_DIR}"/*key.pem
rm -rf "${CA_DIR}"

Verify the handover

until kubectl -n istio-system logs "deploy/istiod-${V2_REVISION}" --since=10m \
  | grep -q "newly added intermediate CA"; do sleep 10; done
kubectl -n istio-system logs "deploy/istiod-${V2_REVISION}" --since=10m | grep -A2 "newly added intermediate CA"
kubectl -n "${MESH_NAMESPACE}" get pod -o jsonpath='{range .items[*]}{.metadata.name} {.status.startTime}{"\n"}{end}'
  • The istiod log reports the new intermediate certificate and prints its subject. The secret is mounted as a file, so this takes up to a minute after the secret is replaced; wait for it before you verify anything else.
  • No pod was restarted: the start times are unchanged.
  • Existing workload certificates keep working. They are reissued from the new intermediate certificate within their normal 24-hour lifetime.

Restart one workload to see the new chain immediately:

kubectl -n "${MESH_NAMESPACE}" rollout restart "deploy/${WORKLOAD}"
kubectl -n "${MESH_NAMESPACE}" rollout status "deploy/${WORKLOAD}" --timeout=300s
POD=$(kubectl -n "${MESH_NAMESPACE}" get pod -l "app=${WORKLOAD}" -o jsonpath='{.items[0].metadata.name}')
istioctl proxy-config secret "${POD}" -n "${MESH_NAMESPACE}" -o json \
  | jq -r '.dynamicActiveSecrets[] | select(.name=="default") | .secret.tlsCertificate.certificateChain.inlineBytes' \
  | base64 -d | openssl x509 -noout -issuer

The issuer is the new intermediate certificate, and calls to workloads that still hold a certificate from the old one keep working, because both are signed by the same root.

Migrate the observability stack

Tracing has been interrupted since Mesh v1 was removed. Follow Migrating the observability stack to install OpenTelemetry v2 and Jaeger v2, keep the existing traces queryable, and repoint the mesh and Kiali at the new pipeline.

Restore the revision deletion grace period

Phase 2 raised inactiveRevisionDeletionGracePeriodSeconds to 30 days so that a revision was not removed during the migration. Once the mesh has been stable for a week or two, restore the default:

kubectl patch istio default --type=json \
  -p='[{"op":"remove","path":"/spec/updateStrategy/inactiveRevisionDeletionGracePeriodSeconds"}]'

Leaving it at 30 days means every future control plane update keeps its previous istiod running for a month. Changing the field does not restart anything.

Final acceptance

kubectl get istio,istiocni,istiorevision,istiorevisiontag
kubectl get pod -A -o json | jq -r '[.items[]
  | select(.metadata.annotations["sidecar.istio.io/status"] != null)
  | .metadata.annotations["sidecar.istio.io/status"] | fromjson | .revision] | unique'
kubectl get crd -o name | sed 's|.*/||' | grep -E '\.asm\.alauda\.io$' | while read -r crd; do
  printf '%s: ' "${crd}"; kubectl get "${crd}" -A --no-headers 2>/dev/null | wc -l
done
  • Every Mesh v2 resource reports Healthy.
  • Every injected pod runs the Mesh v2 revision.
  • No Mesh v1 resources are left.
  • Kiali shows the full traffic graph, and — after the observability stack is migrated — traces are queryable in Jaeger v2.
  • The probes have reported no failures since Phase 5.

Stop the probes, archive the backup with the certificate material, and record the new intermediate certificate in your rotation schedule.