#Upgrade Global Clusters in a DR Environment

#Verify the DR environment before upgrading

Follow your regular global DR inspection procedures to ensure that data in the standby global cluster is consistent with the primary global cluster. For background on the DR topology and synchronization workflow, see Global Cluster Disaster Recovery.

If inconsistencies are detected, do not uninstall the etcd synchronization plugin in the next step, and contact technical support before proceeding. Uninstalling the plugin while the standby global cluster is missing data that the primary holds can cause owner references to resolve incorrectly, and workload-cluster node Machine objects — including immutable-OS clusters, where this destroys the backing virtual machine — may be deleted.

On both global clusters, run the following command to ensure no Machine nodes are in a non-running state:

kubectl get machines.platform.tkestack.io

If any such nodes exist, resolve them before continuing.

#Uninstall the etcd synchronization plugin from the standby global cluster

1. Access the Web Console of the standby global cluster through its IP or VIP.
2. Switch to Administrator view.
3. Navigate to Marketplace > Cluster Plugins and select the global cluster.
4. Find etcd Synchronizer and uninstall it.
5. Wait for the uninstallation to complete before proceeding.

#Sync upgrade artifacts on both global clusters

Complete Sync upgrade artifacts on both the standby global cluster and the primary global cluster. Use the same registry type and exact target-version payload on both clusters. For an external registry, complete both upload modes for each registry endpoint that the two global clusters use.

#Upgrade the standby global cluster

If you will use the Web Console on the standby global cluster, verify that the standby cluster ProductBase includes the standby VIP in spec.alternativeURLs:

apiVersion: product.alauda.io/v1alpha2
kind: ProductBase
metadata:
  name: base
spec:
  alternativeURLs:
    - https://<standby-cluster-vip>

After synchronization completes, run the remaining standard workflow steps on the standby global cluster:

1. Run preflight checks.
2. Deploy the cluster version operator.
3. Request the upgrade.
4. Observe execution until the standby global cluster reaches the desired version.

#Upgrade the primary global cluster

After the standby global cluster reaches the desired version, run the remaining standard workflow steps on the primary global cluster:

1. Run preflight checks.
2. Deploy the cluster version operator.
3. Request the upgrade.
4. Observe execution until the primary global cluster reaches the desired version.

#Reinstall the etcd synchronization plugin and verify sync status

Before reinstalling the plugin, verify that port 2379 is forwarded correctly from both global-cluster VIPs to their control plane nodes when that forwarding mode is used. Port forwarding through a load balancer is not required if the standby global cluster can access the active global cluster directly.

Get the bearer token for the active global cluster API server from the active cluster, then use it to create or update the etcd-sync-active-cluster-token Secret on the standby cluster. Keep the token in the current shell only and clear it after the Secret is created.

# Run on the active cluster and copy the output without storing it in a document.
kubectl -n cpaas-system get secret k8sadmin -o jsonpath='{.data.token}' | base64 -d

Run on the standby cluster:

read -rsp "Active global cluster token: " ACTIVE_CLUSTER_TOKEN
printf '\n'
kubectl -n cpaas-system create secret generic etcd-sync-active-cluster-token \
  --from-literal=token="${ACTIVE_CLUSTER_TOKEN}" \
  --dry-run=client -o yaml | kubectl apply -f -
unset ACTIVE_CLUSTER_TOKEN

To reinstall the plugin:

1. Access the standby global cluster Web Console through its VIP and switch to Administrator view.
2. Navigate to Marketplace > Cluster Plugins and select the global cluster.
3. Find etcd Synchronizer, click Install, and configure the required parameters.

When you configure the plugin:

• Set Active Global Cluster VIP to the VIP of the active global cluster.
• When port 2379 is not forwarded through a load balancer, set Active Global Cluster ETCD Endpoints correctly.
• Set Standby Cluster ETCD Endpoints to the standby cluster etcd address. Use the default value unless the local etcd service is exposed through a different endpoint.
• Set Active Global Cluster Token Secret to etcd-sync-active-cluster-token.
• Use the default value of Data Check Interval.
• Leave Print detail logs disabled unless you are troubleshooting.

During reinstallation, the system runs the etcd-sync-bootstrap Job before the etcd-sync Deployment starts. Verify the bootstrap Job and runtime resources:

kubectl get job -n cpaas-system etcd-sync-bootstrap
kubectl logs -n cpaas-system job/etcd-sync-bootstrap
kubectl get secret -n cpaas-system remote-etcd-ca
kubectl get issuer -n cpaas-system remote-etcd-issuer
kubectl get certificate -n cpaas-system remote-etcd-client
kubectl get secret -n cpaas-system remote-etcd-client

Verify the sync Pods and current leader on the standby global cluster:

kubectl get po -n cpaas-system -l app=etcd-sync
kubectl get lease -n cpaas-system etcd-sync-mirror
leader_pod=$(kubectl get lease -n cpaas-system etcd-sync-mirror -o jsonpath='{.spec.holderIdentity}')
kubectl logs -n cpaas-system "$leader_pod" | grep -E "Acquired leader lease|Start Sync update"

If resources with ownerReference dependencies need to be resynchronized, recreate the current leader Pod after Start Sync update appears:

leader_pod=$(kubectl get lease -n cpaas-system etcd-sync-mirror -o jsonpath='{.spec.holderIdentity}')
kubectl delete po -n cpaas-system "$leader_pod"

Check sync status:

mirror_svc=$(kubectl get svc -n cpaas-system etcd-sync-monitor -o jsonpath='{.spec.clusterIP}')
ipv6_regex="^[0-9a-fA-F:]+$"
if [[ $mirror_svc =~ $ipv6_regex ]]; then
  mirror_host="[$mirror_svc]"
else
  mirror_host="$mirror_svc"
fi
curl -g "http://${mirror_host}/check"

• LOCAL ETCD missed keys: Keys exist in the primary global cluster but are missing from the standby. This often resolves after restarting the current etcd-sync leader Pod.
• LOCAL ETCD surplus keys: Keys exist in the standby global cluster but not in the primary. Review these with your operations team before deleting them.

After verification succeeds, remove any remaining legacy plain-token configuration from the plugin settings or release values. If the active-cluster token or remote etcd-ca changes later, run the plugin upgrade or reinstall workflow again so etcd-sync-bootstrap refreshes the runtime credentials and certificates.