#Upgrade Troubleshooting

Use this page when preflight reports a blocking check, CVO reports Ready=False, Reconciling=True, or Stalled=True, or a Core or Aligned module does not reach its target version.

#Inspect Upgrade State

Replace <cluster> with global or the workload cluster name:

kubectl -n cpaas-system get cvsh <cluster> \
  -o jsonpath='{range .status.conditions[*]}{.type}{"\t"}{.status}{"\t"}{.reason}{"\t"}{.message}{"\n"}{end}'

kubectl -n cpaas-system get cvsh <cluster> \
  -o jsonpath='{.status.preflight.observedAt}{"\n"}{range .status.preflight.checks[*]}{.name}{"\t"}{.policy}{"\t"}{.state}{"\t"}{.reason}{"\t"}{.message}{"\n"}{end}'

kubectl -n cpaas-system get cvsh <cluster> \
  -o jsonpath='{range .status.history[*]}{.version}{"\t"}{.state}{"\t"}{.startedTime}{"\t"}{.completionTime}{"\n"}{end}'

Resolve the first failing preflight check or the relevant Ready, Reconciling, or Stalled condition before changing the upgrade request.

#Handle Preflight Blocks

If ResourcePatchUpgradeable fails with reason=UnexemptResourcePatches, inspect the named ResourcePatch and add the required target-version exemption only after reviewing the patch:

kubectl -n cpaas-system get cvsh <cluster> \
  -o jsonpath='{range .status.preflight.checks[?(@.name=="ResourcePatchUpgradeable")]}{.state}{"\t"}{.reason}{"\t"}{.message}{"\n"}{end}'

kubectl get resourcepatches <rp-name> -o yaml
kubectl annotate resourcepatches <rp-name> \
  config.cpaas.io/exempt-for-ver=<target-version> \
  --overwrite

Common preflight blocks:

Do not disable VersionUpgradePath, KubernetesVersionSupported, or AdminAckRequired to force an unsupported upgrade. If technical support instructs you to disable another check temporarily, disable only the named check in cpaas-system/cvo-config.

#Handle Administrator Acknowledgement Gates

If AdminAckRequired fails, inspect the keys supplied by the target release:

kubectl -n cpaas-system get configmap admin-gates -o yaml

Complete the action described by the applicable gate. For a Kubernetes 1.35 or later node-readiness gate, complete Kubernetes 1.35 or Later Node Readiness on every production node.

Copy the applicable key from admin-gates, record the acknowledgement, and rerun preflight:

ACK_KEY='<key-from-admin-gates>'
kubectl -n cpaas-system patch configmap admin-acks --type merge \
  -p "{\"data\":{\"${ACK_KEY}\":\"true\"}}"

bash upgrade.sh --preflight

#Recover a Missing Aligned Package

An unavailable Aligned package does not change the application's current installed state. Depending on the package type and reconciliation stage, CVO can report the problem through Ready=False or Reconciling=True rather than Stalled=True.

#Aligned Cluster Plugins

If a ClusterVersionShadow condition contains required ready ModulePluginConfig for installed platform-aligned component, identify the target ProductManifest and the named ModulePluginConfig:

kubectl get productmanifest v<target-version> -o yaml

kubectl get modulepluginconfig <modulepluginconfig-name> \
  -o jsonpath='{.spec.image}{"\n"}{range .status.conditions[*]}{.type}{"\t"}{.status}{"\t"}{.reason}{"\t"}{.message}{"\n"}{end}'

For a ModulePlugin channel, artifactStatus: Absent in the target ProductManifest means that the matching target ModulePluginConfig is not fully Ready. It does not mean that the currently installed application became absent, and it does not by itself prove that the image is missing from the registry.

Use the ModulePluginConfig condition and the exact image in .spec.image to select the recovery:

• For an application in ACP Upgrade to v4.4, copy its package into the target Core Package's plugins/ directory. With the platform built-in Registry, rerun upgrade.sh --only-sync-image. With an external registry, rerun both modes in Prepare the Target-Version Payload.
• For another Aligned cluster plugin, publish the target package to the global tier with violet push.

#Aligned Operators

Aligned operators do not use ModulePluginConfig. If an installed operator does not advance, inspect its Subscription, target InstallPlan, catalog, and ModuleInfo on the affected cluster:

kubectl get subscription -A
kubectl get installplan -A
kubectl get moduleinfo -l cpaas.io/cluster-name=<cluster>

If the target operator package is missing, publish it to every cluster where that operator is installed. If only one workload cluster is missing it, push to that cluster with --clusters "<workload-cluster-name>". If the package exists, resolve the Subscription, InstallPlan, catalog, or ModuleInfo condition actually reported by CVO.

Continue observing the existing request after correcting the problem. CVO retries automatically; do not clear or resubmit desiredUpdate.

#Diagnose a Module That Does Not Advance

kubectl get moduleinfo -l cpaas.io/cluster-name=<cluster> \
  -o custom-columns='MODULE:.metadata.labels.cpaas\.io/module-name,CURRENT:.status.version,TARGET:.spec.version,NEW:.status.availableVersions[0].version,PHASE:.status.phase'

kubectl get moduleinfo <name> \
  -o jsonpath='{range .status.conditions[*]}{.type}{"\t"}{.status}{"\t"}{.reason}{"\t"}{.message}{"\n"}{end}'

Inspect the named resource and related Events. For image-pull failures, verify registry reachability, CA trust, pull credentials, and the referenced manifest from the affected node.

#Collect Evidence Before Escalation

Capture the cvsh conditions, preflight checks, stages and history, the affected ModuleInfo or ModulePluginConfig, related Events, and the exact source and target versions. Do not include platform tokens, registry passwords, or Secret contents.