#Troubleshooting

Use symptom-first troubleshooting. This page covers Alauda Build of NPU Operator and its ACP integration points. For hardware health, BIOS, firmware, and vendor driver deep tuning, use Huawei or openFuyao documentation.

Commands on this page use the default Operator namespace npu-operator. If the Operator was installed in another namespace, replace npu-operator with that namespace; operands that use the Operator namespace follow the installation namespace.

#Driver pod is ImagePullBackOff

Check the driver pod that is scheduled to the affected node:

kubectl -n npu-operator get pods -l app.kubernetes.io/component=npu-driver -o wide
kubectl -n npu-operator describe pod <driver-pod-name>

Common causes:

• the tag derived from the HDK version, chip, and complete kernel release was not mirrored;
• ImageWhiteList in the target workload cluster does not include the full image reference;
• registry credentials are missing or expired.

Prepare the image again from Installation. If the pod keeps the old pull failure after the image and whitelist are fixed, delete the failed driver pod during a maintenance window so the DaemonSet can recreate it.

#Node does not show NPU allocatable resources

Check the driver pod, device plugin, node labels, and allocatable resources:

kubectl -n npu-operator get pods -o wide | grep -E "npu-driver|runtime|hccn"
kubectl -n kube-system get pods -l name=ascend-device-plugin-ds -o wide
kubectl get node ${nodeName} --show-labels
kubectl get node ${nodeName} -o jsonpath='{.status.allocatable}'

Common causes:

• shared NFD or NPU Feature Discovery is not ready, or required node labels are missing;
• the driver pod is not ready;
• the device plugin component is disabled;
• the runtime driver tree is not ready at /run/ascend/.ready/driver-ready;
• on Ascend 910 nodes, HCCN device IP configuration is missing or not reconciled;
• the chip is reported unhealthy during upgrade or recovery.

For Ascend 910 nodes, check the Device Plugin logs. If spec.hccnBootstrap.enabled=true, also check the HCCN Bootstrap agents:

kubectl -n kube-system logs ds/ascend-device-plugin --tail=100
kubectl -n npu-operator get ds hccn-bootstrap
kubectl -n npu-operator logs ds/hccn-bootstrap -c reporter --tail=100
kubectl -n npu-operator logs ds/hccn-bootstrap -c executor --tail=100

If the device plugin reports HCCN or device IP errors, confirm that the target node has the required HCCN configuration before treating the device plugin as the root cause.

#NPU component pod is Pending because a placement label is missing

Some managed components select nodes with masterselector=dls-master-node or workerselector=dls-worker-node. NPU Feature Discovery normally maintains these labels automatically.

Check whether discovery runs on the affected node and whether the expected label exists:

kubectl -n npu-operator get daemonset npu-feature-discovery
kubectl -n npu-operator get pods -l app=npu-feature-discovery -o wide
kubectl get nodes \
  -L node-role.kubernetes.io/control-plane,masterselector,workerselector,openfuyao.com/npu.present
kubectl describe pod <pending-component-pod>

For a control-plane node, automatic master labeling requires the standard node-role.kubernetes.io/control-plane label and an npu-feature-discovery Pod that can run on that node. Check node taints and DaemonSet scheduling before changing labels. If the node has only the legacy node-role.kubernetes.io/master role label, add the standard control-plane role label; otherwise, a running discovery Pod treats it as a worker and removes a manually added masterselector.

If discovery cannot cover an intended node, apply the matching placement label as a fallback:

kubectl label node <control-plane-node-id> masterselector=dls-master-node --overwrite
kubectl label node <npu-worker-node-id> workerselector=dls-worker-node --overwrite

Do not use both labels to select unrelated nodes. When discovery later runs on the node, it reconciles these values to the detected node role.

#Pod is Pending

Describe the pod:

kubectl describe pod <pod-name>

Check that the requested resource key matches the node:

• huawei.com/Ascend910
• huawei.com/Ascend310P
• another huawei.com/Ascend* key reported by the installed device plugin

Also check whether all NPU resources are already allocated:

kubectl describe node ${nodeName} | sed -n '/Allocated resources:/,/Events:/p'

If ACP project or namespace quota blocks scheduling, adjust quota in ACP. For quota field visibility, see Accelerator Resource Quota.

#Pod runs but cannot see NPU devices

Check the admitted Pod, RuntimeClass, Device Plugin, and runtime integration component:

kubectl get pod <pod-name> -o jsonpath='{.spec.runtimeClassName}{"\n"}'
kubectl get runtimeclass ascend
kubectl -n npu-operator get pods -o wide | grep -E "runtime|toolkit"
kubectl -n kube-system logs ds/ascend-device-plugin --tail=100
kubectl -n npu-operator logs ds/ascend-runtime-containerd -c prepare-config --tail=100
kubectl -n npu-operator logs ds/ascend-runtime-containerd -c inject --tail=100

The admitted Pod should use the ascend RuntimeClass. The runtime integration DaemonSet stages the ascend-docker-runtime payload and uses npu-container-toolkit to maintain the ascend containerd handler configuration. In precompiled Driver mode, the host-side runtime Driver tree is staged under /run/ascend/driver; in pre-installed Driver and ordinary OS managed install modes, the runtime uses /usr/local/Ascend/driver.

Expected workload-side checks:

kubectl logs <pod-name> --tail=100
kubectl exec <pod-name> -- ls -la /dev/davinci* /dev/davinci_manager
kubectl exec <pod-name> -- /usr/local/bin/npu-smi info

If you need to inspect mounts from inside a debug container, check for workload-side paths such as /usr/local/bin/npu-smi and /usr/local/Ascend/.... Do not confuse these workload paths with the host-side precompiled runtime tree under /run/ascend/driver.

With the default admission webhook enabled, submitted workload manifests do not need to set runtimeClassName; the admitted Pod should contain runtimeClassName: ascend. If the field is missing, check whether the webhook is enabled and ready and whether the Pod requests a recognized huawei.com/Ascend* resource. If the webhook is disabled, set runtimeClassName: ascend explicitly.

If the admitted Pod uses the expected RuntimeClass but still does not receive devices, check the Device Plugin allocation result and the ascend containerd handler. Restart components only according to the runtime integration procedure delivered with the current product version.

#Metrics are missing

Check exporter and ServiceMonitor:

kubectl -n npu-exporter get pods -l app=npu-exporter
kubectl -n monitoring get servicemonitor npu-exporter-servicemonitor

If both exist, continue in ACP monitoring documentation to check dashboard import, Prometheus discovery, and scrape errors.

#Manual Driver upgrade does not start

With spec.driver.upgradePolicy.autoUpgrade=false, v26.6.0 does not create the reboot transaction before manual approval. Approve only the intended node during the maintenance window:

kubectl annotate node ${nodeName} npu.openfuyao.com/approve-reboot=true --overwrite

After the Operator consumes the approval, check the reboot transaction and Rebooter Pod:

kubectl get node ${nodeName} \
  -o custom-columns=NAME:.metadata.name,REBOOT-REQUIRED:.metadata.labels.npu\.openfuyao\.com/reboot-required,REBOOT-REASON:.metadata.annotations.npu\.openfuyao\.com/reboot-reason,TARGET-DRIVER-ID:.metadata.annotations.npu\.openfuyao\.com/target-driver-id
kubectl get nodes -l npu.openfuyao.com/reboot-required=true
kubectl -n npu-operator get pods | grep npu-rebooter
kubectl -n npu-operator logs ds/npu-rebooter --tail=100

The intended node should then have npu.openfuyao.com/reboot-required=true. If the label does not appear, check the NPU Operator controller logs and confirm that the node is in the current Driver target set. Do not bulk-annotate all NPU nodes, because an unused raw approval can remain on a node and approve a later transaction.