#Troubleshooting

Use symptom-first troubleshooting for Alauda Build of HAMi. For vendor driver internals, firmware, and hardware health, continue with the corresponding hardware vendor documentation.

Start by confirming the selected backend path:

• HAMi on NVIDIA GPU
• HAMi on Ascend NPU
• HAMi on Ascend vNPU

#Node does not show HAMi resources

Check the HAMi pods and node labels:

kubectl get pods -n kube-system | grep -E "hami-scheduler|hami-device-plugin"
kubectl get node ${nodeName} --show-labels
kubectl get node ${nodeName} -o jsonpath='{.status.allocatable}'

Common causes:

• the node label used by the HAMi plugin form is missing;
• the vendor driver or runtime is not ready;
• the backend is not enabled in the installed HAMi package;
• another accelerator plugin owns device exposure for the same node or same accelerator devices.

If the node reports only direct vendor resource keys, check whether the vendor device plugin is still the active exposure path. Use the current delivery package and version page to confirm the expected HAMi resource names for Ascend NPU or Ascend vNPU.

For NVIDIA GPU nodes, HAMi commonly selects nodes with gpu=on. The direct NVIDIA GPU Device Plugin commonly selects nodes with nvidia-device-enable=pgpu. If the same node has both labels and reports both nvidia.com/gpu and HAMi resource keys such as nvidia.com/gpualloc, remove one selector label or split the workloads into separate node pools.

For Ascend NPU or Ascend vNPU nodes, HAMi commonly selects nodes with ascend=on. If NPU Operator also prepares the node, check the NPUOperatorCtl component settings. Stop the native Ascend Device Plugin before Alauda Build of HAMi Ascend Device Plugin takes over device exposure. Do not rely on the resource key names to separate the direct NPU path from the HAMi path.

#ACP quota page does not show HAMi resources

Node allocatable resources and ACP quota fields are not the same thing. If the node reports HAMi resources but ACP does not show them in quota pages, register the corresponding resource metadata in kube-public.

For required ConfigMap fields and examples, see Accelerator Resource Quota.

If ACP 3.18.2 does not show HAMi extended resources in forms, use the frontend image build-harbor.alauda.cn/acp/icarus:v3.18.108 to support form-based selection of HAMi extended resources.

#Pod is Pending

Describe the pod:

kubectl describe pod <pod-name>

Check:

• the workload requests the correct HAMi resource keys for the backend;
• project and namespace quota allow the requested resources;
• the target nodes report enough HAMi allocatable resources;
• the node is not still exposed through a conflicting direct vendor plugin path;
• the workload does not mix HAMi resource keys with resource keys from another sharing technology unless the current version page explicitly supports that combination.

#Device plugin pod cannot start on NVIDIA backend

If NVIDIA driver calls are very slow, the HAMi device plugin may fail to start. Check the node with nvidia-smi.

One common mitigation is to enable persistence mode on the node, then restart the HAMi device plugin pod:

nvidia-smi -pm enable

#CUDA reports that devices are busy or unavailable

When multiple workloads share the same NVIDIA GPU, CUDA may report that devices are busy or unavailable if the device compute mode blocks concurrent access.

Check the device compute mode on the node. If the workload plan allows multiple processes, configure the device mode accordingly:

nvidia-smi -i 0 -c 0

#Scheduler waits on a node lock

If a pod is deleted during the bind phase, later pods may wait until the node lock expires. Check the HAMi scheduler logs and confirm whether the workload was deleted during scheduling.

Upgrade to a HAMi version that includes the node-lock cleanup fix when it is available in the Alauda package.

#Metrics are missing

Check the exporter and dashboard path used by your backend. For NVIDIA backend metrics:

kubectl get pods -n kube-system | grep dcgm-exporter

If the exporter is running, continue in ACP monitoring documentation to check dashboard import, Prometheus discovery, and scrape errors.

For Ascend backends, use the metrics path documented by the current Ascend base and HAMi package support matrix. Do not copy NVIDIA DCGM-Exporter checks to Ascend backend troubleshooting.

#HAMi-WebUI returns HTTP 523 or VGPU_DOMAIN_ERROR

The HAMi-WebUI page can load successfully while its Prometheus-backed requests fail. In the browser developer tools or a direct API request, the failure commonly appears on:

POST /api/vgpu/v1/monitor/query/instant-vector

HAMi-WebUI wraps Prometheus query failures as HTTP 523 with reason VGPU_DOMAIN_ERROR. HTTP 523 is the WebUI error code; it is not the original Prometheus status.

If the error contains <!doctype html>, a login page, or an authorization page, the configured Prometheus address was redirected instead of returning Prometheus JSON. A common cause is entering only the base64 payload in the plugin form and omitting the authentication scheme.

1. Regenerate the settings from Install HAMi-WebUI.

2. Confirm that the authentication field contains the complete value:

   Basic <base64(username:password)>

3. Query ${prometheus_address%/}/api/v1/query?query=up with that value as the Authorization header. Continue only after Prometheus returns JSON with "status":"success".

4. Update the Alauda Build of HAMi-WebUI cluster plugin form and wait for the WebUI pod to become Ready.

5. Repeat the WebUI backend instant-vector query. It must return HTTP 200 and a JSON data array.

If the complete Basic value still redirects, verify that the address belongs to the Prometheus API service rather than an interactive platform route, and verify the monitoring Secret selected by the current Feature monitoring resource.

#HAMi Ascend Device Plugin is installed but resources are missing

Check the installation from the owning resources down to the node:

kubectl get csv -A | grep hami-ascend-device-plugin
kubectl -n <hami-namespace> get hamiadp hami-ascend-device-plugin -o yaml
kubectl -n <hami-namespace> get ds hami-ascend-device-plugin -o wide
kubectl get node <ascend-node> --show-labels
kubectl get node <ascend-node> -o jsonpath='{.status.allocatable}'

Common causes include a CSV that has not reached Succeeded, a manually approved InstallPlan that is still pending, a missing ascend=on label, a driver host path that does not match the node, or a native Ascend Device Plugin that is still running.

Use Switch the Ascend Device Plugin Owner to verify the user-facing NPUOperatorCtl setting, generated policy, and active DaemonSet. Do not delete only a generated DaemonSet, because its controller can recreate it.

#Soft slicing is enabled but the node mode remains false

Check the owning custom resource, both ConfigMaps, and the effective node annotation:

kubectl -n <hami-namespace> get hamiadp hami-ascend-device-plugin \
  -o jsonpath='{.spec.config}{"\n"}{.spec.hamiVnpuCore.enabled}{"\n"}{.spec.nodeConfig}{"\n"}'
kubectl -n <hami-namespace> get configmap hami-scheduler-device -o yaml
kubectl -n <hami-namespace> get configmap hami-device-node-config -o yaml
kubectl get node <ascend-node> \
  -o go-template='hami-vnpu-core={{ index .metadata.annotations "hami-vnpu-core" }}{{ "\n" }}'

A node entry in spec.nodeConfig takes precedence over the global value. If the instance reuses hami-scheduler-device with spec.config.create: false, HAMi Ascend Device Plugin reads that HAMi-owned ConfigMap; enabling only spec.hamiVnpuCore.enabled in the Device Plugin form does not rewrite the shared ConfigMap and does not change the effective mode.

Set the target node explicitly in the owning HAMiAscendDevicePlugin.spec.nodeConfig instead of editing a generated ConfigMap. Then wait for daemonset/hami-ascend-device-plugin to roll out. The Device Plugin loads the node configuration at startup; if reconciliation updated the ConfigMap without restarting the DaemonSet, restart only that DaemonSet and verify the node annotation again. See Configure Ascend Slicing Mode.

#Soft-slice Pod exits and the lifecycle hook reports a stopped container

An event such as cannot exec in a stopped state from the PostStart hook is usually secondary: the main container process exited before the hook command could run. Inspect the termination status and the previous container log first:

kubectl -n <namespace> get pod <pod-name> \
  -o jsonpath='{range .status.containerStatuses[*]}{.name}{" exit="}{.state.terminated.exitCode}{" reason="}{.state.terminated.reason}{"\n"}{end}'
kubectl -n <namespace> logs <pod-name> -c <container-name> --previous
kubectl -n <namespace> describe pod <pod-name>

If the process exits with code 127 and reports a missing symbol such as rtStreamGetCaptureInfo while loading /hami-vnpu-core/libvnpu.so, the workload CANN runtime and the injected soft-slice library are ABI-incompatible. Compare the exact workload path and image runtime with the bounded results in Ascend Driver and CANN compatibility.

Select an immutable workload image whose CANN version matches the installed Ascend Driver, record its digest, and run a representative device-open, memory-allocation, or inference operation before accepting the environment.