#Configure NPU Fault Observability

Use this How To to make Ascend NPU card faults observable in an ACP-managed workload cluster.

The main procedure configures the normal production path:

DCMI-reported fault state
  ├─> npu-exporter metrics
  ├─> Prometheus/VictoriaMetrics alert rules
  ├─> ACP alert view
  └─> ACP notification policy
        └─> email, Feishu, DingTalk, Webhook, or another configured channel

Ascend Device Plugin
  └─> marks the affected card unhealthy and reduces Node allocatable resources

The optional validation section explains how the project proved this path with fault-injection images. It is intentionally separate from the production procedure. A healthy cluster does not generate a fault alert; that is the expected state after the alerting path has been configured.

#Before you begin

Make sure that:

• The Alauda Build of NPU Operator is installed on the target cluster.
• The Ascend Device Plugin reports NPU resources on the target node.
• The NPU Exporter is enabled and its ServiceMonitor is scraped by the ACP Monitoring stack.
• You have permission to inspect Nodes, DaemonSets, ConfigMaps, metrics, and create PrometheusRule resources.
• An ACP notification server, receiver group, notification template, and notification policy are available if you need external notifications.

In the v26.6.0 delivery, NPU Exporter runs in npu-exporter, the Ascend Device Plugin runs in kube-system, and the exporter ServiceMonitor runs in monitoring.

For installation and exporter prerequisites, see Install, Verify the installation, and Monitor NPU Usage.

For notification setup, see

and Manage alerts.

#1. Verify that the NPU metrics are available

Find an NPU Exporter Pod and expose its metrics endpoint locally:

kubectl -n npu-exporter get pods -l app=npu-exporter -o wide

kubectl -n npu-exporter port-forward \
  pod/<npu-exporter-pod-name> 18082:8082

In another terminal, query the chip metrics:

curl -fsS http://127.0.0.1:18082/metrics | \
  grep -E 'npu_chip_info_health_status|npu_chip_info_error_code|npu_chip_info_network_status'

The metrics are reported per chip. Keep the labels used to locate a card:

• id
• model_name
• pcie_bus_info
• instance
• node, when the scrape or storage pipeline adds it

Interpret the main metrics as follows:

If these metrics are missing, fix exporter or ServiceMonitor discovery before creating chip fault alerts. See Troubleshooting. The absence of an error-code series alone does not prove that the card is healthy. Also confirm that the health metric for the same card is present and that the exporter is collecting DCMI data without errors.

#2. Record the healthy baseline

Record the NPU capacity and allocatable resources on a target Node:

kubectl get node <node-name> \
  -o jsonpath='capacity={.status.capacity.huawei\.com/Ascend310P} allocatable={.status.allocatable.huawei\.com/Ascend310P}{"\n"}'

Replace huawei.com/Ascend310P with the resource key reported by the target Node, such as huawei.com/Ascend910.

Also inspect the Device Plugin state:

kubectl -n kube-system get cm mindx-dl-deviceinfo-<node-name> \
  -o jsonpath='{.data.DeviceInfoCfg}' | python3 -m json.tool

In a healthy state:

• capacity and allocatable are equal;
• if the target card is not already allocated, it is present in the available device list;
• the target card is absent from the Fault and Unhealthy lists.

This baseline is important because NPUAllocatableReduced detects a difference between capacity and allocatable; it does not, by itself, prove a hardware fault.

#3. Create the production alert rules

Create a PrometheusRule in the monitoring namespace. The following rule set covers the core observability path:

• NPUChipUnhealthy: the exporter reports an unhealthy chip;
• NPUChipErrorCodePresent: the exporter reports a DCMI error code in any of its error-code metric slots;
• NPUAllocatableReduced: the Device Plugin has reduced schedulable NPU resources;
• NPUExporterDown: the exporter scrape target is unavailable.

Apply the following manifest:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: npu-production-fault-rules
  namespace: cpaas-system
  annotations:
    alert.cpaas.io/cluster: <cluster-name>
    alert.cpaas.io/kind: Cluster
    alert.cpaas.io/name: <cluster-name>
    alert.cpaas.io/namespace: cpaas-system
    alert.cpaas.io/notifications: '["<notification-policy-name>"]'
    alert.cpaas.io/rules.version: "26"
    cpaas.io/display-name: npu-production-fault-rules
  labels:
    alert.cpaas.io/cluster: <cluster-name>
    alert.cpaas.io/kind: Cluster
    alert.cpaas.io/name: <cluster-name>
    alert.cpaas.io/namespace: cpaas-system
    alert.cpaas.io/options: Single
    alert.cpaas.io/owner: User
    alert.cpaas.io/project: cpaas-system
    cpaas.io/source: Platform
    prometheus: kube-prometheus
    victoriametrics: victoriametrics
    rule.cpaas.io/cluster: <cluster-name>
    rule.cpaas.io/name: npu-production-fault-rules
    rule.cpaas.io/namespace: cpaas-system
spec:
  groups:
    - name: npu-production-fault
      rules:
        - alert: NPUChipUnhealthy
          expr: npu_chip_info_health_status{vmcluster="<vmcluster-name>"} != 1
          for: 2m
          labels:
            severity: Critical
            vmcluster: <vmcluster-name>
            alert_cluster: <cluster-name>
            alert_name: NPUChipUnhealthy
            alert_resource: npu-production-fault-rules
          annotations:
            alert_current_value: "{{ $value }}"
            alert_notifications: '["<notification-policy-name>"]'
            summary: "NPU chip {{ $labels.id }} on node {{ $labels.node }} is unhealthy; model={{ $labels.model_name }}, PCIe={{ $labels.pcie_bus_info }}"
        - alert: NPUChipErrorCodePresent
          expr: |
            {__name__=~"npu_chip_info_error_code(_[1-9])?",vmcluster="<vmcluster-name>"} > 0
          for: 2m
          labels:
            severity: High
            vmcluster: <vmcluster-name>
            alert_cluster: <cluster-name>
            alert_name: NPUChipErrorCodePresent
            alert_resource: npu-production-fault-rules
          annotations:
            alert_current_value: "{{ $value }}"
            alert_notifications: '["<notification-policy-name>"]'
            summary: "NPU chip {{ $labels.id }} on node {{ $labels.node }} reports DCMI error code {{ $value }}"
        - alert: NPUAllocatableReduced
          expr: |
            (
              kube_node_status_capacity{resource=~"huawei_com_Ascend[0-9A-Za-z]+$",vmcluster="<vmcluster-name>"}
              - on(node, resource, vmcluster)
              kube_node_status_allocatable{resource=~"huawei_com_Ascend[0-9A-Za-z]+$",vmcluster="<vmcluster-name>"}
            ) > 0
          for: 2m
          labels:
            severity: Warning
            vmcluster: <vmcluster-name>
            alert_cluster: <cluster-name>
            alert_name: NPUAllocatableReduced
            alert_resource: npu-production-fault-rules
          annotations:
            alert_current_value: "{{ $value }}"
            alert_notifications: '["<notification-policy-name>"]'
            summary: "NPU allocatable is lower than capacity on node {{ $labels.node }}"
        - alert: NPUExporterDown
          expr: up{vmcluster="<vmcluster-name>",job=~".*npu-exporter.*"} == 0
          for: 3m
          labels:
            severity: High
            vmcluster: <vmcluster-name>
            alert_cluster: <cluster-name>
            alert_name: NPUExporterDown
            alert_resource: npu-production-fault-rules
          annotations:
            alert_current_value: "{{ $value }}"
            alert_notifications: '["<notification-policy-name>"]'
            summary: "NPU Exporter scrape target is unavailable: instance={{ $labels.instance }}, node={{ $labels.node }}"

Replace the following placeholders:

• <cluster-name>: the target workload cluster name.
• <vmcluster-name>: the vmcluster label used by the ACP Monitoring backend. Remove this matcher if the environment does not add a vmcluster label.
• <notification-policy-name>: an existing ACP notification policy. The project validation used cpaas-admin-notification.
• The resource names, scrape job name, and metric labels if they differ in the target environment.

The sample uses alert rule metadata version 26. This is the ACP alert object metadata version; it is not an NPU Operator release version.

Apply and verify the rule:

kubectl apply -f npu-production-fault-rules.yaml

kubectl -n cpaas-system get prometheusrule npu-production-fault-rules \
  -o custom-columns='NAME:.metadata.name,DISPLAY:.metadata.annotations.cpaas\.io/display-name,KIND:.metadata.labels.alert\.cpaas\.io/kind,TARGET:.metadata.labels.alert\.cpaas\.io/name,OPTIONS:.metadata.labels.alert\.cpaas\.io/options'

The expected result is a Cluster-scoped rule for the target cluster. If the rule object exists but is not visible in ACP, check the alert metadata and labels required by the target ACP version.

For Ascend 910 environments, add the following rule when the exporter exposes the expected network metric and model label:

npu_chip_info_network_status{vmcluster="<vmcluster-name>",model_name=~".*910.*"} != 1

You can also add a cluster-wide missing-metrics rule:

absent(npu_chip_info_health_status{vmcluster="<vmcluster-name>"})

The absent rule detects a cluster-wide collection problem. It does not prove that every expected Node has an exporter target.

#4. Configure notification delivery

The PrometheusRule selects an ACP notification policy through:

metadata:
  annotations:
    alert.cpaas.io/notifications: '["<notification-policy-name>"]'

Each alert also carries the selected policy:

annotations:
  alert_notifications: '["<notification-policy-name>"]'

These fields select an existing policy. They do not create a notification server, receiver group, template, or policy.

Configure the delivery objects in ACP Operations Center > Notifications:

1. Configure a notification server.
2. Create a receiver group and add the required email, Feishu, DingTalk, or Webhook destination.
3. Create or select an alert notification template.
4. Create a notification policy and record its name.
5. Put that name in both fields in the PrometheusRule.

See

for the platform procedures.

#5. Verify the configured observability in a healthy cluster

A healthy cluster is useful for validating configuration, even though it does not trigger the fault alerts.

Verify all of the following:

kubectl -n npu-exporter get pods -l app=npu-exporter
kubectl -n monitoring get servicemonitor npu-exporter-servicemonitor
kubectl -n cpaas-system get prometheusrule npu-production-fault-rules

Then query the metrics again and confirm that:

• npu_chip_info_health_status is 1 for each healthy chip;
• every emitted npu_chip_info_error_code metric is 0 or absent, the chip health metric is present, and the exporter reports no DCMI collection error;
• NPU allocatable equals capacity;
• NPUChipUnhealthy, NPUChipErrorCodePresent, and NPUAllocatableReduced are not firing;
• NPUExporterDown is not firing.

The absence of active fault alerts at this point is expected. It proves that the rule object is installed and the healthy-state data path is working, but it does not prove the firing path. The firing path requires a real DCMI fault or a supported non-production fault-injection test.

#6. Observe a real NPU fault

When DCMI reports a real fault, observe the event in this order:

1. Query npu_chip_info_health_status and npu_chip_info_error_code from the exporter and identify the card using id, model_name, pcie_bus_info, and node or instance.
2. Wait for the alert rule duration, such as 2 minutes for NPUChipUnhealthy and NPUChipErrorCodePresent.
3. Open ACP Operation and Maintenance Center > Alerts > Alert Policies and search for npu-production-fault-rules.
4. Open the alert details and verify the card identity, current value, error code, severity, and notification status.
5. Check the Node allocatable resources and DeviceInfoCfg to confirm whether the Device Plugin isolated the card.

For a SeparateNPU fault, the expected isolation result is:

capacity remains unchanged
allocatable decreases
the affected card appears in the Unhealthy list
DeviceInfoCfg records the fault code and SeparateNPU handling

NPUAllocatableReduced is an isolation-result signal. Combine it with NPUChipUnhealthy or NPUChipErrorCodePresent before treating the event as a correlated NPU fault and device-isolation event. Determining whether the root cause is hardware, driver, or another lower-level problem requires additional diagnosis.

A Device Plugin health update does not automatically evict, restart, or migrate a running Pod that has already been allocated the affected card. Identify the affected workload and follow its recovery procedure separately.

Use the following states when evaluating the result:

If the alert is visible in ACP but no external message is delivered, check the notification policy, receiver group, template, and notification server before troubleshooting the NPU metrics.

#7. Optional non-production validation

The project validated the firing path in a non-production environment by injecting a DCMI fault at the shared Device Manager boundary. The validation used two specially built images:

• an Ascend Device Plugin image built with the fault-injection build tag;
• an NPU Exporter image built with the same fault-injection support.

Both images must be replaced together. Injecting the fault into only one component cannot prove the complete detection-and-isolation path.

The internal validation used the following input:

ASCEND_DCMI_FAULT_INJECT=<logicID>:<hexFaultCode>

The test fault 0:80E3A201 was classified as SeparateNPU. The expected result was an exporter health fault, a non-zero error-code metric, reduced Node allocatable resources, and an active ACP alert.

The fault-injection images are not delivered as part of the NPU product package, and this page does not publish their image references or provide copy-and-paste image replacement commands. To run this validation in a non-production environment, contact your cluster administrator to obtain an approved, versioned test bundle that contains:

• the Device Plugin and NPU Exporter image references and digests;
• an image override manifest for both DaemonSets;
• the validated Demo PrometheusRule;
• the target namespace and container names;
• cleanup and recovery commands;
• the supported hardware and NPU Operator version.

Use both images from the same test bundle and follow its cleanup and recovery instructions. Do not use images obtained from an unverified source, and do not set ASCEND_DCMI_FAULT_INJECT in a production DaemonSet. Until an approved test bundle is available, use this section as validation scope and evidence, not as a runnable procedure.

#Troubleshooting

#The exporter metrics are missing

Check the exporter Pod, ServiceMonitor, Monitoring target, and the labels selected by the Prometheus or VictoriaMetrics backend.

#The rule exists but no alert is visible

Check that the expression returns a time series, the vmcluster and job labels match the target environment, and the alert has remained active for its full for duration.

#The card is unhealthy but allocatable does not decrease

Check the Device Plugin logs and fault-code classification. Immediate card isolation in the validated fault-injection path uses a fault classified as SeparateNPU. Other fault classifications follow their configured handling policies. Also allow time for the next Device Plugin health refresh.

#The card is isolated but the Node is still Ready

This is expected for device-level isolation. The Device Plugin removes the affected card from the devices available for new allocation, but it does not cordon the Node. Confirm the reduced allocatable NPU count and the Unhealthy entry in DeviceInfoCfg. Do not expect the Node to show SchedulingDisabled.

#A running Pod remains on the unhealthy card

This procedure does not automatically evict or reschedule a Pod that already uses the affected card. Identify the workload from the card and Node information, then follow the workload's recovery procedure.

#The alert is visible but no notification is delivered

Check that the notification policy name exists, the receiver group and template are valid, and the notification server is reachable. Review the notification status in ACP before troubleshooting the NPU metrics.