扩展 Inference Runtimes

简介

本文将指导你如何添加新的 inference runtimes,用于提供 Large Language Models (LLMs) 以及其他模型的服务,例如图像分类、目标检测和文本分类模型。

Alauda AI 内置了 vLLM inference engine。通过自定义 inference runtimes,你可以引入其他 inference engines,例如 Seldon MLServer、Triton Inference Server。

通过引入自定义 runtimes,你可以扩展平台对更广泛模型类型和 GPU 类型的支持,并针对特定场景优化性能,以满足更广泛的业务需求。

在本节中,我们将演示如何使用自定义 Xinference serving runtime 扩展当前 AI 平台,以部署 LLMs 并暴露一个 OpenAI-compatible API。

场景

如果你遇到以下任一情况,可以考虑扩展 AI Platform inference service runtimes:

  • 支持新的模型类型:你的模型不是当前默认 inference runtime vLLM 原生支持的类型。
  • 兼容其他硬件类型:你需要在 AMD GPUs 或 Huawei Ascend NPUs 等硬件上执行 LLM inference。
  • 针对特定场景进行性能优化:在某些 inference 场景下,新的 runtime(如 Xinference)可能比现有 runtimes 提供更好的性能或资源利用率。
  • 自定义推理逻辑:你需要引入自定义推理逻辑或依赖库,而这些逻辑或库在现有默认 runtimes 中难以实现。

前提条件

开始之前,请确保你满足以下条件:

  1. 你的 ACP cluster 已部署并且正常运行。
  2. 你的 AI Platform 版本为 1.3 或更高。
  3. 你已准备好必要的 inference runtime image(s)。例如,对于 Xinference runtime,镜像可能类似于 xprobe/xinference:v1.2.2(用于 GPU)或 xprobe/xinference:v1.2.2-cpu(用于 CPU)。
  4. 你拥有 cluster administrator privileges(创建 CRD instances 需要此权限)。

标准流程(示例:Xinference)

按照以下步骤扩展平台。我们使用 Xinference 作为基线示例来演示标准流程。

创建 Inference Runtime 资源

你需要根据目标硬件环境(GPU/CPU/NPU)创建相应的 inference runtime ClusterServingRuntime 资源。

  1. 准备 Runtime YAML 配置:

    根据你想要添加的 runtime 类型(例如 Xinference)以及目标硬件环境,准备相应的 YAML 配置文件。以下是不同硬件环境下 Xinference runtime 的示例:

  • GPU Runtime 示例
    # This is a sample YAML for Xinference GPU runtime
    apiVersion: serving.kserve.io/v1alpha1
    kind: ClusterServingRuntime
    metadata:
      name: aml-xinference-cuda-12.1 # Name of the runtime resource
      labels:
        cpaas.io/runtime-class: xinference # required runtime type label
        cpaas.io/accelerator-type: "nvidia"
        cpaas.io/cuda-version: "12.1"
      annotations:
        cpaas.io/display-name: xinference-cuda-12.1 # Display name in the UI
    spec:
      containers:
      - name: kserve-container
        image: xprobe/xinference:v1.2.2  # Replace with your actual GPU runtime image
        env:
        # Required across all runtimes – path to the model directory
        - name: MODEL_PATH
          value: /mnt/models/{{ index .Annotations "aml-model-repo" }}
        # The MODEL_UID parameter is optional for other runtimes.
        - name: MODEL_UID 
          value: '{{ index .Annotations "aml-model-repo" }}'
        # The MODEL_ENGINE parameter is required by the Xinference runtime, while it can be omitted for other runtimes.
        - name: MODEL_ENGINE 
          value: "transformers"
        # Required parameter for xinference runtime, please set it based on your model family, value: "llama" # e.g., "llama", "chatglm", etc.
        - name: MODEL_FAMILY 
          value: ""
        command:
        - bash
        - -c
        - |
            set +e
            if [ "${MODEL_PATH}" == "" ]; then
                echo "Need to set MODEL_PATH!"
                exit 1
            fi
            if [ "${MODEL_ENGINE}" == "" ]; then
                echo "Need to set MODEL_ENGINE!"
                exit 1
            fi
            if [ "${MODEL_UID}" == "" ]; then
                echo "Need to set MODEL_UID!"
                exit 1
            fi
            if [ "${MODEL_FAMILY}" == "" ]; then
                echo "Need to set MODEL_FAMILY!"
                exit 1
            fi
    
            xinference-local --host 0.0.0.0 --port 8080 &
            PID=$!
            while [ true ];
            do
                curl http://127.0.0.1:8080/docs
                if [ $? -eq 0 ]; then
                    break
                else
                    echo "waiting xinference-local server to become ready..."
                    sleep 1
                fi
            done
    
            set -e
            xinference launch --model_path ${MODEL_PATH} --model-engine ${MODEL_ENGINE} -u ${MODEL_UID} -n ${MODEL_FAMILY} -e http://127.0.0.1:8080 $@
            xinference list -e http://127.0.0.1:8080
            echo "model load succeeded, waiting server process: ${PID}..."
            wait ${PID}
        # Add this line to use $@ in the script:
        # see: https://unix.stackexchange.com/questions/144514/add-arguments-to-bash-c
        - bash
        resources:
          limits:
            cpu: 2
            memory: 6Gi
          requests:
            cpu: 2
            memory: 6Gi
        startupProbe:
          httpGet:
            path: /docs
            port: 8080
            scheme: HTTP
          failureThreshold: 60 
          periodSeconds: 10
          timeoutSeconds: 10
      supportedModelFormats:
        - name: transformers # The model format supported by the runtime
          version: "1"
    
    1. 根据环境中的实际模型大小、runtime engine、硬件类型和预期工作负载设置 CPU limit。
    2. 根据环境中的实际模型大小、runtime engine、硬件类型和预期工作负载设置 memory limit。
    • 提示:请务必将 image 字段值替换为你实际准备好的 runtime 镜像路径。你也可以修改 annotations.cpaas.io/display-name 字段,以自定义 AI Platform UI 中该 runtime 的显示名称。
  1. 应用 YAML 文件以创建资源:

    在具有 cluster administrator privileges 的终端中,执行以下命令以应用 YAML 文件并创建 inference runtime 资源:

    kubectl apply -f your-xinference-runtime.yaml
    TIP
    • 重要提示:请参考上述示例,并根据你的实际环境和 inference 需求创建/配置 runtime。 这些示例仅供参考。你需要调整 image、资源 limits 和 requests 等参数,以确保 runtime 与你的模型和硬件环境兼容并高效运行。
    • 注意:只有在创建完成 runtime resource 之后,才能在 inference service 发布页面上使用此自定义 runtime!

发布 Xinference Inference Service 并选择 Runtime

当 Xinference inference runtime 资源成功创建后,你就可以在 AI Platform 上发布 LLM inference service 时选择并配置它。

  1. 为模型配置 Inference Framework:

    确保在即将发布的模型仓库的模型详情页面中,已通过 File Management 元数据编辑功能选择了合适的 framework。此处选择的 framework 值必须与创建 inference service runtime 时 supportedModelFormats 字段中的某个值一致。请确保模型 framework 值已列在 inference runtime 的 supportedModelFormats 字段中。

  2. 进入 Inference Service 发布页面:

    登录 AI Platform,进入 “Inference Services” 或 “Model Deployment” 模块,然后点击 “Publish Inference Service”。

  3. 选择 Xinference Runtime:

    在 inference service 创建向导中,找到 “Runtime” 或 “Inference Framework” 选项。从下拉菜单或列表中,选择你在步骤 1 中创建的 Xinference runtime(例如 “Xinference CPU Runtime” 或 “Xinference GPU Runtime (CUDA)”)。

  4. 设置环境变量: Xinference runtime 需要特定的环境变量才能正常工作。在 inference service 配置页面中,找到 “Environment Variables” 或 “More Settings” 区域,并添加以下环境变量:

    • Environment Variable

      参数名称描述
      MODEL_FAMILY必需。指定你正在部署的 LLM 模型所属的 family 类型。Xinference 使用此参数来识别并加载该模型对应的正确推理逻辑。例如,如果你部署的是 Llama 3 模型,将其设置为 llama;如果是 ChatGLM 模型,则设置为 chatglm。请根据模型的实际 family 进行设置。
    • 示例:

      • 变量名:MODEL_FAMILY
      • 变量值:llama(如果你使用的是 Llama 系列模型,请查看 docs 获取更多详情。或者你也可以运行 xinference registrations -t LLM 来列出所有受支持的模型 family。)

特定 Runtime 示例

理解标准流程后,请参考以下示例,了解其他 runtimes 的具体配置。

MLServer

MLServer runtime 功能丰富,可同时用于 NVIDIA GPUs 和 CPUs。

kind: ClusterServingRuntime
apiVersion: serving.kserve.io/v1alpha1
metadata:
  annotations:
    cpaas.io/display-name: mlserver-cuda11.6-x86-arm
  creationTimestamp: 2026-01-05T07:02:33Z
  generation: 1
  labels:
    cpaas.io/accelerator-type: nvidia
    cpaas.io/cuda-version: "11.6"
    cpaas.io/runtime-class: mlserver
  name: aml-mlserver-cuda-11.6
spec:
  containers:
    - command:
        - /bin/bash
        - -lc
        - |
          if [ "$MODEL_TYPE" = "text-to-image" ]; then
            MODEL_IMPL="mlserver_diffusers.StableDiffusionRuntime"
          else
            MODEL_IMPL="mlserver_huggingface.HuggingFaceRuntime"
          fi            

          MODEL_DIR="${MLSERVER_MODEL_URI}/${MLSERVER_MODEL_NAME}"
          # a. using git lfs storage initializer, model will be in /mnt/models/<model_name>
          # b. using hf storage initializer, model will be in /mnt/models
          if [ ! -d "${MODEL_DIR}" ]; then
              MODEL_DIR="${MLSERVER_MODEL_URI}"
              echo "[WARNING] Model directory ${MODEL_DIR}/${MLSERVER_MODEL_NAME} not found, using ${MODEL_DIR} instead"
          fi

          export MLSERVER_MODEL_IMPLEMENTATION=${MODEL_IMPL}
          export MLSERVER_MODEL_EXTRA="{\"task\":\"${MODEL_TYPE}\",\"pretrained_model\":\"${MODEL_DIR}\"}"            

          mlserver start $MLSERVER_MODEL_URI $@
        - bash
      env:
        - name: MLSERVER_MODEL_URI
          value: /mnt/models
        - name: MLSERVER_MODEL_NAME
          value: '{{ index .Annotations "aml-model-repo" }}'
        - name: MODEL_TYPE
          value: '{{ index .Annotations "aml-pipeline-tag" }}'
      image: alaudadockerhub/seldon-mlserver:1.6.0-cu116-v1.3.1
      name: kserve-container
      resources:
        limits:
          cpu: 2
          memory: 6Gi
        requests:
          cpu: 2
          memory: 6Gi
      securityContext:
        allowPrivilegeEscalation: false
        capabilities:
          drop:
            - ALL
        privileged: false
        runAsNonRoot: true
        runAsUser: 1000
      startupProbe:
        failureThreshold: 60
        httpGet:
          path: /v2/models/{{ index .Annotations "aml-model-repo" }}/ready
          port: 8080
          scheme: HTTP
        periodSeconds: 10
        timeoutSeconds: 10
  labels:
    modelClass: mlserver_sklearn.SKLearnModel
  supportedModelFormats:
    - name: mlflow
      version: "1"
    - name: transformers
      version: "1"
  1. 根据环境中的实际模型大小、runtime engine、硬件类型和预期工作负载设置 CPU limit。
  2. 根据环境中的实际模型大小、runtime engine、硬件类型和预期工作负载设置 memory limit。

Triton Inference Server

Triton Inference Server runtime 面向 NVIDIA GPUs 设计,并支持多种模型格式。与 MLServer 类似,你需要先创建 ClusterServingRuntime 资源,然后再创建 inference service。

apiVersion: serving.kserve.io/v1alpha1
kind: ClusterServingRuntime
metadata:
  annotations:
    cpaas.io/display-name: triton-cuda12-x86
  labels:
    cpaas.io/accelerator-type: nvidia
    cpaas.io/cuda-version: "12.1"
    cpaas.io/runtime-class: triton
  name: aml-triton-cuda-12
spec:
  containers:
    - command:
        - /bin/bash
        - -c
        - >
          tritonserver --log-verbose=1 --http-port=8080
          --model-repository=/mnt/models
      env:
        - name: OMP_NUM_THREADS
          value: "1"
        - name: MODEL_REPO
          value: '{{ index .Annotations "aml-model-repo" }}'
      image: alaudadockerhub/tritonserver:25.02-py3
      name: kserve-container
      resources:
        limits:
          cpu: 2
          memory: 6Gi
        requests:
          cpu: 2
          memory: 6Gi
      securityContext:
        allowPrivilegeEscalation: false
        capabilities:
          drop:
            - ALL
        privileged: false
        runAsNonRoot: true
        runAsUser: 1000
      startupProbe:
        failureThreshold: 60
        httpGet:
          path: /v2/models/{{ index .Annotations "aml-model-repo" }}/ready
          port: 8080
          scheme: HTTP
        periodSeconds: 10
        timeoutSeconds: 10
  supportedModelFormats:
    - name: triton
      version: "1"
  1. 根据环境中的实际模型大小、runtime engine、硬件类型和预期工作负载设置 CPU limit。
  2. 根据环境中的实际模型大小、runtime engine、硬件类型和预期工作负载设置 memory limit。

使用说明:

  1. 创建 ClusterServingRuntime:使用 kubectl apply -f triton-runtime.yaml 应用上述 YAML 配置
  2. 准备你的模型:确保模型格式受 Triton 支持(例如 TensorFlow、PyTorch、ONNX)
  3. 设置模型 Framework:在模型仓库中,将 framework 元数据设置为 triton,以匹配 supportedModelFormats 字段
  4. 创建 Inference Service:在发布 inference service 时,从 runtime 下拉菜单中选择 Triton runtime

vLLM-ascend(Ascend NPU)

vLLM-ascend runtime 适用于 Huawei Ascend NPUs。它保留了 vLLM 的 OpenAI-compatible serving 风格,同时需要为可写路径和组权限额外配置一些 InferenceService 设置。

此示例已在 Ascend 910B4 上验证。它也应适用于其他 Ascend NPU 型号,但你应根据实际环境调整 resource key、image 和相关 version 字段。

Ascend 910 的 Modelcar 权限模式

Alauda AI 默认以非 root UID 1000 运行 KServe Modelcar。该默认值旨在满足平台安全基线,并适用于常见的 GPU 部署,例如 NVIDIA GPU inference services。它同样适用于已验证的 Ascend 910 单卡部署中的社区 vLLM-ascend 镜像。

Ascend 910 单节点多卡 inference 有额外要求,因为 vLLM-ascend 使用 HCCL 进行分布式初始化。在这种场景下,请根据 runtime 镜像和验证结果选择以下某一种 Modelcar 权限模式:

场景Namespace PSA EnforceModelcar UIDRuntime image 要求建议
Ascend 910 单卡 inferencerestricted1000社区 vLLM-ascend 镜像保持默认非 root 模式
Ascend 910 单节点多卡 inference,且镜像兼容 UID 1000restricted1000镜像允许 UID 1000 访问 Ascend 设备并支持 HCCL 初始化在验证后使用非 root 模式
Ascend 910 单节点多卡 inference,使用社区镜像且尚未验证 UID 1000 兼容性baseline0社区 vLLM-ascend 镜像使用 root 模式
WARNING

这些模式在同一个 cluster 中是互斥的,因为 Modelcar UID 是在 cluster-level KServe Modelcar 设置中配置的。对于通用工作负载以及已验证的 Ascend 910 单卡服务,请保持默认的非 root UID 1000。仅在 Ascend 910 多卡 vLLM-ascend 部署确有需要时才切换到 root 模式。

非 root 模式

非 root 模式会保持平台默认的 Modelcar UID 1000,且 namespace 可以继续将 PSA Enforce 保持为 restricted。

对于使用社区镜像的 Ascend 910 单卡 vLLM-ascend inference,请使用此模式。对于 Ascend 910 单节点多卡 inference,仅在验证镜像支持 UID 1000 且支持 HCCL 初始化后再使用此模式。镜像必须允许 runtime process 访问 Ascend device 文件,并且 TP>1 / HCCL 初始化可能要求在 /etc/passwd 中存在与 UID 1000 匹配的用户条目。

Root 模式

当所选镜像无法可靠地以默认 Modelcar UID 1000 运行时,root 模式是 Ascend 910 单节点多卡 vLLM-ascend inference 的一种兼容性选项。

Root 模式会将 cluster-level Modelcar UID 更改为 0。这会影响 cluster 中的 Modelcar workloads,而不仅仅是单个 inference service。除非你的镜像或环境确有需要,否则不要将 root 模式用于单卡部署。

要启用 root 模式:

  1. 在平台控制台中,进入 Project > Namespace,选择 inference service 使用的 namespace,并将 Pod Security Admission Enforce 设置为 baseline。

  2. 在 AmlCluster 配置中将 KServe Modelcar UID 设置为 0:

    spec:
      components:
        kserve:
          values:
            kserve:
              storage:
                uidModelcar: 0

对于 root-mode deployments,不要复用强制使用 non-root UID 的 runtime 配置,例如 runAsNonRoot: true 或 runAsUser: 1000。下面的 ClusterServingRuntime 示例适用于非 root 模式。

当不再需要 root 模式时,请将 cluster-level Modelcar UID 恢复为平台默认的非 root 值 1000,并根据你的 workload 安全要求设置 namespace PSA。

1. ClusterServingRuntime

apiVersion: serving.kserve.io/v1alpha1
kind: ClusterServingRuntime
metadata:
  annotations:
    aml.cpaas.io/model-type: '["generative"]'
    aml.cpaas.io/user-create: "true"
    cpaas.io/display-name: vllm-ascend-cann8.5
    helm.sh/resource-policy: keep
  labels:
    cpaas.io/accelerator-type: ascend
    cpaas.io/cann-version: "8.5.1"
    cpaas.io/runtime-class: vllm
  name: aml-vllm-ascend-cann-8.5.1
spec:
  containers:
    - command:
        - bash
        - -c
        - >
          set -ex

          # 1. check model path

          MODEL_DIR="/mnt/models/${MODEL_NAME}"

          # a. using git lfs storage initializer, model will be in
          /mnt/models/<model_name>

          # b. using hf storage initializer, model will be in /mnt/models

          if [ ! -d "${MODEL_DIR}" ]; then
              MODEL_DIR="/mnt/models"
              echo "[WARNING] Model directory ${MODEL_DIR}/${MODEL_NAME} not found, using ${MODEL_DIR} instead"
          fi


          # 2. check if using gguf models

          c=`find "${MODEL_DIR}" -maxdepth 1 -type f -name '*.gguf' | wc -l`

          echo "find ${c} gguf files"

          if [ "${c}" -gt 1 ]; then
              echo "[ERROR] More than one gguf file found in ${MODEL_DIR}"
              echo "Currently, vllm only supports loading single-file GGUF models. If you have a multi-files GGUF model, you can use gguf-split tool to merge them to a single-file model."
              exit 1
          elif [ "${c}" -eq 1 ]; then
              n=`find "${MODEL_DIR}" -maxdepth 1 -type f -name '*.gguf' -print`
              echo "[INFO] Using GGUF model file: ${n}"
              MODEL_PATH="${n}"
          else
              echo "[INFO] Using standard model directory"
              MODEL_PATH="${MODEL_DIR}"
          fi


          # 3. launch vllm server

          python3 -m vllm.entrypoints.openai.api_server \

          --port 8080 \

          --served-model-name {{.Name}} {{.Namespace}}/{{.Name}} \

          --model ${MODEL_PATH} \

          --gpu-memory-utilization ${GPU_MEMORY_UTILIZATION} \

          $@
        - bash
      env:
        - name: MODEL_NAME
          value: '{{ index .Annotations "aml-model-repo" }}'
        - name: GPU_MEMORY_UTILIZATION
          value: "0.95"
        - name: HOME
          value: /tmp
        - name: USER
          value: vllm
      image: quay.io/ascend/vllm-ascend:v0.18.0rc1
      name: kserve-container
      ports:
        - containerPort: 8080
          name: http1
          protocol: TCP
      resources:
        limits:
          cpu: 2
          memory: 6Gi
        requests:
          cpu: 2
          memory: 6Gi
      securityContext:
        allowPrivilegeEscalation: false
        capabilities:
          drop:
            - ALL
        privileged: false
        runAsNonRoot: true
        runAsUser: 1000
        seccompProfile:
          type: RuntimeDefault
      startupProbe:
        exec:
          command:
            - sh
            - -c
            - >
              curl -s -o /dev/null -w "%{http_code}" -X POST
              "http://127.0.0.1:8080/v1/completions" -H "Content-Type:
              application/json" -d '{"model": "{{ .Name }}", "prompt": "ping"}'
              | grep -q "200"
        failureThreshold: 60
        periodSeconds: 10
        timeoutSeconds: 180
      volumeMounts:
        - mountPath: /dev/shm
          name: devshm
  protocolVersions:
    - v2
  supportedModelFormats:
    - name: transformers
      version: "1"
  volumes:
    - emptyDir:
        medium: Memory
        sizeLimit: 1Gi
      name: devshm
  1. HOME 将临时文件和缓存指向 /tmp,该目录对 runtime container 是可写的。
  2. USER 可防止 Python 的 getpass.getuser() 回退逻辑针对容器 UID 查询 /etc/passwd。这可以避免在以 non-root UID 运行且没有匹配 passwd 条目的镜像中发生启动失败,同时仍允许 torch_npu 正常自动加载。对于 TP>1 / HCCL 初始化,仍需验证镜像是否需要在 /etc/passwd 中存在实际的 UID 1000 条目。
  3. 根据环境中的实际模型大小、runtime engine、硬件类型和预期工作负载设置 CPU limit。
  4. 根据环境中的实际模型大小、runtime engine、硬件类型和预期工作负载设置 memory limit。
  5. runAsUser: 1000 使 runtime container 与平台默认的 Modelcar UID 以及常见的 Ascend 设备权限模型保持一致。

2. InferenceService 示例所需更改

HOME 和 USER 环境变量已在上面的 ClusterServingRuntime 中设置,因此使用该 runtime 的每个 service 都会继承它们。使用 vLLM-ascend 发布 inference service 时,请根据 Modelcar 权限模式选择相应的 InferenceService security context。

对于非 root 模式,添加 fsGroup: 1000 和 supplementalGroups: [1000],以便 service 能通过预期的组权限访问 Ascend device 文件:

kind: InferenceService
apiVersion: serving.kserve.io/v1beta1
metadata:
  name: qwen35
  namespace: demo
  annotations:
    aml-model-repo: Qwen3.5-0.8B
    modelFormat: transformers
    serving.kserve.io/deploymentMode: Standard
  labels:
    aml.cpaas.io/runtime-type: vllm
spec:
  predictor:
    model:
      modelFormat:
        name: transformers
      protocolVersion: v2
      resources:
        limits:
          cpu: "4"
          huawei.com/Ascend910B4: "1"
          memory: 16Gi
        requests:
          cpu: "2"
          memory: 8Gi
      runtime: aml-vllm-ascend-0.18.0rc1
      storageUri: pvc://qwen35/Qwen3.5-0.8B
    securityContext:
      fsGroup: 1000
      seccompProfile:
        type: RuntimeDefault
      supplementalGroups:
        - 1000
  1. 根据环境中的实际模型大小、runtime engine、硬件类型和预期工作负载设置 CPU limit。
  2. 根据环境中的实际模型大小、runtime engine、硬件类型和预期工作负载设置 memory limit。
  3. fsGroup: 1000 会使挂载文件继承组 1000,有助于将文件权限与允许访问 Ascend devices 的组对齐。
  4. supplementalGroups: [1000] 会将容器进程加入组 1000,从而可以通过预期的组权限访问 Ascend devices 及相关挂载文件。

对于 root-mode Modelcar deployments,不需要 UID 1000 组设置。请保持 InferenceService security context 尽可能精简:

spec:
  predictor:
    model:
      runtime: aml-vllm-ascend-0.18.0rc1
      storageUri: oci://<registry>/<repository>:<tag>
    securityContext:
      seccompProfile:
        type: RuntimeDefault

MindIE(Ascend NPU)

MindIE 专为 Huawei Ascend 硬件设计。它在资源管理和元数据方面的配置差异很大。

此示例已在 Ascend 310P 上验证。它也应适用于其他 Ascend NPU 型号,但你应根据实际环境调整 image、资源配置和相关 version 字段。

1.ClusterServingRuntime

# This is a sample YAML for Ascend NPU runtime
kind: ClusterServingRuntime
apiVersion: serving.kserve.io/v1alpha1
metadata:
  annotations:
    cpaas.io/display-name: mindie-2.2RC1
  labels:
    cpaas.io/accelerator-type: npu
    cpaas.io/cann-version: 8.3.0
    cpaas.io/runtime-class: mindie
  name: mindie-2.2rc1-310p
spec:
  containers:
    - command:
        - bash
        - -c
        - |
          REAL_SCRIPT=$(echo "$RAW_SCRIPT" | sed 's/__LT__/\x3c/g')
          echo "$REAL_SCRIPT" > /tmp/startup.sh
          chmod +x /tmp/startup.sh
          
          if [ ! -d "${MODEL_PATH}" ]; then
              MODEL_DIR="/mnt/models"
              echo "[WARNING] Model directory ${MODEL_PATH} not found, using ${MODEL_DIR} instead"
              MODEL_PATH="${MODEL_DIR}"
          fi

          CONFIG_FILE="${MODEL_PATH}/config.json"
          echo "Checking for file: ${CONFIG_FILE}"

          ls -ld "${MODEL_PATH}"
          chmod -R 755 "${MODEL_PATH}"
          echo "Fixing MODEL_PATH permission..."
          ls -ld "${MODEL_PATH}"

          /tmp/startup.sh --model-name "${MODEL_NAME}" --model-path "${MODEL_PATH}" --ip "${MY_POD_IP}"
      env:
        - name: RAW_SCRIPT
          value: |
            #!/bin/bash
            #
            #  Copyright 2024 Huawei Technologies Co., Ltd
            #
            #  Licensed under the Apache License, Version 2.0 (the "License");
            #  you may not use this file except in compliance with the License.
            #  You may obtain a copy of the License at
            #
            #  http://www.apache.org/licenses/LICENSE-2.0
            #
            #  Unless required by applicable law or agreed to in writing, software
            #  distributed under the License is distributed on an "AS IS" BASIS,
            #  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
            #  See the License for the specific language governing permissions and
            #  limitations under the License.
            #  ============================================================================
            #

            ##
            # Script Instruction
            ##
            ### Name:
            ### run_mindie.sh - Use to Start MindIE Service given a specific model
            ###
            ### Usage:
            ###   bash run_mindie.sh --model-name xxx --model-path /path/to/model
            ###
            ### Required:
            ###   --model-name             :Given a model name to identify MindIE Service.
            ###   --model-path             :Given a model path which contain necessary files such as yaml/conf.json/tokenizer/vocab etc.
            ### Options:
            ###   --help                   :Show this message.
            ###   --ip                     :The IP address bound to the MindIE Server business plane RESTful interface,default value: 127.0.0.1.
            ###   --port                   :The port bound to the MindIE Server business plane RESTful interface,default value: 1025.
            ###   --management-ip          :The IP address bound to the MindIE Server management plane RESTful interface,default value: 127.0.0.2.
            ###   --management-port        :The port bound to the MindIE Server management plane RESTful interface,default value: 1026.
            ###   --metrics-port           :The port bound to the performance indicator monitoring interface,default value: 1027.
            ###   --max-seq-len            :Maximum sequence length,default value: 2560.
            ###   --max-iter-times         :The global maximum output length of the model,default value: 512.
            ###   --max-input-token-len    :The maximum length of the token id,default value: 2048.
            ###   --max-prefill-tokens     :Each time prefill occurs, the total number of input tokens in the current batch,default value: 8192
            ###   --truncation             :Whether to perform parameter rationalization check interception,default value: false.
            ###   --template-type          :Reasoning type,default value: "Standard"
            ###   --max-preempt-count      :The upper limit of the maximum number of preemptible requests in each batch,default value: 0.
            ###   --support-select-batch   :Batch selection strategy,default value: false.
            ###   --npu-mem-size           :This can be used to apply for the upper limit of the KV Cache size in the NPU,default value: 8.
            ###   --max-prefill-batch-size :The maximum prefill batch size,default value: 50.
            ###   --world-size             :Enable several cards for inference.
            ###                             1. If it is not set, the parallel config in the YAML file is obtained by default. Set worldsize = dp*mp*pp.
            ###                             2. If set, modify the parallel config in the YAML file. set parallel config: dp:1 mp:worldSize pp:1
            ###   --ms-sched-host          :MS Scheduler IP address,default value: 127.0.0.1.
            ###   --ms-sched-port          :MS Scheduler port,default value: 8119.
            ###   For more details about config description, please check MindIE homepage: https://www.hiascend.com/document/detail/zh/mindie/10RC3/mindiellm/llmdev/mindie_llm0004.html
            help() {
              awk -F'### ' '/^###/ { print $2 }' "$0"
            }

            if [[ $# == 0 ]] || [[ "$1" == "--help" ]]; then
              help
              exit 1
            fi

            ##
            # Get device info
            ##
            total_count=$(npu-smi info -l | grep "Total Count" | awk -F ':' '{print $2}' | xargs)

            if [[ -z "$total_count" ]]; then
                echo "Error: Unable to retrieve device info. Please check if npu-smi is available for current user (id 1001), or if you are specifying an occupied device."
                exit 1
            fi

            echo "$total_count device(s) detected!"

            ##
            # Set toolkit envs
            ##
            echo "Setting toolkit envs..."
            if [[ -f "/usr/local/Ascend/ascend-toolkit/set_env.sh" ]];then
                            source /usr/local/Ascend/ascend-toolkit/set_env.sh
                    else
                            echo "ascend-toolkit package is incomplete please check it."
                            exit 1
                    fi
            echo "Toolkit envs set succeeded!"

            ##
            # Set MindIE envs
            ##
            echo "Setting MindIE envs..."
            if [[ -f "/usr/local/Ascend/mindie/set_env.sh" ]];then
                            source /usr/local/Ascend/mindie/set_env.sh
                    else
                            echo "mindie package is incomplete please check it."
                            exit 1
                    fi
            echo "MindIE envs set succeeded!"

            ##
            # Default MS envs
            ##

            # Set PYTHONPATH
            MF_SCRIPTS_ROOT=$(realpath "$(dirname "$0")")
            export PYTHONPATH=$MF_SCRIPTS_ROOT/../:$PYTHONPATH

            ##
            # Receive args and modify config.json
            ##
            export MIES_INSTALL_PATH=/usr/local/Ascend/mindie/latest/mindie-service
            CONFIG_FILE=${MIES_INSTALL_PATH}/conf/config.json
            echo "MindIE Service config path:$CONFIG_FILE"
            #default config
            BACKEND_TYPE="atb"
            MAX_SEQ_LEN=2560
            MAX_PREFILL_TOKENS=8192
            MAX_ITER_TIMES=512
            MAX_INPUT_TOKEN_LEN=2048
            TRUNCATION=false
            HTTPS_ENABLED=false
            MULTI_NODES_INFER_ENABLED=false
            NPU_MEM_SIZE=8
            MAX_PREFILL_BATCH_SIZE=50
            TEMPLATE_TYPE="Standard"
            MAX_PREEMPT_COUNT=0
            SUPPORT_SELECT_BATCH=false
            IP_ADDRESS="127.0.0.1"
            PORT=8080
            MANAGEMENT_IP_ADDRESS="127.0.0.2"
            MANAGEMENT_PORT=1026
            METRICS_PORT=1027

            #modify config
            while [[ "$#" -gt 0 ]]; do
                case $1 in
                    --model-path) MODEL_WEIGHT_PATH="$2"; shift ;;
                    --model-name) MODEL_NAME="$2"; shift ;;
                    --max-seq-len) MAX_SEQ_LEN="$2"; shift ;;
                    --max-iter-times) MAX_ITER_TIMES="$2"; shift ;;
                    --max-input-token-len) MAX_INPUT_TOKEN_LEN="$2"; shift ;;
                    --max-prefill-tokens) MAX_PREFILL_TOKENS="$2"; shift ;;
                    --truncation) TRUNCATION="$2"; shift ;;
                    --world-size) WORLD_SIZE="$2"; shift ;;
                    --template-type) TEMPLATE_TYPE="$2"; shift ;;
                    --max-preempt-count) MAX_PREEMPT_COUNT="$2"; shift ;;
                    --support-select-batch) SUPPORT_SELECT_BATCH="$2"; shift ;;
                    --npu-mem-size) NPU_MEM_SIZE="$2"; shift ;;
                    --max-prefill-batch-size) MAX_PREFILL_BATCH_SIZE="$2"; shift ;;
                    --ip) IP_ADDRESS="$2"; shift ;;
                    --port) PORT="$2"; shift ;;
                    --management-ip) MANAGEMENT_IP_ADDRESS="$2"; shift ;;
                    --management-port) MANAGEMENT_PORT="$2"; shift ;;
                    --metrics-port) METRICS_PORT="$2"; shift ;;
                    --ms-sched-host) ENV_MS_SCHED_HOST="$2"; shift ;;
                    --ms-sched-port) ENV_MS_SCHED_PORT="$2"; shift ;;
                    *)
                        echo "Unknown parameter: $1"
                        echo "Please check your inputs."
                        exit 1
                        ;;
                esac
                shift
            done

            if [ -z "$MODEL_WEIGHT_PATH" ] || [ -z "$MODEL_NAME" ]; then
                echo "Error: Both --model-path and --model-name are required."
                exit 1
            fi
            MODEL_NAME=${MODEL_NAME:-$(basename "$MODEL_WEIGHT_PATH")}
            echo "MODEL_NAME is set to: $MODEL_NAME"

            WORLD_SIZE=$total_count
            NPU_DEVICE_IDS=$(seq -s, 0 $(($WORLD_SIZE - 1)))

            #validate config
            if [[ "$BACKEND_TYPE" != "atb" ]]; then
                echo "Error: BACKEND must be 'atb'. Current value: $BACKEND_TYPE"
                exit 1
            fi

            if [[ ! "$IP_ADDRESS" =~ ^([0-9]{1,3}\.){3}[0-9]{1,3}$ ]] ||
              [[ ! "$MANAGEMENT_IP_ADDRESS" =~ ^([0-9]{1,3}\.){3}[0-9]{1,3}$ ]]; then
                echo "Error: IP_ADDRESS and MANAGEMENT_IP_ADDRESS must be valid IP addresses. Current values: IP_ADDRESS=$IP_ADDRESS, MANAGEMENT_IP_ADDRESS=$MANAGEMENT_IP_ADDRESS"
                exit 1
            fi

            if [[ ! "$PORT" =~ ^[0-9]+$ ]] || (( PORT __LT__ 1025 || PORT > 65535 )) ||
              [[ ! "$MANAGEMENT_PORT" =~ ^[0-9]+$ ]] || (( MANAGEMENT_PORT __LT__ 1025 || MANAGEMENT_PORT > 65535 )); then
                echo "Error: PORT and MANAGEMENT_PORT must be integers between 1025 and 65535. Current values: PORT=$PORT, MANAGEMENT_PORT=$MANAGEMENT_PORT"
                exit 1
            fi

            if [ "$MAX_PREFILL_TOKENS" -lt "$MAX_SEQ_LEN" ]; then
                MAX_PREFILL_TOKENS=$MAX_SEQ_LEN
                echo "MAX_PREFILL_TOKENS was less than MAX_SEQ_LEN. Setting MAX_PREFILL_TOKENS to $MAX_SEQ_LEN"
            fi

            MODEL_CONFIG_FILE="${MODEL_WEIGHT_PATH}/config.json"
            if [ ! -f "$MODEL_CONFIG_FILE" ]; then
                echo "Error: config.json file not found in $MODEL_WEIGHT_PATH."
                exit 1
            fi
            chmod 600 "$MODEL_CONFIG_FILE"
            #update config file
            chmod u+w ${MIES_INSTALL_PATH}/conf/
            sed -i "s/\"backendType\"\s*:\s*\"[^\"]*\"/\"backendType\": \"$BACKEND_TYPE\"/" $CONFIG_FILE
            sed -i "s/\"modelName\"\s*:\s*\"[^\"]*\"/\"modelName\": \"$MODEL_NAME\"/" $CONFIG_FILE
            sed -i "s|\"modelWeightPath\"\s*:\s*\"[^\"]*\"|\"modelWeightPath\": \"$MODEL_WEIGHT_PATH\"|" $CONFIG_FILE
            sed -i "s/\"maxSeqLen\"\s*:\s*[0-9]*/\"maxSeqLen\": $MAX_SEQ_LEN/" "$CONFIG_FILE"
            sed -i "s/\"maxPrefillTokens\"\s*:\s*[0-9]*/\"maxPrefillTokens\": $MAX_PREFILL_TOKENS/" "$CONFIG_FILE"
            sed -i "s/\"maxIterTimes\"\s*:\s*[0-9]*/\"maxIterTimes\": $MAX_ITER_TIMES/" "$CONFIG_FILE"
            sed -i "s/\"maxInputTokenLen\"\s*:\s*[0-9]*/\"maxInputTokenLen\": $MAX_INPUT_TOKEN_LEN/" "$CONFIG_FILE"
            sed -i "s/\"truncation\"\s*:\s*[a-z]*/\"truncation\": $TRUNCATION/" "$CONFIG_FILE"
            sed -i "s|\(\"npuDeviceIds\"\s*:\s*\[\[\)[^]]*\(]]\)|\1$NPU_DEVICE_IDS\2|" "$CONFIG_FILE"
            sed -i "s/\"worldSize\"\s*:\s*[0-9]*/\"worldSize\": $WORLD_SIZE/" "$CONFIG_FILE"
            sed -i "s/\"httpsEnabled\"\s*:\s*[a-z]*/\"httpsEnabled\": $HTTPS_ENABLED/" "$CONFIG_FILE"
            sed -i "s/\"templateType\"\s*:\s*\"[^\"]*\"/\"templateType\": \"$TEMPLATE_TYPE\"/" $CONFIG_FILE
            sed -i "s/\"maxPreemptCount\"\s*:\s*[0-9]*/\"maxPreemptCount\": $MAX_PREEMPT_COUNT/" $CONFIG_FILE
            sed -i "s/\"supportSelectBatch\"\s*:\s*[a-z]*/\"supportSelectBatch\": $SUPPORT_SELECT_BATCH/" $CONFIG_FILE
            sed -i "s/\"multiNodesInferEnabled\"\s*:\s*[a-z]*/\"multiNodesInferEnabled\": $MULTI_NODES_INFER_ENABLED/" "$CONFIG_FILE"
            sed -i "s/\"maxPrefillBatchSize\"\s*:\s*[0-9]*/\"maxPrefillBatchSize\": $MAX_PREFILL_BATCH_SIZE/" "$CONFIG_FILE"
            sed -i "s/\"ipAddress\"\s*:\s*\"[^\"]*\"/\"ipAddress\": \"$IP_ADDRESS\"/" "$CONFIG_FILE"
            sed -i "s/\"port\"\s*:\s*[0-9]*/\"port\": $PORT/" "$CONFIG_FILE"
            sed -i "s/\"managementIpAddress\"\s*:\s*\"[^\"]*\"/\"managementIpAddress\": \"$MANAGEMENT_IP_ADDRESS\"/" "$CONFIG_FILE"
            sed -i "s/\"managementPort\"\s*:\s*[0-9]*/\"managementPort\": $MANAGEMENT_PORT/" "$CONFIG_FILE"
            sed -i "s/\"metricsPort\"\s*:\s*[0-9]*/\"metricsPort\": $METRICS_PORT/" $CONFIG_FILE
            sed -i "s/\"npuMemSize\"\s*:\s*-*[0-9]*/\"npuMemSize\": $NPU_MEM_SIZE/" "$CONFIG_FILE"

            ##
            # Start service
            ##
            echo "Current configurations are displayed as follows:"
            cat $CONFIG_FILE
            npu-smi info -m > ~/device_info

            ${MIES_INSTALL_PATH}/bin/mindieservice_daemon
        - name: MODEL_NAME
          value: '{{ index .Annotations "aml-model-repo" }}'
        - name: MODEL_PATH
          value: /mnt/models/{{ index .Annotations "aml-model-repo" }}
        - name: MY_POD_IP
          valueFrom:
            fieldRef:
              fieldPath: status.podIP
      image: swr.cn-south-1.myhuaweicloud.com/ascendhub/mindie:2.2.RC1-300I-Duo-py311-openeuler24.03-lts
      name: kserve-container
      resources:
        limits:
          cpu: 2
          memory: 6Gi
        requests:
          cpu: 2
          memory: 6Gi
      volumeMounts:
        - mountPath: /dev/shm
          name: dshm
      startupProbe:
        failureThreshold: 60
        httpGet:
          path: /v1/models
          port: 8080
          scheme: HTTP
        periodSeconds: 10
        timeoutSeconds: 180
  supportedModelFormats:
    - name: transformers
      version: "1"
  volumes:
    - emptyDir:
        medium: Memory
        sizeLimit: 8Gi
      name: dshm
      
  1. 根据环境中的实际模型大小、runtime engine、硬件类型和预期工作负载设置 CPU limit。
  2. 根据环境中的实际模型大小、runtime engine、硬件类型和预期工作负载设置 memory limit。

2.InferenceService 的必需注解

与其他 runtimes 不同,MindIE 在最终发布步骤中必须在 InferenceService 元数据中包含以下注解。这可确保平台调度器正确地将 NPU 硬件绑定到该 service。

配置键值作用
storage.kserve.io/readonly"false"启用对模型存储卷的写入访问。

3.用户权限(Root 访问)

由于 Ascend driver 和硬件抽象层的要求,MindIE 镜像必须以 root 用户运行。请确保你的 ClusterServingRuntime 或 InferenceService security context 已正确配置:

注意:上面的 MindIE ClusterServingRuntime YAML 示例未指定 securityContext,这意味着容器将使用镜像的默认设置运行(通常为 root)。不同于显式设置 runAsNonRoot: true 和 runAsUser: 1000 的 MLServer,MindIE 需要 root privileges 才能访问 NPU 硬件。

Runtime 配置对比

继续之前,请参考下表了解不同 runtimes 的具体要求:

Runtime目标硬件支持的 Frameworks特殊要求
XinferenceCPU / NVIDIA GPUtransformers, pytorch必须设置 MODEL_FAMILY 环境变量
MLServerCPU / NVIDIA GPUsklearn, xgboost, mlflow标准配置
TritonNVIDIA GPUtriton (TensorFlow, PyTorch, ONNX, etc.)标准配置
vLLM-ascendHuawei Ascend NPU(已在 910B4 上验证)transformers必须在 ClusterServingRuntime 中设置 HOME 和 USER,为 Ascend 910 单卡或多卡部署选择合适的 Modelcar 权限模式,并且仅在非 root 模式下添加 UID 1000 组设置
MindIEHuawei Ascend NPU(已在 310P 上验证)mindspore, transformers必须为 InferenceService 添加所需的 NPU 注解