如何定义自定义成本模型

本指南介绍如何通过定义自定义成本模型来添加新的计费项。工作流程分为四个部分:

  • 准备使用量指标
  • 准备标签指标
  • 为 Cost Management Agent 添加采集配置
  • 为 Cost Management Server 添加展示配置

作为一个具体示例,我们将添加两个 GPU 计费项:GPU Compute(cores) 和 GPU Memory,并启用按标签定价。

准备使用量指标

使用量指标表示需要计费的消耗量。请提供一个按步长聚合使用量的 PromQL 查询(例如 5m)。每个步长会生成一个数据点,表示前一个窗口内的使用量;最终使用量是所选时间范围内的总和。

Prometheus 指标类型包括 Counter、Gauge、Summary 和 Histogram。对于使用量,请使用 Counter 或 Gauge。示例:

# Counter example (e.g., GPU core usage time):
vgpu_core_usage_seconds_total{"containeridx":"0","deviceuuid":"GPU-2eec4202-80dc-870f-3ca5-25879d96eca7","endpoint":"monitor","instance":"10.3.0.130:9395","ip":"192.168.131.32","job":"hami-scheduler","namespace":"kube-system","node_name":"192.168.131.32","nodename":"192.168.133.48","pod":"hami-scheduler-86b7ff47c6-sp7kb","podname":"pytorch-cuda12-1-57d9ff4544-bqkvg","podnamespace":"yulin-2","service":"hami-scheduler","zone":"vGPU"}

# Corresponding usage query (step = 5m):
sum by (deviceuuid, podnamespace) (rate(vgpu_core_usage_seconds_total{}[5m]))


# Gauge example (e.g., GPU memory usage in bytes):
vgpu_memory_usage_bytes{"containeridx":"0","deviceuuid":"GPU-2eec4202-80dc-870f-3ca5-25879d96eca7","endpoint":"monitor","instance":"10.3.0.130:9395","ip":"192.168.131.32","job":"hami-scheduler","namespace":"kube-system","node_name":"192.168.131.32","nodename":"192.168.133.48","pod":"hami-scheduler-86b7ff47c6-sp7kb","podname":"pytorch-cuda12-1-57d9ff4544-bqkvg","podnamespace":"yulin-2","service":"hami-scheduler","zone":"vGPU"}

# Corresponding usage query (step = 5m):
sum by (deviceuuid, podnamespace) (avg_over_time(vgpu_memory_usage_bytes{}[5m]))

准备标签指标

标签指标为使用量附加属性,以便按标签定价。它们必须能够与使用量指标进行关联——至少共享一个标签(例如 deviceuuid)——这样系统才能使用这些属性来丰富使用量。

对于标签指标,请使用 Gauge。示例:

# Label metrics (label fields start with label_ in this example):
vgpu_device_labels{"deviceuuid":"GPU-2eec4202-80dc-870f-3ca5-25879d96eca7","label_device":"nvidia","label_modelName":"Tesla T4","podnamespace":"yulin-1"}

# Query for label metrics:
vgpu_device_labels{}

# How the usage and label queries relate (GPU core & memory example):
# 1) The usage query groups by (deviceuuid, podnamespace) and each series includes deviceuuid
# 2) The label query groups by its own labels and each series includes deviceuuid
# 3) Because both include the unique deviceuuid, labels from the label query can be attached to usage

添加采集配置(Cost Management Agent)

在运行 Cost Management Agent 的每个集群中创建一个 ConfigMap,用于声明要采集的内容。

一个典型的采集记录(GPU core/memory)与配置字段的映射关系如下:

{
    "id": "cab9881e380fcf72726ccf45565ffc2d",           # Auto-generated from agreed fields
    "kind": "Vgpu",                                     # Matches kind in config
    "name": "GPU-2eec4202-80dc-870f-3ca5-25879d96eca7", # From usage.mappers.name
    "namespace": "cpaas-system",                        # From usage.mappers.namespace
    "cluster": "default-cluster",                       # Cluster where the agent runs
    "project": "cpaas-system",                          # Derived from namespace
    "labels": {                                          # From labels.query if configured
        "key1": "val1",
        "key2": "val2"
    },
    "date": "2024-04-29T00:00:00Z",                     # Start of that day
    "period": "hourly",                                 # Matches period in config
    "start": "2024-04-29T00:05:00Z",                    # Period start
    "end": "2024-04-29T00:59:59Z",                      # Period end
    "category": "VgpuCore",                             # Matches category in config
    "item": "VgpuCoreUsed",                             # Matches item in config
    "usage": 200                                         # From usage query result
}

现在为 agent(GPU core 和 memory)创建 ConfigMap:

apiVersion: v1
data:
  config: |
    - kind: Vgpu                                        # Required; must be consistent with server config
      category: VgpuCore                                # Required; must be consistent with server config
      item: VgpuCoreUsed                                # Required and unique; must match server config
      period: Hourly                                    # Required; Hourly or Daily; prefer Hourly
      labels:                                           # Optional; enrich usage with labels from this query
        query: "vgpu_core_labels{}"                     # E.g., add GPU model, vendor to labels
        mappers:
          name: deviceuuid                              # Map deviceuuid as name
          namespace: podnamespace                       # Map podnamespace as namespace
          cluster: ""                                   # Leave empty to auto-fill current cluster
          project: ""                                   # Leave empty to auto-fill project from namespace
      usage:                                            # Required; usage query (grouped by deviceuuid,podnamespace)
        query: "sum by (deviceuuid, podnamespace) (rate(vgpu_core_usage_seconds_total{}[5m]))"
        step: 5m                                        # Required; step for sampling points
        mappers:
          name: deviceuuid                              # Map deviceuuid to name
          namespace: podnamespace                       # Map podnamespace to namespace
          cluster: ""                                   # Auto-fill if empty
          project: ""                                   # Auto-fill if empty
    - kind: Vgpu                                        # Second billing item
      category: VgpuMemory                              # Must be consistent with server config
      item: VgpuMemoryUsed                              # Unique item name
      period: Hourly
      labels:
        query: "vgpu_core_labels{}"
        mappers:
          name: deviceuuid
          namespace: podnamespace
          cluster: ""
          project: ""
      usage:
        query: "sum by (deviceuuid, podnamespace) (avg_over_time(vgpu_memory_usage_bytes{}[5m]))"
        step: 5m
        mappers:
          name: deviceuuid
          namespace: podnamespace
          cluster: ""
          project: ""
kind: ConfigMap
metadata:
  labels:
    cpaas.io/slark.collection.config: "true"            # Required; enables collection config
  name: slark-agent-project-config-vgpu
  namespace: cpaas-system                               # Required;

添加 yaml 后,需要重启 Agent Pod 以重新加载配置。

kubectl delete pods -n cpaas-system -l service_name=slark-agent

添加展示/存储配置(Cost Management Server)

在运行 Cost Management Server 的集群中创建一个 ConfigMap,用于声明计费项、计费方式、单位和显示名称。这会告诉服务器要计费什么,以及如何计费。

apiVersion: v1
data:
  config: |
    - name: VgpuCore                                     # Billing item name; must match category above
      displayname:
        zh: "Vgpu"
        en: "Vgpu"
      methods:                                           # List of billing methods (unique names)
        - name: Usage                                    # Method name
          displayname:
            zh: "使用量"
            en: "Used Usage"
          item: VgpuCoreUsed                             # Must match the agent config item
          divisor: 1000                                  # Unit conversion (e.g., mCPU to cores)
          unit:
            zh: "core-hours"
            en: "core-hours"
    - name: VgpuMemory                                   # Second billing item
      displayname:
        zh: "Vgpu 显存"
        en: "VgpuMemory"
      methods:
        - name: Used
          displayname:
            zh: "使用量"
            en: "Used Usage"
          item: VgpuMemoryUsed
          divisor: 1073741824                            # bytes -> Gi
          unit:
            zh: "Gi-hours"
            en: "Gi-hours"
kind: ConfigMap
metadata:
  labels:
    cpaas.io/slark.display.config: "true"                # Required; enables display/storage config
  name: slark-display-config-for-vgpu
  namespace: kube-public                                 # Required;

添加 yaml 后,需要重启 Server Pod 以重新加载配置。

kubectl delete pods -n cpaas-system -l service_name=slark-server

注意事项和最佳实践

  • 保持 agent 和 server 配置中的命名一致:kindcategoryitem 必须匹配。
  • 为了获得更细的粒度和更快的反馈,建议优先使用 Hourly。
  • 确保标签指标可以通过共享标签与使用量关联(例如 deviceuuid)。
  • 在发布前先在本地验证 PromQL;在初次运行时监控数据是否正常。
  • 先从小规模开始(少量集群/计费项),验证后再扩展。