Tekton 组件的指标采集

概述

Tekton 组件通过 HTTP 端点公开 Prometheus 兼容的指标。通过部署 ServiceMonitor 资源,Prometheus(或 VictoriaMetrics)可以自动发现并抓取这些指标。

命名空间说明:本文档使用 tekton-pipelines 作为控制平面组件(Pipelines、Triggers、Results、Chains)的默认命名空间。 主要例外是 EventListener Services,它们运行在创建 EventListener 的应用命名空间中。

如果您的部署使用不同的命名空间,请同时更新以下命令以及下方 ServiceMonitor 资源中的 namespaceSelector 字段。

本文档介绍以下 Tekton 组件的指标:

  • Tekton Pipelines - PipelineRun / TaskRun 执行指标
  • Tekton Triggers - EventListener、TriggerBinding 及相关资源指标
  • Tekton Results - 运行删除和存储指标
  • Tekton Chains - 签名和来源证明指标
  • Controller Framework - 所有 controller 共享的基础设施指标

本文档还介绍:

  • 如何通过 config-observability 配置指标行为
  • 如何部署用于抓取指标的 ServiceMonitor 资源
  • 如何验证指标采集是否正常工作

先决条件

  • Tekton 控制平面组件已安装并运行(至少包括您计划抓取的组件:Pipelines、Triggers、Results 和/或 Chains)。
  • kubectl 已配置为连接目标集群,并且您的账户可以在创建资源的命名空间中创建 ServiceMonitor 资源(本页面中的示例使用 tekton-pipelines)。
  • 已部署监控栈(Prometheus 或兼容的 VictoriaMetrics),并且能够发现/抓取 ServiceMonitor 资源(或平台使用的等效抓取发现对象)。
  • 您的 Prometheus/VictoriaMetrics 实例已配置为发现该命名空间中的 ServiceMonitor 对象(命名空间和标签选择器必须匹配)。
  • 网络策略和防火墙允许抓取器 Pod 访问 Tekton 指标端口(大多数控制平面服务使用 9090,Triggers controller 和 EventListener sink 使用 9000)。
  • 如果您需要 EventListener sink 指标,则目标命名空间中必须存在 EventListener,并公开 http-metrics 端口。
  • 用于直接检查指标端点的 curl CLI,以及用于 runs_not_stored_count 复现的恢复步骤的 jq CLI。在隔离网络的工作站上,请从工作站可以访问的操作系统软件包仓库安装它们,或从已联网的计算机复制二进制文件。
  • 要部署监控面板示例中的监控面板,您的账户必须能够在 cpaas-system 命名空间中创建 MonitorDashboard 资源(ait.alauda.io/v1alpha2)。
  • 更改 config-observability 需要具备更新集群范围 TektonConfig 的权限。在 tekton-pipelines 中读取 ConfigMap 和重启 Pod,以及在那里执行 kubectl port-forward 检查,都需要该命名空间中的相应权限(端口转发需要 pods/portforward)。这些步骤通常由集群管理员执行。直接检查指标端点中的 EventListener sink 检查会在 EventListener 的命名空间中执行端口转发,因此还需要在该命名空间中具备 pods/portforward 权限。

Tekton Pipelines

Tekton Pipelines 组件包含多个在端口 9090 上公开指标的子服务:

服务描述指标端口
tekton-pipelines-controllerPipelineRun / TaskRun 的主协调器9090
tekton-pipelines-webhook准入 webhook9090
tekton-events-controllerCloudEvents controller9090
tekton-pipelines-remote-resolvers远程资源解析9090

Pipeline controller 指标使用前缀 tekton_pipelines_controller_。

PipelineRun 指标

指标名称类型描述标签
pipelinerun_duration_secondsHistogram / LastValuePipelineRun 执行时间(以秒为单位)status、namespace、pipeline*、pipelinerun*、reason*
pipelinerun_totalCounter已完成 PipelineRun 的总数status
running_pipelinerunsLastValue (Gauge)当前正在运行的 PipelineRun 数量由 metrics.running-pipelinerun.level 控制(见下文)
running_pipelineruns_waiting_on_pipeline_resolutionLastValue (Gauge)等待解析 Pipeline 引用的 PipelineRun 数量-
running_pipelineruns_waiting_on_task_resolutionLastValue (Gauge)等待解析 Task 引用的 PipelineRun 数量-

* 标有 * 的标签为可选标签,具体取决于 config-observability 配置。

running_pipelineruns 标签级别

running_pipelineruns 指标的标签由 metrics.running-pipelinerun.level 控制:

级别标签
""(默认,集群级别)无标签
"namespace"namespace
"pipeline"namespace、pipeline
"pipelinerun"namespace、pipeline、pipelinerun

状态标签值

对于 PipelineRun 指标:

  • success - PipelineRun 成功完成
  • failed - PipelineRun 因 Cancelled 之外的原因而失败,包括因 CancelledRunFinally 或 StoppedRunFinally 而取消或停止、且其中某个 task 失败的运行
  • cancelled - PipelineRun 以原因为 Cancelled 结束

对于 TaskRun 指标:

  • success - TaskRun 成功完成
  • failed - TaskRun 因 TaskRunCancelled 之外的原因而失败
  • cancelled - TaskRun 以原因为 TaskRunCancelled 结束

TaskRun 指标

指标名称类型描述标签
taskrun_duration_secondsHistogram / LastValue独立 TaskRun 的执行时间(以秒为单位)status、namespace、task*、taskrun*、reason*
pipelinerun_taskrun_duration_secondsHistogram / LastValue作为 PipelineRun 一部分时的 TaskRun 执行时间status、namespace、task*、taskrun*、pipeline*、pipelinerun*、reason*
taskrun_totalCounter已完成 TaskRun 的总数status
running_taskrunsLastValue (Gauge)当前正在运行的 TaskRun 数量-
running_taskruns_waiting_on_task_resolution_countLastValue (Gauge)等待解析 Task 引用的 TaskRun 数量-
running_taskruns_throttled_by_quotaLastValue (Gauge)受 ResourceQuota 限制的 TaskRun 数量。没有 TaskRun 受到限制时,不会出现在指标输出中namespace*
running_taskruns_throttled_by_nodeLastValue (Gauge)受节点级资源约束限制的 TaskRun 数量。没有 TaskRun 受到限制时,不会出现在指标输出中namespace*
taskruns_pod_latency_millisecondsHistogramTaskRun 的 Pod 调度延迟(以毫秒为单位)namespace、task*、taskrun*

config-observability 配置

tekton-pipelines 命名空间中的 config-observability ConfigMap 控制 Pipeline controller 的指标行为。此 ConfigMap 由 Tekton Operator 管理,应通过 TektonConfig 资源的 spec.pipeline.options.configMaps 字段进行配置。有关详细信息,请参阅调整子组件的可选配置项。直接编辑 ConfigMap 也会在 controller 中生效,并且不会立即被还原,因此这种编辑很容易被忽略;请将值保存在 TektonConfig 中,operator 会将其应用到 ConfigMap。TektonConfig 也会在 spec.pipeline 下直接为其中一些键提供字段,每个字段都使用完整键名(平面键,而不是嵌套路径)命名,例如 spec.pipeline 下的键 metrics.pipelinerun.level;虽然 options.configMaps.config-observability 设置了同一个键,但 ConfigMap 会获得 options.configMaps 值。

重新加载行为:Pipeline controller 会在运行时监视 config-observability。更改级别键(metrics.pipelinerun.level、metrics.taskrun.level 和 metrics.running-pipelinerun.level)以及标签开关 metrics.count.enable-reason 和 metrics.taskrun.throttle.enable-namespace 后,无需重启即可生效。更改 metrics.pipelinerun.level 或 metrics.taskrun.level 后,会显示在更改完成后结束的运行的指标中。请等待一到两个抓取间隔,以便监控面板/查询中显示更改。更改 metrics.pipelinerun.duration-type 或 metrics.taskrun.duration-type 需要重启 tekton-pipelines-controller Pod。更改后立即重启 Pod。如果不重启,正在运行的 controller 会继续使用旧类型的 instrument,并且在下一次运行完成后,指标端点会返回 HTTP 500(响应正文为 An error has occurred while serving metrics: 和 invalid metric);重启 Pod 后指标才会恢复。导出器键 metrics-protocol、metrics-endpoint 和 metrics-export-interval 仅在 controller 启动时读取,因此更改这些键也只有在 Pod 重启后才会生效;重启会短暂暂停 PipelineRun 和 TaskRun 的协调。更改 Pod 规范设置时也需要重启(例如更改 Deployment 中的 CONFIG_OBSERVABILITY_NAME)。

集群管理员通过删除 controller 的 Pod 来重启 controller;Deployment 会创建新的 Pod。然后列出该 Pod:

kubectl delete pods -n tekton-pipelines -l app=tekton-pipelines-controller
kubectl get pods -n tekton-pipelines -l app=tekton-pipelines-controller

新的 Pod 必须处于 Running 状态,并且其 AGE 应短于重启至今经过的时间。执行 metrics-protocol: prometheus,然后按照直接检查指标端点中的说明检查指标。

通过 TektonConfig 配置的示例:

apiVersion: operator.tekton.dev/v1alpha1
kind: TektonConfig
metadata:
  name: config
spec:
  pipeline:
    options:
      disabled: false
      configMaps:
        config-observability:
          data:
            metrics-protocol: prometheus

            # PipelineRun metrics aggregation level.
            # Values: "pipelinerun" | "pipeline" (default) | "namespace"
            #   - "pipelinerun": includes pipeline + pipelinerun labels (new series for every run; with histogram, one per bucket)
            #   - "pipeline": includes pipeline label only
            #   - "namespace": no pipeline/pipelinerun labels
            metrics.pipelinerun.level: "pipeline"

            # TaskRun metrics aggregation level.
            # Values: "taskrun" | "task" (default) | "namespace"
            #   - "taskrun": includes task + taskrun labels (new series for every run; with histogram, one per bucket)
            #   - "task": includes task label only
            #   - "namespace": no task/taskrun labels
            metrics.taskrun.level: "task"

            # Duration metric type for PipelineRun / TaskRun.
            # Values: "histogram" (default) | "lastvalue"
            # Note: The level settings do not change the duration type; it always follows
            #       metrics.*.duration-type. The "pipelinerun" and "taskrun" levels add per-run
            #       labels and significantly increase the number of time series.
            metrics.pipelinerun.duration-type: "histogram"
            metrics.taskrun.duration-type: "histogram"

            # Running PipelineRun metrics aggregation level.
            # Values: "pipelinerun" | "pipeline" | "namespace" | "" (default, cluster-level)
            metrics.running-pipelinerun.level: ""

            # Include reason label on duration metrics (pipelinerun_duration_seconds,
            # taskrun_duration_seconds, pipelinerun_taskrun_duration_seconds).
            # Values: "true" | "false" (default)
            # Warning: Enabling this increases label cardinality.
            # Note: Despite the key name, this does NOT affect count metrics
            # (pipelinerun_total / taskrun_total), only duration metrics.
            metrics.count.enable-reason: "false"

            # Include namespace label on throttled TaskRun metrics.
            # Values: "true" | "false" (default)
            metrics.taskrun.throttle.enable-namespace: "false"

将示例保存到诸如 tektonconfig-observability.yaml 的文件中。若只需更改部分键,请仅在文件中保留这些键,并按照示例将其嵌套在 spec.pipeline.options.configMaps.config-observability.data 下。将该文件合并到现有的 TektonConfig 中。合并补丁只会更改文件中的键,并保留 TektonConfig 的其他字段:

kubectl patch tektonconfig config --type merge --patch-file tektonconfig-observability.yaml

对于仅包含 TektonConfig 部分内容的文件,请勿使用 kubectl apply -f:后续对另一个部分文件执行 kubectl apply -f 时,会删除该文件设置但后续文件中未包含的字段。

合并更改后,检查值是否已写入 ConfigMap:

kubectl get configmap config-observability -n tekton-pipelines -o yaml

data 下、注释的 _example 块之外的键,必须显示您设置的值。

Histogram 存储桶

当持续时间类型为 histogram 时,将使用以下存储桶边界(以秒为单位):

10, 30, 60, 300, 900, 1800, 3600, 5400, 10800, 21600, 43200, 86400

这对应于:10s、30s、1m、5m、15m、30m、1h、1.5h、3h、6h、12h、24h。

对于生产环境,请使用聚合级别来控制标签基数。将以下键以及其后的调试示例中的键设置在上述示例的 spec.pipeline.options.configMaps.config-observability.data 下:

metrics.pipelinerun.level: "pipeline"
metrics.taskrun.level: "task"
metrics.pipelinerun.duration-type: "histogram"
metrics.taskrun.duration-type: "histogram"
metrics.count.enable-reason: "false"

如果需要逐运行粒度进行调试,请临时切换为:

metrics.pipelinerun.level: "pipelinerun"
metrics.taskrun.level: "taskrun"

请注意,这会显著增加时间序列的数量。切换回 pipeline 和 task 不会删除已经创建的逐运行序列;由于 controller 进程会将其保存在内存中,这些序列会一直保留在 controller 的指标输出中,直到 tekton-pipelines-controller Pod 重启。请在测试集群上使用这些级别,或仅在较短的调试窗口内使用。

Tekton Triggers

Tekton Triggers 组件从不同进程中公开两类指标。

Controller 指标(端口 9000)

Triggers controller 每 60 秒报告一次资源数量指标。

Service指标端口
tekton-triggers-controller9000

Triggers controller 指标使用前缀 controller_。

指标名称类型描述标签
eventlistener_countLastValue(Gauge)EventListener 资源数量-
triggerbinding_countLastValue(Gauge)TriggerBinding 资源数量-
clustertriggerbinding_countLastValue(Gauge)ClusterTriggerBinding 资源数量-
triggertemplate_countLastValue(Gauge)TriggerTemplate 资源数量-
clusterinterceptor_countLastValue(Gauge)ClusterInterceptor 资源数量-

EventListener Sink 指标

每个 EventListener Pod 都会公开其他 HTTP 和事件处理指标。这些指标来自 EventListener sink 进程(而非 controller)。Prometheus 指标前缀为 eventlistener_。

指标名称(Prometheus)类型描述标签
eventlistener_http_duration_secondsHistogramEventListener HTTP 请求持续时间-
eventlistener_event_received_totalCountersink 接收的事件总数status
eventlistener_triggered_resources_totalCountertriggers 创建的资源总数kind
  • eventlistener_http_duration_seconds 直方图桶:0.001, 0.01, 0.1, 1, 5, 10(秒)
  • eventlistener_event_received_total 状态值:succeeded、failed
  • eventlistener_triggered_resources_total 类型值:所创建对象的 Kubernetes 资源 Kind(例如 PipelineRun、TaskRun)

这些 sink 指标按 EventListener Pod 公开,而不是来自中央 controller。如果 EventListener Pod 公开了指标端口,可能需要单独的 ServiceMonitor 或 PodMonitor 来抓取这些指标。EventListener 的 Service 会在事件端口旁发布此指标端口(9000、http-metrics);在使用 NodePort 或 LoadBalancer Service 之前,请参阅网络配置。


Tekton Results

Tekton Results 包含两个公开指标的子服务。

服务描述指标端口
tekton-results-watcher监视并清理 PipelineRun/TaskRun 资源9090
tekton-results-api-servicegRPC/REST API 服务器9090

Watcher 指标

Watcher 指标使用前缀 watcher_。Counter 指标使用后缀 _total 导出,例如 watcher_pipelinerun_delete_count_total 和 watcher_runs_not_stored_count_total。

删除指标

指标名称类型描述标签
pipelinerun_delete_countCounter已删除 PipelineRun 的总数status、namespace、pipeline*
pipelinerun_delete_duration_secondsHistogramPipelineRun 完成到删除之间的时间status、namespace、pipeline*
taskrun_delete_countCounter已删除 TaskRun 的总数status、namespace、pipeline*、task*
taskrun_delete_duration_secondsHistogramTaskRun 完成到删除之间的时间status、namespace、pipeline*、task*
customrun_delete_countCounter已删除 CustomRun 的总数status、namespace、pipeline*、customrun*
customrun_delete_duration_secondsHistogramCustomRun 完成到删除之间的时间status、namespace、pipeline*、customrun*

* 可选标签取决于 Results Watcher 的 config-observability 设置。

注意:只有在 Watcher 实际删除 runs 时,才会记录 pipelinerun_delete_count、pipelinerun_delete_duration_seconds、taskrun_delete_count、taskrun_delete_duration_seconds、customrun_delete_count 和 customrun_delete_duration_seconds。除非在 tekton-results-watcher Deployment 上将 --completed_run_grace_period 标志设置为非零值,否则这些指标将保持为空(没有数据点)(具体通过 TektonConfig 设置,在下方复现步骤所示的 watcher 容器的 args 中,并位于 spec.result.options.deployments.tekton-results-watcher 下)。默认情况下,此标志为 0,会禁用自动删除。正数时长(例如 10m)会在该宽限期后删除 runs;负值会立即删除 runs。Watcher 不会等待日志存储完成后再删除 run。

警告:任何非零的 --completed_run_grace_period 都会使 Watcher 删除其监视的每个命名空间中的已完成 PipelineRun、TaskRun 和 CustomRun,默认情况下为所有命名空间,包括其他团队的 runs。Watcher 不会删除具有 owner reference 的 run:属于 PipelineRun 的 TaskRun 和 CustomRun 始终交由其 PipelineRun 管理;而由任何其他对象拥有的 run,在 Watcher 标志 -check_owner 保持默认值 true 时会被保留(随附的 Watcher 参数不会设置该标志)。已删除的 runs 无法恢复。Watcher 的 -label_selector 标志会将删除范围限制为标签匹配的 runs。请仅在专用测试集群上尝试此标志,如下方复现步骤所示。

Results Watcher 的状态标签值:

  • success - Run 已成功完成
  • failed - Run 因以下取消原因之外的其他原因而失败
  • cancelled - Run 以原因 Cancelled(PipelineRun)、TaskRunCancelled(TaskRun)或 CustomRunCancelled(CustomRun)结束

共享指标

这些指标由 Watcher 中的 PipelineRun、TaskRun 和 CustomRun reconciler 注册,用于跟踪存储相关事件。

指标名称类型描述标签
runs_not_stored_countCounter未存储到 Results 就被删除的 runs 数量kind、namespace
run_storage_latency_secondsHistogramRun 完成到成功存储之间的时间kind、namespace

kind 标签用于标识 run 类型。runs_not_stored_count 使用 PipelineRun、TaskRun 和 CustomRun;run_storage_latency_seconds 使用小写值 pipelinerun、taskrun 和 customrun。使用 Watcher 标志 -logs_api=true(随附值)时,仅针对 CustomRun 记录 run_storage_latency_seconds。

注意:只有在 Watcher 持有 finalizer 以协调归档期间,run 被外部删除(例如通过 kubectl delete)时,才会记录 runs_not_stored_count。除非满足以下全部条件,否则该指标将保持为空:

  1. 对于 PipelineRun 和 TaskRun,--logs_api 标志为 false(日志存储已禁用);当 --logs_api 设置为 true 时,Watcher 会跳过针对 PipelineRun 和 TaskRun 的基于 finalizer 的协调。无论哪种情况,CustomRun 都会进行协调。
  2. --disable_crd_update 标志为 false(已启用 annotation 更新)。
  3. 自 run 完成以来,store_deadline 已经过。--store_deadline 标志始终会设置(默认值为 10m);这是 Watcher 在放弃归档并允许删除之前等待归档完成的最长时间。使用 0 时,run 完成后截止时间立即到期。
  4. Run 在完全不带 results.tekton.dev/stored annotation 的情况下被外部删除。仍在运行时已归档的 run 会带有 stored=false,即使其最终状态从未归档,也不会计入。

在正常运行中(在删除前已归档 runs,或 Watcher 自身通过 --completed_run_grace_period 触发删除),此 Counter 保持为零。非零值表示可能发生数据丢失:runs 在其状态保存到 Results API 之前就被删除。

快速复现(仅限专用测试集群): 如果看不到此指标,通常意味着未满足触发条件,而不是指标缺失。 TektonConfig 和 Results Watcher 作用于整个集群:当 Results API 缩放为 0 时,任何命名空间中的 run 都不会被归档,并且在此期间 store_deadline 之后删除的 run 永远不会被存储。请勿在其他团队使用的集群上执行这些步骤。

  1. 保存 TektonConfig 当前的 spec.result(kubectl get tektonconfig config -o jsonpath='{.spec.result}' > result-orig.json)。然后在 spec.result.options.deployments.tekton-results-watcher.spec.template.spec.containers 中设置 Watcher 标志:容器 watcher 使用 args、-logs_api=false、-disable_crd_update=false 以及一个 -store_deadline 值(例如 -store_deadline=30s)。spec.result.logs_api 只设置 Results API 的 LOGS_API 设置,不会更改 Watcher 的 -logs_api 标志。此更改会重启 Watcher:在执行第 2 步之前,等待新的 Watcher Pod 处于 Running(kubectl get pods -n tekton-pipelines -l app.kubernetes.io/name=tekton-results-watcher)状态,因为在 Results API 不可访问时启动的 Watcher 最多等待 1 分钟后就会退出。
  2. 通过 TektonConfig(spec.result.options.deployments.tekton-results-api.spec.replicas: 0)将 Results API 副本临时设置为 0,使 runs 无法归档。
  3. 创建 TaskRun 或 PipelineRun,并等待其完成。
  4. 等待 store_deadline 经过,然后从外部删除 run(kubectl delete taskrun <taskrun-name> -n <namespace> 或 kubectl delete pipelinerun <pipelinerun-name> -n <namespace>)。对于 TaskRun,Watcher 还会等待 run 完成后经过 -forward_buffer(默认值为 150s),因此删除操作和 Counter 可能需要这么长时间;在第 6 步恢复之前,请先检查第 5 步。
  5. 检查 Watcher /metrics 或 Prometheus 中的 watcher_runs_not_stored_count_total(公开名称:组件前缀加上 Counter 后缀 _total);其值应增加。
  6. 通过将 spec.result 替换为第 1 步保存的副本来恢复它,并在路径 /spec/result 上使用 JSON patch op: replace(jq '[{"op":"replace","path":"/spec/result","value":.}]' result-orig.json > restore-patch.json,然后是 kubectl patch tektonconfig config --type json --patch-file restore-patch.json)。Merge patch 不会删除第 1 步和第 2 步添加的键。然后检查 tekton-results-api Deployment 的副本数是否已恢复,并检查 watcher 容器的 args 是否与第 1 步之前相同(默认安装中为 -api_addr、-auth_mode 和 -logs_api=true),且不包含第 1 步中的 -logs_api=false、-disable_crd_update=false 和 -store_deadline 值。

run_storage_latency_seconds 直方图使用以下桶边界(以秒为单位):

0.1, 0.5, 1, 2, 5, 10, 30, 60, 120, 300, 600, 1800

Watcher config-observability

Results Watcher 有自己的 config-observability ConfigMap,即 tekton-results-config-observability(在 CONFIG_OBSERVABILITY_NAME 环境变量中设置的名称)。此 ConfigMap 由 Tekton Operator 管理,应通过 TektonConfig 资源的 spec.result.options.configMaps 字段,在 tekton-results-config-observability 键下进行配置。有关详细信息,请参阅调整子组件的可选配置项。

重新加载行为:Results Watcher 也会监视此 ConfigMap。在下表的键中,两个 level 键(metrics.pipelinerun.level 和 metrics.taskrun.level)的更改无需重启即可生效;两个 duration-type 键会被解析但不会使用,因此更改它们不会产生效果。exporter 键 metrics-protocol(随附值为 prometheus)、metrics-endpoint 和 metrics-export-interval 仅在 Watcher 启动时读取,因此只有在 tekton-results-watcher Pod 重启后,更改才会生效;重启会短暂暂停 Watcher 的 reconciliation,而该过程会将 runs 归档到 Results。更改 Deployment 级别的设置(例如环境变量或参数)时,也需要重启。

以 Pipelines ConfigMap 的相同方式将更改合并到 TektonConfig 中:将键保存在 spec.result.options.configMaps.tekton-results-config-observability.data 下的文件中,然后运行 kubectl patch tektonconfig config --type merge --patch-file <file>。之后检查这些值是否已传递到 Watcher 的 ConfigMap。data 下、注释掉的 _example 块之外的键必须显示你设置的值:

kubectl get configmap tekton-results-config-observability -n tekton-pipelines -o yaml

更改 exporter 键后,集群管理员删除 Watcher 的 Pod 以重启 Watcher,并检查新 Pod 是否为 Running,且其 AGE 小于重启后的时间:

kubectl delete pods -n tekton-pipelines -l app.kubernetes.io/name=tekton-results-watcher
kubectl get pods -n tekton-pipelines -l app.kubernetes.io/name=tekton-results-watcher

除上述介绍的 exporter 键 metrics-protocol、metrics-endpoint 和 metrics-export-interval 外,Watcher 的 ConfigMap 还支持以下 metrics.* 键:

键默认值值描述
metrics.pipelinerun.levelpipelinepipeline、namespace控制删除计数和持续时间指标上的 pipeline 标签
metrics.taskrun.leveltasktask、namespace控制 TaskRun 删除计数和持续时间指标上的 task 标签,以及 CustomRun 指标上的 customrun 标签
metrics.pipelinerun.duration-typehistogramhistogram、lastvalue已解析但未使用:Watcher 始终将删除持续时间记录为直方图
metrics.taskrun.duration-typehistogramhistogram、lastvalue已解析但未使用,原因同上

注意:与 Tekton Pipelines 不同,Results Watcher 不支持 pipelinerun / taskrun 单个 run 的粒度级别。它也没有 metrics.count.enable-reason、metrics.running-pipelinerun.level 或 metrics.taskrun.throttle.enable-namespace 键。

Watcher 将 pipelinerun_delete_duration_seconds、taskrun_delete_duration_seconds 和 customrun_delete_duration_seconds 创建为 OpenTelemetry 直方图,因此将任一 duration-type 键设置为 lastvalue 不会产生效果。

API Server 指标

API server 通过 go-grpc-prometheus 库在端口 9090 上公开标准 gRPC Prometheus 指标。这些指标包括:

  • grpc_server_handled_total - 服务器上完成的 RPC 总数
  • grpc_server_started_total - 服务器上启动的 RPC 总数
  • grpc_server_msg_received_total / grpc_server_msg_sent_total - 消息数量
  • grpc_server_handling_seconds - RPC 处理持续时间,仅在启用 PROMETHEUS_HISTOGRAM 时提供。随附的 tekton-results-api-config ConfigMap 设置了 PROMETHEUS_HISTOGRAM=false,因此 API server 默认不会公开此指标

Tekton Chains

Tekton Chains 是一个安全组件,用于为使用 Tekton Pipelines 构建的制品生成、签名和存储来源证明。它会观察已完成的 TaskRun 和 PipelineRun,然后创建证明和签名。

服务描述指标端口
tekton-chains-metricsChains watcher/controller9090 (http-metrics)

Chains controller 指标使用前缀 watcher_(与 Results Watcher 相同,但自定义指标名称不同,因此不会发生冲突)。

Chains 指标

所有 Chains 指标都是 Counter。只有签名失败计数器带有标签(error_type)。

指标名称(Prometheus)类型描述
watcher_taskrun_sign_created_totalCounterTaskRun 的已签名消息总数
watcher_taskrun_payload_uploaded_totalCounter上传到透明日志的 TaskRun 有效负载总数(仅在执行此类上传时记录)
watcher_taskrun_payload_stored_totalCounterTaskRun 的已存储有效负载总数
watcher_taskrun_marked_signed_totalCounter标记为已签名的 TaskRun 总数
watcher_pipelinerun_sign_created_totalCounterPipelineRun 的已签名消息总数
watcher_pipelinerun_payload_uploaded_totalCounter上传到透明日志的 PipelineRun 有效负载总数(仅在执行此类上传时记录)
watcher_pipelinerun_payload_stored_totalCounterPipelineRun 的已存储有效负载总数
watcher_pipelinerun_marked_signed_totalCounter标记为已签名的 PipelineRun 总数
watcher_taskrun_signing_failures_totalCounterTaskRun 的签名失败总数,按 error_type 标记(例如 storage 或 payload_creation)
watcher_pipelinerun_signing_failures_totalCounterPipelineRun 的签名失败总数,按 error_type 标记

注意:Chains 仅在配置了 signer 时才会对有效负载进行签名和存储;默认情况下,signer 使用 signing-secrets Secret 中的密钥。只有当 Chains 配置中的 generateSigningSecret 为 true 时,Operator 才会创建密钥(默认值为 false);请参阅签名密钥配置。如果没有密钥,Chains 会记录 No signer <signer> configured for <type>,跳过签名,不会将缺少密钥的失败计数添加到 *_signing_failures_total,但仍会将 run 标记为已签名:*_marked_signed_total 会增加,而 *_sign_created_total 和 *_payload_stored_total 不会增加。在 controller 启动后尚未签署任何内容时,这两个计数器根本不会出现在指标输出中,因为 Chains 计数器只有在首次递增后才会出现。

Controller Framework 指标

Tekton controller 还会公开以下 framework 指标。这些指标不使用组件前缀:所有组件中的名称都相同,因此请通过监控堆栈抓取指标时添加的 target 标签来区分组件。workqueue 指标带有 name 标签,用于标识 reconciler,例如 github.com.tektoncd.pipeline.pkg.reconciler.pipelinerun.Reconciler。Pipelines-as-Code controller 和 EventListener sink 仅公开两个 Kubernetes API client 指标。

指标名称(Prometheus)类型描述
http_client_request_duration_secondsHistogramKubernetes API client 请求持续时间(秒)
kn_k8s_client_http_response_status_code_totalCounterKubernetes API client 响应
kn_workqueue_depthGauge当前 workqueue 深度
kn_workqueue_adds_totalCounterworkqueue 添加总数
kn_workqueue_queue_duration_secondsHistogram项目在 workqueue 中等待的时间
kn_workqueue_process_duration_secondsHistogram处理 workqueue 项目所用的时间
kn_workqueue_retries_totalCounterworkqueue 重试总数
kn_workqueue_unfinished_work_secondsGauge未完成 workqueue 项目的持续时间

不存在带有组件前缀的 client_latency、reconcile_count 或 reconcile_latency 序列。


设置 ServiceMonitor

要启用对 Tekton 组件的 Prometheus 抓取,请部署 ServiceMonitor 资源。

前置条件请参阅前置条件。

请根据使用的监控堆栈遵循以下指导:

  • 如果使用 Prometheus(Prometheus Operator),则 metadata.labels.prometheus: kube-prometheus 等标签必须与 Prometheus CR spec.serviceMonitorSelector 匹配;否则不会抓取此 ServiceMonitor。
  • 如果使用 VictoriaMetrics,通常不需要 prometheus: kube-prometheus 等标签;请根据监控设置创建 ServiceMonitor/VMServiceScrape。

使用 Prometheus 时,请使用以下命令查找并验证 selector。这些命令需要对 Prometheus 资源具有读取权限(第一个命令会列出所有命名空间中的资源),通常需要集群管理员权限:

# 1) Locate Prometheus CRs (resource type: monitoring.coreos.com/v1, Kind=Prometheus)
kubectl get prometheus -A

# 2) Check ServiceMonitor selector on the target Prometheus instance
kubectl get prometheus -n <prometheus-namespace> <prometheus-name> -o jsonpath='{.spec.serviceMonitorSelector}'

# 3) Check the ServiceMonitor namespace selector on the same instance
kubectl get prometheus -n <prometheus-namespace> <prometheus-name> -o jsonpath='{.spec.serviceMonitorNamespaceSelector}'

如果第二个命令没有输出,则表示 Prometheus CR 未设置 serviceMonitorSelector。Prometheus Operator API 将未设置的 selector 定义为不匹配任何 ServiceMonitor,将空 selector({})定义为匹配所有资源。因此,在这种情况下,此 Prometheus 实例不会选择下面的 ServiceMonitor 资源,而不论这些资源的标签是什么。

如果第三个命令没有输出,则表示 Prometheus CR 未设置 serviceMonitorNamespaceSelector。Prometheus Operator API 将未设置的命名空间 selector 定义为仅匹配 Prometheus CR 所在的命名空间,将空 selector({})定义为匹配所有命名空间。因此,位于 tekton-pipelines 之外且未设置此字段的 Prometheus CR,不会选择下面的 ServiceMonitor 资源。

如果集群中不存在 Prometheus CR,监控通常由平台管理(例如 VictoriaMetrics)或通过其他方式实现。在这种情况下,通常不需要 prometheus: kube-prometheus 等标签;请遵循平台的抓取规则。

有关更多信息,请参阅 集成外部指标。

Pipeline ServiceMonitor

Pipeline ServiceMonitor YAML
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: tekton-pipelines-metrics
  namespace: tekton-pipelines
  labels:
    app.kubernetes.io/name: tekton-pipelines
    # prometheus: kube-prometheus
spec:
  selector:
    matchLabels:
      app.kubernetes.io/part-of: tekton-pipelines
  endpoints:
  - port: http-metrics
    path: /metrics
    interval: 30s
  namespaceSelector:
    matchNames:
    - tekton-pipelines

此 ServiceMonitor 会匹配带有 app.kubernetes.io/part-of: tekton-pipelines 标签的 Pipeline 服务(包括 remote-resolver),并在 tekton-pipelines 命名空间中抓取这些服务。

Triggers ServiceMonitor

Triggers ServiceMonitor YAML
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: tekton-triggers-metrics
  namespace: tekton-pipelines
  labels:
    app.kubernetes.io/name: tekton-triggers
    # prometheus: kube-prometheus
spec:
  selector:
    matchLabels:
      app.kubernetes.io/part-of: tekton-triggers
      app.kubernetes.io/component: controller
  endpoints:
  - port: http-metrics
    path: /metrics
    interval: 30s
  namespaceSelector:
    matchNames:
    - tekton-pipelines

此 ServiceMonitor 仅收集 Triggers controller 指标(controller_*)。它不包含 EventListener sink 指标。

EventListener Sink ServiceMonitor

EventListener Sink ServiceMonitor YAML
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: tekton-eventlistener-sink-metrics
  namespace: tekton-pipelines
  labels:
    app.kubernetes.io/name: tekton-eventlistener-sink
    # prometheus: kube-prometheus
spec:
  selector:
    matchExpressions:
    - key: eventlistener
      operator: Exists
    - key: app.kubernetes.io/managed-by
      operator: In
      values:
      - EventListener
  endpoints:
  - port: http-metrics
    path: /metrics
    interval: 30s
  namespaceSelector:
    any: true

EventListener Service 通常在应用命名空间中运行,因此此示例使用 namespaceSelector.any: true 进行跨命名空间抓取。如果需要更严格的范围,请切换为 matchNames,并显式列出允许的命名空间。

Results ServiceMonitor

Results 服务同时具有 app.kubernetes.io/part-of: tekton-results 和 app.kubernetes.io/name 标签。为了精确定位 API + Watcher(并排除 Postgres),此示例匹配 app.kubernetes.io/name:

Results ServiceMonitor YAML
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: tekton-results-metrics
  namespace: tekton-pipelines
  labels:
    app.kubernetes.io/name: tekton-results
    # prometheus: kube-prometheus
spec:
  selector:
    matchExpressions:
    - key: app.kubernetes.io/name
      operator: In
      values:
      - tekton-results-api
      - tekton-results-watcher
  endpoints:
  - port: prometheus
    path: /metrics
    interval: 30s
  - port: metrics
    path: /metrics
    interval: 30s
  namespaceSelector:
    matchNames:
    - tekton-pipelines

Results API server 使用端口名称 prometheus(9090),Watcher 使用端口名称 metrics(9090)。每个服务只公开其中一个端口名称,因此只会抓取匹配的 endpoint。

Chains ServiceMonitor

Chains ServiceMonitor YAML
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: tekton-chains-metrics
  namespace: tekton-pipelines
  labels:
    app.kubernetes.io/name: tekton-chains
    # prometheus: kube-prometheus
spec:
  selector:
    matchLabels:
      app.kubernetes.io/part-of: tekton-chains
  endpoints:
  - port: http-metrics
    path: /metrics
    interval: 30s
  namespaceSelector:
    matchNames:
    - tekton-pipelines

验证

部署 ServiceMonitor 资源后,验证 Prometheus 是否正在抓取目标。

直接检查指标端点

下面的每个 kubectl port-forward 命令都会在前台持续运行,直到使用 Ctrl+C 停止。请在第二个终端中运行并保持其运行,然后在第一个终端中运行后续的 curl 命令,最后停止端口转发。

这些检查假设每个组件只有一个副本:kubectl port-forward svc/<service> 会连接到 Service 的一个 Pod。当组件运行多个副本时,你连接到的 Pod 上可能不存在某个 counter;此时请对每个 Pod 将 svc/<service> 替换为 pod/<pod-name> 后重复检查,或者查询 Prometheus。

# Pipeline controller
$ kubectl port-forward -n tekton-pipelines svc/tekton-pipelines-controller 9090:9090
$ curl -s http://localhost:9090/metrics | grep tekton_pipelines_controller_

# HELP tekton_pipelines_controller_pipelinerun_total Number of pipelineruns
# TYPE tekton_pipelines_controller_pipelinerun_total counter
tekton_pipelines_controller_pipelinerun_total{otel_scope_name="tekton_pipelines_controller",otel_scope_schema_url="",otel_scope_version="",status="cancelled"} 11
tekton_pipelines_controller_pipelinerun_total{otel_scope_name="tekton_pipelines_controller",otel_scope_schema_url="",otel_scope_version="",status="failed"} 90

# Triggers controller
$ kubectl port-forward -n tekton-pipelines svc/tekton-triggers-controller 9000:9000
$ curl -s http://localhost:9000/metrics | grep controller_

# HELP controller_clusterinterceptor_count number of clusterinterceptor
# TYPE controller_clusterinterceptor_count gauge
controller_clusterinterceptor_count{otel_scope_name="github.com/tektoncd/triggers/pkg/reconciler/metrics",otel_scope_schema_url="",otel_scope_version=""} 5

# EventListener sink metrics (replace namespace/service)
$ kubectl port-forward -n <eventlistener-namespace> svc/<eventlistener-service> 9094:9000
$ curl -s http://localhost:9094/metrics | grep eventlistener_

# HELP eventlistener_event_received_total number of events received by sink
# TYPE eventlistener_event_received_total counter
eventlistener_event_received_total{otel_scope_name="github.com/tektoncd/triggers/pkg/sink",otel_scope_schema_url="",otel_scope_version="",status="succeeded"} 1

# HELP eventlistener_triggered_resources_total Count of the number of triggered eventlistener resources
# TYPE eventlistener_triggered_resources_total counter
eventlistener_triggered_resources_total{kind="TaskRun",otel_scope_name="github.com/tektoncd/triggers/pkg/sink",otel_scope_schema_url="",otel_scope_version=""} 1

# Results watcher
# With the shipped watcher argument -logs_api=true, the watcher records
# watcher_run_storage_latency_seconds only with kind="customrun", and can record
# watcher_runs_not_stored_count_total with kind="CustomRun" (when a CustomRun
# without the results.tekton.dev/stored annotation is deleted after
# store_deadline; see the runs_not_stored_count note); until a CustomRun
# completes, the grep prints nothing. The deletion metrics appear only when
# --completed_run_grace_period is set (see the note in the Tekton Results section).
$ kubectl port-forward -n tekton-pipelines svc/tekton-results-watcher 9091:9090
$ curl -s http://localhost:9091/metrics | grep watcher_

# HELP watcher_run_storage_latency_seconds time from run completion to successful storage
# TYPE watcher_run_storage_latency_seconds histogram
watcher_run_storage_latency_seconds_bucket{kind="customrun",namespace="<namespace>",otel_scope_name="tekton.dev/results",otel_scope_schema_url="",otel_scope_version="",le="1"} 0
watcher_run_storage_latency_seconds_bucket{kind="customrun",namespace="<namespace>",otel_scope_name="tekton.dev/results",otel_scope_schema_url="",otel_scope_version="",le="2"} 3
watcher_run_storage_latency_seconds_bucket{kind="customrun",namespace="<namespace>",otel_scope_name="tekton.dev/results",otel_scope_schema_url="",otel_scope_version="",le="5"} 3

# To tell an empty watcher_ output from a failed port-forward or endpoint, count
# the target_info line, which the endpoint returns with every scrape: the command
# prints 1 when the endpoint answered and 0 when it did not.
$ curl -s http://localhost:9091/metrics | grep -c '^target_info'

# Results API
$ kubectl port-forward -n tekton-pipelines svc/tekton-results-api-service 9092:9090
$ curl -s http://localhost:9092/metrics | grep grpc_server_

# HELP grpc_server_handled_total Total number of RPCs completed on the server, regardless of success or failure.
# TYPE grpc_server_handled_total counter
grpc_server_handled_total{grpc_code="Aborted",grpc_method="Check",grpc_service="grpc.health.v1.Health",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Aborted",grpc_method="CreateRecord",grpc_service="tekton.results.v1alpha2.Results",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Aborted",grpc_method="CreateResult",grpc_service="tekton.results.v1alpha2.Results",grpc_type="unary"} 0

# HELP grpc_server_started_total Total number of RPCs started on the server.
# TYPE grpc_server_started_total counter
grpc_server_started_total{grpc_method="Check",grpc_service="grpc.health.v1.Health",grpc_type="unary"} 337606
grpc_server_started_total{grpc_method="CreateRecord",grpc_service="tekton.results.v1alpha2.Results",grpc_type="unary"} 10301
grpc_server_started_total{grpc_method="CreateResult",grpc_service="tekton.results.v1alpha2.Results",grpc_type="unary"} 832

# Chains controller
$ kubectl port-forward -n tekton-pipelines svc/tekton-chains-metrics 9093:9090
$ curl -s http://localhost:9093/metrics | grep watcher_

# HELP watcher_pipelinerun_marked_signed_total Total number of objects marked as signed for pipelineruns
# TYPE watcher_pipelinerun_marked_signed_total counter
watcher_pipelinerun_marked_signed_total{otel_scope_name="github.com/tektoncd/chains/pkg/pipelinerunmetrics",otel_scope_schema_url="",otel_scope_version=""} 56

# A Chains counter appears only after its first increment, so the grep prints
# nothing until the controller has counted a run since it started. To tell that
# from a failed port-forward or endpoint, count the target_info line, which the
# endpoint returns with every scrape: the command prints 1 when the endpoint
# answered and 0 when it did not.
$ curl -s http://localhost:9093/metrics | grep -c '^target_info'

EventListener sink 指标(例如 eventlistener_event_received_total 和 eventlistener_http_duration_seconds)由请求驱动。在验证这些指标之前,至少向 EventListener 发送一次请求。

检查 Prometheus 目标

# Verify ServiceMonitor resources exist
$ kubectl get servicemonitor -n tekton-pipelines

NAME                                AGE
tekton-chains-metrics               10m
tekton-eventlistener-sink-metrics   10m
tekton-pipelines-metrics            10m
tekton-results-metrics              10m
tekton-triggers-metrics             10m

# Check Prometheus targets (via Prometheus UI or API)
# Look for targets whose job label is the name of a scraped Service:
# the samples set no spec.jobLabel, so Prometheus Operator uses the
# Service name as the job label, not the ServiceMonitor name

PromQL 查询示例

# PipelineRun cumulative success rate (avoids misinterpretation in empty completion windows)
100 * sum(tekton_pipelines_controller_pipelinerun_total{status="success"}) / clamp_min(sum(tekton_pipelines_controller_pipelinerun_total), 1)

# Completed PipelineRuns in the last 5 minutes (throughput)
round(sum(increase(tekton_pipelines_controller_pipelinerun_total[5m])))

# PipelineRun duration P95 (histogram mode)
histogram_quantile(0.95,
  rate(tekton_pipelines_controller_pipelinerun_duration_seconds_bucket[5m])
)

# TaskRun duration P95, standalone TaskRuns (histogram mode)
histogram_quantile(0.95,
  sum by (le) (rate(tekton_pipelines_controller_taskrun_duration_seconds_bucket[5m]))
)

# TaskRun duration P95, TaskRuns that run in a PipelineRun (histogram mode)
# Each TaskRun is recorded in only one of the two histograms; adding them with `+`
# returns nothing while either histogram has no series in the window.
histogram_quantile(0.95,
  sum by (le) (rate(tekton_pipelines_controller_pipelinerun_taskrun_duration_seconds_bucket[5m]))
)

# PipelineRun duration (lastvalue mode)
avg_over_time(tekton_pipelines_controller_pipelinerun_duration_seconds[5m])

# Currently running PipelineRuns (single series to avoid duplicate legends)
max(tekton_pipelines_controller_running_pipelineruns)

# TaskRuns throttled by resource quota (no data while no TaskRun is throttled).
# This query and the TaskRuns Throttled dashboard panel assume
# metrics.taskrun.throttle.enable-namespace: "false" (the default). With "true" the
# controller reports one series per namespace and no total, so max() returns the
# largest per-namespace count.
max(tekton_pipelines_controller_running_taskruns_throttled_by_quota)

# Trigger resource counts
controller_eventlistener_count
controller_triggertemplate_count

# Chains signing activity
watcher_taskrun_sign_created_total
watcher_pipelinerun_sign_created_total

推荐设置

启用时机。当平台或项目团队需要查看集群中的构建吞吐量和持续时间、签名活动以及事件驱动的触发情况时,启用指标收集。config-observability ConfigMap 控制共享的 Pipeline controller,并通过 TektonConfig 进行管理;控制平面的 ServiceMonitor 资源则以 tekton-pipelines namespace 为目标。

推荐值。从设置 ServiceMonitor部署 Pipeline、Triggers 和 Chains ServiceMonitor 资源;对于运行 EventListener 的 namespace,添加 EventListener sink ServiceMonitor;安装 Results 后,添加 Results ServiceMonitor。在 config-observability 中使用生产环境推荐配置中的值:metrics.pipelinerun.level: "pipeline"、metrics.taskrun.level: "task"、metrics.pipelinerun.duration-type: "histogram"、metrics.taskrun.duration-type: "histogram" 和 metrics.count.enable-reason: "false"。保留 histogram duration type,因为监控面板示例中的 P50/P95 duration 图表会查询 *_bucket series。仅在测试集群或短时间的调试窗口中使用 pipelinerun 和 taskrun levels,因为它们会添加每次运行的 labels,并显著增加 time series 的数量;duration type 仍遵循 metrics.*.duration-type。切换回 pipeline 和 task 不会移除已经创建的每次运行 series:它们会一直保留,直到 tekton-pipelines-controller Pod 重启。在 cpaas-system namespace 中的 tekton folder 下部署一次这些监控面板。

要确认结果,请检查值是否如config-observability 配置中所述达到 config-observability,端点是否能够响应,以及 Prometheus 是否如验证中所述抓取目标;同时,使用监控面板示例中的 kubectl get monitordashboard command 检查监控面板是否存在。在默认安装中,config-observability 已经具有推荐值,因此无需撤销这些设置;请按照生产环境推荐配置中的说明将调试 levels 切换回来。要移除监控设置,请删除你创建的 ServiceMonitor 和 MonitorDashboard 资源,并且只指定这些资源,例如:

kubectl delete servicemonitor -n tekton-pipelines tekton-pipelines-metrics tekton-triggers-metrics tekton-eventlistener-sink-metrics tekton-results-metrics tekton-chains-metrics
kubectl delete monitordashboard -n cpaas-system tekton-pipeline tekton-triggers tekton-results tekton-chains

删除这些资源需要在 tekton-pipelines 和 cpaas-system 中拥有相应资源的删除权限。

前提条件。Prometheus 或 VictoriaMetrics 监控栈必须能够发现你创建的 ServiceMonitor 资源;使用 Prometheus Operator 时,ServiceMonitor labels 必须与 Prometheus 的 spec.serviceMonitorSelector 匹配。指标抓取 Pod 必须能够访问大多数控制平面服务上的 9090 端口,以及 Triggers controller 和 EventListener sink 上的 9000 端口。完整列表请参阅前提条件,其中包括创建 ServiceMonitor 资源所需的权限,以及为监控面板在 cpaas-system 中创建 MonitorDashboard 资源所需的权限。


监控面板示例

以下 MonitorDashboard 资源提供了用于监控 Tekton 组件的开箱即用监控面板。将它们部署到 cpaas-system namespace,并置于 tekton folder 下。应用后,检查它们是否存在:

kubectl get monitordashboard -n cpaas-system -l cpaas.io/dashboard.folder=tekton

输出必须列出你应用的监控面板:tekton-pipeline、tekton-triggers、tekton-results 和 tekton-chains。No resources found in cpaas-system namespace. 表示其中没有任何一个被创建。

重要:每个 panel 必须包含 id(唯一整数)、datasource: prometheus 和 transformations: []。每个 target 必须包含 datasource: prometheus 和 refId。本文档中的 duration P50/P95 panel 使用 *_bucket 查询,并要求 metrics.*.duration-type=histogram;如果使用 lastvalue,请将这些查询替换为 LastValue 风格的表达式,例如 avg_over_time(...)。在 gridPos 中为 y key 加引号("y": 0);未加引号的 y 会被解析为布尔值,导致 panel 丢失其垂直位置。

Tekton Pipeline 监控面板

Tekton Pipeline Dashboard YAML
kind: MonitorDashboard
apiVersion: ait.alauda.io/v1alpha2
metadata:
  labels:
    cpaas.io/dashboard.folder: tekton
    cpaas.io/dashboard.is.home.dashboard: "false"
    cpaas.io/dashboard.tag.tekton: "true"
  name: tekton-pipeline
  namespace: cpaas-system
spec:
  body:
    title: Tekton Pipeline Overview
    titleZh: Tekton Pipeline Overview
    tags:
      - tekton
    time:
      from: now-1h
      to: now
    templating:
      list: []
    panels:
      - id: 1
        title: PipelineRun Total (by status)
        type: timeseries
        datasource: prometheus
        gridPos:
          h: 8
          w: 8
          x: 0
          "y": 0
        targets:
          - datasource: prometheus
            expr: sum by (status) (tekton_pipelines_controller_pipelinerun_total)
            refId: A
        fieldConfig:
          defaults:
            color:
              mode: palette-classic
            custom:
              drawStyle: line
              fillOpacity: 0
              lineWidth: 1
              spanNulls: false
            thresholds:
              mode: absolute
              steps:
                - color: green
                  value: null
          overrides: []
        options:
          legend:
            calcs:
              - latest
            displayMode: list
            placement: bottom
            showLegend: true
          tooltip:
            mode: multi
            sort: desc
        transformations: []
      - id: 2
        title: TaskRun Total (by status)
        type: timeseries
        datasource: prometheus
        gridPos:
          h: 8
          w: 8
          x: 8
          "y": 0
        targets:
          - datasource: prometheus
            expr: sum by (status) (tekton_pipelines_controller_taskrun_total)
            refId: A
        fieldConfig:
          defaults:
            color:
              mode: palette-classic
            custom:
              drawStyle: line
              fillOpacity: 0
              lineWidth: 1
              spanNulls: false
            thresholds:
              mode: absolute
              steps:
                - color: green
                  value: null
          overrides: []
        options:
          legend:
            calcs:
              - latest
            displayMode: list
            placement: bottom
            showLegend: true
          tooltip:
            mode: multi
            sort: desc
        transformations: []
      - id: 3
        title: PipelineRun Success Rate (cumulative)
        type: timeseries
        datasource: prometheus
        gridPos:
          h: 8
          w: 4
          x: 16
          "y": 0
        targets:
          - datasource: prometheus
            expr: "100 * sum(tekton_pipelines_controller_pipelinerun_total{status=\"success\"}) / clamp_min(sum(tekton_pipelines_controller_pipelinerun_total), 1)"
            refId: A
        fieldConfig:
          defaults:
            unit: percent
            color:
              mode: thresholds
            custom:
              drawStyle: line
              fillOpacity: 0
              lineWidth: 1
              spanNulls: false
            thresholds:
              mode: absolute
              steps:
                - color: red
                  value: null
                - color: orange
                  value: 80
                - color: green
                  value: 95
          overrides: []
        options:
          legend:
            calcs:
              - latest
            displayMode: list
            placement: bottom
            showLegend: true
          tooltip:
            mode: multi
            sort: desc
        transformations: []
      - id: 12
        title: Completed PipelineRuns (last 5m)
        type: timeseries
        datasource: prometheus
        gridPos:
          h: 8
          w: 4
          x: 20
          "y": 0
        targets:
          - datasource: prometheus
            expr: "round(sum(increase(tekton_pipelines_controller_pipelinerun_total[5m])))"
            legendFormat: completed
            refId: A
        fieldConfig:
          defaults:
            unit: short
            decimals: 0
            color:
              mode: palette-classic
            custom:
              drawStyle: line
              fillOpacity: 0
              lineWidth: 1
              spanNulls: false
            thresholds:
              mode: absolute
              steps:
                - color: green
                  value: null
          overrides: []
        options:
          legend:
            calcs:
              - latest
            displayMode: list
            placement: bottom
            showLegend: true
          tooltip:
            mode: multi
            sort: desc
        transformations: []
      - id: 4
        title: Running PipelineRuns
        type: timeseries
        datasource: prometheus
        gridPos:
          h: 8
          w: 8
          x: 0
          "y": 8
        targets:
          - datasource: prometheus
            expr: max(tekton_pipelines_controller_running_pipelineruns)
            legendFormat: running
            refId: A
        fieldConfig:
          defaults:
            color:
              mode: palette-classic
            custom:
              drawStyle: line
              fillOpacity: 0
              lineWidth: 1
              spanNulls: false
            thresholds:
              mode: absolute
              steps:
                - color: green
                  value: null
          overrides: []
        options:
          legend:
            calcs:
              - latest
            displayMode: list
            placement: bottom
            showLegend: true
          tooltip:
            mode: multi
            sort: desc
        transformations: []
      - id: 5
        title: Running TaskRuns
        type: timeseries
        datasource: prometheus
        gridPos:
          h: 8
          w: 8
          x: 8
          "y": 8
        targets:
          - datasource: prometheus
            expr: max(tekton_pipelines_controller_running_taskruns)
            legendFormat: running
            refId: A
        fieldConfig:
          defaults:
            color:
              mode: palette-classic
            custom:
              drawStyle: line
              fillOpacity: 0
              lineWidth: 1
              spanNulls: false
            thresholds:
              mode: absolute
              steps:
                - color: green
                  value: null
          overrides: []
        options:
          legend:
            calcs:
              - latest
            displayMode: list
            placement: bottom
            showLegend: true
          tooltip:
            mode: multi
            sort: desc
        transformations: []
      - id: 6
        title: TaskRuns Throttled
        type: timeseries
        datasource: prometheus
        gridPos:
          h: 8
          w: 8
          x: 16
          "y": 8
        targets:
          - datasource: prometheus
            expr: max(tekton_pipelines_controller_running_taskruns_throttled_by_quota)
            legendFormat: by quota
            refId: A
          - datasource: prometheus
            expr: max(tekton_pipelines_controller_running_taskruns_throttled_by_node)
            legendFormat: by node
            refId: B
        fieldConfig:
          defaults:
            color:
              mode: palette-classic
            custom:
              drawStyle: line
              fillOpacity: 0
              lineWidth: 1
              spanNulls: false
            thresholds:
              mode: absolute
              steps:
                - color: orange
                  value: null
          overrides: []
        options:
          legend:
            calcs:
              - latest
            displayMode: list
            placement: bottom
            showLegend: true
          tooltip:
            mode: multi
            sort: desc
        transformations: []
      - id: 7
        title: PipelineRun Duration P50 / P95
        type: timeseries
        datasource: prometheus
        gridPos:
          h: 8
          w: 8
          x: 0
          "y": 16
        targets:
          - datasource: prometheus
            expr: (histogram_quantile(0.5, sum by (le) (rate(tekton_pipelines_controller_pipelinerun_duration_seconds_bucket[5m])))) and on() (sum(rate(tekton_pipelines_controller_pipelinerun_duration_seconds_bucket{le="+Inf"}[5m])) > 0)
            legendFormat: P50
            refId: A
          - datasource: prometheus
            expr: (histogram_quantile(0.95, sum by (le) (rate(tekton_pipelines_controller_pipelinerun_duration_seconds_bucket[5m])))) and on() (sum(rate(tekton_pipelines_controller_pipelinerun_duration_seconds_bucket{le="+Inf"}[5m])) > 0)
            legendFormat: P95
            refId: B
        fieldConfig:
          defaults:
            unit: s
            color:
              mode: palette-classic
            custom:
              drawStyle: line
              fillOpacity: 0
              lineWidth: 1
              spanNulls: false
            thresholds:
              mode: absolute
              steps:
                - color: green
                  value: null
          overrides: []
        options:
          legend:
            calcs:
              - latest
            displayMode: list
            placement: bottom
            showLegend: true
          tooltip:
            mode: multi
            sort: desc
        transformations: []
      - id: 8
        title: TaskRun Duration P50 / P95 (Standalone)
        type: timeseries
        datasource: prometheus
        gridPos:
          h: 8
          w: 8
          x: 8
          "y": 16
        targets:
          - datasource: prometheus
            expr: (histogram_quantile(0.5, sum by (le) (rate(tekton_pipelines_controller_taskrun_duration_seconds_bucket[5m])))) and on() (sum(rate(tekton_pipelines_controller_taskrun_duration_seconds_bucket{le="+Inf"}[5m])) > 0)
            legendFormat: P50
            refId: A
          - datasource: prometheus
            expr: (histogram_quantile(0.95, sum by (le) (rate(tekton_pipelines_controller_taskrun_duration_seconds_bucket[5m])))) and on() (sum(rate(tekton_pipelines_controller_taskrun_duration_seconds_bucket{le="+Inf"}[5m])) > 0)
            legendFormat: P95
            refId: B
        fieldConfig:
          defaults:
            unit: s
            color:
              mode: palette-classic
            custom:
              drawStyle: line
              fillOpacity: 0
              lineWidth: 1
              spanNulls: false
            thresholds:
              mode: absolute
              steps:
                - color: green
                  value: null
          overrides: []
        options:
          legend:
            calcs:
              - latest
            displayMode: list
            placement: bottom
            showLegend: true
          tooltip:
            mode: multi
            sort: desc
        transformations: []
      - id: 13
        title: TaskRun Duration P50 / P95 (In-Pipeline)
        type: timeseries
        datasource: prometheus
        gridPos:
          h: 8
          w: 8
          x: 16
          "y": 16
        targets:
          - datasource: prometheus
            expr: (histogram_quantile(0.5, sum by (le) (rate(tekton_pipelines_controller_pipelinerun_taskrun_duration_seconds_bucket[5m])))) and on() (sum(rate(tekton_pipelines_controller_pipelinerun_taskrun_duration_seconds_bucket{le="+Inf"}[5m])) > 0)
            legendFormat: P50
            refId: A
          - datasource: prometheus
            expr: (histogram_quantile(0.95, sum by (le) (rate(tekton_pipelines_controller_pipelinerun_taskrun_duration_seconds_bucket[5m])))) and on() (sum(rate(tekton_pipelines_controller_pipelinerun_taskrun_duration_seconds_bucket{le="+Inf"}[5m])) > 0)
            legendFormat: P95
            refId: B
        fieldConfig:
          defaults:
            unit: s
            color:
              mode: palette-classic
            custom:
              drawStyle: line
              fillOpacity: 0
              lineWidth: 1
              spanNulls: false
            thresholds:
              mode: absolute
              steps:
                - color: green
                  value: null
          overrides: []
        options:
          legend:
            calcs:
              - latest
            displayMode: list
            placement: bottom
            showLegend: true
          tooltip:
            mode: multi
            sort: desc
        transformations: []
      - id: 9
        title: Workqueue Depth
        type: timeseries
        datasource: prometheus
        gridPos:
          h: 8
          w: 8
          x: 0
          "y": 24
        targets:
          - datasource: prometheus
            expr: max(kn_workqueue_depth{name=~"github.com.tektoncd.pipeline.pkg.reconciler.(pipelinerun|taskrun).Reconciler"})
            legendFormat: depth
            refId: A
        fieldConfig:
          defaults:
            color:
              mode: palette-classic
            custom:
              drawStyle: line
              fillOpacity: 0
              lineWidth: 1
              spanNulls: false
            thresholds:
              mode: absolute
              steps:
                - color: green
                  value: null
          overrides: []
        options:
          legend:
            calcs:
              - latest
            displayMode: list
            placement: bottom
            showLegend: true
          tooltip:
            mode: multi
            sort: desc
        transformations: []
      - id: 10
        title: Workqueue Adds and Retries
        type: timeseries
        datasource: prometheus
        gridPos:
          h: 8
          w: 8
          x: 8
          "y": 24
        targets:
          - datasource: prometheus
            expr: sum(increase(kn_workqueue_adds_total{name=~"github.com.tektoncd.pipeline.pkg.reconciler.(pipelinerun|taskrun).Reconciler"}[5m]))
            legendFormat: adds
            refId: A
          - datasource: prometheus
            expr: sum(increase(kn_workqueue_retries_total{name=~"github.com.tektoncd.pipeline.pkg.reconciler.(pipelinerun|taskrun).Reconciler"}[5m]))
            legendFormat: retries
            refId: B
        fieldConfig:
          defaults:
            color:
              mode: palette-classic
            custom:
              drawStyle: line
              fillOpacity: 0
              lineWidth: 1
              spanNulls: false
            thresholds:
              mode: absolute
              steps:
                - color: green
                  value: null
          overrides: []
        options:
          legend:
            calcs:
              - latest
            displayMode: list
            placement: bottom
            showLegend: true
          tooltip:
            mode: multi
            sort: desc
        transformations: []
      - id: 11
        title: Resolution Waiting
        type: timeseries
        datasource: prometheus
        gridPos:
          h: 8
          w: 8
          x: 16
          "y": 24
        targets:
          - datasource: prometheus
            expr: max(tekton_pipelines_controller_running_pipelineruns_waiting_on_pipeline_resolution)
            legendFormat: PR waiting pipeline
            refId: A
          - datasource: prometheus
            expr: max(tekton_pipelines_controller_running_pipelineruns_waiting_on_task_resolution)
            legendFormat: PR waiting task
            refId: B
          - datasource: prometheus
            expr: max(tekton_pipelines_controller_running_taskruns_waiting_on_task_resolution_count)
            legendFormat: TR waiting task
            refId: C
        fieldConfig:
          defaults:
            color:
              mode: palette-classic
            custom:
              drawStyle: line
              fillOpacity: 0
              lineWidth: 1
              spanNulls: false
            thresholds:
              mode: absolute
              steps:
                - color: orange
                  value: null
          overrides: []
        options:
          legend:
            calcs:
              - latest
            displayMode: list
            placement: bottom
            showLegend: true
          tooltip:
            mode: multi
            sort: desc
        transformations: []

Tekton Pipeline 监控面板说明(常见问题)

  • PipelineRun Total (by status) 是由 controller 记录的完成事件计数器,而不是 PipelineRun 对象的总数。以 Cancelled 为 reason 结束的 PipelineRun(包括使用 spec.status: Cancelled 取消的 PipelineRun)会计入 status="cancelled"。使用 spec.status: CancelledRunFinally 或 StoppedRunFinally 取消或停止的 PipelineRun 中,如果某个 task 或 finally task 失败,该 PipelineRun 会以 Failed 为 reason 结束,并计入 status="failed"。
  • Running PipelineRuns 是实时快照(当前正在运行的数量)。它可以独立于 PipelineRun Total 发生变化。
  • Completed PipelineRuns (last 5m) 表示吞吐量(过去 5 分钟内新完成的运行次数)。在流量较低或空闲期间看到 0 是预期行为。
  • PipelineRun Success Rate (cumulative) 是自 controller 启动以来的累计值,而不是 5 分钟时间窗口内的成功率。短期失败不会立即导致其发生较大变化。
  • Workqueue Adds and Retries 统计过去 5 分钟内添加到 PipelineRun 和 TaskRun reconciler 工作队列以及在其中重试的项目数,而不是 PipelineRun 数量。
  • 仅当选定时间范围内实际存在相应 label 值的样本时,才会显示状态 series。如果某个状态在该时间窗口内没有样本,则不会显示其曲线或图例。
  • TaskRun Duration P50 / P95 (Standalone) 和 TaskRun Duration P50 / P95 (In-Pipeline) 被拆分,以避免混合查询不稳定。在只公开其中一个 histogram family 的环境中,另一个 panel 可能为空,这是预期行为。

Tekton Triggers 监控面板

Tekton Triggers Dashboard YAML
kind: MonitorDashboard
apiVersion: ait.alauda.io/v1alpha2
metadata:
  labels:
    cpaas.io/dashboard.folder: tekton
    cpaas.io/dashboard.is.home.dashboard: "false"
    cpaas.io/dashboard.tag.tekton: "true"
  name: tekton-triggers
  namespace: cpaas-system
spec:
  body:
    title: Tekton Triggers Overview
    titleZh: Tekton Triggers Overview
    tags:
      - tekton
    time:
      from: now-1h
      to: now
    templating:
      list: []
    panels:
      - id: 1
        title: EventListener Count
        type: timeseries
        datasource: prometheus
        gridPos:
          h: 6
          w: 5
          x: 0
          "y": 0
        targets:
          - datasource: prometheus
            expr: controller_eventlistener_count
            legendFormat: EventListener
            refId: A
        fieldConfig:
          defaults:
            color:
              mode: palette-classic
            custom:
              drawStyle: line
              fillOpacity: 0
              lineWidth: 1
              spanNulls: false
            thresholds:
              mode: absolute
              steps:
                - color: green
                  value: null
          overrides: []
        options:
          legend:
            calcs:
              - latest
            displayMode: list
            placement: bottom
            showLegend: true
          tooltip:
            mode: multi
            sort: desc
        transformations: []
      - id: 2
        title: TriggerTemplate Count
        type: timeseries
        datasource: prometheus
        gridPos:
          h: 6
          w: 5
          x: 5
          "y": 0
        targets:
          - datasource: prometheus
            expr: controller_triggertemplate_count
            legendFormat: TriggerTemplate
            refId: A
        fieldConfig:
          defaults:
            color:
              mode: palette-classic
            custom:
              drawStyle: line
              fillOpacity: 0
              lineWidth: 1
              spanNulls: false
            thresholds:
              mode: absolute
              steps:
                - color: green
                  value: null
          overrides: []
        options:
          legend:
            calcs:
              - latest
            displayMode: list
            placement: bottom
            showLegend: true
          tooltip:
            mode: multi
            sort: desc
        transformations: []
      - id: 3
        title: TriggerBinding Count
        type: timeseries
        datasource: prometheus
        gridPos:
          h: 6
          w: 5
          x: 10
          "y": 0
        targets:
          - datasource: prometheus
            expr: controller_triggerbinding_count
            legendFormat: TriggerBinding
            refId: A
        fieldConfig:
          defaults:
            color:
              mode: palette-classic
            custom:
              drawStyle: line
              fillOpacity: 0
              lineWidth: 1
              spanNulls: false
            thresholds:
              mode: absolute
              steps:
                - color: green
                  value: null
          overrides: []
        options:
          legend:
            calcs:
              - latest
            displayMode: list
            placement: bottom
            showLegend: true
          tooltip:
            mode: multi
            sort: desc
        transformations: []
      - id: 4
        title: ClusterTriggerBinding
        type: timeseries
        datasource: prometheus
        gridPos:
          h: 6
          w: 5
          x: 15
          "y": 0
        targets:
          - datasource: prometheus
            expr: controller_clustertriggerbinding_count
            legendFormat: ClusterTriggerBinding
            refId: A
        fieldConfig:
          defaults:
            color:
              mode: palette-classic
            custom:
              drawStyle: line
              fillOpacity: 0
              lineWidth: 1
              spanNulls: false
            thresholds:
              mode: absolute
              steps:
                - color: green
                  value: null
          overrides: []
        options:
          legend:
            calcs:
              - latest
            displayMode: list
            placement: bottom
            showLegend: true
          tooltip:
            mode: multi
            sort: desc
        transformations: []
      - id: 5
        title: ClusterInterceptor
        type: timeseries
        datasource: prometheus
        gridPos:
          h: 6
          w: 4
          x: 20
          "y": 0
        targets:
          - datasource: prometheus
            expr: controller_clusterinterceptor_count
            legendFormat: ClusterInterceptor
            refId: A
        fieldConfig:
          defaults:
            color:
              mode: palette-classic
            custom:
              drawStyle: line
              fillOpacity: 0
              lineWidth: 1
              spanNulls: false
            thresholds:
              mode: absolute
              steps:
                - color: green
                  value: null
          overrides: []
        options:
          legend:
            calcs:
              - latest
            displayMode: list
            placement: bottom
            showLegend: true
          tooltip:
            mode: multi
            sort: desc
        transformations: []
      - id: 6
        title: All Trigger Resource Counts (trend)
        type: timeseries
        datasource: prometheus
        gridPos:
          h: 8
          w: 24
          x: 0
          "y": 6
        targets:
          - datasource: prometheus
            expr: controller_eventlistener_count
            legendFormat: EventListener
            refId: A
          - datasource: prometheus
            expr: controller_triggertemplate_count
            legendFormat: TriggerTemplate
            refId: B
          - datasource: prometheus
            expr: controller_triggerbinding_count
            legendFormat: TriggerBinding
            refId: C
          - datasource: prometheus
            expr: controller_clustertriggerbinding_count
            legendFormat: ClusterTriggerBinding
            refId: D
          - datasource: prometheus
            expr: controller_clusterinterceptor_count
            legendFormat: ClusterInterceptor
            refId: E
        fieldConfig:
          defaults:
            color:
              mode: palette-classic
            custom:
              drawStyle: line
              fillOpacity: 0
              lineWidth: 1
              spanNulls: false
            thresholds:
              mode: absolute
              steps:
                - color: green
                  value: null
          overrides: []
        options:
          legend:
            calcs:
              - latest
            displayMode: list
            placement: bottom
            showLegend: true
          tooltip:
            mode: multi
            sort: desc
        transformations: []

Tekton Triggers 监控面板说明(常见问题)

  • EventListener Count、TriggerTemplate Count、TriggerBinding Count、ClusterTriggerBinding 和 ClusterInterceptor 是对象数量快照,而不是请求量或事件处理吞吐量。
  • All Trigger Resource Counts (trend) 显示相同资源数量的合并趋势。与单资源趋势 panel 相比,在一次抓取间隔内出现短暂偏差是预期行为。
  • 在不存在 Triggers 资源时显示 0 是正常现象,并不表示抓取失败。

Tekton Results 监控面板

Tekton Results Dashboard YAML
kind: MonitorDashboard
apiVersion: ait.alauda.io/v1alpha2
metadata:
  labels:
    cpaas.io/dashboard.folder: tekton
    cpaas.io/dashboard.is.home.dashboard: "false"
    cpaas.io/dashboard.tag.tekton: "true"
  name: tekton-results
  namespace: cpaas-system
spec:
  body:
    title: Tekton Results Overview
    titleZh: Tekton Results Overview
    tags:
      - tekton
    time:
      from: now-1h
      to: now
    templating:
      list: []
    panels:
      - id: 1
        title: PipelineRun Reconciles and Retries (last 5m)
        type: timeseries
        datasource: prometheus
        gridPos:
          h: 8
          w: 12
          x: 0
          "y": 0
        targets:
          - datasource: prometheus
            expr: round(sum(increase(kn_workqueue_process_duration_seconds_count{name="github.com.tektoncd.results.pkg.watcher.reconciler.pipelinerun.Reconciler"}[5m])))
            legendFormat: processed
            refId: A
          - datasource: prometheus
            expr: round(sum(increase(kn_workqueue_retries_total{name="github.com.tektoncd.results.pkg.watcher.reconciler.pipelinerun.Reconciler"}[5m])))
            legendFormat: retries
            refId: B
        fieldConfig:
          defaults:
            color:
              mode: palette-classic
            custom:
              drawStyle: line
              fillOpacity: 0
              lineWidth: 1
              spanNulls: false
            thresholds:
              mode: absolute
              steps:
                - color: green
                  value: null
          overrides: []
        options:
          legend:
            calcs:
              - latest
            displayMode: list
            placement: bottom
            showLegend: true
          tooltip:
            mode: multi
            sort: desc
        transformations: []
      - id: 2
        title: TaskRun Reconciles and Retries (last 5m)
        type: timeseries
        datasource: prometheus
        gridPos:
          h: 8
          w: 12
          x: 12
          "y": 0
        targets:
          - datasource: prometheus
            expr: round(sum(increase(kn_workqueue_process_duration_seconds_count{name="github.com.tektoncd.results.pkg.watcher.reconciler.taskrun.Reconciler"}[5m])))
            legendFormat: processed
            refId: A
          - datasource: prometheus
            expr: round(sum(increase(kn_workqueue_retries_total{name="github.com.tektoncd.results.pkg.watcher.reconciler.taskrun.Reconciler"}[5m])))
            legendFormat: retries
            refId: B
        fieldConfig:
          defaults:
            color:
              mode: palette-classic
            custom:
              drawStyle: line
              fillOpacity: 0
              lineWidth: 1
              spanNulls: false
            thresholds:
              mode: absolute
              steps:
                - color: green
                  value: null
          overrides: []
        options:
          legend:
            calcs:
              - latest
            displayMode: list
            placement: bottom
            showLegend: true
          tooltip:
            mode: multi
            sort: desc
        transformations: []
      - id: 3
        title: PipelineRun Reconcile Latency P95
        type: timeseries
        datasource: prometheus
        gridPos:
          h: 8
          w: 12
          x: 0
          "y": 8
        targets:
          - datasource: prometheus
            expr: histogram_quantile(0.95, sum by (le) (rate(kn_workqueue_process_duration_seconds_bucket{name="github.com.tektoncd.results.pkg.watcher.reconciler.pipelinerun.Reconciler"}[5m])))
            legendFormat: P95
            refId: A
        fieldConfig:
          defaults:
            unit: s
            color:
              mode: palette-classic
            custom:
              drawStyle: line
              fillOpacity: 0
              lineWidth: 1
              spanNulls: false
            thresholds:
              mode: absolute
              steps:
                - color: green
                  value: null
          overrides: []
        options:
          legend:
            calcs:
              - latest
            displayMode: list
            placement: bottom
            showLegend: true
          tooltip:
            mode: multi
            sort: desc
        transformations: []
      - id: 4
        title: TaskRun Reconcile Latency P95
        type: timeseries
        datasource: prometheus
        gridPos:
          h: 8
          w: 12
          x: 12
          "y": 8
        targets:
          - datasource: prometheus
            expr: histogram_quantile(0.95, sum by (le) (rate(kn_workqueue_process_duration_seconds_bucket{name="github.com.tektoncd.results.pkg.watcher.reconciler.taskrun.Reconciler"}[5m])))
            legendFormat: P95
            refId: A
        fieldConfig:
          defaults:
            unit: s
            color:
              mode: palette-classic
            custom:
              drawStyle: line
              fillOpacity: 0
              lineWidth: 1
              spanNulls: false
            thresholds:
              mode: absolute
              steps:
                - color: green
                  value: null
          overrides: []
        options:
          legend:
            calcs:
              - latest
            displayMode: list
            placement: bottom
            showLegend: true
          tooltip:
            mode: multi
            sort: desc
        transformations: []
      - id: 5
        title: Workqueue Depth (PipelineRun vs TaskRun)
        type: timeseries
        datasource: prometheus
        gridPos:
          h: 8
          w: 12
          x: 0
          "y": 16
        targets:
          - datasource: prometheus
            expr: sum(kn_workqueue_depth{name="github.com.tektoncd.results.pkg.watcher.reconciler.pipelinerun.Reconciler"})
            legendFormat: pipelinerun
            refId: A
          - datasource: prometheus
            expr: sum(kn_workqueue_depth{name="github.com.tektoncd.results.pkg.watcher.reconciler.taskrun.Reconciler"})
            legendFormat: taskrun
            refId: B
        fieldConfig:
          defaults:
            color:
              mode: palette-classic
            custom:
              drawStyle: line
              fillOpacity: 0
              lineWidth: 1
              spanNulls: false
            thresholds:
              mode: absolute
              steps:
                - color: green
                  value: null
          overrides: []
        options:
          legend:
            calcs:
              - latest
            displayMode: list
            placement: bottom
            showLegend: true
          tooltip:
            mode: multi
            sort: desc
        transformations: []
      - id: 6
        title: Workqueue Adds (last 5m)
        type: timeseries
        datasource: prometheus
        gridPos:
          h: 8
          w: 12
          x: 12
          "y": 16
        targets:
          - datasource: prometheus
            expr: round(sum(increase(kn_workqueue_adds_total{name="github.com.tektoncd.results.pkg.watcher.reconciler.pipelinerun.Reconciler"}[5m])))
            legendFormat: pipelinerun adds
            refId: A
          - datasource: prometheus
            expr: round(sum(increase(kn_workqueue_adds_total{name="github.com.tektoncd.results.pkg.watcher.reconciler.taskrun.Reconciler"}[5m])))
            legendFormat: taskrun adds
            refId: B
        fieldConfig:
          defaults:
            color:
              mode: palette-classic
            custom:
              drawStyle: line
              fillOpacity: 0
              lineWidth: 1
              spanNulls: false
            thresholds:
              mode: absolute
              steps:
                - color: green
                  value: null
          overrides: []
        options:
          legend:
            calcs:
              - latest
            displayMode: list
            placement: bottom
            showLegend: true
          tooltip:
            mode: multi
            sort: desc
        transformations: []
      - id: 7
        title: gRPC Request Rate (Results API)
        type: timeseries
        datasource: prometheus
        gridPos:
          h: 8
          w: 12
          x: 0
          "y": 24
        targets:
          - datasource: prometheus
            expr: "sum(rate(grpc_server_handled_total{grpc_service=~\"tekton.results.*\"}[5m]))"
            legendFormat: requests
            refId: A
        fieldConfig:
          defaults:
            color:
              mode: palette-classic
            custom:
              drawStyle: line
              fillOpacity: 0
              lineWidth: 1
              spanNulls: false
            thresholds:
              mode: absolute
              steps:
                - color: green
                  value: null
          overrides: []
        options:
          legend:
            calcs:
              - latest
            displayMode: list
            placement: bottom
            showLegend: true
          tooltip:
            mode: multi
            sort: desc
        transformations: []
      - id: 8
        title: gRPC Error Percentage (Results API, excl. NotFound/Canceled)
        type: timeseries
        datasource: prometheus
        gridPos:
          h: 8
          w: 12
          x: 12
          "y": 24
        targets:
          - datasource: prometheus
            expr: "100 * ((sum(rate(grpc_server_handled_total{grpc_service=~\"tekton.results.*\",grpc_code!~\"OK|NotFound|Canceled\"}[5m])) or vector(0)) / clamp_min((sum(rate(grpc_server_handled_total{grpc_service=~\"tekton.results.*\"}[5m])) or vector(0)), 0.001))"
            legendFormat: error %
            refId: A
        fieldConfig:
          defaults:
            unit: percent
            color:
              mode: palette-classic
            custom:
              drawStyle: line
              fillOpacity: 0
              lineWidth: 1
              spanNulls: false
            thresholds:
              mode: absolute
              steps:
                - color: red
                  value: null
          overrides: []
        options:
          legend:
            calcs:
              - latest
            displayMode: list
            placement: bottom
            showLegend: true
          tooltip:
            mode: multi
            sort: desc
        transformations: []

Tekton Results 监控面板说明(常见问题)

  • 此监控面板版本基于 Results Watcher 工作队列指标(kn_workqueue_*,name label 中每个 reconciler 对应一个 series)以及 Results API gRPC 指标,因此在常见部署(logs_api=true,禁用自动删除)下仍会显示数据。
  • PipelineRun Reconciles and Retries (last 5m) 和 TaskRun Reconciles and Retries (last 5m) 按每个 5 分钟时间窗口显示 reconciler 处理的项目数(kn_workqueue_process_duration_seconds_count)以及重新添加到队列以进行下一次尝试的项目数(kn_workqueue_retries_total)。
  • PipelineRun Reconcile Latency P95 和 TaskRun Reconcile Latency P95 根据 watcher 工作队列处理时间 histogram(kn_workqueue_process_duration_seconds,单位为秒)计算。在流量较低时,曲线可能较为稀疏。
  • Workqueue Depth 显示当前队列深度,Workqueue Adds (last 5m) 显示过去 5 分钟内的入队量。
  • gRPC Error Percentage (Results API, excl. NotFound/Canceled) 表示异常错误占总请求数的百分比,其中排除了常见的业务返回码(NotFound、Canceled)。

Tekton Chains 监控面板

Tekton Chains Dashboard YAML
kind: MonitorDashboard
apiVersion: ait.alauda.io/v1alpha2
metadata:
  labels:
    cpaas.io/dashboard.folder: tekton
    cpaas.io/dashboard.is.home.dashboard: "false"
    cpaas.io/dashboard.tag.tekton: "true"
  name: tekton-chains
  namespace: cpaas-system
spec:
  body:
    title: Tekton Chains Overview
    titleZh: Tekton Chains Overview
    tags:
      - tekton
    time:
      from: now-1h
      to: now
    templating:
      list: []
    panels:
      - id: 1
        title: TaskRun Signatures Created (last 5m)
        type: timeseries
        datasource: prometheus
        gridPos:
          h: 8
          w: 12
          x: 0
          "y": 0
        targets:
          - datasource: prometheus
            expr: round(increase(watcher_taskrun_sign_created_total[5m]))
            legendFormat: sign created
            refId: A
        fieldConfig:
          defaults:
            color:
              mode: palette-classic
            custom:
              drawStyle: line
              fillOpacity: 0
              lineWidth: 1
              spanNulls: false
            thresholds:
              mode: absolute
              steps:
                - color: green
                  value: null
          overrides: []
        options:
          legend:
            calcs:
              - latest
            displayMode: list
            placement: bottom
            showLegend: true
          tooltip:
            mode: multi
            sort: desc
        transformations: []
      - id: 2
        title: PipelineRun Signatures Created (last 5m)
        type: timeseries
        datasource: prometheus
        gridPos:
          h: 8
          w: 12
          x: 12
          "y": 0
        targets:
          - datasource: prometheus
            expr: round(increase(watcher_pipelinerun_sign_created_total[5m]))
            legendFormat: sign created
            refId: A
        fieldConfig:
          defaults:
            color:
              mode: palette-classic
            custom:
              drawStyle: line
              fillOpacity: 0
              lineWidth: 1
              spanNulls: false
            thresholds:
              mode: absolute
              steps:
                - color: green
                  value: null
          overrides: []
        options:
          legend:
            calcs:
              - latest
            displayMode: list
            placement: bottom
            showLegend: true
          tooltip:
            mode: multi
            sort: desc
        transformations: []
      - id: 3
        title: Payloads Stored (last 5m, TaskRun vs PipelineRun)
        type: timeseries
        datasource: prometheus
        gridPos:
          h: 8
          w: 12
          x: 0
          "y": 8
        targets:
          - datasource: prometheus
            expr: round(increase(watcher_taskrun_payload_stored_total[5m]))
            legendFormat: TaskRun
            refId: A
          - datasource: prometheus
            expr: round(increase(watcher_pipelinerun_payload_stored_total[5m]))
            legendFormat: PipelineRun
            refId: B
        fieldConfig:
          defaults:
            color:
              mode: palette-classic
            custom:
              drawStyle: line
              fillOpacity: 0
              lineWidth: 1
              spanNulls: false
            thresholds:
              mode: absolute
              steps:
                - color: green
                  value: null
          overrides: []
        options:
          legend:
            calcs:
              - latest
            displayMode: list
            placement: bottom
            showLegend: true
          tooltip:
            mode: multi
            sort: desc
        transformations: []
      - id: 4
        title: Marked Signed (last 5m, TaskRun vs PipelineRun)
        type: timeseries
        datasource: prometheus
        gridPos:
          h: 8
          w: 12
          x: 12
          "y": 8
        targets:
          - datasource: prometheus
            expr: round(increase(watcher_taskrun_marked_signed_total[5m]))
            legendFormat: TaskRun
            refId: A
          - datasource: prometheus
            expr: round(increase(watcher_pipelinerun_marked_signed_total[5m]))
            legendFormat: PipelineRun
            refId: B
        fieldConfig:
          defaults:
            color:
              mode: palette-classic
            custom:
              drawStyle: line
              fillOpacity: 0
              lineWidth: 1
              spanNulls: false
            thresholds:
              mode: absolute
              steps:
                - color: green
                  value: null
          overrides: []
        options:
          legend:
            calcs:
              - latest
            displayMode: list
            placement: bottom
            showLegend: true
          tooltip:
            mode: multi
            sort: desc
        transformations: []

Tekton Chains 监控面板说明(常见问题)

  • TaskRun Signatures Created (last 5m)、PipelineRun Signatures Created (last 5m)、Payloads Stored (last 5m) 和 Marked Signed (last 5m) 使用 increase(...[5m]),表示过去五分钟内的增量。
  • 没有新的签名或存储活动时,这些曲线会降至 0;这并不表示组件发生故障。如果某个 counter 自 Chains controller 启动以来从未递增,则不会显示数据,因为该 counter 尚未出现在指标输出中。
  • Payloads Stored 和 Marked Signed 表示不同的处理阶段,因此它们的值不应始终匹配。Payloads Stored 还会统计这样一种 payload:oci storage backend 未找到 image subject,因此未向 registry 上传任何内容(请参阅签名密钥配置中的前提条件)。
  • 如果 Marked Signed 增长,而 TaskRun Signatures Created 和 PipelineRun Signatures Created 没有数据显示(或者保持为 0,即 controller 之前已创建签名),并且 *_signing_failures_total 没有增长,则 Chains 会将运行标记为已签名,但不会实际进行签名(请参阅Chains 指标下的说明)。当未配置 signer 时会发生这种情况(默认情况下,signing-secrets 不包含密钥),或者禁用了对 TaskRun 和 PipelineRun 工件的签名(例如使用 artifacts.taskrun.storage: "" 和 artifacts.pipelinerun.storage: "",请参阅Chains 配置)。没有任何监控面板显示 *_signing_failures_total;请在 Prometheus 或 Chains 指标输出中查询它。如果在这种情况下该指标增长,则表示 Chains 创建、编码或签署 payload 失败(error_type、payload_creation、marshal_payload 或 signing),但仍将运行标记为已签名。