Tekton 组件的指标采集
概述
Tekton 组件通过 HTTP 端点公开 Prometheus 兼容的指标。通过部署 ServiceMonitor 资源,Prometheus(或 VictoriaMetrics)可以自动发现并抓取这些指标。
命名空间说明:本文档使用 tekton-pipelines 作为控制平面组件(Pipelines、Triggers、Results、Chains)的默认命名空间。
主要例外是 EventListener Services,它们运行在创建 EventListener 的应用命名空间中。
如果您的部署使用不同的命名空间,请同时更新以下命令以及下方 ServiceMonitor 资源中的 namespaceSelector 字段。
本文档介绍以下 Tekton 组件的指标:
- Tekton Pipelines - PipelineRun / TaskRun 执行指标
- Tekton Triggers - EventListener、TriggerBinding 及相关资源指标
- Tekton Results - 运行删除和存储指标
- Tekton Chains - 签名和来源证明指标
- Controller Framework - 所有 controller 共享的基础设施指标
本文档还介绍:
- 如何通过
config-observability 配置指标行为
- 如何部署用于抓取指标的
ServiceMonitor 资源
- 如何验证指标采集是否正常工作
先决条件
- Tekton 控制平面组件已安装并运行(至少包括您计划抓取的组件:Pipelines、Triggers、Results 和/或 Chains)。
kubectl 已配置为连接目标集群,并且您的账户可以在创建资源的命名空间中创建 ServiceMonitor 资源(本页面中的示例使用 tekton-pipelines)。
- 已部署监控栈(Prometheus 或兼容的 VictoriaMetrics),并且能够发现/抓取
ServiceMonitor 资源(或平台使用的等效抓取发现对象)。
- 您的 Prometheus/VictoriaMetrics 实例已配置为发现该命名空间中的
ServiceMonitor 对象(命名空间和标签选择器必须匹配)。
- 网络策略和防火墙允许抓取器 Pod 访问 Tekton 指标端口(大多数控制平面服务使用
9090,Triggers controller 和 EventListener sink 使用 9000)。
- 如果您需要 EventListener sink 指标,则目标命名空间中必须存在 EventListener,并公开
http-metrics 端口。
- 用于直接检查指标端点的
curl CLI,以及用于 runs_not_stored_count 复现的恢复步骤的 jq CLI。在隔离网络的工作站上,请从工作站可以访问的操作系统软件包仓库安装它们,或从已联网的计算机复制二进制文件。
- 要部署监控面板示例中的监控面板,您的账户必须能够在
cpaas-system 命名空间中创建 MonitorDashboard 资源(ait.alauda.io/v1alpha2)。
- 更改
config-observability 需要具备更新集群范围 TektonConfig 的权限。在 tekton-pipelines 中读取 ConfigMap 和重启 Pod,以及在那里执行 kubectl port-forward 检查,都需要该命名空间中的相应权限(端口转发需要 pods/portforward)。这些步骤通常由集群管理员执行。直接检查指标端点中的 EventListener sink 检查会在 EventListener 的命名空间中执行端口转发,因此还需要在该命名空间中具备 pods/portforward 权限。
Tekton Pipelines
Tekton Pipelines 组件包含多个在端口 9090 上公开指标的子服务:
Pipeline controller 指标使用前缀 tekton_pipelines_controller_。
PipelineRun 指标
* 标有 * 的标签为可选标签,具体取决于 config-observability 配置。
running_pipelineruns 标签级别
running_pipelineruns 指标的标签由 metrics.running-pipelinerun.level 控制:
状态标签值
对于 PipelineRun 指标:
success - PipelineRun 成功完成
failed - PipelineRun 因 Cancelled 之外的原因而失败,包括因 CancelledRunFinally 或 StoppedRunFinally 而取消或停止、且其中某个 task 失败的运行
cancelled - PipelineRun 以原因为 Cancelled 结束
对于 TaskRun 指标:
success - TaskRun 成功完成
failed - TaskRun 因 TaskRunCancelled 之外的原因而失败
cancelled - TaskRun 以原因为 TaskRunCancelled 结束
TaskRun 指标
config-observability 配置
tekton-pipelines 命名空间中的 config-observability ConfigMap 控制 Pipeline controller 的指标行为。此 ConfigMap 由 Tekton Operator 管理,应通过 TektonConfig 资源的 spec.pipeline.options.configMaps 字段进行配置。有关详细信息,请参阅调整子组件的可选配置项。直接编辑 ConfigMap 也会在 controller 中生效,并且不会立即被还原,因此这种编辑很容易被忽略;请将值保存在 TektonConfig 中,operator 会将其应用到 ConfigMap。TektonConfig 也会在 spec.pipeline 下直接为其中一些键提供字段,每个字段都使用完整键名(平面键,而不是嵌套路径)命名,例如 spec.pipeline 下的键 metrics.pipelinerun.level;虽然 options.configMaps.config-observability 设置了同一个键,但 ConfigMap 会获得 options.configMaps 值。
重新加载行为:Pipeline controller 会在运行时监视 config-observability。更改级别键(metrics.pipelinerun.level、metrics.taskrun.level 和 metrics.running-pipelinerun.level)以及标签开关 metrics.count.enable-reason 和 metrics.taskrun.throttle.enable-namespace 后,无需重启即可生效。更改 metrics.pipelinerun.level 或 metrics.taskrun.level 后,会显示在更改完成后结束的运行的指标中。请等待一到两个抓取间隔,以便监控面板/查询中显示更改。更改 metrics.pipelinerun.duration-type 或 metrics.taskrun.duration-type 需要重启 tekton-pipelines-controller Pod。更改后立即重启 Pod。如果不重启,正在运行的 controller 会继续使用旧类型的 instrument,并且在下一次运行完成后,指标端点会返回 HTTP 500(响应正文为 An error has occurred while serving metrics: 和 invalid metric);重启 Pod 后指标才会恢复。导出器键 metrics-protocol、metrics-endpoint 和 metrics-export-interval 仅在 controller 启动时读取,因此更改这些键也只有在 Pod 重启后才会生效;重启会短暂暂停 PipelineRun 和 TaskRun 的协调。更改 Pod 规范设置时也需要重启(例如更改 Deployment 中的 CONFIG_OBSERVABILITY_NAME)。
集群管理员通过删除 controller 的 Pod 来重启 controller;Deployment 会创建新的 Pod。然后列出该 Pod:
kubectl delete pods -n tekton-pipelines -l app=tekton-pipelines-controller
kubectl get pods -n tekton-pipelines -l app=tekton-pipelines-controller
新的 Pod 必须处于 Running 状态,并且其 AGE 应短于重启至今经过的时间。执行 metrics-protocol: prometheus,然后按照直接检查指标端点中的说明检查指标。
通过 TektonConfig 配置的示例:
apiVersion: operator.tekton.dev/v1alpha1
kind: TektonConfig
metadata:
name: config
spec:
pipeline:
options:
disabled: false
configMaps:
config-observability:
data:
metrics-protocol: prometheus
# PipelineRun metrics aggregation level.
# Values: "pipelinerun" | "pipeline" (default) | "namespace"
# - "pipelinerun": includes pipeline + pipelinerun labels (new series for every run; with histogram, one per bucket)
# - "pipeline": includes pipeline label only
# - "namespace": no pipeline/pipelinerun labels
metrics.pipelinerun.level: "pipeline"
# TaskRun metrics aggregation level.
# Values: "taskrun" | "task" (default) | "namespace"
# - "taskrun": includes task + taskrun labels (new series for every run; with histogram, one per bucket)
# - "task": includes task label only
# - "namespace": no task/taskrun labels
metrics.taskrun.level: "task"
# Duration metric type for PipelineRun / TaskRun.
# Values: "histogram" (default) | "lastvalue"
# Note: The level settings do not change the duration type; it always follows
# metrics.*.duration-type. The "pipelinerun" and "taskrun" levels add per-run
# labels and significantly increase the number of time series.
metrics.pipelinerun.duration-type: "histogram"
metrics.taskrun.duration-type: "histogram"
# Running PipelineRun metrics aggregation level.
# Values: "pipelinerun" | "pipeline" | "namespace" | "" (default, cluster-level)
metrics.running-pipelinerun.level: ""
# Include reason label on duration metrics (pipelinerun_duration_seconds,
# taskrun_duration_seconds, pipelinerun_taskrun_duration_seconds).
# Values: "true" | "false" (default)
# Warning: Enabling this increases label cardinality.
# Note: Despite the key name, this does NOT affect count metrics
# (pipelinerun_total / taskrun_total), only duration metrics.
metrics.count.enable-reason: "false"
# Include namespace label on throttled TaskRun metrics.
# Values: "true" | "false" (default)
metrics.taskrun.throttle.enable-namespace: "false"
将示例保存到诸如 tektonconfig-observability.yaml 的文件中。若只需更改部分键,请仅在文件中保留这些键,并按照示例将其嵌套在 spec.pipeline.options.configMaps.config-observability.data 下。将该文件合并到现有的 TektonConfig 中。合并补丁只会更改文件中的键,并保留 TektonConfig 的其他字段:
kubectl patch tektonconfig config --type merge --patch-file tektonconfig-observability.yaml
对于仅包含 TektonConfig 部分内容的文件,请勿使用 kubectl apply -f:后续对另一个部分文件执行 kubectl apply -f 时,会删除该文件设置但后续文件中未包含的字段。
合并更改后,检查值是否已写入 ConfigMap:
kubectl get configmap config-observability -n tekton-pipelines -o yaml
data 下、注释的 _example 块之外的键,必须显示您设置的值。
Histogram 存储桶
当持续时间类型为 histogram 时,将使用以下存储桶边界(以秒为单位):
10, 30, 60, 300, 900, 1800, 3600, 5400, 10800, 21600, 43200, 86400
这对应于:10s、30s、1m、5m、15m、30m、1h、1.5h、3h、6h、12h、24h。
生产环境推荐配置
对于生产环境,请使用聚合级别来控制标签基数。将以下键以及其后的调试示例中的键设置在上述示例的 spec.pipeline.options.configMaps.config-observability.data 下:
metrics.pipelinerun.level: "pipeline"
metrics.taskrun.level: "task"
metrics.pipelinerun.duration-type: "histogram"
metrics.taskrun.duration-type: "histogram"
metrics.count.enable-reason: "false"
如果需要逐运行粒度进行调试,请临时切换为:
metrics.pipelinerun.level: "pipelinerun"
metrics.taskrun.level: "taskrun"
请注意,这会显著增加时间序列的数量。切换回 pipeline 和 task 不会删除已经创建的逐运行序列;由于 controller 进程会将其保存在内存中,这些序列会一直保留在 controller 的指标输出中,直到 tekton-pipelines-controller Pod 重启。请在测试集群上使用这些级别,或仅在较短的调试窗口内使用。
Tekton Triggers
Tekton Triggers 组件从不同进程中公开两类指标。
Controller 指标(端口 9000)
Triggers controller 每 60 秒报告一次资源数量指标。
Triggers controller 指标使用前缀 controller_。
EventListener Sink 指标
每个 EventListener Pod 都会公开其他 HTTP 和事件处理指标。这些指标来自 EventListener sink 进程(而非 controller)。Prometheus 指标前缀为 eventlistener_。
eventlistener_http_duration_seconds 直方图桶:0.001, 0.01, 0.1, 1, 5, 10(秒)
eventlistener_event_received_total 状态值:succeeded、failed
eventlistener_triggered_resources_total 类型值:所创建对象的 Kubernetes 资源 Kind(例如 PipelineRun、TaskRun)
这些 sink 指标按 EventListener Pod 公开,而不是来自中央 controller。如果 EventListener Pod 公开了指标端口,可能需要单独的 ServiceMonitor 或 PodMonitor 来抓取这些指标。EventListener 的 Service 会在事件端口旁发布此指标端口(9000、http-metrics);在使用 NodePort 或 LoadBalancer Service 之前,请参阅网络配置。
Tekton Results
Tekton Results 包含两个公开指标的子服务。
Watcher 指标
Watcher 指标使用前缀 watcher_。Counter 指标使用后缀 _total 导出,例如 watcher_pipelinerun_delete_count_total 和 watcher_runs_not_stored_count_total。
删除指标
* 可选标签取决于 Results Watcher 的 config-observability 设置。
注意:只有在 Watcher 实际删除 runs 时,才会记录 pipelinerun_delete_count、pipelinerun_delete_duration_seconds、taskrun_delete_count、taskrun_delete_duration_seconds、customrun_delete_count 和 customrun_delete_duration_seconds。除非在 tekton-results-watcher Deployment 上将 --completed_run_grace_period 标志设置为非零值,否则这些指标将保持为空(没有数据点)(具体通过 TektonConfig 设置,在下方复现步骤所示的 watcher 容器的 args 中,并位于 spec.result.options.deployments.tekton-results-watcher 下)。默认情况下,此标志为 0,会禁用自动删除。正数时长(例如 10m)会在该宽限期后删除 runs;负值会立即删除 runs。Watcher 不会等待日志存储完成后再删除 run。
警告:任何非零的 --completed_run_grace_period 都会使 Watcher 删除其监视的每个命名空间中的已完成 PipelineRun、TaskRun 和 CustomRun,默认情况下为所有命名空间,包括其他团队的 runs。Watcher 不会删除具有 owner reference 的 run:属于 PipelineRun 的 TaskRun 和 CustomRun 始终交由其 PipelineRun 管理;而由任何其他对象拥有的 run,在 Watcher 标志 -check_owner 保持默认值 true 时会被保留(随附的 Watcher 参数不会设置该标志)。已删除的 runs 无法恢复。Watcher 的 -label_selector 标志会将删除范围限制为标签匹配的 runs。请仅在专用测试集群上尝试此标志,如下方复现步骤所示。
Results Watcher 的状态标签值:
success - Run 已成功完成
failed - Run 因以下取消原因之外的其他原因而失败
cancelled - Run 以原因 Cancelled(PipelineRun)、TaskRunCancelled(TaskRun)或 CustomRunCancelled(CustomRun)结束
共享指标
这些指标由 Watcher 中的 PipelineRun、TaskRun 和 CustomRun reconciler 注册,用于跟踪存储相关事件。
kind 标签用于标识 run 类型。runs_not_stored_count 使用 PipelineRun、TaskRun 和 CustomRun;run_storage_latency_seconds 使用小写值 pipelinerun、taskrun 和 customrun。使用 Watcher 标志 -logs_api=true(随附值)时,仅针对 CustomRun 记录 run_storage_latency_seconds。
注意:只有在 Watcher 持有 finalizer 以协调归档期间,run 被外部删除(例如通过 kubectl delete)时,才会记录 runs_not_stored_count。除非满足以下全部条件,否则该指标将保持为空:
- 对于 PipelineRun 和 TaskRun,
--logs_api 标志为 false(日志存储已禁用);当 --logs_api 设置为 true 时,Watcher 会跳过针对 PipelineRun 和 TaskRun 的基于 finalizer 的协调。无论哪种情况,CustomRun 都会进行协调。
--disable_crd_update 标志为 false(已启用 annotation 更新)。
- 自 run 完成以来,
store_deadline 已经过。--store_deadline 标志始终会设置(默认值为 10m);这是 Watcher 在放弃归档并允许删除之前等待归档完成的最长时间。使用 0 时,run 完成后截止时间立即到期。
- Run 在完全不带
results.tekton.dev/stored annotation 的情况下被外部删除。仍在运行时已归档的 run 会带有 stored=false,即使其最终状态从未归档,也不会计入。
在正常运行中(在删除前已归档 runs,或 Watcher 自身通过 --completed_run_grace_period 触发删除),此 Counter 保持为零。非零值表示可能发生数据丢失:runs 在其状态保存到 Results API 之前就被删除。
快速复现(仅限专用测试集群):
如果看不到此指标,通常意味着未满足触发条件,而不是指标缺失。
TektonConfig 和 Results Watcher 作用于整个集群:当 Results API 缩放为 0 时,任何命名空间中的 run 都不会被归档,并且在此期间 store_deadline 之后删除的 run 永远不会被存储。请勿在其他团队使用的集群上执行这些步骤。
- 保存
TektonConfig 当前的 spec.result(kubectl get tektonconfig config -o jsonpath='{.spec.result}' > result-orig.json)。然后在 spec.result.options.deployments.tekton-results-watcher.spec.template.spec.containers 中设置 Watcher 标志:容器 watcher 使用 args、-logs_api=false、-disable_crd_update=false 以及一个 -store_deadline 值(例如 -store_deadline=30s)。spec.result.logs_api 只设置 Results API 的 LOGS_API 设置,不会更改 Watcher 的 -logs_api 标志。此更改会重启 Watcher:在执行第 2 步之前,等待新的 Watcher Pod 处于 Running(kubectl get pods -n tekton-pipelines -l app.kubernetes.io/name=tekton-results-watcher)状态,因为在 Results API 不可访问时启动的 Watcher 最多等待 1 分钟后就会退出。
- 通过
TektonConfig(spec.result.options.deployments.tekton-results-api.spec.replicas: 0)将 Results API 副本临时设置为 0,使 runs 无法归档。
- 创建 TaskRun 或 PipelineRun,并等待其完成。
- 等待
store_deadline 经过,然后从外部删除 run(kubectl delete taskrun <taskrun-name> -n <namespace> 或 kubectl delete pipelinerun <pipelinerun-name> -n <namespace>)。对于 TaskRun,Watcher 还会等待 run 完成后经过 -forward_buffer(默认值为 150s),因此删除操作和 Counter 可能需要这么长时间;在第 6 步恢复之前,请先检查第 5 步。
- 检查 Watcher
/metrics 或 Prometheus 中的 watcher_runs_not_stored_count_total(公开名称:组件前缀加上 Counter 后缀 _total);其值应增加。
- 通过将
spec.result 替换为第 1 步保存的副本来恢复它,并在路径 /spec/result 上使用 JSON patch op: replace(jq '[{"op":"replace","path":"/spec/result","value":.}]' result-orig.json > restore-patch.json,然后是 kubectl patch tektonconfig config --type json --patch-file restore-patch.json)。Merge patch 不会删除第 1 步和第 2 步添加的键。然后检查 tekton-results-api Deployment 的副本数是否已恢复,并检查 watcher 容器的 args 是否与第 1 步之前相同(默认安装中为 -api_addr、-auth_mode 和 -logs_api=true),且不包含第 1 步中的 -logs_api=false、-disable_crd_update=false 和 -store_deadline 值。
run_storage_latency_seconds 直方图使用以下桶边界(以秒为单位):
0.1, 0.5, 1, 2, 5, 10, 30, 60, 120, 300, 600, 1800
Watcher config-observability
Results Watcher 有自己的 config-observability ConfigMap,即 tekton-results-config-observability(在 CONFIG_OBSERVABILITY_NAME 环境变量中设置的名称)。此 ConfigMap 由 Tekton Operator 管理,应通过 TektonConfig 资源的 spec.result.options.configMaps 字段,在 tekton-results-config-observability 键下进行配置。有关详细信息,请参阅调整子组件的可选配置项。
重新加载行为:Results Watcher 也会监视此 ConfigMap。在下表的键中,两个 level 键(metrics.pipelinerun.level 和 metrics.taskrun.level)的更改无需重启即可生效;两个 duration-type 键会被解析但不会使用,因此更改它们不会产生效果。exporter 键 metrics-protocol(随附值为 prometheus)、metrics-endpoint 和 metrics-export-interval 仅在 Watcher 启动时读取,因此只有在 tekton-results-watcher Pod 重启后,更改才会生效;重启会短暂暂停 Watcher 的 reconciliation,而该过程会将 runs 归档到 Results。更改 Deployment 级别的设置(例如环境变量或参数)时,也需要重启。
以 Pipelines ConfigMap 的相同方式将更改合并到 TektonConfig 中:将键保存在 spec.result.options.configMaps.tekton-results-config-observability.data 下的文件中,然后运行 kubectl patch tektonconfig config --type merge --patch-file <file>。之后检查这些值是否已传递到 Watcher 的 ConfigMap。data 下、注释掉的 _example 块之外的键必须显示你设置的值:
kubectl get configmap tekton-results-config-observability -n tekton-pipelines -o yaml
更改 exporter 键后,集群管理员删除 Watcher 的 Pod 以重启 Watcher,并检查新 Pod 是否为 Running,且其 AGE 小于重启后的时间:
kubectl delete pods -n tekton-pipelines -l app.kubernetes.io/name=tekton-results-watcher
kubectl get pods -n tekton-pipelines -l app.kubernetes.io/name=tekton-results-watcher
除上述介绍的 exporter 键 metrics-protocol、metrics-endpoint 和 metrics-export-interval 外,Watcher 的 ConfigMap 还支持以下 metrics.* 键:
注意:与 Tekton Pipelines 不同,Results Watcher 不支持 pipelinerun / taskrun 单个 run 的粒度级别。它也没有 metrics.count.enable-reason、metrics.running-pipelinerun.level 或 metrics.taskrun.throttle.enable-namespace 键。
Watcher 将 pipelinerun_delete_duration_seconds、taskrun_delete_duration_seconds 和 customrun_delete_duration_seconds 创建为 OpenTelemetry 直方图,因此将任一 duration-type 键设置为 lastvalue 不会产生效果。
API Server 指标
API server 通过 go-grpc-prometheus 库在端口 9090 上公开标准 gRPC Prometheus 指标。这些指标包括:
grpc_server_handled_total - 服务器上完成的 RPC 总数
grpc_server_started_total - 服务器上启动的 RPC 总数
grpc_server_msg_received_total / grpc_server_msg_sent_total - 消息数量
grpc_server_handling_seconds - RPC 处理持续时间,仅在启用 PROMETHEUS_HISTOGRAM 时提供。随附的 tekton-results-api-config ConfigMap 设置了 PROMETHEUS_HISTOGRAM=false,因此 API server 默认不会公开此指标
Tekton Chains
Tekton Chains 是一个安全组件,用于为使用 Tekton Pipelines 构建的制品生成、签名和存储来源证明。它会观察已完成的 TaskRun 和 PipelineRun,然后创建证明和签名。
Chains controller 指标使用前缀 watcher_(与 Results Watcher 相同,但自定义指标名称不同,因此不会发生冲突)。
Chains 指标
所有 Chains 指标都是 Counter。只有签名失败计数器带有标签(error_type)。
注意:Chains 仅在配置了 signer 时才会对有效负载进行签名和存储;默认情况下,signer 使用 signing-secrets Secret 中的密钥。只有当 Chains 配置中的 generateSigningSecret 为 true 时,Operator 才会创建密钥(默认值为 false);请参阅签名密钥配置。如果没有密钥,Chains 会记录 No signer <signer> configured for <type>,跳过签名,不会将缺少密钥的失败计数添加到 *_signing_failures_total,但仍会将 run 标记为已签名:*_marked_signed_total 会增加,而 *_sign_created_total 和 *_payload_stored_total 不会增加。在 controller 启动后尚未签署任何内容时,这两个计数器根本不会出现在指标输出中,因为 Chains 计数器只有在首次递增后才会出现。
Controller Framework 指标
Tekton controller 还会公开以下 framework 指标。这些指标不使用组件前缀:所有组件中的名称都相同,因此请通过监控堆栈抓取指标时添加的 target 标签来区分组件。workqueue 指标带有 name 标签,用于标识 reconciler,例如 github.com.tektoncd.pipeline.pkg.reconciler.pipelinerun.Reconciler。Pipelines-as-Code controller 和 EventListener sink 仅公开两个 Kubernetes API client 指标。
不存在带有组件前缀的 client_latency、reconcile_count 或 reconcile_latency 序列。
设置 ServiceMonitor
要启用对 Tekton 组件的 Prometheus 抓取,请部署 ServiceMonitor 资源。
前置条件请参阅前置条件。
请根据使用的监控堆栈遵循以下指导:
- 如果使用 Prometheus(Prometheus Operator),则
metadata.labels.prometheus: kube-prometheus 等标签必须与 Prometheus CR spec.serviceMonitorSelector 匹配;否则不会抓取此 ServiceMonitor。
- 如果使用 VictoriaMetrics,通常不需要
prometheus: kube-prometheus 等标签;请根据监控设置创建 ServiceMonitor/VMServiceScrape。
使用 Prometheus 时,请使用以下命令查找并验证 selector。这些命令需要对 Prometheus 资源具有读取权限(第一个命令会列出所有命名空间中的资源),通常需要集群管理员权限:
# 1) Locate Prometheus CRs (resource type: monitoring.coreos.com/v1, Kind=Prometheus)
kubectl get prometheus -A
# 2) Check ServiceMonitor selector on the target Prometheus instance
kubectl get prometheus -n <prometheus-namespace> <prometheus-name> -o jsonpath='{.spec.serviceMonitorSelector}'
# 3) Check the ServiceMonitor namespace selector on the same instance
kubectl get prometheus -n <prometheus-namespace> <prometheus-name> -o jsonpath='{.spec.serviceMonitorNamespaceSelector}'
如果第二个命令没有输出,则表示 Prometheus CR 未设置 serviceMonitorSelector。Prometheus Operator API 将未设置的 selector 定义为不匹配任何 ServiceMonitor,将空 selector({})定义为匹配所有资源。因此,在这种情况下,此 Prometheus 实例不会选择下面的 ServiceMonitor 资源,而不论这些资源的标签是什么。
如果第三个命令没有输出,则表示 Prometheus CR 未设置 serviceMonitorNamespaceSelector。Prometheus Operator API 将未设置的命名空间 selector 定义为仅匹配 Prometheus CR 所在的命名空间,将空 selector({})定义为匹配所有命名空间。因此,位于 tekton-pipelines 之外且未设置此字段的 Prometheus CR,不会选择下面的 ServiceMonitor 资源。
如果集群中不存在 Prometheus CR,监控通常由平台管理(例如 VictoriaMetrics)或通过其他方式实现。在这种情况下,通常不需要 prometheus: kube-prometheus 等标签;请遵循平台的抓取规则。
有关更多信息,请参阅 集成外部指标。
Pipeline ServiceMonitor
Pipeline ServiceMonitor YAML
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: tekton-pipelines-metrics
namespace: tekton-pipelines
labels:
app.kubernetes.io/name: tekton-pipelines
# prometheus: kube-prometheus
spec:
selector:
matchLabels:
app.kubernetes.io/part-of: tekton-pipelines
endpoints:
- port: http-metrics
path: /metrics
interval: 30s
namespaceSelector:
matchNames:
- tekton-pipelines
此 ServiceMonitor 会匹配带有 app.kubernetes.io/part-of: tekton-pipelines 标签的 Pipeline 服务(包括 remote-resolver),并在 tekton-pipelines 命名空间中抓取这些服务。
Triggers ServiceMonitor
Triggers ServiceMonitor YAML
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: tekton-triggers-metrics
namespace: tekton-pipelines
labels:
app.kubernetes.io/name: tekton-triggers
# prometheus: kube-prometheus
spec:
selector:
matchLabels:
app.kubernetes.io/part-of: tekton-triggers
app.kubernetes.io/component: controller
endpoints:
- port: http-metrics
path: /metrics
interval: 30s
namespaceSelector:
matchNames:
- tekton-pipelines
此 ServiceMonitor 仅收集 Triggers controller 指标(controller_*)。它不包含 EventListener sink 指标。
EventListener Sink ServiceMonitor
EventListener Sink ServiceMonitor YAML
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: tekton-eventlistener-sink-metrics
namespace: tekton-pipelines
labels:
app.kubernetes.io/name: tekton-eventlistener-sink
# prometheus: kube-prometheus
spec:
selector:
matchExpressions:
- key: eventlistener
operator: Exists
- key: app.kubernetes.io/managed-by
operator: In
values:
- EventListener
endpoints:
- port: http-metrics
path: /metrics
interval: 30s
namespaceSelector:
any: true
EventListener Service 通常在应用命名空间中运行,因此此示例使用 namespaceSelector.any: true 进行跨命名空间抓取。如果需要更严格的范围,请切换为 matchNames,并显式列出允许的命名空间。
Results ServiceMonitor
Results 服务同时具有 app.kubernetes.io/part-of: tekton-results 和 app.kubernetes.io/name 标签。为了精确定位 API + Watcher(并排除 Postgres),此示例匹配 app.kubernetes.io/name:
Results ServiceMonitor YAML
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: tekton-results-metrics
namespace: tekton-pipelines
labels:
app.kubernetes.io/name: tekton-results
# prometheus: kube-prometheus
spec:
selector:
matchExpressions:
- key: app.kubernetes.io/name
operator: In
values:
- tekton-results-api
- tekton-results-watcher
endpoints:
- port: prometheus
path: /metrics
interval: 30s
- port: metrics
path: /metrics
interval: 30s
namespaceSelector:
matchNames:
- tekton-pipelines
Results API server 使用端口名称 prometheus(9090),Watcher 使用端口名称 metrics(9090)。每个服务只公开其中一个端口名称,因此只会抓取匹配的 endpoint。
Chains ServiceMonitor
Chains ServiceMonitor YAML
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: tekton-chains-metrics
namespace: tekton-pipelines
labels:
app.kubernetes.io/name: tekton-chains
# prometheus: kube-prometheus
spec:
selector:
matchLabels:
app.kubernetes.io/part-of: tekton-chains
endpoints:
- port: http-metrics
path: /metrics
interval: 30s
namespaceSelector:
matchNames:
- tekton-pipelines
验证
部署 ServiceMonitor 资源后,验证 Prometheus 是否正在抓取目标。
直接检查指标端点
下面的每个 kubectl port-forward 命令都会在前台持续运行,直到使用 Ctrl+C 停止。请在第二个终端中运行并保持其运行,然后在第一个终端中运行后续的 curl 命令,最后停止端口转发。
这些检查假设每个组件只有一个副本:kubectl port-forward svc/<service> 会连接到 Service 的一个 Pod。当组件运行多个副本时,你连接到的 Pod 上可能不存在某个 counter;此时请对每个 Pod 将 svc/<service> 替换为 pod/<pod-name> 后重复检查,或者查询 Prometheus。
# Pipeline controller
$ kubectl port-forward -n tekton-pipelines svc/tekton-pipelines-controller 9090:9090
$ curl -s http://localhost:9090/metrics | grep tekton_pipelines_controller_
# HELP tekton_pipelines_controller_pipelinerun_total Number of pipelineruns
# TYPE tekton_pipelines_controller_pipelinerun_total counter
tekton_pipelines_controller_pipelinerun_total{otel_scope_name="tekton_pipelines_controller",otel_scope_schema_url="",otel_scope_version="",status="cancelled"} 11
tekton_pipelines_controller_pipelinerun_total{otel_scope_name="tekton_pipelines_controller",otel_scope_schema_url="",otel_scope_version="",status="failed"} 90
# Triggers controller
$ kubectl port-forward -n tekton-pipelines svc/tekton-triggers-controller 9000:9000
$ curl -s http://localhost:9000/metrics | grep controller_
# HELP controller_clusterinterceptor_count number of clusterinterceptor
# TYPE controller_clusterinterceptor_count gauge
controller_clusterinterceptor_count{otel_scope_name="github.com/tektoncd/triggers/pkg/reconciler/metrics",otel_scope_schema_url="",otel_scope_version=""} 5
# EventListener sink metrics (replace namespace/service)
$ kubectl port-forward -n <eventlistener-namespace> svc/<eventlistener-service> 9094:9000
$ curl -s http://localhost:9094/metrics | grep eventlistener_
# HELP eventlistener_event_received_total number of events received by sink
# TYPE eventlistener_event_received_total counter
eventlistener_event_received_total{otel_scope_name="github.com/tektoncd/triggers/pkg/sink",otel_scope_schema_url="",otel_scope_version="",status="succeeded"} 1
# HELP eventlistener_triggered_resources_total Count of the number of triggered eventlistener resources
# TYPE eventlistener_triggered_resources_total counter
eventlistener_triggered_resources_total{kind="TaskRun",otel_scope_name="github.com/tektoncd/triggers/pkg/sink",otel_scope_schema_url="",otel_scope_version=""} 1
# Results watcher
# With the shipped watcher argument -logs_api=true, the watcher records
# watcher_run_storage_latency_seconds only with kind="customrun", and can record
# watcher_runs_not_stored_count_total with kind="CustomRun" (when a CustomRun
# without the results.tekton.dev/stored annotation is deleted after
# store_deadline; see the runs_not_stored_count note); until a CustomRun
# completes, the grep prints nothing. The deletion metrics appear only when
# --completed_run_grace_period is set (see the note in the Tekton Results section).
$ kubectl port-forward -n tekton-pipelines svc/tekton-results-watcher 9091:9090
$ curl -s http://localhost:9091/metrics | grep watcher_
# HELP watcher_run_storage_latency_seconds time from run completion to successful storage
# TYPE watcher_run_storage_latency_seconds histogram
watcher_run_storage_latency_seconds_bucket{kind="customrun",namespace="<namespace>",otel_scope_name="tekton.dev/results",otel_scope_schema_url="",otel_scope_version="",le="1"} 0
watcher_run_storage_latency_seconds_bucket{kind="customrun",namespace="<namespace>",otel_scope_name="tekton.dev/results",otel_scope_schema_url="",otel_scope_version="",le="2"} 3
watcher_run_storage_latency_seconds_bucket{kind="customrun",namespace="<namespace>",otel_scope_name="tekton.dev/results",otel_scope_schema_url="",otel_scope_version="",le="5"} 3
# To tell an empty watcher_ output from a failed port-forward or endpoint, count
# the target_info line, which the endpoint returns with every scrape: the command
# prints 1 when the endpoint answered and 0 when it did not.
$ curl -s http://localhost:9091/metrics | grep -c '^target_info'
# Results API
$ kubectl port-forward -n tekton-pipelines svc/tekton-results-api-service 9092:9090
$ curl -s http://localhost:9092/metrics | grep grpc_server_
# HELP grpc_server_handled_total Total number of RPCs completed on the server, regardless of success or failure.
# TYPE grpc_server_handled_total counter
grpc_server_handled_total{grpc_code="Aborted",grpc_method="Check",grpc_service="grpc.health.v1.Health",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Aborted",grpc_method="CreateRecord",grpc_service="tekton.results.v1alpha2.Results",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Aborted",grpc_method="CreateResult",grpc_service="tekton.results.v1alpha2.Results",grpc_type="unary"} 0
# HELP grpc_server_started_total Total number of RPCs started on the server.
# TYPE grpc_server_started_total counter
grpc_server_started_total{grpc_method="Check",grpc_service="grpc.health.v1.Health",grpc_type="unary"} 337606
grpc_server_started_total{grpc_method="CreateRecord",grpc_service="tekton.results.v1alpha2.Results",grpc_type="unary"} 10301
grpc_server_started_total{grpc_method="CreateResult",grpc_service="tekton.results.v1alpha2.Results",grpc_type="unary"} 832
# Chains controller
$ kubectl port-forward -n tekton-pipelines svc/tekton-chains-metrics 9093:9090
$ curl -s http://localhost:9093/metrics | grep watcher_
# HELP watcher_pipelinerun_marked_signed_total Total number of objects marked as signed for pipelineruns
# TYPE watcher_pipelinerun_marked_signed_total counter
watcher_pipelinerun_marked_signed_total{otel_scope_name="github.com/tektoncd/chains/pkg/pipelinerunmetrics",otel_scope_schema_url="",otel_scope_version=""} 56
# A Chains counter appears only after its first increment, so the grep prints
# nothing until the controller has counted a run since it started. To tell that
# from a failed port-forward or endpoint, count the target_info line, which the
# endpoint returns with every scrape: the command prints 1 when the endpoint
# answered and 0 when it did not.
$ curl -s http://localhost:9093/metrics | grep -c '^target_info'
EventListener sink 指标(例如 eventlistener_event_received_total 和 eventlistener_http_duration_seconds)由请求驱动。在验证这些指标之前,至少向 EventListener 发送一次请求。
检查 Prometheus 目标
# Verify ServiceMonitor resources exist
$ kubectl get servicemonitor -n tekton-pipelines
NAME AGE
tekton-chains-metrics 10m
tekton-eventlistener-sink-metrics 10m
tekton-pipelines-metrics 10m
tekton-results-metrics 10m
tekton-triggers-metrics 10m
# Check Prometheus targets (via Prometheus UI or API)
# Look for targets whose job label is the name of a scraped Service:
# the samples set no spec.jobLabel, so Prometheus Operator uses the
# Service name as the job label, not the ServiceMonitor name
PromQL 查询示例
# PipelineRun cumulative success rate (avoids misinterpretation in empty completion windows)
100 * sum(tekton_pipelines_controller_pipelinerun_total{status="success"}) / clamp_min(sum(tekton_pipelines_controller_pipelinerun_total), 1)
# Completed PipelineRuns in the last 5 minutes (throughput)
round(sum(increase(tekton_pipelines_controller_pipelinerun_total[5m])))
# PipelineRun duration P95 (histogram mode)
histogram_quantile(0.95,
rate(tekton_pipelines_controller_pipelinerun_duration_seconds_bucket[5m])
)
# TaskRun duration P95, standalone TaskRuns (histogram mode)
histogram_quantile(0.95,
sum by (le) (rate(tekton_pipelines_controller_taskrun_duration_seconds_bucket[5m]))
)
# TaskRun duration P95, TaskRuns that run in a PipelineRun (histogram mode)
# Each TaskRun is recorded in only one of the two histograms; adding them with `+`
# returns nothing while either histogram has no series in the window.
histogram_quantile(0.95,
sum by (le) (rate(tekton_pipelines_controller_pipelinerun_taskrun_duration_seconds_bucket[5m]))
)
# PipelineRun duration (lastvalue mode)
avg_over_time(tekton_pipelines_controller_pipelinerun_duration_seconds[5m])
# Currently running PipelineRuns (single series to avoid duplicate legends)
max(tekton_pipelines_controller_running_pipelineruns)
# TaskRuns throttled by resource quota (no data while no TaskRun is throttled).
# This query and the TaskRuns Throttled dashboard panel assume
# metrics.taskrun.throttle.enable-namespace: "false" (the default). With "true" the
# controller reports one series per namespace and no total, so max() returns the
# largest per-namespace count.
max(tekton_pipelines_controller_running_taskruns_throttled_by_quota)
# Trigger resource counts
controller_eventlistener_count
controller_triggertemplate_count
# Chains signing activity
watcher_taskrun_sign_created_total
watcher_pipelinerun_sign_created_total
推荐设置
启用时机。当平台或项目团队需要查看集群中的构建吞吐量和持续时间、签名活动以及事件驱动的触发情况时,启用指标收集。config-observability ConfigMap 控制共享的 Pipeline controller,并通过 TektonConfig 进行管理;控制平面的 ServiceMonitor 资源则以 tekton-pipelines namespace 为目标。
推荐值。从设置 ServiceMonitor部署 Pipeline、Triggers 和 Chains ServiceMonitor 资源;对于运行 EventListener 的 namespace,添加 EventListener sink ServiceMonitor;安装 Results 后,添加 Results ServiceMonitor。在 config-observability 中使用生产环境推荐配置中的值:metrics.pipelinerun.level: "pipeline"、metrics.taskrun.level: "task"、metrics.pipelinerun.duration-type: "histogram"、metrics.taskrun.duration-type: "histogram" 和 metrics.count.enable-reason: "false"。保留 histogram duration type,因为监控面板示例中的 P50/P95 duration 图表会查询 *_bucket series。仅在测试集群或短时间的调试窗口中使用 pipelinerun 和 taskrun levels,因为它们会添加每次运行的 labels,并显著增加 time series 的数量;duration type 仍遵循 metrics.*.duration-type。切换回 pipeline 和 task 不会移除已经创建的每次运行 series:它们会一直保留,直到 tekton-pipelines-controller Pod 重启。在 cpaas-system namespace 中的 tekton folder 下部署一次这些监控面板。
要确认结果,请检查值是否如config-observability 配置中所述达到 config-observability,端点是否能够响应,以及 Prometheus 是否如验证中所述抓取目标;同时,使用监控面板示例中的 kubectl get monitordashboard command 检查监控面板是否存在。在默认安装中,config-observability 已经具有推荐值,因此无需撤销这些设置;请按照生产环境推荐配置中的说明将调试 levels 切换回来。要移除监控设置,请删除你创建的 ServiceMonitor 和 MonitorDashboard 资源,并且只指定这些资源,例如:
kubectl delete servicemonitor -n tekton-pipelines tekton-pipelines-metrics tekton-triggers-metrics tekton-eventlistener-sink-metrics tekton-results-metrics tekton-chains-metrics
kubectl delete monitordashboard -n cpaas-system tekton-pipeline tekton-triggers tekton-results tekton-chains
删除这些资源需要在 tekton-pipelines 和 cpaas-system 中拥有相应资源的删除权限。
前提条件。Prometheus 或 VictoriaMetrics 监控栈必须能够发现你创建的 ServiceMonitor 资源;使用 Prometheus Operator 时,ServiceMonitor labels 必须与 Prometheus 的 spec.serviceMonitorSelector 匹配。指标抓取 Pod 必须能够访问大多数控制平面服务上的 9090 端口,以及 Triggers controller 和 EventListener sink 上的 9000 端口。完整列表请参阅前提条件,其中包括创建 ServiceMonitor 资源所需的权限,以及为监控面板在 cpaas-system 中创建 MonitorDashboard 资源所需的权限。
监控面板示例
以下 MonitorDashboard 资源提供了用于监控 Tekton 组件的开箱即用监控面板。将它们部署到 cpaas-system namespace,并置于 tekton folder 下。应用后,检查它们是否存在:
kubectl get monitordashboard -n cpaas-system -l cpaas.io/dashboard.folder=tekton
输出必须列出你应用的监控面板:tekton-pipeline、tekton-triggers、tekton-results 和 tekton-chains。No resources found in cpaas-system namespace. 表示其中没有任何一个被创建。
重要:每个 panel 必须包含 id(唯一整数)、datasource: prometheus 和 transformations: []。每个 target 必须包含 datasource: prometheus 和 refId。本文档中的 duration P50/P95 panel 使用 *_bucket 查询,并要求 metrics.*.duration-type=histogram;如果使用 lastvalue,请将这些查询替换为 LastValue 风格的表达式,例如 avg_over_time(...)。在 gridPos 中为 y key 加引号("y": 0);未加引号的 y 会被解析为布尔值,导致 panel 丢失其垂直位置。
Tekton Pipeline 监控面板
Tekton Pipeline Dashboard YAML
kind: MonitorDashboard
apiVersion: ait.alauda.io/v1alpha2
metadata:
labels:
cpaas.io/dashboard.folder: tekton
cpaas.io/dashboard.is.home.dashboard: "false"
cpaas.io/dashboard.tag.tekton: "true"
name: tekton-pipeline
namespace: cpaas-system
spec:
body:
title: Tekton Pipeline Overview
titleZh: Tekton Pipeline Overview
tags:
- tekton
time:
from: now-1h
to: now
templating:
list: []
panels:
- id: 1
title: PipelineRun Total (by status)
type: timeseries
datasource: prometheus
gridPos:
h: 8
w: 8
x: 0
"y": 0
targets:
- datasource: prometheus
expr: sum by (status) (tekton_pipelines_controller_pipelinerun_total)
refId: A
fieldConfig:
defaults:
color:
mode: palette-classic
custom:
drawStyle: line
fillOpacity: 0
lineWidth: 1
spanNulls: false
thresholds:
mode: absolute
steps:
- color: green
value: null
overrides: []
options:
legend:
calcs:
- latest
displayMode: list
placement: bottom
showLegend: true
tooltip:
mode: multi
sort: desc
transformations: []
- id: 2
title: TaskRun Total (by status)
type: timeseries
datasource: prometheus
gridPos:
h: 8
w: 8
x: 8
"y": 0
targets:
- datasource: prometheus
expr: sum by (status) (tekton_pipelines_controller_taskrun_total)
refId: A
fieldConfig:
defaults:
color:
mode: palette-classic
custom:
drawStyle: line
fillOpacity: 0
lineWidth: 1
spanNulls: false
thresholds:
mode: absolute
steps:
- color: green
value: null
overrides: []
options:
legend:
calcs:
- latest
displayMode: list
placement: bottom
showLegend: true
tooltip:
mode: multi
sort: desc
transformations: []
- id: 3
title: PipelineRun Success Rate (cumulative)
type: timeseries
datasource: prometheus
gridPos:
h: 8
w: 4
x: 16
"y": 0
targets:
- datasource: prometheus
expr: "100 * sum(tekton_pipelines_controller_pipelinerun_total{status=\"success\"}) / clamp_min(sum(tekton_pipelines_controller_pipelinerun_total), 1)"
refId: A
fieldConfig:
defaults:
unit: percent
color:
mode: thresholds
custom:
drawStyle: line
fillOpacity: 0
lineWidth: 1
spanNulls: false
thresholds:
mode: absolute
steps:
- color: red
value: null
- color: orange
value: 80
- color: green
value: 95
overrides: []
options:
legend:
calcs:
- latest
displayMode: list
placement: bottom
showLegend: true
tooltip:
mode: multi
sort: desc
transformations: []
- id: 12
title: Completed PipelineRuns (last 5m)
type: timeseries
datasource: prometheus
gridPos:
h: 8
w: 4
x: 20
"y": 0
targets:
- datasource: prometheus
expr: "round(sum(increase(tekton_pipelines_controller_pipelinerun_total[5m])))"
legendFormat: completed
refId: A
fieldConfig:
defaults:
unit: short
decimals: 0
color:
mode: palette-classic
custom:
drawStyle: line
fillOpacity: 0
lineWidth: 1
spanNulls: false
thresholds:
mode: absolute
steps:
- color: green
value: null
overrides: []
options:
legend:
calcs:
- latest
displayMode: list
placement: bottom
showLegend: true
tooltip:
mode: multi
sort: desc
transformations: []
- id: 4
title: Running PipelineRuns
type: timeseries
datasource: prometheus
gridPos:
h: 8
w: 8
x: 0
"y": 8
targets:
- datasource: prometheus
expr: max(tekton_pipelines_controller_running_pipelineruns)
legendFormat: running
refId: A
fieldConfig:
defaults:
color:
mode: palette-classic
custom:
drawStyle: line
fillOpacity: 0
lineWidth: 1
spanNulls: false
thresholds:
mode: absolute
steps:
- color: green
value: null
overrides: []
options:
legend:
calcs:
- latest
displayMode: list
placement: bottom
showLegend: true
tooltip:
mode: multi
sort: desc
transformations: []
- id: 5
title: Running TaskRuns
type: timeseries
datasource: prometheus
gridPos:
h: 8
w: 8
x: 8
"y": 8
targets:
- datasource: prometheus
expr: max(tekton_pipelines_controller_running_taskruns)
legendFormat: running
refId: A
fieldConfig:
defaults:
color:
mode: palette-classic
custom:
drawStyle: line
fillOpacity: 0
lineWidth: 1
spanNulls: false
thresholds:
mode: absolute
steps:
- color: green
value: null
overrides: []
options:
legend:
calcs:
- latest
displayMode: list
placement: bottom
showLegend: true
tooltip:
mode: multi
sort: desc
transformations: []
- id: 6
title: TaskRuns Throttled
type: timeseries
datasource: prometheus
gridPos:
h: 8
w: 8
x: 16
"y": 8
targets:
- datasource: prometheus
expr: max(tekton_pipelines_controller_running_taskruns_throttled_by_quota)
legendFormat: by quota
refId: A
- datasource: prometheus
expr: max(tekton_pipelines_controller_running_taskruns_throttled_by_node)
legendFormat: by node
refId: B
fieldConfig:
defaults:
color:
mode: palette-classic
custom:
drawStyle: line
fillOpacity: 0
lineWidth: 1
spanNulls: false
thresholds:
mode: absolute
steps:
- color: orange
value: null
overrides: []
options:
legend:
calcs:
- latest
displayMode: list
placement: bottom
showLegend: true
tooltip:
mode: multi
sort: desc
transformations: []
- id: 7
title: PipelineRun Duration P50 / P95
type: timeseries
datasource: prometheus
gridPos:
h: 8
w: 8
x: 0
"y": 16
targets:
- datasource: prometheus
expr: (histogram_quantile(0.5, sum by (le) (rate(tekton_pipelines_controller_pipelinerun_duration_seconds_bucket[5m])))) and on() (sum(rate(tekton_pipelines_controller_pipelinerun_duration_seconds_bucket{le="+Inf"}[5m])) > 0)
legendFormat: P50
refId: A
- datasource: prometheus
expr: (histogram_quantile(0.95, sum by (le) (rate(tekton_pipelines_controller_pipelinerun_duration_seconds_bucket[5m])))) and on() (sum(rate(tekton_pipelines_controller_pipelinerun_duration_seconds_bucket{le="+Inf"}[5m])) > 0)
legendFormat: P95
refId: B
fieldConfig:
defaults:
unit: s
color:
mode: palette-classic
custom:
drawStyle: line
fillOpacity: 0
lineWidth: 1
spanNulls: false
thresholds:
mode: absolute
steps:
- color: green
value: null
overrides: []
options:
legend:
calcs:
- latest
displayMode: list
placement: bottom
showLegend: true
tooltip:
mode: multi
sort: desc
transformations: []
- id: 8
title: TaskRun Duration P50 / P95 (Standalone)
type: timeseries
datasource: prometheus
gridPos:
h: 8
w: 8
x: 8
"y": 16
targets:
- datasource: prometheus
expr: (histogram_quantile(0.5, sum by (le) (rate(tekton_pipelines_controller_taskrun_duration_seconds_bucket[5m])))) and on() (sum(rate(tekton_pipelines_controller_taskrun_duration_seconds_bucket{le="+Inf"}[5m])) > 0)
legendFormat: P50
refId: A
- datasource: prometheus
expr: (histogram_quantile(0.95, sum by (le) (rate(tekton_pipelines_controller_taskrun_duration_seconds_bucket[5m])))) and on() (sum(rate(tekton_pipelines_controller_taskrun_duration_seconds_bucket{le="+Inf"}[5m])) > 0)
legendFormat: P95
refId: B
fieldConfig:
defaults:
unit: s
color:
mode: palette-classic
custom:
drawStyle: line
fillOpacity: 0
lineWidth: 1
spanNulls: false
thresholds:
mode: absolute
steps:
- color: green
value: null
overrides: []
options:
legend:
calcs:
- latest
displayMode: list
placement: bottom
showLegend: true
tooltip:
mode: multi
sort: desc
transformations: []
- id: 13
title: TaskRun Duration P50 / P95 (In-Pipeline)
type: timeseries
datasource: prometheus
gridPos:
h: 8
w: 8
x: 16
"y": 16
targets:
- datasource: prometheus
expr: (histogram_quantile(0.5, sum by (le) (rate(tekton_pipelines_controller_pipelinerun_taskrun_duration_seconds_bucket[5m])))) and on() (sum(rate(tekton_pipelines_controller_pipelinerun_taskrun_duration_seconds_bucket{le="+Inf"}[5m])) > 0)
legendFormat: P50
refId: A
- datasource: prometheus
expr: (histogram_quantile(0.95, sum by (le) (rate(tekton_pipelines_controller_pipelinerun_taskrun_duration_seconds_bucket[5m])))) and on() (sum(rate(tekton_pipelines_controller_pipelinerun_taskrun_duration_seconds_bucket{le="+Inf"}[5m])) > 0)
legendFormat: P95
refId: B
fieldConfig:
defaults:
unit: s
color:
mode: palette-classic
custom:
drawStyle: line
fillOpacity: 0
lineWidth: 1
spanNulls: false
thresholds:
mode: absolute
steps:
- color: green
value: null
overrides: []
options:
legend:
calcs:
- latest
displayMode: list
placement: bottom
showLegend: true
tooltip:
mode: multi
sort: desc
transformations: []
- id: 9
title: Workqueue Depth
type: timeseries
datasource: prometheus
gridPos:
h: 8
w: 8
x: 0
"y": 24
targets:
- datasource: prometheus
expr: max(kn_workqueue_depth{name=~"github.com.tektoncd.pipeline.pkg.reconciler.(pipelinerun|taskrun).Reconciler"})
legendFormat: depth
refId: A
fieldConfig:
defaults:
color:
mode: palette-classic
custom:
drawStyle: line
fillOpacity: 0
lineWidth: 1
spanNulls: false
thresholds:
mode: absolute
steps:
- color: green
value: null
overrides: []
options:
legend:
calcs:
- latest
displayMode: list
placement: bottom
showLegend: true
tooltip:
mode: multi
sort: desc
transformations: []
- id: 10
title: Workqueue Adds and Retries
type: timeseries
datasource: prometheus
gridPos:
h: 8
w: 8
x: 8
"y": 24
targets:
- datasource: prometheus
expr: sum(increase(kn_workqueue_adds_total{name=~"github.com.tektoncd.pipeline.pkg.reconciler.(pipelinerun|taskrun).Reconciler"}[5m]))
legendFormat: adds
refId: A
- datasource: prometheus
expr: sum(increase(kn_workqueue_retries_total{name=~"github.com.tektoncd.pipeline.pkg.reconciler.(pipelinerun|taskrun).Reconciler"}[5m]))
legendFormat: retries
refId: B
fieldConfig:
defaults:
color:
mode: palette-classic
custom:
drawStyle: line
fillOpacity: 0
lineWidth: 1
spanNulls: false
thresholds:
mode: absolute
steps:
- color: green
value: null
overrides: []
options:
legend:
calcs:
- latest
displayMode: list
placement: bottom
showLegend: true
tooltip:
mode: multi
sort: desc
transformations: []
- id: 11
title: Resolution Waiting
type: timeseries
datasource: prometheus
gridPos:
h: 8
w: 8
x: 16
"y": 24
targets:
- datasource: prometheus
expr: max(tekton_pipelines_controller_running_pipelineruns_waiting_on_pipeline_resolution)
legendFormat: PR waiting pipeline
refId: A
- datasource: prometheus
expr: max(tekton_pipelines_controller_running_pipelineruns_waiting_on_task_resolution)
legendFormat: PR waiting task
refId: B
- datasource: prometheus
expr: max(tekton_pipelines_controller_running_taskruns_waiting_on_task_resolution_count)
legendFormat: TR waiting task
refId: C
fieldConfig:
defaults:
color:
mode: palette-classic
custom:
drawStyle: line
fillOpacity: 0
lineWidth: 1
spanNulls: false
thresholds:
mode: absolute
steps:
- color: orange
value: null
overrides: []
options:
legend:
calcs:
- latest
displayMode: list
placement: bottom
showLegend: true
tooltip:
mode: multi
sort: desc
transformations: []
Tekton Pipeline 监控面板说明(常见问题)
PipelineRun Total (by status) 是由 controller 记录的完成事件计数器,而不是 PipelineRun 对象的总数。以 Cancelled 为 reason 结束的 PipelineRun(包括使用 spec.status: Cancelled 取消的 PipelineRun)会计入 status="cancelled"。使用 spec.status: CancelledRunFinally 或 StoppedRunFinally 取消或停止的 PipelineRun 中,如果某个 task 或 finally task 失败,该 PipelineRun 会以 Failed 为 reason 结束,并计入 status="failed"。
Running PipelineRuns 是实时快照(当前正在运行的数量)。它可以独立于 PipelineRun Total 发生变化。
Completed PipelineRuns (last 5m) 表示吞吐量(过去 5 分钟内新完成的运行次数)。在流量较低或空闲期间看到 0 是预期行为。
PipelineRun Success Rate (cumulative) 是自 controller 启动以来的累计值,而不是 5 分钟时间窗口内的成功率。短期失败不会立即导致其发生较大变化。
Workqueue Adds and Retries 统计过去 5 分钟内添加到 PipelineRun 和 TaskRun reconciler 工作队列以及在其中重试的项目数,而不是 PipelineRun 数量。
- 仅当选定时间范围内实际存在相应 label 值的样本时,才会显示状态 series。如果某个状态在该时间窗口内没有样本,则不会显示其曲线或图例。
TaskRun Duration P50 / P95 (Standalone) 和 TaskRun Duration P50 / P95 (In-Pipeline) 被拆分,以避免混合查询不稳定。在只公开其中一个 histogram family 的环境中,另一个 panel 可能为空,这是预期行为。
Tekton Triggers 监控面板
Tekton Triggers Dashboard YAML
kind: MonitorDashboard
apiVersion: ait.alauda.io/v1alpha2
metadata:
labels:
cpaas.io/dashboard.folder: tekton
cpaas.io/dashboard.is.home.dashboard: "false"
cpaas.io/dashboard.tag.tekton: "true"
name: tekton-triggers
namespace: cpaas-system
spec:
body:
title: Tekton Triggers Overview
titleZh: Tekton Triggers Overview
tags:
- tekton
time:
from: now-1h
to: now
templating:
list: []
panels:
- id: 1
title: EventListener Count
type: timeseries
datasource: prometheus
gridPos:
h: 6
w: 5
x: 0
"y": 0
targets:
- datasource: prometheus
expr: controller_eventlistener_count
legendFormat: EventListener
refId: A
fieldConfig:
defaults:
color:
mode: palette-classic
custom:
drawStyle: line
fillOpacity: 0
lineWidth: 1
spanNulls: false
thresholds:
mode: absolute
steps:
- color: green
value: null
overrides: []
options:
legend:
calcs:
- latest
displayMode: list
placement: bottom
showLegend: true
tooltip:
mode: multi
sort: desc
transformations: []
- id: 2
title: TriggerTemplate Count
type: timeseries
datasource: prometheus
gridPos:
h: 6
w: 5
x: 5
"y": 0
targets:
- datasource: prometheus
expr: controller_triggertemplate_count
legendFormat: TriggerTemplate
refId: A
fieldConfig:
defaults:
color:
mode: palette-classic
custom:
drawStyle: line
fillOpacity: 0
lineWidth: 1
spanNulls: false
thresholds:
mode: absolute
steps:
- color: green
value: null
overrides: []
options:
legend:
calcs:
- latest
displayMode: list
placement: bottom
showLegend: true
tooltip:
mode: multi
sort: desc
transformations: []
- id: 3
title: TriggerBinding Count
type: timeseries
datasource: prometheus
gridPos:
h: 6
w: 5
x: 10
"y": 0
targets:
- datasource: prometheus
expr: controller_triggerbinding_count
legendFormat: TriggerBinding
refId: A
fieldConfig:
defaults:
color:
mode: palette-classic
custom:
drawStyle: line
fillOpacity: 0
lineWidth: 1
spanNulls: false
thresholds:
mode: absolute
steps:
- color: green
value: null
overrides: []
options:
legend:
calcs:
- latest
displayMode: list
placement: bottom
showLegend: true
tooltip:
mode: multi
sort: desc
transformations: []
- id: 4
title: ClusterTriggerBinding
type: timeseries
datasource: prometheus
gridPos:
h: 6
w: 5
x: 15
"y": 0
targets:
- datasource: prometheus
expr: controller_clustertriggerbinding_count
legendFormat: ClusterTriggerBinding
refId: A
fieldConfig:
defaults:
color:
mode: palette-classic
custom:
drawStyle: line
fillOpacity: 0
lineWidth: 1
spanNulls: false
thresholds:
mode: absolute
steps:
- color: green
value: null
overrides: []
options:
legend:
calcs:
- latest
displayMode: list
placement: bottom
showLegend: true
tooltip:
mode: multi
sort: desc
transformations: []
- id: 5
title: ClusterInterceptor
type: timeseries
datasource: prometheus
gridPos:
h: 6
w: 4
x: 20
"y": 0
targets:
- datasource: prometheus
expr: controller_clusterinterceptor_count
legendFormat: ClusterInterceptor
refId: A
fieldConfig:
defaults:
color:
mode: palette-classic
custom:
drawStyle: line
fillOpacity: 0
lineWidth: 1
spanNulls: false
thresholds:
mode: absolute
steps:
- color: green
value: null
overrides: []
options:
legend:
calcs:
- latest
displayMode: list
placement: bottom
showLegend: true
tooltip:
mode: multi
sort: desc
transformations: []
- id: 6
title: All Trigger Resource Counts (trend)
type: timeseries
datasource: prometheus
gridPos:
h: 8
w: 24
x: 0
"y": 6
targets:
- datasource: prometheus
expr: controller_eventlistener_count
legendFormat: EventListener
refId: A
- datasource: prometheus
expr: controller_triggertemplate_count
legendFormat: TriggerTemplate
refId: B
- datasource: prometheus
expr: controller_triggerbinding_count
legendFormat: TriggerBinding
refId: C
- datasource: prometheus
expr: controller_clustertriggerbinding_count
legendFormat: ClusterTriggerBinding
refId: D
- datasource: prometheus
expr: controller_clusterinterceptor_count
legendFormat: ClusterInterceptor
refId: E
fieldConfig:
defaults:
color:
mode: palette-classic
custom:
drawStyle: line
fillOpacity: 0
lineWidth: 1
spanNulls: false
thresholds:
mode: absolute
steps:
- color: green
value: null
overrides: []
options:
legend:
calcs:
- latest
displayMode: list
placement: bottom
showLegend: true
tooltip:
mode: multi
sort: desc
transformations: []
Tekton Triggers 监控面板说明(常见问题)
EventListener Count、TriggerTemplate Count、TriggerBinding Count、ClusterTriggerBinding 和 ClusterInterceptor 是对象数量快照,而不是请求量或事件处理吞吐量。
All Trigger Resource Counts (trend) 显示相同资源数量的合并趋势。与单资源趋势 panel 相比,在一次抓取间隔内出现短暂偏差是预期行为。
- 在不存在 Triggers 资源时显示
0 是正常现象,并不表示抓取失败。
Tekton Results 监控面板
Tekton Results Dashboard YAML
kind: MonitorDashboard
apiVersion: ait.alauda.io/v1alpha2
metadata:
labels:
cpaas.io/dashboard.folder: tekton
cpaas.io/dashboard.is.home.dashboard: "false"
cpaas.io/dashboard.tag.tekton: "true"
name: tekton-results
namespace: cpaas-system
spec:
body:
title: Tekton Results Overview
titleZh: Tekton Results Overview
tags:
- tekton
time:
from: now-1h
to: now
templating:
list: []
panels:
- id: 1
title: PipelineRun Reconciles and Retries (last 5m)
type: timeseries
datasource: prometheus
gridPos:
h: 8
w: 12
x: 0
"y": 0
targets:
- datasource: prometheus
expr: round(sum(increase(kn_workqueue_process_duration_seconds_count{name="github.com.tektoncd.results.pkg.watcher.reconciler.pipelinerun.Reconciler"}[5m])))
legendFormat: processed
refId: A
- datasource: prometheus
expr: round(sum(increase(kn_workqueue_retries_total{name="github.com.tektoncd.results.pkg.watcher.reconciler.pipelinerun.Reconciler"}[5m])))
legendFormat: retries
refId: B
fieldConfig:
defaults:
color:
mode: palette-classic
custom:
drawStyle: line
fillOpacity: 0
lineWidth: 1
spanNulls: false
thresholds:
mode: absolute
steps:
- color: green
value: null
overrides: []
options:
legend:
calcs:
- latest
displayMode: list
placement: bottom
showLegend: true
tooltip:
mode: multi
sort: desc
transformations: []
- id: 2
title: TaskRun Reconciles and Retries (last 5m)
type: timeseries
datasource: prometheus
gridPos:
h: 8
w: 12
x: 12
"y": 0
targets:
- datasource: prometheus
expr: round(sum(increase(kn_workqueue_process_duration_seconds_count{name="github.com.tektoncd.results.pkg.watcher.reconciler.taskrun.Reconciler"}[5m])))
legendFormat: processed
refId: A
- datasource: prometheus
expr: round(sum(increase(kn_workqueue_retries_total{name="github.com.tektoncd.results.pkg.watcher.reconciler.taskrun.Reconciler"}[5m])))
legendFormat: retries
refId: B
fieldConfig:
defaults:
color:
mode: palette-classic
custom:
drawStyle: line
fillOpacity: 0
lineWidth: 1
spanNulls: false
thresholds:
mode: absolute
steps:
- color: green
value: null
overrides: []
options:
legend:
calcs:
- latest
displayMode: list
placement: bottom
showLegend: true
tooltip:
mode: multi
sort: desc
transformations: []
- id: 3
title: PipelineRun Reconcile Latency P95
type: timeseries
datasource: prometheus
gridPos:
h: 8
w: 12
x: 0
"y": 8
targets:
- datasource: prometheus
expr: histogram_quantile(0.95, sum by (le) (rate(kn_workqueue_process_duration_seconds_bucket{name="github.com.tektoncd.results.pkg.watcher.reconciler.pipelinerun.Reconciler"}[5m])))
legendFormat: P95
refId: A
fieldConfig:
defaults:
unit: s
color:
mode: palette-classic
custom:
drawStyle: line
fillOpacity: 0
lineWidth: 1
spanNulls: false
thresholds:
mode: absolute
steps:
- color: green
value: null
overrides: []
options:
legend:
calcs:
- latest
displayMode: list
placement: bottom
showLegend: true
tooltip:
mode: multi
sort: desc
transformations: []
- id: 4
title: TaskRun Reconcile Latency P95
type: timeseries
datasource: prometheus
gridPos:
h: 8
w: 12
x: 12
"y": 8
targets:
- datasource: prometheus
expr: histogram_quantile(0.95, sum by (le) (rate(kn_workqueue_process_duration_seconds_bucket{name="github.com.tektoncd.results.pkg.watcher.reconciler.taskrun.Reconciler"}[5m])))
legendFormat: P95
refId: A
fieldConfig:
defaults:
unit: s
color:
mode: palette-classic
custom:
drawStyle: line
fillOpacity: 0
lineWidth: 1
spanNulls: false
thresholds:
mode: absolute
steps:
- color: green
value: null
overrides: []
options:
legend:
calcs:
- latest
displayMode: list
placement: bottom
showLegend: true
tooltip:
mode: multi
sort: desc
transformations: []
- id: 5
title: Workqueue Depth (PipelineRun vs TaskRun)
type: timeseries
datasource: prometheus
gridPos:
h: 8
w: 12
x: 0
"y": 16
targets:
- datasource: prometheus
expr: sum(kn_workqueue_depth{name="github.com.tektoncd.results.pkg.watcher.reconciler.pipelinerun.Reconciler"})
legendFormat: pipelinerun
refId: A
- datasource: prometheus
expr: sum(kn_workqueue_depth{name="github.com.tektoncd.results.pkg.watcher.reconciler.taskrun.Reconciler"})
legendFormat: taskrun
refId: B
fieldConfig:
defaults:
color:
mode: palette-classic
custom:
drawStyle: line
fillOpacity: 0
lineWidth: 1
spanNulls: false
thresholds:
mode: absolute
steps:
- color: green
value: null
overrides: []
options:
legend:
calcs:
- latest
displayMode: list
placement: bottom
showLegend: true
tooltip:
mode: multi
sort: desc
transformations: []
- id: 6
title: Workqueue Adds (last 5m)
type: timeseries
datasource: prometheus
gridPos:
h: 8
w: 12
x: 12
"y": 16
targets:
- datasource: prometheus
expr: round(sum(increase(kn_workqueue_adds_total{name="github.com.tektoncd.results.pkg.watcher.reconciler.pipelinerun.Reconciler"}[5m])))
legendFormat: pipelinerun adds
refId: A
- datasource: prometheus
expr: round(sum(increase(kn_workqueue_adds_total{name="github.com.tektoncd.results.pkg.watcher.reconciler.taskrun.Reconciler"}[5m])))
legendFormat: taskrun adds
refId: B
fieldConfig:
defaults:
color:
mode: palette-classic
custom:
drawStyle: line
fillOpacity: 0
lineWidth: 1
spanNulls: false
thresholds:
mode: absolute
steps:
- color: green
value: null
overrides: []
options:
legend:
calcs:
- latest
displayMode: list
placement: bottom
showLegend: true
tooltip:
mode: multi
sort: desc
transformations: []
- id: 7
title: gRPC Request Rate (Results API)
type: timeseries
datasource: prometheus
gridPos:
h: 8
w: 12
x: 0
"y": 24
targets:
- datasource: prometheus
expr: "sum(rate(grpc_server_handled_total{grpc_service=~\"tekton.results.*\"}[5m]))"
legendFormat: requests
refId: A
fieldConfig:
defaults:
color:
mode: palette-classic
custom:
drawStyle: line
fillOpacity: 0
lineWidth: 1
spanNulls: false
thresholds:
mode: absolute
steps:
- color: green
value: null
overrides: []
options:
legend:
calcs:
- latest
displayMode: list
placement: bottom
showLegend: true
tooltip:
mode: multi
sort: desc
transformations: []
- id: 8
title: gRPC Error Percentage (Results API, excl. NotFound/Canceled)
type: timeseries
datasource: prometheus
gridPos:
h: 8
w: 12
x: 12
"y": 24
targets:
- datasource: prometheus
expr: "100 * ((sum(rate(grpc_server_handled_total{grpc_service=~\"tekton.results.*\",grpc_code!~\"OK|NotFound|Canceled\"}[5m])) or vector(0)) / clamp_min((sum(rate(grpc_server_handled_total{grpc_service=~\"tekton.results.*\"}[5m])) or vector(0)), 0.001))"
legendFormat: error %
refId: A
fieldConfig:
defaults:
unit: percent
color:
mode: palette-classic
custom:
drawStyle: line
fillOpacity: 0
lineWidth: 1
spanNulls: false
thresholds:
mode: absolute
steps:
- color: red
value: null
overrides: []
options:
legend:
calcs:
- latest
displayMode: list
placement: bottom
showLegend: true
tooltip:
mode: multi
sort: desc
transformations: []
Tekton Results 监控面板说明(常见问题)
- 此监控面板版本基于 Results Watcher 工作队列指标(
kn_workqueue_*,name label 中每个 reconciler 对应一个 series)以及 Results API gRPC 指标,因此在常见部署(logs_api=true,禁用自动删除)下仍会显示数据。
PipelineRun Reconciles and Retries (last 5m) 和 TaskRun Reconciles and Retries (last 5m) 按每个 5 分钟时间窗口显示 reconciler 处理的项目数(kn_workqueue_process_duration_seconds_count)以及重新添加到队列以进行下一次尝试的项目数(kn_workqueue_retries_total)。
PipelineRun Reconcile Latency P95 和 TaskRun Reconcile Latency P95 根据 watcher 工作队列处理时间 histogram(kn_workqueue_process_duration_seconds,单位为秒)计算。在流量较低时,曲线可能较为稀疏。
Workqueue Depth 显示当前队列深度,Workqueue Adds (last 5m) 显示过去 5 分钟内的入队量。
gRPC Error Percentage (Results API, excl. NotFound/Canceled) 表示异常错误占总请求数的百分比,其中排除了常见的业务返回码(NotFound、Canceled)。
Tekton Chains 监控面板
Tekton Chains Dashboard YAML
kind: MonitorDashboard
apiVersion: ait.alauda.io/v1alpha2
metadata:
labels:
cpaas.io/dashboard.folder: tekton
cpaas.io/dashboard.is.home.dashboard: "false"
cpaas.io/dashboard.tag.tekton: "true"
name: tekton-chains
namespace: cpaas-system
spec:
body:
title: Tekton Chains Overview
titleZh: Tekton Chains Overview
tags:
- tekton
time:
from: now-1h
to: now
templating:
list: []
panels:
- id: 1
title: TaskRun Signatures Created (last 5m)
type: timeseries
datasource: prometheus
gridPos:
h: 8
w: 12
x: 0
"y": 0
targets:
- datasource: prometheus
expr: round(increase(watcher_taskrun_sign_created_total[5m]))
legendFormat: sign created
refId: A
fieldConfig:
defaults:
color:
mode: palette-classic
custom:
drawStyle: line
fillOpacity: 0
lineWidth: 1
spanNulls: false
thresholds:
mode: absolute
steps:
- color: green
value: null
overrides: []
options:
legend:
calcs:
- latest
displayMode: list
placement: bottom
showLegend: true
tooltip:
mode: multi
sort: desc
transformations: []
- id: 2
title: PipelineRun Signatures Created (last 5m)
type: timeseries
datasource: prometheus
gridPos:
h: 8
w: 12
x: 12
"y": 0
targets:
- datasource: prometheus
expr: round(increase(watcher_pipelinerun_sign_created_total[5m]))
legendFormat: sign created
refId: A
fieldConfig:
defaults:
color:
mode: palette-classic
custom:
drawStyle: line
fillOpacity: 0
lineWidth: 1
spanNulls: false
thresholds:
mode: absolute
steps:
- color: green
value: null
overrides: []
options:
legend:
calcs:
- latest
displayMode: list
placement: bottom
showLegend: true
tooltip:
mode: multi
sort: desc
transformations: []
- id: 3
title: Payloads Stored (last 5m, TaskRun vs PipelineRun)
type: timeseries
datasource: prometheus
gridPos:
h: 8
w: 12
x: 0
"y": 8
targets:
- datasource: prometheus
expr: round(increase(watcher_taskrun_payload_stored_total[5m]))
legendFormat: TaskRun
refId: A
- datasource: prometheus
expr: round(increase(watcher_pipelinerun_payload_stored_total[5m]))
legendFormat: PipelineRun
refId: B
fieldConfig:
defaults:
color:
mode: palette-classic
custom:
drawStyle: line
fillOpacity: 0
lineWidth: 1
spanNulls: false
thresholds:
mode: absolute
steps:
- color: green
value: null
overrides: []
options:
legend:
calcs:
- latest
displayMode: list
placement: bottom
showLegend: true
tooltip:
mode: multi
sort: desc
transformations: []
- id: 4
title: Marked Signed (last 5m, TaskRun vs PipelineRun)
type: timeseries
datasource: prometheus
gridPos:
h: 8
w: 12
x: 12
"y": 8
targets:
- datasource: prometheus
expr: round(increase(watcher_taskrun_marked_signed_total[5m]))
legendFormat: TaskRun
refId: A
- datasource: prometheus
expr: round(increase(watcher_pipelinerun_marked_signed_total[5m]))
legendFormat: PipelineRun
refId: B
fieldConfig:
defaults:
color:
mode: palette-classic
custom:
drawStyle: line
fillOpacity: 0
lineWidth: 1
spanNulls: false
thresholds:
mode: absolute
steps:
- color: green
value: null
overrides: []
options:
legend:
calcs:
- latest
displayMode: list
placement: bottom
showLegend: true
tooltip:
mode: multi
sort: desc
transformations: []
Tekton Chains 监控面板说明(常见问题)
TaskRun Signatures Created (last 5m)、PipelineRun Signatures Created (last 5m)、Payloads Stored (last 5m) 和 Marked Signed (last 5m) 使用 increase(...[5m]),表示过去五分钟内的增量。
- 没有新的签名或存储活动时,这些曲线会降至
0;这并不表示组件发生故障。如果某个 counter 自 Chains controller 启动以来从未递增,则不会显示数据,因为该 counter 尚未出现在指标输出中。
Payloads Stored 和 Marked Signed 表示不同的处理阶段,因此它们的值不应始终匹配。Payloads Stored 还会统计这样一种 payload:oci storage backend 未找到 image subject,因此未向 registry 上传任何内容(请参阅签名密钥配置中的前提条件)。
- 如果
Marked Signed 增长,而 TaskRun Signatures Created 和 PipelineRun Signatures Created 没有数据显示(或者保持为 0,即 controller 之前已创建签名),并且 *_signing_failures_total 没有增长,则 Chains 会将运行标记为已签名,但不会实际进行签名(请参阅Chains 指标下的说明)。当未配置 signer 时会发生这种情况(默认情况下,signing-secrets 不包含密钥),或者禁用了对 TaskRun 和 PipelineRun 工件的签名(例如使用 artifacts.taskrun.storage: "" 和 artifacts.pipelinerun.storage: "",请参阅Chains 配置)。没有任何监控面板显示 *_signing_failures_total;请在 Prometheus 或 Chains 指标输出中查询它。如果在这种情况下该指标增长,则表示 Chains 创建、编码或签署 payload 失败(error_type、payload_creation、marshal_payload 或 signing),但仍将运行标记为已签名。