Tekton 组件的指标采集

概述

Tekton 组件通过 HTTP 端点公开 Prometheus 兼容的指标。部署 ServiceMonitor 资源后,Prometheus(或 VictoriaMetrics)可以自动发现并抓取这些指标。

命名空间说明:本文档使用 tekton-pipelines 作为控制平面组件(Pipelines、Triggers、Results、Chains)的默认命名空间。 主要例外是 EventListener Service,它们运行在创建 EventListener 的应用命名空间中。

如果您的部署使用不同的命名空间,请同时更新以下命令以及下方 ServiceMonitor 资源中的 namespaceSelector 字段。

本文档涵盖以下 Tekton 组件的指标:

  • Tekton Pipelines - PipelineRun / TaskRun 执行指标
  • Tekton Triggers - EventListener、TriggerBinding 及相关资源指标
  • Tekton Results - 运行删除和存储指标
  • Tekton Chains - 签名和来源证明指标
  • Controller Framework - 所有 controller 共享的基础设施指标

本文档还介绍:

  • 如何通过 config-observability 配置指标行为
  • 如何部署用于抓取指标的 ServiceMonitor 资源
  • 如何验证指标采集是否正常工作

前提条件

  • 已安装并运行 Tekton 控制平面组件(至少包括您计划抓取的组件:Pipelines、Triggers、Results 和/或 Chains)。
  • 已针对目标集群配置 kubectl,且您的账户可以在监控命名空间中创建 ServiceMonitor 资源。
  • 已部署监控栈(Prometheus 或兼容的 VictoriaMetrics),并且能够发现/抓取 ServiceMonitor 资源(或平台使用的等效抓取发现对象)。
  • 您的 Prometheus/VictoriaMetrics 实例已配置为发现您创建的 ServiceMonitor 对象(命名空间和标签选择器必须匹配)。
  • 网络策略和防火墙允许抓取器 Pod 访问 Tekton 指标端口(大多数控制平面服务使用 9090,Triggers controller 和 EventListener sink 使用 9000)。
  • 如果需要 EventListener sink 指标,则目标命名空间中必须存在 EventListener,并公开 http-metrics 端口。

Tekton Pipelines

Tekton Pipelines 组件包含多个子服务,这些服务在端口 9090 上公开指标:

服务描述指标端口
tekton-pipelines-controllerPipelineRun / TaskRun 的主 reconciler9090
tekton-pipelines-webhookAdmission webhook9090
tekton-events-controllerCloudEvents controller9090
tekton-pipelines-remote-resolvers远程资源解析9090

Pipeline controller 指标使用前缀 tekton_pipelines_controller_。

PipelineRun 指标

指标名称类型描述标签
pipelinerun_duration_secondsHistogram / LastValuePipelineRun 执行时间(秒)status、namespace、pipeline*、pipelinerun*、reason*
pipelinerun_totalCounter已完成 PipelineRun 的总数status
running_pipelinerunsLastValue (Gauge)当前正在运行的 PipelineRun 数量由 metrics.running-pipelinerun.level 控制(见下文)
running_pipelineruns_waiting_on_pipeline_resolutionLastValue (Gauge)等待 Pipeline 引用解析的 PipelineRun 数量-
running_pipelineruns_waiting_on_task_resolutionLastValue (Gauge)等待 Task 引用解析的 PipelineRun 数量-

* 带有 * 标记的标签为可选标签,具体取决于 config-observability 配置。

running_pipelineruns 标签级别

running_pipelineruns 指标的标签由 metrics.running-pipelinerun.level 控制:

级别标签
""(默认,集群级别)无标签
"namespace"namespace
"pipeline"namespace、pipeline
"pipelinerun"namespace、pipeline、pipelinerun

状态标签值

对于 PipelineRun 指标:

  • success - PipelineRun 已成功完成
  • failed - PipelineRun 失败
  • cancelled - PipelineRun 已取消

对于 TaskRun 指标:

  • success - TaskRun 已成功完成
  • failed - TaskRun 失败

TaskRun 指标

指标名称类型描述标签
taskrun_duration_secondsHistogram / LastValue独立 TaskRun 的执行时间(秒)status、namespace、task*、taskrun*、reason*
pipelinerun_taskrun_duration_secondsHistogram / LastValue作为 PipelineRun 一部分时的 TaskRun 执行时间status、namespace、task*、taskrun*、pipeline*、pipelinerun*、reason*
taskrun_totalCounter已完成 TaskRun 的总数status
running_taskrunsLastValue (Gauge)当前正在运行的 TaskRun 数量-
running_taskruns_waiting_on_task_resolution_countLastValue (Gauge)等待 Task 引用解析的 TaskRun 数量-
running_taskruns_throttled_by_quotaLastValue (Gauge)受 ResourceQuota 限制的 TaskRun 数量namespace*
running_taskruns_throttled_by_nodeLastValue (Gauge)受节点级资源约束限制的 TaskRun 数量namespace*
taskruns_pod_latency_millisecondsLastValueTaskRun 的 Pod 调度延迟(毫秒)namespace、pod、task*、taskrun*

config-observability 配置

config-observability 命名空间中的 tekton-pipelines ConfigMap 控制 Pipeline controller 的指标行为。此 ConfigMap 由 Tekton Operator 管理,应通过 TektonConfig 资源的 spec.pipeline.options.configMaps 字段进行配置。详情请参阅调整子组件的可选配置项。

热加载行为:config-observability 会在运行时受到监视。大多数关键更改(例如 metrics.*)无需重启 Pod 即可生效。请等待一到两个抓取间隔,以便监控面板/查询中显示更改。只有在 Pod 规范设置发生更改时才需要重启(例如更改 Deployment 中的 CONFIG_OBSERVABILITY_NAME)。

通过 TektonConfig 配置的示例:

apiVersion: operator.tekton.dev/v1alpha1
kind: TektonConfig
metadata:
  name: config
spec:
  pipeline:
    options:
      disabled: false
      configMaps:
        config-observability:
          data:
            metrics.backend-destination: prometheus

            # PipelineRun metrics aggregation level.
            # Values: "pipelinerun" | "pipeline" (default) | "namespace"
            #   - "pipelinerun": includes pipeline + pipelinerun labels; duration uses LastValue
            #   - "pipeline": includes pipeline label only
            #   - "namespace": no pipeline/pipelinerun labels
            metrics.pipelinerun.level: "pipeline"

            # TaskRun metrics aggregation level.
            # Values: "taskrun" | "task" (default) | "namespace"
            #   - "taskrun": includes task + taskrun labels; duration uses LastValue
            #   - "task": includes task label only
            #   - "namespace": no task/taskrun labels
            metrics.taskrun.level: "task"

            # Duration metric type for PipelineRun / TaskRun.
            # Values: "histogram" (default) | "lastvalue"
            # Note: When pipelinerun.level is "pipelinerun" or taskrun.level is "taskrun",
            #       duration type is forced to "lastvalue" regardless of this setting.
            metrics.pipelinerun.duration-type: "histogram"
            metrics.taskrun.duration-type: "histogram"

            # Running PipelineRun metrics aggregation level.
            # Values: "pipelinerun" | "pipeline" | "namespace" | "" (default, cluster-level)
            metrics.running-pipelinerun.level: ""

            # Include reason label on duration metrics (pipelinerun_duration_seconds,
            # taskrun_duration_seconds, pipelinerun_taskrun_duration_seconds).
            # Values: "true" | "false" (default)
            # Warning: Enabling this increases label cardinality.
            # Note: Despite the key name, this does NOT affect count metrics
            # (pipelinerun_total / taskrun_total), only duration metrics.
            metrics.count.enable-reason: "false"

            # Include namespace label on throttled TaskRun metrics.
            # Values: "true" | "false" (default)
            metrics.taskrun.throttle.enable-namespace: "false"

Histogram 桶

当持续时间类型为 histogram 时,将使用以下桶边界(单位为秒):

10, 30, 60, 300, 900, 1800, 3600, 5400, 10800, 21600, 43200, 86400

这对应于:10s、30s、1m、5m、15m、30m、1h、1.5h、3h、6h、12h、24h。

推荐的生产配置

对于生产环境,请使用聚合级别来控制标签基数:

metrics.pipelinerun.level: "pipeline"
metrics.taskrun.level: "task"
metrics.pipelinerun.duration-type: "histogram"
metrics.taskrun.duration-type: "histogram"
metrics.count.enable-reason: "false"

如果需要逐个运行的粒度以进行调试,可以临时切换为:

metrics.pipelinerun.level: "pipelinerun"
metrics.taskrun.level: "taskrun"

请注意,这将显著增加时间序列的数量。


Tekton Triggers

Tekton Triggers 组件从不同进程中公开两类指标。

Controller 指标(端口 9000)

Triggers controller 每 60 秒报告一次资源数量指标。

Service指标端口
tekton-triggers-controller9000

Triggers controller 指标使用前缀 controller_。

指标名称类型描述标签
eventlistener_countLastValue (Gauge)EventListener 资源数量-
triggerbinding_countLastValue (Gauge)TriggerBinding 资源数量-
clustertriggerbinding_countLastValue (Gauge)ClusterTriggerBinding 资源数量-
triggertemplate_countLastValue (Gauge)TriggerTemplate 资源数量-
clusterinterceptor_countLastValue (Gauge)ClusterInterceptor 资源数量-

EventListener Sink 指标

每个 EventListener pod 都会公开额外的 HTTP 和事件处理指标。这些指标来自 EventListener sink 进程(而非 controller)。Prometheus 指标前缀为 eventlistener_。

指标名称(Prometheus)类型描述标签
eventlistener_http_duration_secondsHistogramEventListener HTTP 请求持续时间-
eventlistener_event_received_countCountersink 接收的事件总数status
eventlistener_triggered_resourcesCountertriggers 创建的资源总数kind
  • eventlistener_http_duration_seconds 直方图桶:0.001, 0.01, 0.1, 1, 10(秒)
  • eventlistener_event_received_count 状态值:succeeded、failed
  • eventlistener_triggered_resources 类型值:所创建对象的 Kubernetes 资源 Kind(例如 PipelineRun、TaskRun)

这些 sink 指标按 EventListener pod 分别公开,而不是由中心 controller 公开。如果 EventListener pod 暴露了指标端口,可能需要单独的 ServiceMonitor 或 PodMonitor 来抓取这些指标。

Tekton Results

Tekton Results 包含两个公开指标的子服务。

服务描述指标端口
tekton-results-watcher监视并清理 PipelineRun/TaskRun 资源9090
tekton-results-apigRPC/REST API server9090

Watcher 指标

Watcher 指标使用前缀 watcher_。

删除指标

指标名称类型描述标签
pipelinerun_delete_countCounter已删除 PipelineRun 的总数status、namespace
pipelinerun_delete_duration_secondsHistogram / LastValue从 PipelineRun 完成到删除所经过的时间status、namespace、pipeline*
taskrun_delete_countCounter已删除 TaskRun 的总数status、namespace
taskrun_delete_duration_secondsHistogram / LastValue从 TaskRun 完成到删除所经过的时间status、namespace、pipeline*、task*

* 可选标签取决于 Results Watcher 的 config-observability 设置。

注意:仅当 Watcher 实际删除运行时,才会记录 pipelinerun_delete_count、pipelinerun_delete_duration_seconds、taskrun_delete_count 和 taskrun_delete_duration_seconds。除非在 tekton-results-watcher Deployment 上将 --completed_run_grace_period 标志设置为非零值,否则这些指标将保持为空(没有数据点)。默认情况下,此标志为 0,会禁用自动删除。将其设置为正时长(例如 10m)可启用宽限期后的删除,或将其设置为负值,以便在归档后立即删除。

Results Watcher 的状态标签值:

  • success - 运行成功完成
  • failed - 运行失败
  • cancelled - 运行已取消

共享指标

Watcher 中的 PipelineRun 和 TaskRun reconciler 都会注册这些指标,用于跟踪与存储相关的事件。

指标名称类型描述标签
runs_not_stored_countCounter未存储到 Results 就被删除的运行数kind、namespace
run_storage_latency_secondsHistogram从运行完成到成功存储所经过的时间kind、namespace

kind 标签用于标识运行类型(某些指标序列中为 PipelineRun / TaskRun,其他指标序列中为 pipelinerun / taskrun)。

注意:仅当运行在 Watcher 持有 finalizer 以协调归档期间被外部删除(例如通过 kubectl delete)时,才会记录 runs_not_stored_count。只有满足以下所有条件时,该指标才不会为空:

  1. --logs_api 标志为 false(禁用日志存储);如果启用了日志,Watcher 会完全跳过基于 finalizer 的协调。
  2. --disable_crd_update 标志为 false(启用 annotation 更新)。
  3. --store_deadline 标志设置为非零时长;这是 Watcher 在放弃并允许删除之前等待归档完成的最长时间。
  4. 运行在成功归档之前被外部删除(没有 results.tekton.dev/stored=true annotation),并且 store_deadline 已经过期。

在正常运行中(运行在删除前已完成归档,或删除由 Watcher 通过 --completed_run_grace_period 自身触发),该计数器保持为零。非零值表示可能存在数据丢失:运行在其状态保存到 Results API 之前就被删除。

快速复现(测试环境): 如果看不到此指标,通常意味着未满足触发条件,而不是指标缺失。

  1. 通过 TektonConfig 配置 Results Watcher,使 logs_api=false、disable_crd_update=false 和 store_deadline 为非零值(例如 30s)。
  2. 通过 TektonConfig(spec.result.options.deployments.tekton-results-api.spec.replicas: 0)暂时将 Results API 副本数设置为 0,使运行无法归档。
  3. 创建一个 TaskRun 或 PipelineRun,并等待其完成。
  4. 等待 store_deadline 经过,然后从外部删除该运行(kubectl delete ...)。
  5. 检查 Watcher 的 /metrics,或在 Prometheus 中检查 watcher_runs_not_stored_count(exposition 格式中的组件前缀名称);其值应增加。
  6. 恢复原始的 TektonConfig(重新启用 Results API 副本并恢复正常的 logs_api 设置)。

run_storage_latency_seconds 直方图使用以下桶边界(以秒为单位):

0.1, 0.5, 1, 2, 5, 10, 30, 60, 120, 300, 600, 1800

Watcher config-observability

Results Watcher 有自己的 config-observability ConfigMap(其名称通过 CONFIG_OBSERVABILITY_NAME 环境变量指定,通常为 tekton-results-config-observability)。此 ConfigMap 由 Tekton Operator 管理,应通过 TektonConfig 资源的 spec.result.options.configMaps 字段进行配置。详情请参阅调整子组件的可选配置项。

热重载行为:Results Watcher 也会监视此 ConfigMap,并在无需重启 Pod 的情况下应用大多数键的更改。只有更改 Deployment 级别的设置(例如 env/args)时才需要重启。

它支持以下键:

键默认值值描述
metrics.pipelinerun.levelpipelinepipeline、namespace控制删除持续时间指标中的 pipeline 标签
metrics.taskrun.leveltasktask、namespace控制删除持续时间指标中的 task 标签
metrics.pipelinerun.duration-typehistogramhistogram、lastvaluePipelineRun 和 TaskRun 删除的持续时间指标聚合类型
metrics.taskrun.duration-typehistogramhistogram、lastvalue已解析但当前未使用;metrics.pipelinerun.duration-type 控制两者

注意:与 Tekton Pipelines 不同,Results Watcher 不支持 pipelinerun / taskrun 这种单个运行粒度级别。它也没有 metrics.count.enable-reason、metrics.running-pipelinerun.level 或 metrics.taskrun.throttle.enable-namespace 键。

上游已知问题:taskrun_delete_duration_seconds 使用 metrics.pipelinerun.duration-type(而不是 metrics.taskrun.duration-type)来确定聚合类型。这似乎是 Results 源代码中的复制粘贴错误。

API Server 指标

API server 通过 go-grpc-prometheus 库在端口 9090 上公开标准 gRPC Prometheus 指标。这些指标包括:

  • grpc_server_handled_total - server 上完成的 RPC 总数
  • grpc_server_started_total - server 上启动的 RPC 总数
  • grpc_server_msg_received_total / grpc_server_msg_sent_total - 消息数量
  • grpc_server_handling_seconds(如果启用了 PROMETHEUS_HISTOGRAM)- RPC 处理持续时间

Tekton Chains

Tekton Chains 是一个安全组件,用于为使用 Tekton Pipelines 构建的制品生成、签名并存储来源信息。它会监视已完成的 TaskRun 和 PipelineRun,然后创建证明和签名。

ServiceDescriptionMetrics Port
tekton-chains-metricsChains watcher/controller9090 (http-metrics)

Chains controller 指标使用前缀 watcher_(与 Results Watcher 相同,但自定义指标名称不同,因此不会发生冲突)。

Chains 指标

所有 Chains 指标均为不带标签的 Counter。

Metric Name (Prometheus)TypeDescription
watcher_taskrun_sign_created_totalCounterTaskRun 的已签名消息总数
watcher_taskrun_payload_stored_totalCounterTaskRun 的已存储负载总数
watcher_taskrun_marked_signed_totalCounter标记为已签名的 TaskRun 总数
watcher_pipelinerun_sign_created_totalCounterPipelineRun 的已签名消息总数
watcher_pipelinerun_payload_stored_totalCounterPipelineRun 的已存储负载总数
watcher_pipelinerun_marked_signed_totalCounter标记为已签名的 PipelineRun 总数

注意:官方 Tekton Chains 文档还提到了 TaskRun 和 PipelineRun 的 *_signing_failures_total Counter,但当前上游源代码中并不存在这些指标。请根据已部署的版本进行确认。


Controller Framework 指标

所有 Tekton controller 都会自动公开以下基础设施指标。这些指标使用与组件自定义指标相同的前缀(例如 tekton_pipelines_controller_、controller_、watcher_)。

Metric Name (without prefix)TypeDescription
client_latencyHistogramKubernetes API client 请求延迟(秒)
client_resultsCounterKubernetes API 请求数(按状态码统计)
workqueue_depthGauge当前 workqueue 深度
workqueue_adds_totalCounterworkqueue 添加总数
workqueue_queue_latency_secondsHistogram项目在 workqueue 中等待的时间
workqueue_work_duration_secondsHistogram处理 workqueue 项目所用的时间
workqueue_retries_totalCounterworkqueue 重试总数
workqueue_unfinished_work_secondsHistogram未完成 workqueue 项目的持续时间
workqueue_longest_running_processor_secondsHistogram运行时间最长的 workqueue 处理器的持续时间
reconcile_countCounterreconciler 调用总数(按 reconciler、success、namespace_name 标签统计)
reconcile_latencyHistogramreconciler 调用延迟(按 reconciler、success、namespace_name 标签统计)

设置 ServiceMonitor

要启用对 Tekton 组件的 Prometheus 抓取,请部署 ServiceMonitor 资源。

前置条件请参阅前置条件。

请根据您使用的监控栈参阅以下指导:

  • 如果您使用 Prometheus(Prometheus Operator),则 metadata.labels.prometheus: kube-prometheus 等标签必须与 Prometheus CR spec.serviceMonitorSelector 匹配;否则,此 ServiceMonitor 不会被抓取。
  • 如果您使用 VictoriaMetrics,通常不需要 prometheus: kube-prometheus 等标签;请根据您的监控设置创建 ServiceMonitor/VMServiceScrape。

使用 Prometheus 时,请使用以下命令查找并验证 selector:

# 1) Locate Prometheus CRs (resource type: monitoring.coreos.com/v1, Kind=Prometheus)
$ kubectl get prometheus -A

# 2) Check ServiceMonitor selector on the target Prometheus instance
$ kubectl get prometheus -n <prometheus-namespace> <prometheus-name> -o yaml | yq '.spec.serviceMonitorSelector'

如果集群中不存在 Prometheus CR,则监控通常由平台管理(例如 VictoriaMetrics),或采用了其他实现方式。在这种情况下,通常不需要 prometheus: kube-prometheus 等标签;请遵循平台的抓取规则。

有关更多信息,请参阅 集成外部指标。

Pipeline ServiceMonitor

Pipeline ServiceMonitor YAML
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: tekton-pipelines-metrics
  namespace: tekton-pipelines
  labels:
    app.kubernetes.io/name: tekton-pipelines
    # prometheus: kube-prometheus
spec:
  selector:
    matchLabels:
      app.kubernetes.io/part-of: tekton-pipelines
  endpoints:
  - port: http-metrics
    path: /metrics
    interval: 30s
  namespaceSelector:
    matchNames:
    - tekton-pipelines

此 ServiceMonitor 会匹配带有 app.kubernetes.io/part-of: tekton-pipelines 标签的 Pipeline 服务(包括 remote-resolvers),并抓取 tekton-pipelines 命名空间中的服务。

Triggers ServiceMonitor

Triggers ServiceMonitor YAML
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: tekton-triggers-metrics
  namespace: tekton-pipelines
  labels:
    app.kubernetes.io/name: tekton-triggers
    # prometheus: kube-prometheus
spec:
  selector:
    matchLabels:
      app.kubernetes.io/part-of: tekton-triggers
      app.kubernetes.io/component: controller
  endpoints:
  - port: http-metrics
    path: /metrics
    interval: 30s
  namespaceSelector:
    matchNames:
    - tekton-pipelines

此 ServiceMonitor 仅收集 Triggers controller 指标(controller_*)。它不包含 EventListener sink 指标。

EventListener Sink ServiceMonitor

EventListener Sink ServiceMonitor YAML
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: tekton-eventlistener-sink-metrics
  namespace: tekton-pipelines
  labels:
    app.kubernetes.io/name: tekton-eventlistener-sink
    # prometheus: kube-prometheus
spec:
  selector:
    matchExpressions:
    - key: eventlistener
      operator: Exists
    - key: app.kubernetes.io/managed-by
      operator: In
      values:
      - EventListener
  endpoints:
  - port: http-metrics
    path: /metrics
    interval: 30s
  namespaceSelector:
    any: true

EventListener Service 通常运行在应用命名空间中,因此此示例使用 namespaceSelector.any: true 进行跨命名空间抓取。如果需要更严格的范围,请切换为 matchNames,并显式列出允许的命名空间。

Results ServiceMonitor

Results 服务同时具有 app.kubernetes.io/part-of: tekton-results 和 app.kubernetes.io/name 标签。为了精确定位 API + Watcher 并排除 Postgres,此示例匹配 app.kubernetes.io/name:

Results ServiceMonitor YAML
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: tekton-results-metrics
  namespace: tekton-pipelines
  labels:
    app.kubernetes.io/name: tekton-results
    # prometheus: kube-prometheus
spec:
  selector:
    matchExpressions:
    - key: app.kubernetes.io/name
      operator: In
      values:
      - tekton-results-api
      - tekton-results-watcher
  endpoints:
  - port: prometheus
    path: /metrics
    interval: 30s
  - port: metrics
    path: /metrics
    interval: 30s
  namespaceSelector:
    matchNames:
    - tekton-pipelines

Results API server 使用端口名称 prometheus(9090),Watcher 使用端口名称 metrics(9090)。每个服务只公开其中一个端口名称,因此只会抓取匹配的端点。

Chains ServiceMonitor

Chains ServiceMonitor YAML
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: tekton-chains-metrics
  namespace: tekton-pipelines
  labels:
    app.kubernetes.io/name: tekton-chains
    # prometheus: kube-prometheus
spec:
  selector:
    matchLabels:
      app.kubernetes.io/part-of: tekton-chains
  endpoints:
  - port: http-metrics
    path: /metrics
    interval: 30s
  namespaceSelector:
    matchNames:
    - tekton-pipelines

验证

部署 ServiceMonitor 资源后,验证 Prometheus 是否正在抓取目标。

直接检查指标端点

# Pipeline controller
$ kubectl port-forward -n tekton-pipelines svc/tekton-pipelines-controller 9090:9090
$ curl -s http://localhost:9090/metrics | grep tekton_pipelines_controller_

# HELP tekton_pipelines_controller_client_latency How long Kubernetes API requests take
# TYPE tekton_pipelines_controller_client_latency histogram
tekton_pipelines_controller_client_latency_bucket{name="",le="1e-05"} 0
tekton_pipelines_controller_client_latency_bucket{name="",le="0.0001"} 0
tekton_pipelines_controller_client_latency_bucket{name="",le="0.001"} 0

# Triggers controller
$ kubectl port-forward -n tekton-pipelines svc/tekton-triggers-controller 9000:9000
$ curl -s http://localhost:9000/metrics | grep controller_

# HELP controller_client_latency How long Kubernetes API requests take
# TYPE controller_client_latency histogram
controller_client_latency_bucket{name="",le="1e-05"} 0
controller_client_latency_bucket{name="",le="0.0001"} 1
controller_client_latency_bucket{name="",le="0.001"} 2

# EventListener sink metrics (replace namespace/service)
$ kubectl port-forward -n <eventlistener-namespace> svc/<eventlistener-service> 9000:9000
$ curl -s http://localhost:9000/metrics | grep eventlistener_

# HELP eventlistener_client_latency How long Kubernetes API requests take
# TYPE eventlistener_client_latency histogram
eventlistener_client_latency_bucket{name="",le="1e-05"} 0
eventlistener_client_latency_bucket{name="",le="0.0001"} 0
eventlistener_client_latency_bucket{name="",le="0.001"} 0

# HELP eventlistener_triggered_resources Count of the number of triggered eventlistener resources
# TYPE eventlistener_triggered_resources counter
eventlistener_triggered_resources{kind="PipelineRun"} 10

# Results watcher
$ kubectl port-forward -n tekton-pipelines svc/tekton-results-watcher 9091:9090
$ curl -s http://localhost:9091/metrics | grep watcher_

# HELP watcher_client_latency How long Kubernetes API requests take
# TYPE watcher_client_latency histogram
watcher_client_latency_bucket{name="",le="1e-05"} 0
watcher_client_latency_bucket{name="",le="0.0001"} 0
watcher_client_latency_bucket{name="",le="0.001"} 0

# Results API
$ kubectl port-forward -n tekton-pipelines svc/tekton-results-api-service 9092:9090
$ curl -s http://localhost:9092/metrics | grep grpc_server_

# HELP grpc_server_handled_total Total number of RPCs completed on the server, regardless of success or failure.
# TYPE grpc_server_handled_total counter
grpc_server_handled_total{grpc_code="Aborted",grpc_method="Check",grpc_service="grpc.health.v1.Health",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Aborted",grpc_method="CreateRecord",grpc_service="tekton.results.v1alpha2.Results",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Aborted",grpc_method="CreateResult",grpc_service="tekton.results.v1alpha2.Results",grpc_type="unary"} 0

# HELP grpc_server_started_total Total number of RPCs started on the server.
# TYPE grpc_server_started_total counter
grpc_server_started_total{grpc_method="Check",grpc_service="grpc.health.v1.Health",grpc_type="unary"} 337606
grpc_server_started_total{grpc_method="CreateRecord",grpc_service="tekton.results.v1alpha2.Results",grpc_type="unary"} 10301
grpc_server_started_total{grpc_method="CreateResult",grpc_service="tekton.results.v1alpha2.Results",grpc_type="unary"} 832

# Chains controller
$ kubectl port-forward -n tekton-pipelines svc/tekton-chains-metrics 9093:9090
$ curl -s http://localhost:9093/metrics | grep watcher_

# HELP watcher_client_latency How long Kubernetes API requests take
# TYPE watcher_client_latency histogram
watcher_client_latency_bucket{name="",le="1e-05"} 0
watcher_client_latency_bucket{name="",le="0.0001"} 0
watcher_client_latency_bucket{name="",le="0.001"} 0

EventListener sink 指标(例如 eventlistener_event_received_count 和 eventlistener_http_duration_seconds)由请求驱动。在验证这些指标之前,至少向 EventListener 发送一个请求。

检查 Prometheus 目标

# Verify ServiceMonitor resources exist
$ kubectl get servicemonitor -n tekton-pipelines

NAME                                AGE
tekton-chains-metrics               10m
tekton-eventlistener-sink-metrics   10m
tekton-pipelines-metrics            10m
tekton-results-metrics              10m
tekton-triggers-metrics             10m

# Check Prometheus targets (via Prometheus UI or API)
# Look for targets with job labels matching the ServiceMonitor names

PromQL 查询示例

# PipelineRun cumulative success rate (avoids misinterpretation in empty completion windows)
100 * sum(tekton_pipelines_controller_pipelinerun_total{status="success"}) / clamp_min(sum(tekton_pipelines_controller_pipelinerun_total), 1)

# Completed PipelineRuns in the last 5 minutes (throughput)
round(sum(increase(tekton_pipelines_controller_pipelinerun_total[5m])))

# PipelineRun duration P95 (histogram mode)
histogram_quantile(0.95,
  rate(tekton_pipelines_controller_pipelinerun_duration_seconds_bucket[5m])
)

# TaskRun duration P95 (histogram mode, includes standalone + in-pipeline TaskRuns)
histogram_quantile(0.95,
  (
    sum by (le) (rate(tekton_pipelines_controller_taskrun_duration_seconds_bucket[5m]))
    +
    sum by (le) (rate(tekton_pipelines_controller_pipelinerun_taskrun_duration_seconds_bucket[5m]))
  )
)

# PipelineRun duration (lastvalue mode)
avg_over_time(tekton_pipelines_controller_pipelinerun_duration_seconds[5m])

# Currently running PipelineRuns (single series to avoid duplicate legends)
max(tekton_pipelines_controller_running_pipelineruns)

# TaskRuns throttled by resource quota
max(tekton_pipelines_controller_running_taskruns_throttled_by_quota)

# Trigger resource counts
controller_eventlistener_count
controller_triggertemplate_count

# Chains signing activity
watcher_taskrun_sign_created_total
watcher_pipelinerun_sign_created_total

监控面板示例

以下 MonitorDashboard 资源提供了用于监控 Tekton 组件的开箱即用监控面板。将其部署到 cpaas-system 命名空间下的 tekton 文件夹中。

重要:每个图表都必须包含 id(唯一整数)、datasource: prometheus 和 transformations: []。每个目标都必须包含 datasource: prometheus 和 refId。本文档中的持续时间 P50/P95 图表使用 *_bucket 查询,并要求使用 metrics.*.duration-type=histogram;如果使用 lastvalue,请将这些查询替换为 LastValue 风格的表达式,例如 avg_over_time(...)。

Tekton Pipeline 监控面板

Tekton Pipeline Dashboard YAML
kind: MonitorDashboard
apiVersion: ait.alauda.io/v1alpha2
metadata:
  labels:
    cpaas.io/dashboard.folder: tekton
    cpaas.io/dashboard.is.home.dashboard: "false"
    cpaas.io/dashboard.tag.tekton: "true"
  name: tekton-pipeline
  namespace: cpaas-system
spec:
  body:
    titleZh: Tekton Pipeline Overview
    tags:
      - tekton
    time:
      from: now-1h
      to: now
    templating:
      list: []
    panels:
      - id: 1
        title: PipelineRun Total (by status)
        type: timeseries
        datasource: prometheus
        gridPos: { h: 8, w: 8, x: 0, y: 0 }
        targets:
          - datasource: prometheus
            expr: sum by (status) (tekton_pipelines_controller_pipelinerun_total)
            refId: A
        fieldConfig:
          defaults:
            color: { mode: palette-classic }
            custom: { drawStyle: line, fillOpacity: 0, lineWidth: 1, spanNulls: false }
            thresholds: { mode: absolute, steps: [{ color: green, value: null }] }
          overrides: []
        options:
          legend: { calcs: [latest], displayMode: list, placement: bottom, showLegend: true }
          tooltip: { mode: multi, sort: desc }
        transformations: []
      - id: 2
        title: TaskRun Total (by status)
        type: timeseries
        datasource: prometheus
        gridPos: { h: 8, w: 8, x: 8, y: 0 }
        targets:
          - datasource: prometheus
            expr: sum by (status) (tekton_pipelines_controller_taskrun_total)
            refId: A
        fieldConfig:
          defaults:
            color: { mode: palette-classic }
            custom: { drawStyle: line, fillOpacity: 0, lineWidth: 1, spanNulls: false }
            thresholds: { mode: absolute, steps: [{ color: green, value: null }] }
          overrides: []
        options:
          legend: { calcs: [latest], displayMode: list, placement: bottom, showLegend: true }
          tooltip: { mode: multi, sort: desc }
        transformations: []
      - id: 3
        title: PipelineRun Success Rate (cumulative)
        type: timeseries
        datasource: prometheus
        gridPos: { h: 8, w: 4, x: 16, y: 0 }
        targets:
          - datasource: prometheus
            expr: "100 * sum(tekton_pipelines_controller_pipelinerun_total{status=\"success\"}) / clamp_min(sum(tekton_pipelines_controller_pipelinerun_total), 1)"
            refId: A
        fieldConfig:
          defaults:
            unit: percent
            color: { mode: thresholds }
            custom: { drawStyle: line, fillOpacity: 0, lineWidth: 1, spanNulls: false }
            thresholds:
              mode: absolute
              steps:
                - { color: red, value: null }
                - { color: orange, value: 80 }
                - { color: green, value: 95 }
          overrides: []
        options:
          legend: { calcs: [latest], displayMode: list, placement: bottom, showLegend: true }
          tooltip: { mode: multi, sort: desc }
        transformations: []
      - id: 12
        title: Completed PipelineRuns (last 5m)
        type: timeseries
        datasource: prometheus
        gridPos: { h: 8, w: 4, x: 20, y: 0 }
        targets:
          - datasource: prometheus
            expr: "round(sum(increase(tekton_pipelines_controller_pipelinerun_total[5m])))"
            legendFormat: completed
            refId: A
        fieldConfig:
          defaults:
            unit: short
            decimals: 0
            color: { mode: palette-classic }
            custom: { drawStyle: line, fillOpacity: 0, lineWidth: 1, spanNulls: false }
            thresholds: { mode: absolute, steps: [{ color: green, value: null }] }
          overrides: []
        options:
          legend: { calcs: [latest], displayMode: list, placement: bottom, showLegend: true }
          tooltip: { mode: multi, sort: desc }
        transformations: []
      - id: 4
        title: Running PipelineRuns
        type: timeseries
        datasource: prometheus
        gridPos: { h: 8, w: 8, x: 0, y: 8 }
        targets:
          - datasource: prometheus
            expr: max(tekton_pipelines_controller_running_pipelineruns)
            legendFormat: running
            refId: A
        fieldConfig:
          defaults:
            color: { mode: palette-classic }
            custom: { drawStyle: line, fillOpacity: 0, lineWidth: 1, spanNulls: false }
            thresholds: { mode: absolute, steps: [{ color: green, value: null }] }
          overrides: []
        options:
          legend: { calcs: [latest], displayMode: list, placement: bottom, showLegend: true }
          tooltip: { mode: multi, sort: desc }
        transformations: []
      - id: 5
        title: Running TaskRuns
        type: timeseries
        datasource: prometheus
        gridPos: { h: 8, w: 8, x: 8, y: 8 }
        targets:
          - datasource: prometheus
            expr: max(tekton_pipelines_controller_running_taskruns)
            legendFormat: running
            refId: A
        fieldConfig:
          defaults:
            color: { mode: palette-classic }
            custom: { drawStyle: line, fillOpacity: 0, lineWidth: 1, spanNulls: false }
            thresholds: { mode: absolute, steps: [{ color: green, value: null }] }
          overrides: []
        options:
          legend: { calcs: [latest], displayMode: list, placement: bottom, showLegend: true }
          tooltip: { mode: multi, sort: desc }
        transformations: []
      - id: 6
        title: TaskRuns Throttled
        type: timeseries
        datasource: prometheus
        gridPos: { h: 8, w: 8, x: 16, y: 8 }
        targets:
          - datasource: prometheus
            expr: max(tekton_pipelines_controller_running_taskruns_throttled_by_quota)
            legendFormat: by quota
            refId: A
          - datasource: prometheus
            expr: max(tekton_pipelines_controller_running_taskruns_throttled_by_node)
            legendFormat: by node
            refId: B
        fieldConfig:
          defaults:
            color: { mode: palette-classic }
            custom: { drawStyle: line, fillOpacity: 0, lineWidth: 1, spanNulls: false }
            thresholds: { mode: absolute, steps: [{ color: orange, value: null }] }
          overrides: []
        options:
          legend: { calcs: [latest], displayMode: list, placement: bottom, showLegend: true }
          tooltip: { mode: multi, sort: desc }
        transformations: []
      - id: 7
        title: PipelineRun Duration P50 / P95
        type: timeseries
        datasource: prometheus
        gridPos: { h: 8, w: 8, x: 0, y: 16 }
        targets:
          - datasource: prometheus
            expr: (histogram_quantile(0.5, sum by (le) (rate(tekton_pipelines_controller_pipelinerun_duration_seconds_bucket[5m])))) and on() (sum(rate(tekton_pipelines_controller_pipelinerun_duration_seconds_bucket{le="+Inf"}[5m])) > 0)
            legendFormat: P50
            refId: A
          - datasource: prometheus
            expr: (histogram_quantile(0.95, sum by (le) (rate(tekton_pipelines_controller_pipelinerun_duration_seconds_bucket[5m])))) and on() (sum(rate(tekton_pipelines_controller_pipelinerun_duration_seconds_bucket{le="+Inf"}[5m])) > 0)
            legendFormat: P95
            refId: B
        fieldConfig:
          defaults:
            unit: s
            color: { mode: palette-classic }
            custom: { drawStyle: line, fillOpacity: 0, lineWidth: 1, spanNulls: false }
            thresholds: { mode: absolute, steps: [{ color: green, value: null }] }
          overrides: []
        options:
          legend: { calcs: [latest], displayMode: list, placement: bottom, showLegend: true }
          tooltip: { mode: multi, sort: desc }
        transformations: []
      - id: 8
        title: TaskRun Duration P50 / P95 (Standalone)
        type: timeseries
        datasource: prometheus
        gridPos: { h: 8, w: 8, x: 8, y: 16 }
        targets:
          - datasource: prometheus
            expr: (histogram_quantile(0.5, sum by (le) (rate(tekton_pipelines_controller_taskrun_duration_seconds_bucket[5m])))) and on() (sum(rate(tekton_pipelines_controller_taskrun_duration_seconds_bucket{le="+Inf"}[5m])) > 0)
            legendFormat: P50
            refId: A
          - datasource: prometheus
            expr: (histogram_quantile(0.95, sum by (le) (rate(tekton_pipelines_controller_taskrun_duration_seconds_bucket[5m])))) and on() (sum(rate(tekton_pipelines_controller_taskrun_duration_seconds_bucket{le="+Inf"}[5m])) > 0)
            legendFormat: P95
            refId: B
        fieldConfig:
          defaults:
            unit: s
            color: { mode: palette-classic }
            custom: { drawStyle: line, fillOpacity: 0, lineWidth: 1, spanNulls: false }
            thresholds: { mode: absolute, steps: [{ color: green, value: null }] }
          overrides: []
        options:
          legend: { calcs: [latest], displayMode: list, placement: bottom, showLegend: true }
          tooltip: { mode: multi, sort: desc }
        transformations: []
      - id: 13
        title: TaskRun Duration P50 / P95 (In-Pipeline)
        type: timeseries
        datasource: prometheus
        gridPos: { h: 8, w: 8, x: 16, y: 16 }
        targets:
          - datasource: prometheus
            expr: (histogram_quantile(0.5, sum by (le) (rate(tekton_pipelines_controller_pipelinerun_taskrun_duration_seconds_bucket[5m])))) and on() (sum(rate(tekton_pipelines_controller_pipelinerun_taskrun_duration_seconds_bucket{le="+Inf"}[5m])) > 0)
            legendFormat: P50
            refId: A
          - datasource: prometheus
            expr: (histogram_quantile(0.95, sum by (le) (rate(tekton_pipelines_controller_pipelinerun_taskrun_duration_seconds_bucket[5m])))) and on() (sum(rate(tekton_pipelines_controller_pipelinerun_taskrun_duration_seconds_bucket{le="+Inf"}[5m])) > 0)
            legendFormat: P95
            refId: B
        fieldConfig:
          defaults:
            unit: s
            color: { mode: palette-classic }
            custom: { drawStyle: line, fillOpacity: 0, lineWidth: 1, spanNulls: false }
            thresholds: { mode: absolute, steps: [{ color: green, value: null }] }
          overrides: []
        options:
          legend: { calcs: [latest], displayMode: list, placement: bottom, showLegend: true }
          tooltip: { mode: multi, sort: desc }
        transformations: []
      - id: 9
        title: Workqueue Depth
        type: timeseries
        datasource: prometheus
        gridPos: { h: 8, w: 8, x: 0, y: 24 }
        targets:
          - datasource: prometheus
            expr: max(tekton_pipelines_controller_workqueue_depth)
            legendFormat: depth
            refId: A
        fieldConfig:
          defaults:
            color: { mode: palette-classic }
            custom: { drawStyle: line, fillOpacity: 0, lineWidth: 1, spanNulls: false }
            thresholds: { mode: absolute, steps: [{ color: green, value: null }] }
          overrides: []
        options:
          legend: { calcs: [latest], displayMode: list, placement: bottom, showLegend: true }
          tooltip: { mode: multi, sort: desc }
        transformations: []
      - id: 10
        title: Reconcile Count (by success)
        type: timeseries
        datasource: prometheus
        gridPos: { h: 8, w: 8, x: 8, y: 24 }
        targets:
          - datasource: prometheus
            expr: sum(increase(tekton_pipelines_controller_reconcile_count{success="true"}[5m]))
            legendFormat: success=true
            refId: A
          - datasource: prometheus
            expr: sum(increase(tekton_pipelines_controller_reconcile_count{success="false"}[5m]))
            legendFormat: success=false
            refId: B
        fieldConfig:
          defaults:
            color: { mode: palette-classic }
            custom: { drawStyle: line, fillOpacity: 0, lineWidth: 1, spanNulls: false }
            thresholds: { mode: absolute, steps: [{ color: green, value: null }] }
          overrides: []
        options:
          legend: { calcs: [latest], displayMode: list, placement: bottom, showLegend: true }
          tooltip: { mode: multi, sort: desc }
        transformations: []
      - id: 11
        title: Resolution Waiting
        type: timeseries
        datasource: prometheus
        gridPos: { h: 8, w: 8, x: 16, y: 24 }
        targets:
          - datasource: prometheus
            expr: max(tekton_pipelines_controller_running_pipelineruns_waiting_on_pipeline_resolution)
            legendFormat: PR waiting pipeline
            refId: A
          - datasource: prometheus
            expr: max(tekton_pipelines_controller_running_pipelineruns_waiting_on_task_resolution)
            legendFormat: PR waiting task
            refId: B
          - datasource: prometheus
            expr: max(tekton_pipelines_controller_running_taskruns_waiting_on_task_resolution_count)
            legendFormat: TR waiting task
            refId: C
        fieldConfig:
          defaults:
            color: { mode: palette-classic }
            custom: { drawStyle: line, fillOpacity: 0, lineWidth: 1, spanNulls: false }
            thresholds: { mode: absolute, steps: [{ color: orange, value: null }] }
          overrides: []
        options:
          legend: { calcs: [latest], displayMode: list, placement: bottom, showLegend: true }
          tooltip: { mode: multi, sort: desc }
        transformations: []

Tekton Pipeline 监控面板解读(常见问题)

  • PipelineRun Total (by status) 是由 controller 记录的完成事件计数器,而不是 PipelineRun 对象的总数。在当前实现中,用户触发的取消操作(spec.status=Cancelled)可能不会进入此计数路径,因此 cancelled 系列可能不存在。要验证取消量,请检查 PipelineRun 对象和事件。
  • Running PipelineRuns 是实时快照(当前正在运行的数量)。它可以独立于 PipelineRun Total 发生变化。
  • Completed PipelineRuns (last 5m) 是吞吐量(过去 5 分钟内新完成的运行次数)。在低流量或空闲期间看到 0 是预期行为。
  • PipelineRun Success Rate (cumulative) 是 controller 启动以来的累计值,而不是 5 分钟窗口内的成功率。短期故障不会立即导致数值大幅变化。
  • Reconcile Count (by success) 衡量的是 controller reconcile 循环,而不是 PipelineRun 数量。
  • 只有在所选时间范围内实际存在样本的标签值才会显示状态系列。如果某个状态在该时间窗口内没有样本,则不会显示其曲线或图例。
  • TaskRun Duration P50 / P95 (Standalone) 和 TaskRun Duration P50 / P95 (In-Pipeline) 分开显示,以避免混合查询不稳定。在仅公开其中一个直方图系列的环境中,另一个图表可能为空,这是预期行为。

Tekton Triggers 监控面板

Tekton Triggers Dashboard YAML
kind: MonitorDashboard
apiVersion: ait.alauda.io/v1alpha2
metadata:
  labels:
    cpaas.io/dashboard.folder: tekton
    cpaas.io/dashboard.is.home.dashboard: "false"
    cpaas.io/dashboard.tag.tekton: "true"
  name: tekton-triggers
  namespace: cpaas-system
spec:
  body:
    titleZh: Tekton Triggers Overview
    tags:
      - tekton
    time:
      from: now-1h
      to: now
    templating:
      list: []
    panels:
      - id: 1
        title: EventListener Count
        type: timeseries
        datasource: prometheus
        gridPos: { h: 6, w: 5, x: 0, y: 0 }
        targets:
          - datasource: prometheus
            expr: controller_eventlistener_count
            legendFormat: EventListener
            refId: A
        fieldConfig:
          defaults:
            color: { mode: palette-classic }
            custom: { drawStyle: line, fillOpacity: 0, lineWidth: 1, spanNulls: false }
            thresholds: { mode: absolute, steps: [{ color: green, value: null }] }
          overrides: []
        options:
          legend: { calcs: [latest], displayMode: list, placement: bottom, showLegend: true }
          tooltip: { mode: multi, sort: desc }
        transformations: []
      - id: 2
        title: TriggerTemplate Count
        type: timeseries
        datasource: prometheus
        gridPos: { h: 6, w: 5, x: 5, y: 0 }
        targets:
          - datasource: prometheus
            expr: controller_triggertemplate_count
            legendFormat: TriggerTemplate
            refId: A
        fieldConfig:
          defaults:
            color: { mode: palette-classic }
            custom: { drawStyle: line, fillOpacity: 0, lineWidth: 1, spanNulls: false }
            thresholds: { mode: absolute, steps: [{ color: green, value: null }] }
          overrides: []
        options:
          legend: { calcs: [latest], displayMode: list, placement: bottom, showLegend: true }
          tooltip: { mode: multi, sort: desc }
        transformations: []
      - id: 3
        title: TriggerBinding Count
        type: timeseries
        datasource: prometheus
        gridPos: { h: 6, w: 5, x: 10, y: 0 }
        targets:
          - datasource: prometheus
            expr: controller_triggerbinding_count
            legendFormat: TriggerBinding
            refId: A
        fieldConfig:
          defaults:
            color: { mode: palette-classic }
            custom: { drawStyle: line, fillOpacity: 0, lineWidth: 1, spanNulls: false }
            thresholds: { mode: absolute, steps: [{ color: green, value: null }] }
          overrides: []
        options:
          legend: { calcs: [latest], displayMode: list, placement: bottom, showLegend: true }
          tooltip: { mode: multi, sort: desc }
        transformations: []
      - id: 4
        title: ClusterTriggerBinding
        type: timeseries
        datasource: prometheus
        gridPos: { h: 6, w: 5, x: 15, y: 0 }
        targets:
          - datasource: prometheus
            expr: controller_clustertriggerbinding_count
            legendFormat: ClusterTriggerBinding
            refId: A
        fieldConfig:
          defaults:
            color: { mode: palette-classic }
            custom: { drawStyle: line, fillOpacity: 0, lineWidth: 1, spanNulls: false }
            thresholds: { mode: absolute, steps: [{ color: green, value: null }] }
          overrides: []
        options:
          legend: { calcs: [latest], displayMode: list, placement: bottom, showLegend: true }
          tooltip: { mode: multi, sort: desc }
        transformations: []
      - id: 5
        title: ClusterInterceptor
        type: timeseries
        datasource: prometheus
        gridPos: { h: 6, w: 4, x: 20, y: 0 }
        targets:
          - datasource: prometheus
            expr: controller_clusterinterceptor_count
            legendFormat: ClusterInterceptor
            refId: A
        fieldConfig:
          defaults:
            color: { mode: palette-classic }
            custom: { drawStyle: line, fillOpacity: 0, lineWidth: 1, spanNulls: false }
            thresholds: { mode: absolute, steps: [{ color: green, value: null }] }
          overrides: []
        options:
          legend: { calcs: [latest], displayMode: list, placement: bottom, showLegend: true }
          tooltip: { mode: multi, sort: desc }
        transformations: []
      - id: 6
        title: All Trigger Resource Counts (trend)
        type: timeseries
        datasource: prometheus
        gridPos: { h: 8, w: 24, x: 0, y: 6 }
        targets:
          - datasource: prometheus
            expr: controller_eventlistener_count
            legendFormat: EventListener
            refId: A
          - datasource: prometheus
            expr: controller_triggertemplate_count
            legendFormat: TriggerTemplate
            refId: B
          - datasource: prometheus
            expr: controller_triggerbinding_count
            legendFormat: TriggerBinding
            refId: C
          - datasource: prometheus
            expr: controller_clustertriggerbinding_count
            legendFormat: ClusterTriggerBinding
            refId: D
          - datasource: prometheus
            expr: controller_clusterinterceptor_count
            legendFormat: ClusterInterceptor
            refId: E
        fieldConfig:
          defaults:
            color: { mode: palette-classic }
            custom: { drawStyle: line, fillOpacity: 0, lineWidth: 1, spanNulls: false }
            thresholds: { mode: absolute, steps: [{ color: green, value: null }] }
          overrides: []
        options:
          legend: { calcs: [latest], displayMode: list, placement: bottom, showLegend: true }
          tooltip: { mode: multi, sort: desc }
        transformations: []

Tekton Triggers 监控面板解读(常见问题)

  • EventListener Count、TriggerTemplate Count、TriggerBinding Count、ClusterTriggerBinding 和 ClusterInterceptor 是对象数量快照,而不是请求量或事件处理吞吐量。
  • All Trigger Resource Counts (trend) 显示相同资源数量的综合趋势。与单资源趋势图表相比,在一个抓取间隔内出现短暂偏差是预期行为。
  • 在不存在 Triggers 资源时显示 0 属于正常现象,并不表示抓取失败。

Tekton Results 监控面板

Tekton Results Dashboard YAML
kind: MonitorDashboard
apiVersion: ait.alauda.io/v1alpha2
metadata:
  labels:
    cpaas.io/dashboard.folder: tekton
    cpaas.io/dashboard.is.home.dashboard: "false"
    cpaas.io/dashboard.tag.tekton: "true"
  name: tekton-results
  namespace: cpaas-system
spec:
  body:
    titleZh: Tekton Results Overview
    tags:
      - tekton
    time:
      from: now-1h
      to: now
    templating:
      list: []
    panels:
      - id: 1
        title: PipelineRun Reconcile Count (last 5m)
        type: timeseries
        datasource: prometheus
        gridPos: { h: 8, w: 12, x: 0, y: 0 }
        targets:
          - datasource: prometheus
            expr: round(sum(increase(watcher_reconcile_count{reconciler="github.com.tektoncd.results.pkg.watcher.reconciler.pipelinerun.Reconciler",success="true"}[5m])))
            legendFormat: success=true
            refId: A
          - datasource: prometheus
            expr: round(sum(increase(watcher_reconcile_count{reconciler="github.com.tektoncd.results.pkg.watcher.reconciler.pipelinerun.Reconciler",success="false"}[5m])))
            legendFormat: success=false
            refId: B
        fieldConfig:
          defaults:
            color: { mode: palette-classic }
            custom: { drawStyle: line, fillOpacity: 0, lineWidth: 1, spanNulls: false }
            thresholds: { mode: absolute, steps: [{ color: green, value: null }] }
          overrides: []
        options:
          legend: { calcs: [latest], displayMode: list, placement: bottom, showLegend: true }
          tooltip: { mode: multi, sort: desc }
        transformations: []
      - id: 2
        title: TaskRun Reconcile Count (last 5m)
        type: timeseries
        datasource: prometheus
        gridPos: { h: 8, w: 12, x: 12, y: 0 }
        targets:
          - datasource: prometheus
            expr: round(sum(increase(watcher_reconcile_count{reconciler="github.com.tektoncd.results.pkg.watcher.reconciler.taskrun.Reconciler",success="true"}[5m])))
            legendFormat: success=true
            refId: A
          - datasource: prometheus
            expr: round(sum(increase(watcher_reconcile_count{reconciler="github.com.tektoncd.results.pkg.watcher.reconciler.taskrun.Reconciler",success="false"}[5m])))
            legendFormat: success=false
            refId: B
        fieldConfig:
          defaults:
            color: { mode: palette-classic }
            custom: { drawStyle: line, fillOpacity: 0, lineWidth: 1, spanNulls: false }
            thresholds: { mode: absolute, steps: [{ color: green, value: null }] }
          overrides: []
        options:
          legend: { calcs: [latest], displayMode: list, placement: bottom, showLegend: true }
          tooltip: { mode: multi, sort: desc }
        transformations: []
      - id: 3
        title: PipelineRun Reconcile Latency P95
        type: timeseries
        datasource: prometheus
        gridPos: { h: 8, w: 12, x: 0, y: 8 }
        targets:
          - datasource: prometheus
            expr: histogram_quantile(0.95, sum by (le) (rate(watcher_reconcile_latency_bucket{reconciler="github.com.tektoncd.results.pkg.watcher.reconciler.pipelinerun.Reconciler"}[5m])))
            legendFormat: P95
            refId: A
        fieldConfig:
          defaults:
            unit: ms
            color: { mode: palette-classic }
            custom: { drawStyle: line, fillOpacity: 0, lineWidth: 1, spanNulls: false }
            thresholds: { mode: absolute, steps: [{ color: green, value: null }] }
          overrides: []
        options:
          legend: { calcs: [latest], displayMode: list, placement: bottom, showLegend: true }
          tooltip: { mode: multi, sort: desc }
        transformations: []
      - id: 4
        title: TaskRun Reconcile Latency P95
        type: timeseries
        datasource: prometheus
        gridPos: { h: 8, w: 12, x: 12, y: 8 }
        targets:
          - datasource: prometheus
            expr: histogram_quantile(0.95, sum by (le) (rate(watcher_reconcile_latency_bucket{reconciler="github.com.tektoncd.results.pkg.watcher.reconciler.taskrun.Reconciler"}[5m])))
            legendFormat: P95
            refId: A
        fieldConfig:
          defaults:
            unit: ms
            color: { mode: palette-classic }
            custom: { drawStyle: line, fillOpacity: 0, lineWidth: 1, spanNulls: false }
            thresholds: { mode: absolute, steps: [{ color: green, value: null }] }
          overrides: []
        options:
          legend: { calcs: [latest], displayMode: list, placement: bottom, showLegend: true }
          tooltip: { mode: multi, sort: desc }
        transformations: []
      - id: 5
        title: Workqueue Depth (PipelineRun vs TaskRun)
        type: timeseries
        datasource: prometheus
        gridPos: { h: 8, w: 12, x: 0, y: 16 }
        targets:
          - datasource: prometheus
            expr: sum(watcher_work_queue_depth{reconciler="github.com.tektoncd.results.pkg.watcher.reconciler.pipelinerun.Reconciler"})
            legendFormat: pipelinerun
            refId: A
          - datasource: prometheus
            expr: sum(watcher_work_queue_depth{reconciler="github.com.tektoncd.results.pkg.watcher.reconciler.taskrun.Reconciler"})
            legendFormat: taskrun
            refId: B
        fieldConfig:
          defaults:
            color: { mode: palette-classic }
            custom: { drawStyle: line, fillOpacity: 0, lineWidth: 1, spanNulls: false }
            thresholds: { mode: absolute, steps: [{ color: green, value: null }] }
          overrides: []
        options:
          legend: { calcs: [latest], displayMode: list, placement: bottom, showLegend: true }
          tooltip: { mode: multi, sort: desc }
        transformations: []
      - id: 6
        title: Workqueue Adds (last 5m)
        type: timeseries
        datasource: prometheus
        gridPos: { h: 8, w: 12, x: 12, y: 16 }
        targets:
          - datasource: prometheus
            expr: round(sum(increase(watcher_workqueue_adds_total{name=~"github.com.tektoncd.results.pkg.watcher.reconciler.pipelinerun.Reconciler-(consumer|fast|slow)"}[5m])))
            legendFormat: pipelinerun adds
            refId: A
          - datasource: prometheus
            expr: round(sum(increase(watcher_workqueue_adds_total{name=~"github.com.tektoncd.results.pkg.watcher.reconciler.taskrun.Reconciler-(consumer|fast|slow)"}[5m])))
            legendFormat: taskrun adds
            refId: B
        fieldConfig:
          defaults:
            color: { mode: palette-classic }
            custom: { drawStyle: line, fillOpacity: 0, lineWidth: 1, spanNulls: false }
            thresholds: { mode: absolute, steps: [{ color: green, value: null }] }
          overrides: []
        options:
          legend: { calcs: [latest], displayMode: list, placement: bottom, showLegend: true }
          tooltip: { mode: multi, sort: desc }
        transformations: []
      - id: 7
        title: gRPC Request Rate (Results API)
        type: timeseries
        datasource: prometheus
        gridPos: { h: 8, w: 12, x: 0, y: 24 }
        targets:
          - datasource: prometheus
            expr: "sum(rate(grpc_server_handled_total{grpc_service=~\"tekton.results.*\"}[5m]))"
            legendFormat: requests
            refId: A
        fieldConfig:
          defaults:
            color: { mode: palette-classic }
            custom: { drawStyle: line, fillOpacity: 0, lineWidth: 1, spanNulls: false }
            thresholds: { mode: absolute, steps: [{ color: green, value: null }] }
          overrides: []
        options:
          legend: { calcs: [latest], displayMode: list, placement: bottom, showLegend: true }
          tooltip: { mode: multi, sort: desc }
        transformations: []
      - id: 8
        title: gRPC Error Percentage (Results API, excl. NotFound/Canceled)
        type: timeseries
        datasource: prometheus
        gridPos: { h: 8, w: 12, x: 12, y: 24 }
        targets:
          - datasource: prometheus
            expr: "100 * ((sum(rate(grpc_server_handled_total{grpc_service=~\"tekton.results.*\",grpc_code!~\"OK|NotFound|Canceled\"}[5m])) or vector(0)) / clamp_min((sum(rate(grpc_server_handled_total{grpc_service=~\"tekton.results.*\"}[5m])) or vector(0)), 0.001))"
            legendFormat: error %
            refId: A
        fieldConfig:
          defaults:
            unit: percent
            color: { mode: palette-classic }
            custom: { drawStyle: line, fillOpacity: 0, lineWidth: 1, spanNulls: false }
            thresholds: { mode: absolute, steps: [{ color: red, value: null }] }
          overrides: []
        options:
          legend: { calcs: [latest], displayMode: list, placement: bottom, showLegend: true }
          tooltip: { mode: multi, sort: desc }
        transformations: []

Tekton Results 监控面板解读(常见问题)

  • 此监控面板版本基于 Results Watcher reconcile/workqueue 指标以及 Results API gRPC 指标,因此在常见部署配置(logs_api=true、禁用自动删除)下仍会显示数据。
  • PipelineRun Reconcile Count (last 5m) 和 TaskRun Reconcile Count (last 5m) 分别显示 success=true 和 success=false 在 5 分钟内的增量。
  • PipelineRun Reconcile Latency P95 和 TaskRun Reconcile Latency P95 根据 watcher reconcile 延迟直方图计算。在低流量情况下,曲线可能较为稀疏。
  • Workqueue Depth 显示当前队列深度,Workqueue Adds (last 5m) 显示过去 5 分钟内的入队量。
  • gRPC Error Percentage (Results API, excl. NotFound/Canceled) 是异常错误占全部请求的百分比,其中排除了常见业务返回码(NotFound、Canceled)。

Tekton Chains 监控面板

Tekton Chains Dashboard YAML
kind: MonitorDashboard
apiVersion: ait.alauda.io/v1alpha2
metadata:
  labels:
    cpaas.io/dashboard.folder: tekton
    cpaas.io/dashboard.is.home.dashboard: "false"
    cpaas.io/dashboard.tag.tekton: "true"
  name: tekton-chains
  namespace: cpaas-system
spec:
  body:
    titleZh: Tekton Chains Overview
    tags:
      - tekton
    time:
      from: now-1h
      to: now
    templating:
      list: []
    panels:
      - id: 1
        title: TaskRun Signatures Created (last 5m)
        type: timeseries
        datasource: prometheus
        gridPos: { h: 8, w: 12, x: 0, y: 0 }
        targets:
          - datasource: prometheus
            expr: round(increase(watcher_taskrun_sign_created_total[5m]))
            legendFormat: sign created
            refId: A
        fieldConfig:
          defaults:
            color: { mode: palette-classic }
            custom: { drawStyle: line, fillOpacity: 0, lineWidth: 1, spanNulls: false }
            thresholds: { mode: absolute, steps: [{ color: green, value: null }] }
          overrides: []
        options:
          legend: { calcs: [latest], displayMode: list, placement: bottom, showLegend: true }
          tooltip: { mode: multi, sort: desc }
        transformations: []
      - id: 2
        title: PipelineRun Signatures Created (last 5m)
        type: timeseries
        datasource: prometheus
        gridPos: { h: 8, w: 12, x: 12, y: 0 }
        targets:
          - datasource: prometheus
            expr: round(increase(watcher_pipelinerun_sign_created_total[5m]))
            legendFormat: sign created
            refId: A
        fieldConfig:
          defaults:
            color: { mode: palette-classic }
            custom: { drawStyle: line, fillOpacity: 0, lineWidth: 1, spanNulls: false }
            thresholds: { mode: absolute, steps: [{ color: green, value: null }] }
          overrides: []
        options:
          legend: { calcs: [latest], displayMode: list, placement: bottom, showLegend: true }
          tooltip: { mode: multi, sort: desc }
        transformations: []
      - id: 3
        title: Payloads Stored (last 5m, TaskRun vs PipelineRun)
        type: timeseries
        datasource: prometheus
        gridPos: { h: 8, w: 12, x: 0, y: 8 }
        targets:
          - datasource: prometheus
            expr: round(increase(watcher_taskrun_payload_stored_total[5m]))
            legendFormat: TaskRun
            refId: A
          - datasource: prometheus
            expr: round(increase(watcher_pipelinerun_payload_stored_total[5m]))
            legendFormat: PipelineRun
            refId: B
        fieldConfig:
          defaults:
            color: { mode: palette-classic }
            custom: { drawStyle: line, fillOpacity: 0, lineWidth: 1, spanNulls: false }
            thresholds: { mode: absolute, steps: [{ color: green, value: null }] }
          overrides: []
        options:
          legend: { calcs: [latest], displayMode: list, placement: bottom, showLegend: true }
          tooltip: { mode: multi, sort: desc }
        transformations: []
      - id: 4
        title: Marked Signed (last 5m, TaskRun vs PipelineRun)
        type: timeseries
        datasource: prometheus
        gridPos: { h: 8, w: 12, x: 12, y: 8 }
        targets:
          - datasource: prometheus
            expr: round(increase(watcher_taskrun_marked_signed_total[5m]))
            legendFormat: TaskRun
            refId: A
          - datasource: prometheus
            expr: round(increase(watcher_pipelinerun_marked_signed_total[5m]))
            legendFormat: PipelineRun
            refId: B
        fieldConfig:
          defaults:
            color: { mode: palette-classic }
            custom: { drawStyle: line, fillOpacity: 0, lineWidth: 1, spanNulls: false }
            thresholds: { mode: absolute, steps: [{ color: green, value: null }] }
          overrides: []
        options:
          legend: { calcs: [latest], displayMode: list, placement: bottom, showLegend: true }
          tooltip: { mode: multi, sort: desc }
        transformations: []

Tekton Chains 监控面板解读(常见问题)

  • TaskRun Signatures Created (last 5m)、PipelineRun Signatures Created (last 5m)、Payloads Stored (last 5m) 和 Marked Signed (last 5m) 使用 increase(...[5m]),表示过去五分钟内的增量。
  • 没有新的签名或存储活动时,这些曲线会降至 0;这并不表示组件发生故障。
  • Payloads Stored 和 Marked Signed 表示不同的处理阶段,因此其值不一定始终匹配。