Tekton 组件的指标采集

概述前提条件 Tekton Pipelines PipelineRun 指标 running_pipelineruns 标签级别状态标签值 TaskRun 指标 config-observability 配置 Histogram Buckets 推荐的生产环境配置 Tekton Triggers Controller 指标（端口 9000）EventListener Sink 指标 Tekton Results Watcher 指标删除指标共享指标 Watcher config-observability API Server 指标 Tekton Chains Chains 指标 Controller Framework 指标设置 ServiceMonitor Pipeline ServiceMonitor Triggers ServiceMonitor EventListener Sink ServiceMonitor Results ServiceMonitor Chains ServiceMonitor 验证直接检查指标 endpoint 检查 Prometheus Targets PromQL 查询示例 MonitorDashboard 示例 Tekton Pipeline 监控面板 Tekton Pipeline 监控面板说明（常见问题）Tekton Triggers 监控面板 Tekton Triggers 监控面板说明（常见问题）Tekton Results 监控面板 Tekton Results 监控面板说明（常见问题）Tekton Chains 监控面板 Tekton Chains 监控面板说明（常见问题）

概述

Tekton 组件通过 HTTP endpoints 暴露与 Prometheus 兼容的指标。通过部署 ServiceMonitor 资源，Prometheus（或 VictoriaMetrics）可以自动发现并抓取这些指标。

命名空间说明：本文档将 tekton-pipelines 作为控制平面组件（Pipelines、Triggers、Results、Chains）的默认 namespace。主要例外是 EventListener Services，它们运行在创建 EventListeners 的应用程序 namespace 中。

如果你的部署使用不同的 namespace，请同时更新下面命令中的 namespace 以及 ServiceMonitor 资源中的 namespaceSelector 字段。

本文档涵盖以下 Tekton 组件的指标：

Tekton Pipelines - PipelineRun / TaskRun 执行指标
Tekton Triggers - EventListener、TriggerBinding 及相关资源指标
Tekton Results - Run 删除和存储指标
Tekton Chains - 签名和 provenance 指标
Controller Framework - 所有 controller 共享的基础设施指标

此外还涵盖：

如何通过 config-observability 配置指标行为
如何部署用于抓取的 ServiceMonitor 资源
如何验证指标采集是否正常工作

前提条件

Tekton 控制平面组件已安装并运行（至少包括你计划抓取的组件：Pipelines、Triggers、Results 和/或 Chains）。
kubectl 已针对目标集群完成配置，并且你的账号可以在监控 namespace 中创建 ServiceMonitor 资源。
已部署监控栈（Prometheus 或兼容的 VictoriaMetrics），并且能够发现/抓取 ServiceMonitor 资源（或平台使用的等效抓取发现对象）。
你的 Prometheus/VictoriaMetrics 实例已配置为发现你创建的 ServiceMonitor 对象（namespace 和 label 选择器必须匹配）。
网络策略和防火墙允许抓取 Pod 访问 Tekton 指标端口（大多数控制平面服务使用 9090，Triggers controller 和 EventListener sink 使用 9000）。
如果你需要 EventListener sink 指标，EventListeners 必须存在于其目标 namespace 中，并暴露 http-metrics 端口。

Tekton Pipelines

Tekton Pipelines 组件包含多个在 9090 端口暴露指标的子服务：

Service	Description	Metrics Port
`tekton-pipelines-controller`	PipelineRun / TaskRun 的主 reconciler	9090
`tekton-pipelines-webhook`	Admission webhook	9090
`tekton-events-controller`	CloudEvents controller	9090
`tekton-pipelines-remote-resolvers`	远程资源解析	9090

Pipeline controller 指标使用前缀 tekton_pipelines_controller_。

PipelineRun 指标

Metric Name	Type	Description	Labels
`pipelinerun_duration_seconds`	Histogram / LastValue	PipelineRun 执行时间（秒）	`status`, `namespace`, `pipeline`, `pipelinerun`, `reason`*
`pipelinerun_total`	Counter	已完成 PipelineRun 总数	`status`
`running_pipelineruns`	LastValue (Gauge)	当前正在运行的 PipelineRun 数量	由 `metrics.running-pipelinerun.level` 控制（见下文）
`running_pipelineruns_waiting_on_pipeline_resolution`	LastValue (Gauge)	正在等待 Pipeline 引用解析的 PipelineRun	-
`running_pipelineruns_waiting_on_task_resolution`	LastValue (Gauge)	正在等待 Task 引用解析的 PipelineRun	-

* 带有 * 标记的标签是可选的，取决于 config-observability 配置。

`running_pipelineruns` 标签级别

running_pipelineruns 指标的标签由 metrics.running-pipelinerun.level 控制：

Level	Labels
`""` (default, cluster)	无标签
`"namespace"`	`namespace`
`"pipeline"`	`namespace`, `pipeline`
`"pipelinerun"`	`namespace`, `pipeline`, `pipelinerun`

状态标签值

对于 PipelineRun 指标：

success - PipelineRun 已成功完成
failed - PipelineRun 失败
cancelled - PipelineRun 已取消

对于 TaskRun 指标：

success - TaskRun 已成功完成
failed - TaskRun 失败

TaskRun 指标

Metric Name	Type	Description	Labels
`taskrun_duration_seconds`	Histogram / LastValue	独立 TaskRun 执行时间（秒）	`status`, `namespace`, `task`, `taskrun`, `reason`*
`pipelinerun_taskrun_duration_seconds`	Histogram / LastValue	作为 PipelineRun 一部分时的 TaskRun 执行时间	`status`, `namespace`, `task`, `taskrun`, `pipeline`, `pipelinerun`, `reason`*
`taskrun_total`	Counter	已完成 TaskRun 总数	`status`
`running_taskruns`	LastValue (Gauge)	当前正在运行的 TaskRun 数量	-
`running_taskruns_waiting_on_task_resolution_count`	LastValue (Gauge)	正在等待 Task 引用解析的 TaskRun	-
`running_taskruns_throttled_by_quota`	LastValue (Gauge)	被 ResourceQuota 限流的 TaskRun	`namespace`*
`running_taskruns_throttled_by_node`	LastValue (Gauge)	被节点级资源约束限流的 TaskRun	`namespace`*
`taskruns_pod_latency_milliseconds`	LastValue	TaskRun 的 Pod 调度延迟，单位毫秒	`namespace`, `pod`, `task`, `taskrun`

config-observability 配置

tekton-pipelines namespace 中的 config-observability ConfigMap 控制 Pipeline controller 的指标行为。该 ConfigMap 由 Tekton Operator 管理，应通过 TektonConfig 资源的 spec.pipeline.options.configMaps 字段进行配置。详细信息请参见为子组件调整可选配置项。

热加载行为：运行时会监视 config-observability。大多数 key 的变更（例如 metrics.*）会在不重启 Pod 的情况下生效。请等待一个或两个抓取周期，以便监控面板/查询变化显示出来。只有在 Pod spec 设置变更时才需要重启（例如在 Deployment 中更改 CONFIG_OBSERVABILITY_NAME）。

通过 TektonConfig 配置的示例：

apiVersion: operator.tekton.dev/v1alpha1
kind: TektonConfig
metadata:
  name: config
spec:
  pipeline:
    options:
      disabled: false
      configMaps:
        config-observability:
          data:
            metrics.backend-destination: prometheus

            # PipelineRun 指标聚合级别。
            # Values: "pipelinerun" | "pipeline" (default) | "namespace"
            #   - "pipelinerun": 包含 pipeline + pipelinerun 标签；duration 使用 LastValue
            #   - "pipeline": 仅包含 pipeline 标签
            #   - "namespace": 不包含 pipeline/pipelinerun 标签
            metrics.pipelinerun.level: "pipeline"

            # TaskRun 指标聚合级别。
            # Values: "taskrun" | "task" (default) | "namespace"
            #   - "taskrun": 包含 task + taskrun 标签；duration 使用 LastValue
            #   - "task": 仅包含 task 标签
            #   - "namespace": 不包含 task/taskrun 标签
            metrics.taskrun.level: "task"

            # PipelineRun / TaskRun 的 duration metric 类型。
            # Values: "histogram" (default) | "lastvalue"
            # Note: When pipelinerun.level is "pipelinerun" or taskrun.level is "taskrun",
            #       duration type is forced to "lastvalue" regardless of this setting.
            metrics.pipelinerun.duration-type: "histogram"
            metrics.taskrun.duration-type: "histogram"

            # Running PipelineRun 指标聚合级别。
            # Values: "pipelinerun" | "pipeline" | "namespace" | "" (default, cluster-level)
            metrics.running-pipelinerun.level: ""

            # 在 duration 指标上包含 reason 标签（pipelinerun_duration_seconds、
            # taskrun_duration_seconds、pipelinerun_taskrun_duration_seconds）。
            # Values: "true" | "false" (default)
            # Warning: 启用此项会增加 label cardinality。
            # Note: 尽管 key 名称如此，但这不会影响 count 指标
            # (pipelinerun_total / taskrun_total)，只影响 duration 指标。
            metrics.count.enable-reason: "false"

            # 在被限流的 TaskRun 指标上包含 namespace 标签。
            # Values: "true" | "false" (default)
            metrics.taskrun.throttle.enable-namespace: "false"

Histogram Buckets

当 duration 类型为 histogram 时，使用以下 bucket 边界（单位：秒）：

10, 30, 60, 300, 900, 1800, 3600, 5400, 10800, 21600, 43200, 86400

对应为：10s、30s、1m、5m、15m、30m、1h、1.5h、3h、6h、12h、24h。

Tekton Triggers

Tekton Triggers 组件从不同进程暴露两类指标。

Controller 指标（端口 9000）

Triggers controller 每 60 秒上报一次资源计数指标。

Service	Metrics Port
`tekton-triggers-controller`	9000

Triggers controller 指标使用前缀 controller_。

Metric Name	Type	Description	Labels
`eventlistener_count`	LastValue (Gauge)	EventListener 资源数量	-
`triggerbinding_count`	LastValue (Gauge)	TriggerBinding 资源数量	-
`clustertriggerbinding_count`	LastValue (Gauge)	ClusterTriggerBinding 资源数量	-
`triggertemplate_count`	LastValue (Gauge)	TriggerTemplate 资源数量	-
`clusterinterceptor_count`	LastValue (Gauge)	ClusterInterceptor 资源数量	-

EventListener Sink 指标

每个 EventListener Pod 还会暴露额外的 HTTP 和事件处理指标。这些指标来自 EventListener sink 进程（不是 controller）。Prometheus 指标前缀为 eventlistener_。

Metric Name (Prometheus)	Type	Description	Labels
`eventlistener_http_duration_seconds`	Histogram	EventListener HTTP 请求耗时	-
`eventlistener_event_received_count`	Counter	sink 接收到的事件总数	`status`
`eventlistener_triggered_resources`	Counter	由 triggers 创建的资源总数	`kind`

eventlistener_http_duration_seconds histogram buckets: 0.001, 0.01, 0.1, 1, 10（秒）
eventlistener_event_received_count status values: succeeded, failed
eventlistener_triggered_resources kind values: 已创建对象的 Kubernetes resource Kind（例如 PipelineRun、TaskRun）

这些 sink 指标是按每个 EventListener Pod 暴露的，而不是来自中心 controller。如果 EventListener Pod 暴露了 metrics port，你可能需要单独的 ServiceMonitor 或 PodMonitor 来抓取它们。

Tekton Results

Tekton Results 有两个暴露指标的子服务。

Service	Description	Metrics Port
`tekton-results-watcher`	监视并清理 PipelineRun/TaskRun 资源	9090
`tekton-results-api`	gRPC/REST API server	9090

Watcher 指标

Watcher 指标使用前缀 watcher_。

删除指标

Metric Name	Type	Description	Labels
`pipelinerun_delete_count`	Counter	已删除 PipelineRun 总数	`status`, `namespace`
`pipelinerun_delete_duration_seconds`	Histogram / LastValue	从 PipelineRun 完成到删除的时间	`status`, `namespace`, `pipeline`*
`taskrun_delete_count`	Counter	已删除 TaskRun 总数	`status`, `namespace`
`taskrun_delete_duration_seconds`	Histogram / LastValue	从 TaskRun 完成到删除的时间	`status`, `namespace`, `pipeline`, `task`

* 可选标签取决于 Results Watcher 的 config-observability 设置。

注意：只有在 Watcher 实际删除 run 时，才会记录 pipelinerun_delete_count、pipelinerun_delete_duration_seconds、taskrun_delete_count 和 taskrun_delete_duration_seconds。除非在 tekton-results-watcher Deployment 上将 --completed_run_grace_period 参数设置为非零值，否则这些指标将保持为空（没有数据点）。默认情况下该参数为 0，会禁用自动删除。将其设置为正持续时间（例如 10m）可在宽限期后启用删除，或者设置为负值以在归档后立即删除。

Results Watcher 的状态标签值：

success - Run 已成功完成
failed - Run 失败
cancelled - Run 已取消

共享指标

这些指标由 Watcher 中的 PipelineRun 和 TaskRun reconciler 共同注册，用于跟踪与存储相关的事件。

Metric Name	Type	Description	Labels
`runs_not_stored_count`	Counter	删除但未存储到 Results 的 runs	`kind`, `namespace`
`run_storage_latency_seconds`	Histogram	从 run 完成到成功存储的时间	`kind`, `namespace`

kind 标签标识 run 类型（某些 metric series 中为 PipelineRun / TaskRun，另一些中为 pipelinerun / taskrun）。

注意：只有当某个 run 在 Watcher 持有 finalizer 用于协调归档期间被外部删除时（例如通过 kubectl delete），才会记录 runs_not_stored_count。除非同时满足以下所有条件，否则该指标将保持为空：

--logs_api 参数为 false（禁用日志存储）——如果启用了日志，Watcher 会完全跳过基于 finalizer 的协调。

--disable_crd_update 参数为 false（启用 annotation 更新）。

--store_deadline 参数设置为非零持续时间——这是 Watcher 在放弃并允许删除之前，等待归档完成的最长时间。

某个 run 在成功归档之前被外部删除（没有 results.tekton.dev/stored=true annotation），并且 store_deadline 已经过期。

在正常运行中（run 在删除前已归档，或者删除由 Watcher 通过 --completed_run_grace_period 自身触发），这个计数器会保持为 0。非零值表示可能存在数据丢失：run 在其状态保存到 Results API 之前就已被删除。

快速复现（测试环境）：如果你没有看到该指标，通常意味着触发条件未满足，而不是指标缺失。

通过 TektonConfig 配置 Results Watcher，使 logs_api=false、disable_crd_update=false，并且 store_deadline 为非零值（例如 30s）。

临时通过 TektonConfig 将 Results API 副本数设置为 0（spec.result.options.deployments.tekton-results-api.spec.replicas: 0），这样 run 将无法被归档。

创建一个 TaskRun 或 PipelineRun，并等待其完成。

等待 store_deadline 经过，然后外部删除该 run（kubectl delete ...）。

检查 Watcher 的 /metrics 或 Prometheus 中的 watcher_runs_not_stored_count（在暴露格式中为 component-prefixed 名称）；它应当增加。

恢复原始 TektonConfig（重新启用 Results API 副本并恢复正常的 logs_api 设置）。

run_storage_latency_seconds histogram 使用以下 bucket 边界（单位：秒）：

0.1, 0.5, 1, 2, 5, 10, 30, 60, 120, 300, 600, 1800

Watcher config-observability

Results Watcher 拥有自己的 config-observability ConfigMap（通过 CONFIG_OBSERVABILITY_NAME 环境变量指定，通常为 tekton-results-config-observability）。该 ConfigMap 由 Tekton Operator 管理，应通过 TektonConfig 资源的 spec.result.options.configMaps 字段进行配置。详细信息请参见为子组件调整可选配置项。

热加载行为：Results Watcher 也会监视这个 ConfigMap，并在不重启 Pod 的情况下应用大多数 key 变更。只有在 Deployment 级别设置（例如 env vars/args）发生变化时才需要重启。

它支持以下 keys：

Key	Default	Values	Description
`metrics.pipelinerun.level`	`pipeline`	`pipeline`, `namespace`	控制删除 duration 指标上的 `pipeline` 标签
`metrics.taskrun.level`	`task`	`task`, `namespace`	控制删除 duration 指标上的 `task` 标签
`metrics.pipelinerun.duration-type`	`histogram`	`histogram`, `lastvalue`	PipelineRun 和 TaskRun 删除的 duration metric 聚合类型
`metrics.taskrun.duration-type`	`histogram`	`histogram`, `lastvalue`	已解析，但当前未使用；由 `metrics.pipelinerun.duration-type` 同时控制

注意：与 Tekton Pipelines 不同，Results Watcher 不支持 pipelinerun / taskrun 这种单次运行粒度级别。它也没有 metrics.count.enable-reason、metrics.running-pipelinerun.level 或 metrics.taskrun.throttle.enable-namespace 这些 key。

上游已知问题：taskrun_delete_duration_seconds 使用 metrics.pipelinerun.duration-type（而不是 metrics.taskrun.duration-type）来确定聚合类型。这看起来是 Results 源码中的一次复制粘贴 bug。

API Server 指标

API server 通过 go-grpc-prometheus 库在 9090 端口暴露标准的 gRPC Prometheus 指标。这些指标包括：

grpc_server_handled_total - server 上已完成的 RPC 总数
grpc_server_started_total - server 上已开始的 RPC 总数
grpc_server_msg_received_total / grpc_server_msg_sent_total - 消息计数
grpc_server_handling_seconds（如果启用了 PROMETHEUS_HISTOGRAM）- RPC 处理耗时

Tekton Chains

Tekton Chains 是一个安全组件，用于为使用 Tekton Pipelines 构建的制品生成、签名并存储 provenance。它会观察已完成的 TaskRun 和 PipelineRun，然后创建 attestations 和 signatures。

Service	Description	Metrics Port
`tekton-chains-metrics`	Chains watcher/controller	9090 (`http-metrics`)

Chains controller 指标使用前缀 watcher_（与 Results Watcher 相同，但自定义 metric 名称不同，因此不会冲突）。

Chains 指标

所有 Chains 指标都是不带标签的 Counters。

Metric Name (Prometheus)	Type	Description
`watcher_taskrun_sign_created_total`	Counter	TaskRun 的签名消息总数
`watcher_taskrun_payload_stored_total`	Counter	TaskRun 的已存储 payload 总数
`watcher_taskrun_marked_signed_total`	Counter	被标记为已签名的 TaskRun 总数
`watcher_pipelinerun_sign_created_total`	Counter	PipelineRun 的签名消息总数
`watcher_pipelinerun_payload_stored_total`	Counter	PipelineRun 的已存储 payload 总数
`watcher_pipelinerun_marked_signed_total`	Counter	被标记为已签名的 PipelineRun 总数

注意：官方 Tekton Chains 文档还提到了 TaskRun 和 PipelineRun 的 *_signing_failures_total counters，但它们并不存在于当前的上游源码中。请根据你部署的版本进行验证。

Controller Framework 指标

所有 Tekton controller 会自动暴露以下基础设施指标。这些指标使用与组件自定义指标相同的前缀（例如 tekton_pipelines_controller_、controller_、watcher_）。

Metric Name (without prefix)	Type	Description
`client_latency`	Histogram	Kubernetes API client 请求延迟（秒）
`client_results`	Counter	Kubernetes API 请求计数（按状态码）
`workqueue_depth`	Gauge	当前 workqueue 深度
`workqueue_adds_total`	Counter	workqueue 添加总数
`workqueue_queue_latency_seconds`	Histogram	条目在 workqueue 中等待的时间
`workqueue_work_duration_seconds`	Histogram	处理 workqueue 条目所花费的时间
`workqueue_retries_total`	Counter	workqueue 重试总数
`workqueue_unfinished_work_seconds`	Histogram	未完成的 workqueue 条目持续时间
`workqueue_longest_running_processor_seconds`	Histogram	运行时间最长的 workqueue processor 持续时间
`reconcile_count`	Counter	reconciler 调用总数（按 `reconciler`、`success`、`namespace_name` 打标签）
`reconcile_latency`	Histogram	reconciler 调用延迟（按 `reconciler`、`success`、`namespace_name` 打标签）

设置 ServiceMonitor

要为 Tekton 组件启用 Prometheus 抓取，请部署 ServiceMonitor 资源。

前提条件请参见前提条件。

请根据你的监控栈使用以下指导：

如果你使用 Prometheus（Prometheus Operator），诸如 metadata.labels.prometheus: kube-prometheus 之类的标签必须与 Prometheus CR 的 spec.serviceMonitorSelector 匹配；否则不会抓取该 ServiceMonitor。
如果你使用 VictoriaMetrics，通常不需要像 prometheus: kube-prometheus 这样的标签；请根据你的监控方案创建 ServiceMonitor/VMServiceScrape。

使用 Prometheus 时，可使用以下命令查找并验证 selector：

# 1) 定位 Prometheus CR（resource type: monitoring.coreos.com/v1, Kind=Prometheus）
$ kubectl get prometheus -A

# 2) 检查目标 Prometheus 实例上的 ServiceMonitor selector
$ kubectl get prometheus -n <prometheus-namespace> <prometheus-name> -o yaml | yq '.spec.serviceMonitorSelector'

如果集群中不存在 Prometheus CR，监控通常由平台托管（例如 VictoriaMetrics）或以其他方式实现。在这种情况下，通常不需要像 prometheus: kube-prometheus 这样的标签；请遵循你平台的抓取规则。

更多信息请参见集成外部指标。

Pipeline ServiceMonitor

Pipeline ServiceMonitor YAML

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: tekton-pipelines-metrics
  namespace: tekton-pipelines
  labels:
    app.kubernetes.io/name: tekton-pipelines
    # prometheus: kube-prometheus
spec:
  selector:
    matchLabels:
      app.kubernetes.io/part-of: tekton-pipelines
  endpoints:
  - port: http-metrics
    path: /metrics
    interval: 30s
  namespaceSelector:
    matchNames:
    - tekton-pipelines

该 ServiceMonitor 会匹配带有 app.kubernetes.io/part-of: tekton-pipelines 标签的 Pipeline services（包括 remote-resolvers），并抓取 tekton-pipelines namespace 中的指标。

Triggers ServiceMonitor

Triggers ServiceMonitor YAML

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: tekton-triggers-metrics
  namespace: tekton-pipelines
  labels:
    app.kubernetes.io/name: tekton-triggers
    # prometheus: kube-prometheus
spec:
  selector:
    matchLabels:
      app.kubernetes.io/part-of: tekton-triggers
      app.kubernetes.io/component: controller
  endpoints:
  - port: http-metrics
    path: /metrics
    interval: 30s
  namespaceSelector:
    matchNames:
    - tekton-pipelines

该 ServiceMonitor 仅收集 Triggers controller 指标（controller_*）。它不包含 EventListener sink 指标。

EventListener Sink ServiceMonitor

EventListener Sink ServiceMonitor YAML

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: tekton-eventlistener-sink-metrics
  namespace: tekton-pipelines
  labels:
    app.kubernetes.io/name: tekton-eventlistener-sink
    # prometheus: kube-prometheus
spec:
  selector:
    matchExpressions:
    - key: eventlistener
      operator: Exists
    - key: app.kubernetes.io/managed-by
      operator: In
      values:
      - EventListener
  endpoints:
  - port: http-metrics
    path: /metrics
    interval: 30s
  namespaceSelector:
    any: true

EventListener Services 通常运行在应用程序 namespace 中，因此此示例使用 namespaceSelector.any: true 进行跨 namespace 抓取。如果你需要更严格的范围，请改用 matchNames 并显式列出允许的 namespace。

Results ServiceMonitor

Results services 同时具有 app.kubernetes.io/part-of: tekton-results 和 app.kubernetes.io/name 标签。为了精确定位 API + Watcher（并排除 Postgres），此示例匹配 app.kubernetes.io/name：

Results ServiceMonitor YAML

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: tekton-results-metrics
  namespace: tekton-pipelines
  labels:
    app.kubernetes.io/name: tekton-results
    # prometheus: kube-prometheus
spec:
  selector:
    matchExpressions:
    - key: app.kubernetes.io/name
      operator: In
      values:
      - tekton-results-api
      - tekton-results-watcher
  endpoints:
  - port: prometheus
    path: /metrics
    interval: 30s
  - port: metrics
    path: /metrics
    interval: 30s
  namespaceSelector:
    matchNames:
    - tekton-pipelines

Results API server 使用端口名 prometheus（9090），Watcher 使用端口名 metrics（9090）。每个 service 只暴露其中一个端口名，因此只有匹配的 endpoint 会被抓取。

Chains ServiceMonitor

Chains ServiceMonitor YAML

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: tekton-chains-metrics
  namespace: tekton-pipelines
  labels:
    app.kubernetes.io/name: tekton-chains
    # prometheus: kube-prometheus
spec:
  selector:
    matchLabels:
      app.kubernetes.io/part-of: tekton-chains
  endpoints:
  - port: http-metrics
    path: /metrics
    interval: 30s
  namespaceSelector:
    matchNames:
    - tekton-pipelines

验证

部署 ServiceMonitor 资源后，验证 Prometheus 是否正在抓取这些 target。

直接检查指标 endpoint

# Pipeline controller
$ kubectl port-forward -n tekton-pipelines svc/tekton-pipelines-controller 9090:9090
$ curl -s http://localhost:9090/metrics | grep tekton_pipelines_controller_

# HELP tekton_pipelines_controller_client_latency How long Kubernetes API requests take
# TYPE tekton_pipelines_controller_client_latency histogram
tekton_pipelines_controller_client_latency_bucket{name="",le="1e-05"} 0
tekton_pipelines_controller_client_latency_bucket{name="",le="0.0001"} 0
tekton_pipelines_controller_client_latency_bucket{name="",le="0.001"} 0

# Triggers controller
$ kubectl port-forward -n tekton-pipelines svc/tekton-triggers-controller 9000:9000
$ curl -s http://localhost:9000/metrics | grep controller_

# HELP controller_client_latency How long Kubernetes API requests take
# TYPE controller_client_latency histogram
controller_client_latency_bucket{name="",le="1e-05"} 0
controller_client_latency_bucket{name="",le="0.0001"} 1
controller_client_latency_bucket{name="",le="0.001"} 2

# EventListener sink metrics (replace namespace/service)
$ kubectl port-forward -n <eventlistener-namespace> svc/<eventlistener-service> 9000:9000
$ curl -s http://localhost:9000/metrics | grep eventlistener_

# HELP eventlistener_client_latency How long Kubernetes API requests take
# TYPE eventlistener_client_latency histogram
eventlistener_client_latency_bucket{name="",le="1e-05"} 0
eventlistener_client_latency_bucket{name="",le="0.0001"} 0
eventlistener_client_latency_bucket{name="",le="0.001"} 0

# HELP eventlistener_triggered_resources Count of the number of triggered eventlistener resources
# TYPE eventlistener_triggered_resources counter
eventlistener_triggered_resources{kind="PipelineRun"} 10

# Results watcher
$ kubectl port-forward -n tekton-pipelines svc/tekton-results-watcher 9091:9090
$ curl -s http://localhost:9091/metrics | grep watcher_

# HELP watcher_client_latency How long Kubernetes API requests take
# TYPE watcher_client_latency histogram
watcher_client_latency_bucket{name="",le="1e-05"} 0
watcher_client_latency_bucket{name="",le="0.0001"} 0
watcher_client_latency_bucket{name="",le="0.001"} 0

# Results API
$ kubectl port-forward -n tekton-pipelines svc/tekton-results-api-service 9092:9090
$ curl -s http://localhost:9092/metrics | grep grpc_server_

# HELP grpc_server_handled_total Total number of RPCs completed on the server, regardless of success or failure.
# TYPE grpc_server_handled_total counter
grpc_server_handled_total{grpc_code="Aborted",grpc_method="Check",grpc_service="grpc.health.v1.Health",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Aborted",grpc_method="CreateRecord",grpc_service="tekton.results.v1alpha2.Results",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Aborted",grpc_method="CreateResult",grpc_service="tekton.results.v1alpha2.Results",grpc_type="unary"} 0

# HELP grpc_server_started_total Total number of RPCs started on the server.
# TYPE grpc_server_started_total counter
grpc_server_started_total{grpc_method="Check",grpc_service="grpc.health.v1.Health",grpc_type="unary"} 337606
grpc_server_started_total{grpc_method="CreateRecord",grpc_service="tekton.results.v1alpha2.Results",grpc_type="unary"} 10301
grpc_server_started_total{grpc_method="CreateResult",grpc_service="tekton.results.v1alpha2.Results",grpc_type="unary"} 832

# Chains controller
$ kubectl port-forward -n tekton-pipelines svc/tekton-chains-metrics 9093:9090
$ curl -s http://localhost:9093/metrics | grep watcher_

# HELP watcher_client_latency How long Kubernetes API requests take
# TYPE watcher_client_latency histogram
watcher_client_latency_bucket{name="",le="1e-05"} 0
watcher_client_latency_bucket{name="",le="0.0001"} 0
watcher_client_latency_bucket{name="",le="0.001"} 0

EventListener sink 指标，例如 eventlistener_event_received_count 和 eventlistener_http_duration_seconds，是按请求驱动的。在验证这些指标之前，请先向 EventListener 发送至少一个请求。

检查 Prometheus Targets

# 验证 ServiceMonitor 资源是否存在
$ kubectl get servicemonitor -n tekton-pipelines

NAME                                AGE
tekton-chains-metrics               10m
tekton-eventlistener-sink-metrics   10m
tekton-pipelines-metrics            10m
tekton-results-metrics              10m
tekton-triggers-metrics             10m

# 检查 Prometheus targets（通过 Prometheus UI 或 API）
# 查找 job labels 与 ServiceMonitor 名称匹配的 targets

PromQL 查询示例

# PipelineRun 累计成功率（避免在空完成窗口中产生误解）
100 * sum(tekton_pipelines_controller_pipelinerun_total{status="success"}) / clamp_min(sum(tekton_pipelines_controller_pipelinerun_total), 1)

# 最近 5 分钟内完成的 PipelineRun（吞吐量）
round(sum(increase(tekton_pipelines_controller_pipelinerun_total[5m])))

# PipelineRun duration P95（histogram 模式）
histogram_quantile(0.95,
  rate(tekton_pipelines_controller_pipelinerun_duration_seconds_bucket[5m])
)

# TaskRun duration P95（histogram 模式，包含独立 TaskRun + Pipeline 内 TaskRun）
histogram_quantile(0.95,
  (
    sum by (le) (rate(tekton_pipelines_controller_taskrun_duration_seconds_bucket[5m]))
    +
    sum by (le) (rate(tekton_pipelines_controller_pipelinerun_taskrun_duration_seconds_bucket[5m]))
  )
)

# PipelineRun duration（lastvalue 模式）
avg_over_time(tekton_pipelines_controller_pipelinerun_duration_seconds[5m])

# 当前正在运行的 PipelineRun（单一 series，避免重复 legend）
max(tekton_pipelines_controller_running_pipelineruns)

# 被 resource quota 限流的 TaskRun
max(tekton_pipelines_controller_running_taskruns_throttled_by_quota)

# Trigger resource counts
controller_eventlistener_count
controller_triggertemplate_count

# Chains signing 活动
watcher_taskrun_sign_created_total
watcher_pipelinerun_sign_created_total

MonitorDashboard 示例

以下 MonitorDashboard resources 提供了可直接使用的 Tekton 组件监控面板。请将它们部署到 cpaas-system namespace 下的 tekton 文件夹中。

重要：每个 panel 都必须包含 id（唯一整数）、datasource: prometheus 和 transformations: []。每个 target 都必须包含 datasource: prometheus 和 refId。本文档中的 duration P50/P95 panels 使用 *_bucket 查询，并要求 metrics.*.duration-type=histogram；如果你使用 lastvalue，请将这些查询替换为 LastValue 风格表达式，例如 avg_over_time(...)。

Tekton Pipeline 监控面板

Tekton Pipeline 监控面板 YAML

kind: MonitorDashboard
apiVersion: ait.alauda.io/v1alpha2
metadata:
  labels:
    cpaas.io/dashboard.folder: tekton
    cpaas.io/dashboard.is.home.dashboard: "false"
    cpaas.io/dashboard.tag.tekton: "true"
  name: tekton-pipeline
  namespace: cpaas-system
spec:
  body:
    titleZh: Tekton Pipeline Overview
    tags:
      - tekton
    time:
      from: now-1h
      to: now
    templating:
      list: []
    panels:
      - id: 1
        title: PipelineRun Total (by status)
        type: timeseries
        datasource: prometheus
        gridPos: { h: 8, w: 8, x: 0, y: 0 }
        targets:
          - datasource: prometheus
            expr: sum by (status) (tekton_pipelines_controller_pipelinerun_total)
            refId: A
        fieldConfig:
          defaults:
            color: { mode: palette-classic }
            custom: { drawStyle: line, fillOpacity: 0, lineWidth: 1, spanNulls: false }
            thresholds: { mode: absolute, steps: [{ color: green, value: null }] }
          overrides: []
        options:
          legend: { calcs: [latest], displayMode: list, placement: bottom, showLegend: true }
          tooltip: { mode: multi, sort: desc }
        transformations: []
      - id: 2
        title: TaskRun Total (by status)
        type: timeseries
        datasource: prometheus
        gridPos: { h: 8, w: 8, x: 8, y: 0 }
        targets:
          - datasource: prometheus
            expr: sum by (status) (tekton_pipelines_controller_taskrun_total)
            refId: A
        fieldConfig:
          defaults:
            color: { mode: palette-classic }
            custom: { drawStyle: line, fillOpacity: 0, lineWidth: 1, spanNulls: false }
            thresholds: { mode: absolute, steps: [{ color: green, value: null }] }
          overrides: []
        options:
          legend: { calcs: [latest], displayMode: list, placement: bottom, showLegend: true }
          tooltip: { mode: multi, sort: desc }
        transformations: []
      - id: 3
        title: PipelineRun Success Rate (cumulative)
        type: timeseries
        datasource: prometheus
        gridPos: { h: 8, w: 4, x: 16, y: 0 }
        targets:
          - datasource: prometheus
            expr: "100 * sum(tekton_pipelines_controller_pipelinerun_total{status=\"success\"}) / clamp_min(sum(tekton_pipelines_controller_pipelinerun_total), 1)"
            refId: A
        fieldConfig:
          defaults:
            unit: percent
            color: { mode: thresholds }
            custom: { drawStyle: line, fillOpacity: 0, lineWidth: 1, spanNulls: false }
            thresholds:
              mode: absolute
              steps:
                - { color: red, value: null }
                - { color: orange, value: 80 }
                - { color: green, value: 95 }
          overrides: []
        options:
          legend: { calcs: [latest], displayMode: list, placement: bottom, showLegend: true }
          tooltip: { mode: multi, sort: desc }
        transformations: []
      - id: 12
        title: Completed PipelineRuns (last 5m)
        type: timeseries
        datasource: prometheus
        gridPos: { h: 8, w: 4, x: 20, y: 0 }
        targets:
          - datasource: prometheus
            expr: "round(sum(increase(tekton_pipelines_controller_pipelinerun_total[5m])))"
            legendFormat: completed
            refId: A
        fieldConfig:
          defaults:
            unit: short
            decimals: 0
            color: { mode: palette-classic }
            custom: { drawStyle: line, fillOpacity: 0, lineWidth: 1, spanNulls: false }
            thresholds: { mode: absolute, steps: [{ color: green, value: null }] }
          overrides: []
        options:
          legend: { calcs: [latest], displayMode: list, placement: bottom, showLegend: true }
          tooltip: { mode: multi, sort: desc }
        transformations: []
      - id: 4
        title: Running PipelineRuns
        type: timeseries
        datasource: prometheus
        gridPos: { h: 8, w: 8, x: 0, y: 8 }
        targets:
          - datasource: prometheus
            expr: max(tekton_pipelines_controller_running_pipelineruns)
            legendFormat: running
            refId: A
        fieldConfig:
          defaults:
            color: { mode: palette-classic }
            custom: { drawStyle: line, fillOpacity: 0, lineWidth: 1, spanNulls: false }
            thresholds: { mode: absolute, steps: [{ color: green, value: null }] }
          overrides: []
        options:
          legend: { calcs: [latest], displayMode: list, placement: bottom, showLegend: true }
          tooltip: { mode: multi, sort: desc }
        transformations: []
      - id: 5
        title: Running TaskRuns
        type: timeseries
        datasource: prometheus
        gridPos: { h: 8, w: 8, x: 8, y: 8 }
        targets:
          - datasource: prometheus
            expr: max(tekton_pipelines_controller_running_taskruns)
            legendFormat: running
            refId: A
        fieldConfig:
          defaults:
            color: { mode: palette-classic }
            custom: { drawStyle: line, fillOpacity: 0, lineWidth: 1, spanNulls: false }
            thresholds: { mode: absolute, steps: [{ color: green, value: null }] }
          overrides: []
        options:
          legend: { calcs: [latest], displayMode: list, placement: bottom, showLegend: true }
          tooltip: { mode: multi, sort: desc }
        transformations: []
      - id: 6
        title: TaskRuns Throttled
        type: timeseries
        datasource: prometheus
        gridPos: { h: 8, w: 8, x: 16, y: 8 }
        targets:
          - datasource: prometheus
            expr: max(tekton_pipelines_controller_running_taskruns_throttled_by_quota)
            legendFormat: by quota
            refId: A
          - datasource: prometheus
            expr: max(tekton_pipelines_controller_running_taskruns_throttled_by_node)
            legendFormat: by node
            refId: B
        fieldConfig:
          defaults:
            color: { mode: palette-classic }
            custom: { drawStyle: line, fillOpacity: 0, lineWidth: 1, spanNulls: false }
            thresholds: { mode: absolute, steps: [{ color: orange, value: null }] }
          overrides: []
        options:
          legend: { calcs: [latest], displayMode: list, placement: bottom, showLegend: true }
          tooltip: { mode: multi, sort: desc }
        transformations: []
      - id: 7
        title: PipelineRun Duration P50 / P95
        type: timeseries
        datasource: prometheus
        gridPos: { h: 8, w: 8, x: 0, y: 16 }
        targets:
          - datasource: prometheus
            expr: (histogram_quantile(0.5, sum by (le) (rate(tekton_pipelines_controller_pipelinerun_duration_seconds_bucket[5m])))) and on() (sum(rate(tekton_pipelines_controller_pipelinerun_duration_seconds_bucket{le="+Inf"}[5m])) > 0)
            legendFormat: P50
            refId: A
          - datasource: prometheus
            expr: (histogram_quantile(0.95, sum by (le) (rate(tekton_pipelines_controller_pipelinerun_duration_seconds_bucket[5m])))) and on() (sum(rate(tekton_pipelines_controller_pipelinerun_duration_seconds_bucket{le="+Inf"}[5m])) > 0)
            legendFormat: P95
            refId: B
        fieldConfig:
          defaults:
            unit: s
            color: { mode: palette-classic }
            custom: { drawStyle: line, fillOpacity: 0, lineWidth: 1, spanNulls: false }
            thresholds: { mode: absolute, steps: [{ color: green, value: null }] }
          overrides: []
        options:
          legend: { calcs: [latest], displayMode: list, placement: bottom, showLegend: true }
          tooltip: { mode: multi, sort: desc }
        transformations: []
      - id: 8
        title: TaskRun Duration P50 / P95 (Standalone)
        type: timeseries
        datasource: prometheus
        gridPos: { h: 8, w: 8, x: 8, y: 16 }
        targets:
          - datasource: prometheus
            expr: (histogram_quantile(0.5, sum by (le) (rate(tekton_pipelines_controller_taskrun_duration_seconds_bucket[5m])))) and on() (sum(rate(tekton_pipelines_controller_taskrun_duration_seconds_bucket{le="+Inf"}[5m])) > 0)
            legendFormat: P50
            refId: A
          - datasource: prometheus
            expr: (histogram_quantile(0.95, sum by (le) (rate(tekton_pipelines_controller_taskrun_duration_seconds_bucket[5m])))) and on() (sum(rate(tekton_pipelines_controller_taskrun_duration_seconds_bucket{le="+Inf"}[5m])) > 0)
            legendFormat: P95
            refId: B
        fieldConfig:
          defaults:
            unit: s
            color: { mode: palette-classic }
            custom: { drawStyle: line, fillOpacity: 0, lineWidth: 1, spanNulls: false }
            thresholds: { mode: absolute, steps: [{ color: green, value: null }] }
          overrides: []
        options:
          legend: { calcs: [latest], displayMode: list, placement: bottom, showLegend: true }
          tooltip: { mode: multi, sort: desc }
        transformations: []
      - id: 13
        title: TaskRun Duration P50 / P95 (In-Pipeline)
        type: timeseries
        datasource: prometheus
        gridPos: { h: 8, w: 8, x: 16, y: 16 }
        targets:
          - datasource: prometheus
            expr: (histogram_quantile(0.5, sum by (le) (rate(tekton_pipelines_controller_pipelinerun_taskrun_duration_seconds_bucket[5m])))) and on() (sum(rate(tekton_pipelines_controller_pipelinerun_taskrun_duration_seconds_bucket{le="+Inf"}[5m])) > 0)
            legendFormat: P50
            refId: A
          - datasource: prometheus
            expr: (histogram_quantile(0.95, sum by (le) (rate(tekton_pipelines_controller_pipelinerun_taskrun_duration_seconds_bucket[5m])))) and on() (sum(rate(tekton_pipelines_controller_pipelinerun_taskrun_duration_seconds_bucket{le="+Inf"}[5m])) > 0)
            legendFormat: P95
            refId: B
        fieldConfig:
          defaults:
            unit: s
            color: { mode: palette-classic }
            custom: { drawStyle: line, fillOpacity: 0, lineWidth: 1, spanNulls: false }
            thresholds: { mode: absolute, steps: [{ color: green, value: null }] }
          overrides: []
        options:
          legend: { calcs: [latest], displayMode: list, placement: bottom, showLegend: true }
          tooltip: { mode: multi, sort: desc }
        transformations: []
      - id: 9
        title: Workqueue Depth
        type: timeseries
        datasource: prometheus
        gridPos: { h: 8, w: 8, x: 0, y: 24 }
        targets:
          - datasource: prometheus
            expr: max(tekton_pipelines_controller_workqueue_depth)
            legendFormat: depth
            refId: A
        fieldConfig:
          defaults:
            color: { mode: palette-classic }
            custom: { drawStyle: line, fillOpacity: 0, lineWidth: 1, spanNulls: false }
            thresholds: { mode: absolute, steps: [{ color: green, value: null }] }
          overrides: []
        options:
          legend: { calcs: [latest], displayMode: list, placement: bottom, showLegend: true }
          tooltip: { mode: multi, sort: desc }
        transformations: []
      - id: 10
        title: Reconcile Count (by success)
        type: timeseries
        datasource: prometheus
        gridPos: { h: 8, w: 8, x: 8, y: 24 }
        targets:
          - datasource: prometheus
            expr: sum(increase(tekton_pipelines_controller_reconcile_count{success="true"}[5m]))
            legendFormat: success=true
            refId: A
          - datasource: prometheus
            expr: sum(increase(tekton_pipelines_controller_reconcile_count{success="false"}[5m]))
            legendFormat: success=false
            refId: B
        fieldConfig:
          defaults:
            color: { mode: palette-classic }
            custom: { drawStyle: line, fillOpacity: 0, lineWidth: 1, spanNulls: false }
            thresholds: { mode: absolute, steps: [{ color: green, value: null }] }
          overrides: []
        options:
          legend: { calcs: [latest], displayMode: list, placement: bottom, showLegend: true }
          tooltip: { mode: multi, sort: desc }
        transformations: []
      - id: 11
        title: Resolution Waiting
        type: timeseries
        datasource: prometheus
        gridPos: { h: 8, w: 8, x: 16, y: 24 }
        targets:
          - datasource: prometheus
            expr: max(tekton_pipelines_controller_running_pipelineruns_waiting_on_pipeline_resolution)
            legendFormat: PR waiting pipeline
            refId: A
          - datasource: prometheus
            expr: max(tekton_pipelines_controller_running_pipelineruns_waiting_on_task_resolution)
            legendFormat: PR waiting task
            refId: B
          - datasource: prometheus
            expr: max(tekton_pipelines_controller_running_taskruns_waiting_on_task_resolution_count)
            legendFormat: TR waiting task
            refId: C
        fieldConfig:
          defaults:
            color: { mode: palette-classic }
            custom: { drawStyle: line, fillOpacity: 0, lineWidth: 1, spanNulls: false }
            thresholds: { mode: absolute, steps: [{ color: orange, value: null }] }
          overrides: []
        options:
          legend: { calcs: [latest], displayMode: list, placement: bottom, showLegend: true }
          tooltip: { mode: multi, sort: desc }
        transformations: []

Tekton Pipeline 监控面板说明（常见问题）

PipelineRun Total (by status) 是 controller 记录的完成事件计数器，而不是 PipelineRun 对象总数。在当前实现中，用户触发的取消（spec.status=Cancelled）可能不会进入该计数路径，因此 cancelled series 可能不会出现。要验证取消量，请检查 PipelineRun 对象和 events。
Running PipelineRuns 是实时快照（当前正在运行的数量）。它可以独立于 PipelineRun Total 变化。
Completed PipelineRuns (last 5m) 表示吞吐量（最近 5 分钟内新完成的 run 数量）。在低流量或空闲时段看到 0 是正常的。
PipelineRun Success Rate (cumulative) 是从 controller 启动以来的累计值，不是 5 分钟窗口成功率。短期失败不会立刻导致大幅变化。
Reconcile Count (by success) 衡量的是 controller reconcile 循环，而不是 PipelineRun 数量。
只有在所选时间范围内实际有样本的 label 值才会显示对应的状态 series。如果某个状态在窗口内没有样本，其曲线/图例不会出现。
TaskRun Duration P50 / P95 (Standalone) 与 TaskRun Duration P50 / P95 (In-Pipeline) 被拆分是为了避免混合查询不稳定。在只暴露一种 histogram family 的环境中，另一个图表可能为空，这是正常现象。

Tekton Triggers 监控面板

Tekton Triggers 监控面板 YAML

kind: MonitorDashboard
apiVersion: ait.alauda.io/v1alpha2
metadata:
  labels:
    cpaas.io/dashboard.folder: tekton
    cpaas.io/dashboard.is.home.dashboard: "false"
    cpaas.io/dashboard.tag.tekton: "true"
  name: tekton-triggers
  namespace: cpaas-system
spec:
  body:
    titleZh: Tekton Triggers Overview
    tags:
      - tekton
    time:
      from: now-1h
      to: now
    templating:
      list: []
    panels:
      - id: 1
        title: EventListener Count
        type: timeseries
        datasource: prometheus
        gridPos: { h: 6, w: 5, x: 0, y: 0 }
        targets:
          - datasource: prometheus
            expr: controller_eventlistener_count
            legendFormat: EventListener
            refId: A
        fieldConfig:
          defaults:
            color: { mode: palette-classic }
            custom: { drawStyle: line, fillOpacity: 0, lineWidth: 1, spanNulls: false }
            thresholds: { mode: absolute, steps: [{ color: green, value: null }] }
          overrides: []
        options:
          legend: { calcs: [latest], displayMode: list, placement: bottom, showLegend: true }
          tooltip: { mode: multi, sort: desc }
        transformations: []
      - id: 2
        title: TriggerTemplate Count
        type: timeseries
        datasource: prometheus
        gridPos: { h: 6, w: 5, x: 5, y: 0 }
        targets:
          - datasource: prometheus
            expr: controller_triggertemplate_count
            legendFormat: TriggerTemplate
            refId: A
        fieldConfig:
          defaults:
            color: { mode: palette-classic }
            custom: { drawStyle: line, fillOpacity: 0, lineWidth: 1, spanNulls: false }
            thresholds: { mode: absolute, steps: [{ color: green, value: null }] }
          overrides: []
        options:
          legend: { calcs: [latest], displayMode: list, placement: bottom, showLegend: true }
          tooltip: { mode: multi, sort: desc }
        transformations: []
      - id: 3
        title: TriggerBinding Count
        type: timeseries
        datasource: prometheus
        gridPos: { h: 6, w: 5, x: 10, y: 0 }
        targets:
          - datasource: prometheus
            expr: controller_triggerbinding_count
            legendFormat: TriggerBinding
            refId: A
        fieldConfig:
          defaults:
            color: { mode: palette-classic }
            custom: { drawStyle: line, fillOpacity: 0, lineWidth: 1, spanNulls: false }
            thresholds: { mode: absolute, steps: [{ color: green, value: null }] }
          overrides: []
        options:
          legend: { calcs: [latest], displayMode: list, placement: bottom, showLegend: true }
          tooltip: { mode: multi, sort: desc }
        transformations: []
      - id: 4
        title: ClusterTriggerBinding
        type: timeseries
        datasource: prometheus
        gridPos: { h: 6, w: 5, x: 15, y: 0 }
        targets:
          - datasource: prometheus
            expr: controller_clustertriggerbinding_count
            legendFormat: ClusterTriggerBinding
            refId: A
        fieldConfig:
          defaults:
            color: { mode: palette-classic }
            custom: { drawStyle: line, fillOpacity: 0, lineWidth: 1, spanNulls: false }
            thresholds: { mode: absolute, steps: [{ color: green, value: null }] }
          overrides: []
        options:
          legend: { calcs: [latest], displayMode: list, placement: bottom, showLegend: true }
          tooltip: { mode: multi, sort: desc }
        transformations: []
      - id: 5
        title: ClusterInterceptor
        type: timeseries
        datasource: prometheus
        gridPos: { h: 6, w: 4, x: 20, y: 0 }
        targets:
          - datasource: prometheus
            expr: controller_clusterinterceptor_count
            legendFormat: ClusterInterceptor
            refId: A
        fieldConfig:
          defaults:
            color: { mode: palette-classic }
            custom: { drawStyle: line, fillOpacity: 0, lineWidth: 1, spanNulls: false }
            thresholds: { mode: absolute, steps: [{ color: green, value: null }] }
          overrides: []
        options:
          legend: { calcs: [latest], displayMode: list, placement: bottom, showLegend: true }
          tooltip: { mode: multi, sort: desc }
        transformations: []
      - id: 6
        title: All Trigger Resource Counts (trend)
        type: timeseries
        datasource: prometheus
        gridPos: { h: 8, w: 24, x: 0, y: 6 }
        targets:
          - datasource: prometheus
            expr: controller_eventlistener_count
            legendFormat: EventListener
            refId: A
          - datasource: prometheus
            expr: controller_triggertemplate_count
            legendFormat: TriggerTemplate
            refId: B
          - datasource: prometheus
            expr: controller_triggerbinding_count
            legendFormat: TriggerBinding
            refId: C
          - datasource: prometheus
            expr: controller_clustertriggerbinding_count
            legendFormat: ClusterTriggerBinding
            refId: D
          - datasource: prometheus
            expr: controller_clusterinterceptor_count
            legendFormat: ClusterInterceptor
            refId: E
        fieldConfig:
          defaults:
            color: { mode: palette-classic }
            custom: { drawStyle: line, fillOpacity: 0, lineWidth: 1, spanNulls: false }
            thresholds: { mode: absolute, steps: [{ color: green, value: null }] }
          overrides: []
        options:
          legend: { calcs: [latest], displayMode: list, placement: bottom, showLegend: true }
          tooltip: { mode: multi, sort: desc }
        transformations: []

Tekton Triggers 监控面板说明（常见问题）

EventListener Count、TriggerTemplate Count、TriggerBinding Count、ClusterTriggerBinding 和 ClusterInterceptor 是对象计数快照，不是请求量或事件处理吞吐量。
All Trigger Resource Counts (trend) 展示的是相同资源计数的组合趋势。与单资源趋势图在一个抓取周期内出现短暂偏差是正常的。
当不存在任何 Triggers 资源时显示 0 是正常现象，并不表示抓取失败。

Tekton Results 监控面板

Tekton Results 监控面板 YAML

kind: MonitorDashboard
apiVersion: ait.alauda.io/v1alpha2
metadata:
  labels:
    cpaas.io/dashboard.folder: tekton
    cpaas.io/dashboard.is.home.dashboard: "false"
    cpaas.io/dashboard.tag.tekton: "true"
  name: tekton-results
  namespace: cpaas-system
spec:
  body:
    titleZh: Tekton Results Overview
    tags:
      - tekton
    time:
      from: now-1h
      to: now
    templating:
      list: []
    panels:
      - id: 1
        title: PipelineRun Reconcile Count (last 5m)
        type: timeseries
        datasource: prometheus
        gridPos: { h: 8, w: 12, x: 0, y: 0 }
        targets:
          - datasource: prometheus
            expr: round(sum(increase(watcher_reconcile_count{reconciler="github.com.tektoncd.results.pkg.watcher.reconciler.pipelinerun.Reconciler",success="true"}[5m])))
            legendFormat: success=true
            refId: A
          - datasource: prometheus
            expr: round(sum(increase(watcher_reconcile_count{reconciler="github.com.tektoncd.results.pkg.watcher.reconciler.pipelinerun.Reconciler",success="false"}[5m])))
            legendFormat: success=false
            refId: B
        fieldConfig:
          defaults:
            color: { mode: palette-classic }
            custom: { drawStyle: line, fillOpacity: 0, lineWidth: 1, spanNulls: false }
            thresholds: { mode: absolute, steps: [{ color: green, value: null }] }
          overrides: []
        options:
          legend: { calcs: [latest], displayMode: list, placement: bottom, showLegend: true }
          tooltip: { mode: multi, sort: desc }
        transformations: []
      - id: 2
        title: TaskRun Reconcile Count (last 5m)
        type: timeseries
        datasource: prometheus
        gridPos: { h: 8, w: 12, x: 12, y: 0 }
        targets:
          - datasource: prometheus
            expr: round(sum(increase(watcher_reconcile_count{reconciler="github.com.tektoncd.results.pkg.watcher.reconciler.taskrun.Reconciler",success="true"}[5m])))
            legendFormat: success=true
            refId: A
          - datasource: prometheus
            expr: round(sum(increase(watcher_reconcile_count{reconciler="github.com.tektoncd.results.pkg.watcher.reconciler.taskrun.Reconciler",success="false"}[5m])))
            legendFormat: success=false
            refId: B
        fieldConfig:
          defaults:
            color: { mode: palette-classic }
            custom: { drawStyle: line, fillOpacity: 0, lineWidth: 1, spanNulls: false }
            thresholds: { mode: absolute, steps: [{ color: green, value: null }] }
          overrides: []
        options:
          legend: { calcs: [latest], displayMode: list, placement: bottom, showLegend: true }
          tooltip: { mode: multi, sort: desc }
        transformations: []
      - id: 3
        title: PipelineRun Reconcile Latency P95
        type: timeseries
        datasource: prometheus
        gridPos: { h: 8, w: 12, x: 0, y: 8 }
        targets:
          - datasource: prometheus
            expr: histogram_quantile(0.95, sum by (le) (rate(watcher_reconcile_latency_bucket{reconciler="github.com.tektoncd.results.pkg.watcher.reconciler.pipelinerun.Reconciler"}[5m])))
            legendFormat: P95
            refId: A
        fieldConfig:
          defaults:
            unit: ms
            color: { mode: palette-classic }
            custom: { drawStyle: line, fillOpacity: 0, lineWidth: 1, spanNulls: false }
            thresholds: { mode: absolute, steps: [{ color: green, value: null }] }
          overrides: []
        options:
          legend: { calcs: [latest], displayMode: list, placement: bottom, showLegend: true }
          tooltip: { mode: multi, sort: desc }
        transformations: []
      - id: 4
        title: TaskRun Reconcile Latency P95
        type: timeseries
        datasource: prometheus
        gridPos: { h: 8, w: 12, x: 12, y: 8 }
        targets:
          - datasource: prometheus
            expr: histogram_quantile(0.95, sum by (le) (rate(watcher_reconcile_latency_bucket{reconciler="github.com.tektoncd.results.pkg.watcher.reconciler.taskrun.Reconciler"}[5m])))
            legendFormat: P95
            refId: A
        fieldConfig:
          defaults:
            unit: ms
            color: { mode: palette-classic }
            custom: { drawStyle: line, fillOpacity: 0, lineWidth: 1, spanNulls: false }
            thresholds: { mode: absolute, steps: [{ color: green, value: null }] }
          overrides: []
        options:
          legend: { calcs: [latest], displayMode: list, placement: bottom, showLegend: true }
          tooltip: { mode: multi, sort: desc }
        transformations: []
      - id: 5
        title: Workqueue Depth (PipelineRun vs TaskRun)
        type: timeseries
        datasource: prometheus
        gridPos: { h: 8, w: 12, x: 0, y: 16 }
        targets:
          - datasource: prometheus
            expr: sum(watcher_work_queue_depth{reconciler="github.com.tektoncd.results.pkg.watcher.reconciler.pipelinerun.Reconciler"})
            legendFormat: pipelinerun
            refId: A
          - datasource: prometheus
            expr: sum(watcher_work_queue_depth{reconciler="github.com.tektoncd.results.pkg.watcher.reconciler.taskrun.Reconciler"})
            legendFormat: taskrun
            refId: B
        fieldConfig:
          defaults:
            color: { mode: palette-classic }
            custom: { drawStyle: line, fillOpacity: 0, lineWidth: 1, spanNulls: false }
            thresholds: { mode: absolute, steps: [{ color: green, value: null }] }
          overrides: []
        options:
          legend: { calcs: [latest], displayMode: list, placement: bottom, showLegend: true }
          tooltip: { mode: multi, sort: desc }
        transformations: []
      - id: 6
        title: Workqueue Adds (last 5m)
        type: timeseries
        datasource: prometheus
        gridPos: { h: 8, w: 12, x: 12, y: 16 }
        targets:
          - datasource: prometheus
            expr: round(sum(increase(watcher_workqueue_adds_total{name=~"github.com.tektoncd.results.pkg.watcher.reconciler.pipelinerun.Reconciler-(consumer|fast|slow)"}[5m])))
            legendFormat: pipelinerun adds
            refId: A
          - datasource: prometheus
            expr: round(sum(increase(watcher_workqueue_adds_total{name=~"github.com.tektoncd.results.pkg.watcher.reconciler.taskrun.Reconciler-(consumer|fast|slow)"}[5m])))
            legendFormat: taskrun adds
            refId: B
        fieldConfig:
          defaults:
            color: { mode: palette-classic }
            custom: { drawStyle: line, fillOpacity: 0, lineWidth: 1, spanNulls: false }
            thresholds: { mode: absolute, steps: [{ color: green, value: null }] }
          overrides: []
        options:
          legend: { calcs: [latest], displayMode: list, placement: bottom, showLegend: true }
          tooltip: { mode: multi, sort: desc }
        transformations: []
      - id: 7
        title: gRPC Request Rate (Results API)
        type: timeseries
        datasource: prometheus
        gridPos: { h: 8, w: 12, x: 0, y: 24 }
        targets:
          - datasource: prometheus
            expr: "sum(rate(grpc_server_handled_total{grpc_service=~\"tekton.results.*\"}[5m]))"
            legendFormat: requests
            refId: A
        fieldConfig:
          defaults:
            color: { mode: palette-classic }
            custom: { drawStyle: line, fillOpacity: 0, lineWidth: 1, spanNulls: false }
            thresholds: { mode: absolute, steps: [{ color: green, value: null }] }
          overrides: []
        options:
          legend: { calcs: [latest], displayMode: list, placement: bottom, showLegend: true }
          tooltip: { mode: multi, sort: desc }
        transformations: []
      - id: 8
        title: gRPC Error Percentage (Results API, excl. NotFound/Canceled)
        type: timeseries
        datasource: prometheus
        gridPos: { h: 8, w: 12, x: 12, y: 24 }
        targets:
          - datasource: prometheus
            expr: "100 * ((sum(rate(grpc_server_handled_total{grpc_service=~\"tekton.results.*\",grpc_code!~\"OK|NotFound|Canceled\"}[5m])) or vector(0)) / clamp_min((sum(rate(grpc_server_handled_total{grpc_service=~\"tekton.results.*\"}[5m])) or vector(0)), 0.001))"
            legendFormat: error %
            refId: A
        fieldConfig:
          defaults:
            unit: percent
            color: { mode: palette-classic }
            custom: { drawStyle: line, fillOpacity: 0, lineWidth: 1, spanNulls: false }
            thresholds: { mode: absolute, steps: [{ color: red, value: null }] }
          overrides: []
        options:
          legend: { calcs: [latest], displayMode: list, placement: bottom, showLegend: true }
          tooltip: { mode: multi, sort: desc }
        transformations: []

Tekton Results 监控面板说明（常见问题）

该 dashboard 版本基于 Results Watcher 的 reconcile/workqueue 指标以及 Results API gRPC 指标，因此在常见部署（logs_api=true、禁用自动删除）中仍会显示数据。
PipelineRun Reconcile Count (last 5m) 和 TaskRun Reconcile Count (last 5m) 会分别显示 success=true 与 success=false 在最近 5 分钟内的增量。
PipelineRun Reconcile Latency P95 和 TaskRun Reconcile Latency P95 基于 watcher reconcile latency histogram 计算。在低流量下，曲线可能较稀疏。
Workqueue Depth 显示当前队列深度，而 Workqueue Adds (last 5m) 显示最近 5 分钟的入队量。
gRPC Error Percentage (Results API, excl. NotFound/Canceled) 是异常错误占总请求数的百分比，排除了常见业务返回码（NotFound、Canceled）。

Tekton Chains 监控面板

Tekton Chains 监控面板 YAML

kind: MonitorDashboard
apiVersion: ait.alauda.io/v1alpha2
metadata:
  labels:
    cpaas.io/dashboard.folder: tekton
    cpaas.io/dashboard.is.home.dashboard: "false"
    cpaas.io/dashboard.tag.tekton: "true"
  name: tekton-chains
  namespace: cpaas-system
spec:
  body:
    titleZh: Tekton Chains Overview
    tags:
      - tekton
    time:
      from: now-1h
      to: now
    templating:
      list: []
    panels:
      - id: 1
        title: TaskRun Signatures Created (last 5m)
        type: timeseries
        datasource: prometheus
        gridPos: { h: 8, w: 12, x: 0, y: 0 }
        targets:
          - datasource: prometheus
            expr: round(increase(watcher_taskrun_sign_created_total[5m]))
            legendFormat: sign created
            refId: A
        fieldConfig:
          defaults:
            color: { mode: palette-classic }
            custom: { drawStyle: line, fillOpacity: 0, lineWidth: 1, spanNulls: false }
            thresholds: { mode: absolute, steps: [{ color: green, value: null }] }
          overrides: []
        options:
          legend: { calcs: [latest], displayMode: list, placement: bottom, showLegend: true }
          tooltip: { mode: multi, sort: desc }
        transformations: []
      - id: 2
        title: PipelineRun Signatures Created (last 5m)
        type: timeseries
        datasource: prometheus
        gridPos: { h: 8, w: 12, x: 12, y: 0 }
        targets:
          - datasource: prometheus
            expr: round(increase(watcher_pipelinerun_sign_created_total[5m]))
            legendFormat: sign created
            refId: A
        fieldConfig:
          defaults:
            color: { mode: palette-classic }
            custom: { drawStyle: line, fillOpacity: 0, lineWidth: 1, spanNulls: false }
            thresholds: { mode: absolute, steps: [{ color: green, value: null }] }
          overrides: []
        options:
          legend: { calcs: [latest], displayMode: list, placement: bottom, showLegend: true }
          tooltip: { mode: multi, sort: desc }
        transformations: []
      - id: 3
        title: Payloads Stored (last 5m, TaskRun vs PipelineRun)
        type: timeseries
        datasource: prometheus
        gridPos: { h: 8, w: 12, x: 0, y: 8 }
        targets:
          - datasource: prometheus
            expr: round(increase(watcher_taskrun_payload_stored_total[5m]))
            legendFormat: TaskRun
            refId: A
          - datasource: prometheus
            expr: round(increase(watcher_pipelinerun_payload_stored_total[5m]))
            legendFormat: PipelineRun
            refId: B
        fieldConfig:
          defaults:
            color: { mode: palette-classic }
            custom: { drawStyle: line, fillOpacity: 0, lineWidth: 1, spanNulls: false }
            thresholds: { mode: absolute, steps: [{ color: green, value: null }] }
          overrides: []
        options:
          legend: { calcs: [latest], displayMode: list, placement: bottom, showLegend: true }
          tooltip: { mode: multi, sort: desc }
        transformations: []
      - id: 4
        title: Marked Signed (last 5m, TaskRun vs PipelineRun)
        type: timeseries
        datasource: prometheus
        gridPos: { h: 8, w: 12, x: 12, y: 8 }
        targets:
          - datasource: prometheus
            expr: round(increase(watcher_taskrun_marked_signed_total[5m]))
            legendFormat: TaskRun
            refId: A
          - datasource: prometheus
            expr: round(increase(watcher_pipelinerun_marked_signed_total[5m]))
            legendFormat: PipelineRun
            refId: B
        fieldConfig:
          defaults:
            color: { mode: palette-classic }
            custom: { drawStyle: line, fillOpacity: 0, lineWidth: 1, spanNulls: false }
            thresholds: { mode: absolute, steps: [{ color: green, value: null }] }
          overrides: []
        options:
          legend: { calcs: [latest], displayMode: list, placement: bottom, showLegend: true }
          tooltip: { mode: multi, sort: desc }
        transformations: []

Tekton Chains 监控面板说明（常见问题）

TaskRun Signatures Created (last 5m)、PipelineRun Signatures Created (last 5m)、Payloads Stored (last 5m) 和 Marked Signed (last 5m) 使用 increase(...[5m])，表示最近五分钟内的增量。
当没有新的签名或存储活动时，这些曲线会降为 0；这并不表示组件故障。
Payloads Stored 和 Marked Signed 表示不同的处理阶段，因此它们的值不应始终一致。

#Tekton 组件的指标采集

#目录

#概述

#前提条件

#Tekton Pipelines

#PipelineRun 指标

#running_pipelineruns 标签级别

#状态标签值

#TaskRun 指标

#config-observability 配置

#Histogram Buckets

#推荐的生产环境配置

#Tekton Triggers

#Controller 指标（端口 9000）

#EventListener Sink 指标

#Tekton Results

#Watcher 指标

#删除指标

#共享指标

#Watcher config-observability

#API Server 指标

#Tekton Chains

#Chains 指标

#Controller Framework 指标

#设置 ServiceMonitor

#Pipeline ServiceMonitor

#Triggers ServiceMonitor

#EventListener Sink ServiceMonitor

#Results ServiceMonitor

#Chains ServiceMonitor

#验证

#直接检查指标 endpoint

#检查 Prometheus Targets

#PromQL 查询示例

#MonitorDashboard 示例

#Tekton Pipeline 监控面板

#Tekton Pipeline 监控面板说明（常见问题）

#Tekton Triggers 监控面板

#Tekton Triggers 监控面板说明（常见问题）

#Tekton Results 监控面板

#Tekton Results 监控面板说明（常见问题）

#Tekton Chains 监控面板

#Tekton Chains 监控面板说明（常见问题）