Metrics Collection for Tekton Components

Overview

Tekton components expose Prometheus-compatible metrics via HTTP endpoints. By deploying ServiceMonitor resources, Prometheus (or VictoriaMetrics) can automatically discover and scrape these metrics.

Namespace Note: This document uses tekton-pipelines as the default namespace for control-plane components (Pipelines, Triggers, Results, Chains). The primary exception is EventListener Services, which run in application namespaces where EventListeners are created.

If your deployment uses different namespaces, update both the commands and the namespaceSelector fields in ServiceMonitor resources below.

This document covers metrics for the following Tekton components:

  • Tekton Pipelines - PipelineRun / TaskRun execution metrics
  • Tekton Triggers - EventListener, TriggerBinding, and related resource metrics
  • Tekton Results - Run deletion and storage metrics
  • Tekton Chains - Signing and provenance metrics
  • Controller Framework - Infrastructure metrics shared by all controllers

It also covers:

  • How to configure metrics behavior via config-observability
  • How to deploy ServiceMonitor resources for scraping
  • How to verify that metrics collection is working

Prerequisites

  • Tekton control-plane components are installed and running (at minimum, the components you plan to scrape: Pipelines, Triggers, Results, and/or Chains).
  • kubectl is configured against the target cluster, and your account can create ServiceMonitor resources in the namespace where you create them (the samples on this page use tekton-pipelines).
  • A monitoring stack is deployed (Prometheus or compatible VictoriaMetrics) and can discover/scrape ServiceMonitor resources (or equivalent scrape-discovery objects used by your platform).
  • Your Prometheus/VictoriaMetrics instance is configured to discover the ServiceMonitor objects in that namespace (namespace and label selectors must match).
  • Network policies and firewalls allow scraper pods to reach Tekton metrics ports (9090 for most control-plane services, 9000 for Triggers controller and EventListener sink).
  • If you want EventListener sink metrics, EventListeners must exist in their target namespaces and expose the http-metrics port.
  • The curl CLI for the endpoint checks in Check Metrics Endpoints Directly, and the jq CLI for the restore step of the runs_not_stored_count reproduction. On an air-gapped workstation, install them from an operating system package repository that the workstation can reach, or copy the binaries from a connected machine.
  • To deploy the dashboards in MonitorDashboard Examples, your account can create MonitorDashboard resources (ait.alauda.io/v1alpha2) in the cpaas-system namespace.
  • Changing config-observability needs the right to update the cluster-scoped TektonConfig. Reading the ConfigMaps and restarting Pods in tekton-pipelines, and the kubectl port-forward checks there, need the matching rights in that namespace (pods/portforward for the port-forward). These steps are usually done by a cluster administrator. The EventListener sink check in Check Metrics Endpoints Directly port-forwards in the EventListener's namespace and needs pods/portforward there as well.

Tekton Pipelines

The Tekton Pipelines component includes multiple sub-services that expose metrics on port 9090:

ServiceDescriptionMetrics Port
tekton-pipelines-controllerMain reconciler for PipelineRun / TaskRun9090
tekton-pipelines-webhookAdmission webhook9090
tekton-events-controllerCloudEvents controller9090
tekton-pipelines-remote-resolversRemote resource resolution9090

The Pipeline controller metrics use the prefix tekton_pipelines_controller_.

PipelineRun Metrics

Metric NameTypeDescriptionLabels
pipelinerun_duration_secondsHistogram / LastValuePipelineRun execution time in secondsstatus, namespace, pipeline*, pipelinerun*, reason*
pipelinerun_totalCounterTotal number of completed PipelineRunsstatus
running_pipelinerunsLastValue (Gauge)Number of currently running PipelineRunsControlled by metrics.running-pipelinerun.level (see below)
running_pipelineruns_waiting_on_pipeline_resolutionLastValue (Gauge)PipelineRuns waiting on Pipeline reference resolution-
running_pipelineruns_waiting_on_task_resolutionLastValue (Gauge)PipelineRuns waiting on Task reference resolution-

* Labels marked with * are optional and depend on the config-observability configuration.

running_pipelineruns Label Levels

The running_pipelineruns metric labels are controlled by metrics.running-pipelinerun.level:

LevelLabels
"" (default, cluster)No labels
"namespace"namespace
"pipeline"namespace, pipeline
"pipelinerun"namespace, pipeline, pipelinerun

Status Label Values

For PipelineRun metrics:

  • success - PipelineRun completed successfully
  • failed - PipelineRun failed with a reason other than Cancelled, including a run cancelled or stopped with CancelledRunFinally or StoppedRunFinally in which a task failed
  • cancelled - PipelineRun ended with reason Cancelled

For TaskRun metrics:

  • success - TaskRun completed successfully
  • failed - TaskRun failed with a reason other than TaskRunCancelled
  • cancelled - TaskRun ended with reason TaskRunCancelled

TaskRun Metrics

Metric NameTypeDescriptionLabels
taskrun_duration_secondsHistogram / LastValueStandalone TaskRun execution time in secondsstatus, namespace, task*, taskrun*, reason*
pipelinerun_taskrun_duration_secondsHistogram / LastValueTaskRun execution time when part of a PipelineRunstatus, namespace, task*, taskrun*, pipeline*, pipelinerun*, reason*
taskrun_totalCounterTotal number of completed TaskRunsstatus
running_taskrunsLastValue (Gauge)Number of currently running TaskRuns-
running_taskruns_waiting_on_task_resolution_countLastValue (Gauge)TaskRuns waiting on Task reference resolution-
running_taskruns_throttled_by_quotaLastValue (Gauge)TaskRuns throttled by ResourceQuota. Absent from the metrics output while no TaskRun is throttlednamespace*
running_taskruns_throttled_by_nodeLastValue (Gauge)TaskRuns throttled by node-level resource constraints. Absent from the metrics output while no TaskRun is throttlednamespace*
taskruns_pod_latency_millisecondsHistogramPod scheduling latency for TaskRuns in millisecondsnamespace, task*, taskrun*

config-observability Configuration

The config-observability ConfigMap in the tekton-pipelines namespace controls metrics behavior for the Pipeline controller. This ConfigMap is managed by the Tekton Operator and should be configured via the TektonConfig resource's spec.pipeline.options.configMaps field. See Adjusting Optional Configuration Items for Subcomponents for details. A direct edit of the ConfigMap also takes effect in the controller and is not reverted at once, which makes such an edit easy to miss; keep the values in TektonConfig, which the operator applies to the ConfigMap. TektonConfig also has fields for some of these keys directly under spec.pipeline, each named with the full key (a flat key, not a nested path), such as the key metrics.pipelinerun.level under spec.pipeline; while options.configMaps.config-observability sets the same key, the ConfigMap gets the options.configMaps value.

Reload behavior: The Pipeline controller watches config-observability at runtime. Changes to the level keys (metrics.pipelinerun.level, metrics.taskrun.level, and metrics.running-pipelinerun.level) and to the label switches metrics.count.enable-reason and metrics.taskrun.throttle.enable-namespace take effect without a restart. A change to metrics.pipelinerun.level or metrics.taskrun.level shows in the metrics of runs that complete after the change. Allow one or two scrape intervals for dashboard/query changes to appear. A change to metrics.pipelinerun.duration-type or metrics.taskrun.duration-type needs a restart of the tekton-pipelines-controller Pod. Restart the Pod right after the change. Without the restart, the running controller keeps the instrument of the old type, and after the next run completes, the metrics endpoint answers HTTP 500 (body An error has occurred while serving metrics: and invalid metric); restart the Pod to get the metrics back. The exporter keys metrics-protocol, metrics-endpoint, and metrics-export-interval are read only when the controller starts, so a change to them also takes effect only after the Pod restarts; the restart briefly pauses the reconciliation of PipelineRuns and TaskRuns. A restart is also needed when Pod spec settings change (for example changing CONFIG_OBSERVABILITY_NAME in the Deployment).

A cluster administrator restarts the controller by deleting its Pod; the Deployment creates a new one. Then list the Pod:

kubectl delete pods -n tekton-pipelines -l app=tekton-pipelines-controller
kubectl get pods -n tekton-pipelines -l app=tekton-pipelines-controller

The new Pod must be Running, with an AGE shorter than the time since the restart. With metrics-protocol: prometheus, then check the metrics as described in Check Metrics Endpoints Directly.

Example configuration via TektonConfig:

apiVersion: operator.tekton.dev/v1alpha1
kind: TektonConfig
metadata:
  name: config
spec:
  pipeline:
    options:
      disabled: false
      configMaps:
        config-observability:
          data:
            metrics-protocol: prometheus

            # PipelineRun metrics aggregation level.
            # Values: "pipelinerun" | "pipeline" (default) | "namespace"
            #   - "pipelinerun": includes pipeline + pipelinerun labels (new series for every run; with histogram, one per bucket)
            #   - "pipeline": includes pipeline label only
            #   - "namespace": no pipeline/pipelinerun labels
            metrics.pipelinerun.level: "pipeline"

            # TaskRun metrics aggregation level.
            # Values: "taskrun" | "task" (default) | "namespace"
            #   - "taskrun": includes task + taskrun labels (new series for every run; with histogram, one per bucket)
            #   - "task": includes task label only
            #   - "namespace": no task/taskrun labels
            metrics.taskrun.level: "task"

            # Duration metric type for PipelineRun / TaskRun.
            # Values: "histogram" (default) | "lastvalue"
            # Note: The level settings do not change the duration type; it always follows
            #       metrics.*.duration-type. The "pipelinerun" and "taskrun" levels add per-run
            #       labels and significantly increase the number of time series.
            metrics.pipelinerun.duration-type: "histogram"
            metrics.taskrun.duration-type: "histogram"

            # Running PipelineRun metrics aggregation level.
            # Values: "pipelinerun" | "pipeline" | "namespace" | "" (default, cluster-level)
            metrics.running-pipelinerun.level: ""

            # Include reason label on duration metrics (pipelinerun_duration_seconds,
            # taskrun_duration_seconds, pipelinerun_taskrun_duration_seconds).
            # Values: "true" | "false" (default)
            # Warning: Enabling this increases label cardinality.
            # Note: Despite the key name, this does NOT affect count metrics
            # (pipelinerun_total / taskrun_total), only duration metrics.
            metrics.count.enable-reason: "false"

            # Include namespace label on throttled TaskRun metrics.
            # Values: "true" | "false" (default)
            metrics.taskrun.throttle.enable-namespace: "false"

Save the sample in a file such as tektonconfig-observability.yaml. To change only some keys, keep only those keys in the file, still nested under spec.pipeline.options.configMaps.config-observability.data as in the sample. Merge the file into the existing TektonConfig. A merge patch changes only the keys in the file and keeps the other fields of TektonConfig:

kubectl patch tektonconfig config --type merge --patch-file tektonconfig-observability.yaml

Do not use kubectl apply -f for a file that holds only part of TektonConfig: a later kubectl apply -f of another partial file deletes the fields that this file set and the later file leaves out.

After you merge the change, check that the values reached the ConfigMap:

kubectl get configmap config-observability -n tekton-pipelines -o yaml

The keys under data, outside the commented _example block, must show the values that you set.

Histogram Buckets

When the duration type is histogram, the following bucket boundaries (in seconds) are used:

10, 30, 60, 300, 900, 1800, 3600, 5400, 10800, 21600, 43200, 86400

This corresponds to: 10s, 30s, 1m, 5m, 15m, 30m, 1h, 1.5h, 3h, 6h, 12h, 24h.

For production environments, use aggregated levels to control label cardinality. Set the keys below, and those of the debugging example after them, under spec.pipeline.options.configMaps.config-observability.data of the sample above:

metrics.pipelinerun.level: "pipeline"
metrics.taskrun.level: "task"
metrics.pipelinerun.duration-type: "histogram"
metrics.taskrun.duration-type: "histogram"
metrics.count.enable-reason: "false"

If you need per-run granularity for debugging, temporarily switch to:

metrics.pipelinerun.level: "pipelinerun"
metrics.taskrun.level: "taskrun"

Note that this will significantly increase the number of time series. Switching back to pipeline and task does not remove the per-run series that were already created; they stay in the controller's metrics output until the tekton-pipelines-controller Pod restarts, because the controller process holds them in memory. Use these levels on a test cluster or for a short debugging window.


Tekton Triggers

The Tekton Triggers component exposes two categories of metrics from different processes.

Controller Metrics (port 9000)

The Triggers controller reports resource count metrics every 60 seconds.

ServiceMetrics Port
tekton-triggers-controller9000

The Triggers controller metrics use the prefix controller_.

Metric NameTypeDescriptionLabels
eventlistener_countLastValue (Gauge)Number of EventListener resources-
triggerbinding_countLastValue (Gauge)Number of TriggerBinding resources-
clustertriggerbinding_countLastValue (Gauge)Number of ClusterTriggerBinding resources-
triggertemplate_countLastValue (Gauge)Number of TriggerTemplate resources-
clusterinterceptor_countLastValue (Gauge)Number of ClusterInterceptor resources-

EventListener Sink Metrics

Each EventListener pod exposes additional HTTP and event processing metrics. These metrics come from the EventListener sink process (not the controller). The Prometheus metric prefix is eventlistener_.

Metric Name (Prometheus)TypeDescriptionLabels
eventlistener_http_duration_secondsHistogramEventListener HTTP request duration-
eventlistener_event_received_totalCounterTotal events received by the sinkstatus
eventlistener_triggered_resources_totalCounterTotal resources created by triggerskind
  • eventlistener_http_duration_seconds histogram buckets: 0.001, 0.01, 0.1, 1, 5, 10 (seconds)
  • eventlistener_event_received_total status values: succeeded, failed
  • eventlistener_triggered_resources_total kind values: the Kubernetes resource Kind of the created object (e.g., PipelineRun, TaskRun)

These sink metrics are exposed per EventListener pod, not from the central controller. You may need a separate ServiceMonitor or PodMonitor to scrape them if the EventListener pods expose a metrics port. The Service of an EventListener publishes this metrics port (9000, http-metrics) next to the event port; before you use a NodePort or LoadBalancer Service, see Network Configuration.


Tekton Results

Tekton Results has two sub-services that expose metrics.

ServiceDescriptionMetrics Port
tekton-results-watcherWatches and cleans up PipelineRun/TaskRun resources9090
tekton-results-api-servicegRPC/REST API server9090

Watcher Metrics

The Watcher metrics use the prefix watcher_. Counters are exported with the suffix _total, for example watcher_pipelinerun_delete_count_total and watcher_runs_not_stored_count_total.

Deletion Metrics

Metric NameTypeDescriptionLabels
pipelinerun_delete_countCounterTotal number of deleted PipelineRunsstatus, namespace, pipeline*
pipelinerun_delete_duration_secondsHistogramTime from PipelineRun completion to deletionstatus, namespace, pipeline*
taskrun_delete_countCounterTotal number of deleted TaskRunsstatus, namespace, pipeline*, task*
taskrun_delete_duration_secondsHistogramTime from TaskRun completion to deletionstatus, namespace, pipeline*, task*
customrun_delete_countCounterTotal number of deleted CustomRunsstatus, namespace, pipeline*, customrun*
customrun_delete_duration_secondsHistogramTime from CustomRun completion to deletionstatus, namespace, pipeline*, customrun*

* Optional labels depend on config-observability settings for the Results Watcher.

Note: pipelinerun_delete_count, pipelinerun_delete_duration_seconds, taskrun_delete_count, taskrun_delete_duration_seconds, customrun_delete_count, and customrun_delete_duration_seconds are only recorded when the Watcher actually deletes runs. These metrics will remain empty (no data points) unless the --completed_run_grace_period flag is set to a non-zero value on the tekton-results-watcher Deployment (through TektonConfig, in the args of the watcher container under spec.result.options.deployments.tekton-results-watcher, as in the reproduction steps below). By default this flag is 0, which disables automatic deletion. A positive duration (e.g. 10m) deletes runs after that grace period; a negative value deletes them right away. The Watcher does not wait for log storage to finish before it deletes a run.

Warning: Any non-zero --completed_run_grace_period makes the Watcher delete completed PipelineRuns, TaskRuns, and CustomRuns in every namespace it watches, by default all namespaces, including the runs of other teams. It does not delete a run that has an owner reference: TaskRuns and CustomRuns that are part of a PipelineRun are always left to their PipelineRun, and a run owned by any other object is kept while the Watcher flag -check_owner has its default true (the shipped Watcher arguments do not set it). Deleted runs cannot be restored. The -label_selector flag of the Watcher limits deletion to runs whose labels match. Try the flag only on a dedicated test cluster, as in the reproduction steps below.

Status label values for Results Watcher:

  • success - Run completed successfully
  • failed - Run failed with a reason other than the cancelled reasons below
  • cancelled - Run ended with reason Cancelled (PipelineRun), TaskRunCancelled (TaskRun), or CustomRunCancelled (CustomRun)

Shared Metrics

These metrics are registered by the PipelineRun, TaskRun, and CustomRun reconcilers in the Watcher, tracking storage-related events.

Metric NameTypeDescriptionLabels
runs_not_stored_countCounterRuns deleted without being stored to Resultskind, namespace
run_storage_latency_secondsHistogramTime from run completion to successful storagekind, namespace

The kind label identifies the run type. runs_not_stored_count uses PipelineRun, TaskRun, and CustomRun; run_storage_latency_seconds uses the lower-case values pipelinerun, taskrun, and customrun. With the Watcher flag -logs_api=true (the shipped value), run_storage_latency_seconds is recorded only for CustomRuns.

Note: runs_not_stored_count is only recorded when a run is externally deleted (e.g. via kubectl delete) while the Watcher is holding a finalizer to coordinate archiving. It will remain empty unless all of the following conditions are met:

  1. For PipelineRuns and TaskRuns, the --logs_api flag is false (log storage disabled) — with --logs_api set to true, the Watcher skips finalizer-based coordination for PipelineRuns and TaskRuns. CustomRuns are coordinated either way.
  2. The --disable_crd_update flag is false (annotation updates enabled).
  3. The store_deadline has elapsed since the run completed. The --store_deadline flag is always set (default 10m); it is the maximum time the Watcher waits for archiving to complete before giving up and allowing deletion. With 0, the deadline has passed as soon as the run completes.
  4. The run is externally deleted while it carries no results.tekton.dev/stored annotation at all. A run that was archived while it was still running carries stored=false and is not counted, even if its final state was never archived.

In normal operation (runs archived before deletion, or deletion triggered by the Watcher itself via --completed_run_grace_period), this counter stays at zero. A non-zero value indicates potential data loss: runs were deleted before their state could be saved to the Results API.

Quick reproduction (dedicated test cluster only): If you do not see this metric, that usually means the trigger conditions were not met, not that the metric is missing. TektonConfig and the Results Watcher are cluster-wide: while the Results API is scaled to 0, no run in any namespace is archived, and a run deleted after store_deadline in that time is never stored. Do not run these steps on a cluster that other teams use.

  1. Save the current spec.result of TektonConfig (kubectl get tektonconfig config -o jsonpath='{.spec.result}' > result-orig.json). Then set the Watcher flags in spec.result.options.deployments.tekton-results-watcher.spec.template.spec.containers: container watcher with args -logs_api=false, -disable_crd_update=false, and a -store_deadline value (for example -store_deadline=30s). spec.result.logs_api sets only the LOGS_API setting of the Results API and does not change the Watcher's -logs_api flag. The change restarts the Watcher: wait until the new Watcher Pod is Running (kubectl get pods -n tekton-pipelines -l app.kubernetes.io/name=tekton-results-watcher) before step 2, because a Watcher that starts while the Results API is unreachable waits up to 1 minute for it and then exits.
  2. Temporarily set Results API replicas to 0 via TektonConfig (spec.result.options.deployments.tekton-results-api.spec.replicas: 0) so runs cannot be archived.
  3. Create a TaskRun or PipelineRun and wait until it completes.
  4. Wait until store_deadline has elapsed, then externally delete the run (kubectl delete taskrun <taskrun-name> -n <namespace> or kubectl delete pipelinerun <pipelinerun-name> -n <namespace>). For a TaskRun, the Watcher also waits until -forward_buffer (default 150s) has passed since the run completed, so the deletion and the counter can take that long; check step 5 before you restore in step 6.
  5. Check Watcher /metrics or Prometheus for watcher_runs_not_stored_count_total (the exposition name: component prefix plus the counter suffix _total); it should increase.
  6. Restore spec.result by replacing it with the copy saved in step 1, with a JSON patch op: replace on the path /spec/result (jq '[{"op":"replace","path":"/spec/result","value":.}]' result-orig.json > restore-patch.json, then kubectl patch tektonconfig config --type json --patch-file restore-patch.json). A merge patch does not remove the keys added in steps 1 and 2. Then check that the tekton-results-api Deployment has its replicas back and that the watcher container args are the ones they had before step 1 (on a default installation -api_addr, -auth_mode and -logs_api=true), without the -logs_api=false, -disable_crd_update=false and -store_deadline values of step 1.

The run_storage_latency_seconds histogram uses the following bucket boundaries (in seconds):

0.1, 0.5, 1, 2, 5, 10, 30, 60, 120, 300, 600, 1800

Watcher config-observability

The Results Watcher has its own config-observability ConfigMap, tekton-results-config-observability (the name set in the CONFIG_OBSERVABILITY_NAME environment variable). This ConfigMap is managed by the Tekton Operator and should be configured via the TektonConfig resource's spec.result.options.configMaps field, under the key tekton-results-config-observability. See Adjusting Optional Configuration Items for Subcomponents for details.

Reload behavior: The Results Watcher also watches this ConfigMap. Of the keys in the table below, changes to the two level keys (metrics.pipelinerun.level and metrics.taskrun.level) take effect without a restart; the two duration-type keys are parsed but not used, so changing them has no effect. The exporter keys metrics-protocol (shipped as prometheus), metrics-endpoint, and metrics-export-interval are read only when the Watcher starts, so a change to them takes effect only after the tekton-results-watcher Pod restarts; the restart briefly pauses the Watcher's reconciliation, which archives runs to Results. A restart is also needed when Deployment-level settings (such as env vars/args) are changed.

Merge the change into TektonConfig in the same way as for the Pipelines ConfigMap: save the keys under spec.result.options.configMaps.tekton-results-config-observability.data in a file and run kubectl patch tektonconfig config --type merge --patch-file <file>. Then check that the values reached the Watcher's ConfigMap. The keys under data, outside the commented _example block, must show the values that you set:

kubectl get configmap tekton-results-config-observability -n tekton-pipelines -o yaml

After a change to an exporter key, a cluster administrator restarts the Watcher by deleting its Pod and checks that the new Pod is Running, with an AGE shorter than the time since the restart:

kubectl delete pods -n tekton-pipelines -l app.kubernetes.io/name=tekton-results-watcher
kubectl get pods -n tekton-pipelines -l app.kubernetes.io/name=tekton-results-watcher

Besides the exporter keys metrics-protocol, metrics-endpoint, and metrics-export-interval described above, the Watcher's ConfigMap supports the following metrics.* keys:

KeyDefaultValuesDescription
metrics.pipelinerun.levelpipelinepipeline, namespaceControls pipeline label on the deletion count and duration metrics
metrics.taskrun.leveltasktask, namespaceControls task label on the TaskRun deletion count and duration metrics, and customrun label on the CustomRun ones
metrics.pipelinerun.duration-typehistogramhistogram, lastvalueParsed but not used: the Watcher always records the deletion durations as histograms
metrics.taskrun.duration-typehistogramhistogram, lastvalueParsed but not used, as above

Note: Unlike Tekton Pipelines, the Results Watcher does not support pipelinerun / taskrun individual-run granularity levels. It also does not have the metrics.count.enable-reason, metrics.running-pipelinerun.level, or metrics.taskrun.throttle.enable-namespace keys.

The Watcher creates pipelinerun_delete_duration_seconds, taskrun_delete_duration_seconds and customrun_delete_duration_seconds as OpenTelemetry histograms, so setting either duration-type key to lastvalue has no effect.

API Server Metrics

The API server exposes standard gRPC Prometheus metrics via the go-grpc-prometheus library on port 9090. These include:

  • grpc_server_handled_total - Total RPCs completed on the server
  • grpc_server_started_total - Total RPCs started on the server
  • grpc_server_msg_received_total / grpc_server_msg_sent_total - Message counts
  • grpc_server_handling_seconds - RPC handling duration, only if PROMETHEUS_HISTOGRAM is enabled. The shipped tekton-results-api-config ConfigMap sets PROMETHEUS_HISTOGRAM=false, so the API server does not expose this metric by default

Tekton Chains

Tekton Chains is a security component that generates, signs, and stores provenance for artifacts built with Tekton Pipelines. It observes completed TaskRuns and PipelineRuns, then creates attestations and signatures.

ServiceDescriptionMetrics Port
tekton-chains-metricsChains watcher/controller9090 (http-metrics)

The Chains controller metrics use the prefix watcher_ (same as Results Watcher, but the custom metric names are different, so there are no collisions).

Chains Metrics

All Chains metrics are Counters. Only the signing-failure counters have a label (error_type).

Metric Name (Prometheus)TypeDescription
watcher_taskrun_sign_created_totalCounterTotal signed messages for TaskRuns
watcher_taskrun_payload_uploaded_totalCounterTotal payloads uploaded to a transparency log for TaskRuns (recorded only on such an upload)
watcher_taskrun_payload_stored_totalCounterTotal stored payloads for TaskRuns
watcher_taskrun_marked_signed_totalCounterTotal TaskRuns marked as signed
watcher_pipelinerun_sign_created_totalCounterTotal signed messages for PipelineRuns
watcher_pipelinerun_payload_uploaded_totalCounterTotal payloads uploaded to a transparency log for PipelineRuns (recorded only on such an upload)
watcher_pipelinerun_payload_stored_totalCounterTotal stored payloads for PipelineRuns
watcher_pipelinerun_marked_signed_totalCounterTotal PipelineRuns marked as signed
watcher_taskrun_signing_failures_totalCounterTotal signing failures for TaskRuns, labeled by error_type (for example storage or payload_creation)
watcher_pipelinerun_signing_failures_totalCounterTotal signing failures for PipelineRuns, labeled by error_type

Note: Chains signs and stores payloads only when a signer is configured; by default the signer uses a key in the signing-secrets Secret. The Operator creates no key unless generateSigningSecret is true in the Chains configuration (the default is false); see Signing Key Configuration. Without a key, Chains logs No signer <signer> configured for <type>, skips the signature, adds no failure count for the missing key to *_signing_failures_total, and still marks the run as signed: *_marked_signed_total grows, while *_sign_created_total and *_payload_stored_total do not. On a controller that has signed nothing since it started, these two counters do not appear in the metrics output at all, because a Chains counter appears only after its first increment.


Controller Framework Metrics

Tekton controllers also expose the following framework metrics. These metrics do not use the component prefix: the names are the same on every component, so distinguish the components by the target labels that your monitoring stack adds when it scrapes them. The workqueue metrics carry a name label with the reconciler, for example github.com.tektoncd.pipeline.pkg.reconciler.pipelinerun.Reconciler. The Pipelines-as-Code controller and EventListener sinks expose only the two Kubernetes API client metrics.

Metric Name (Prometheus)TypeDescription
http_client_request_duration_secondsHistogramKubernetes API client request duration (seconds)
kn_k8s_client_http_response_status_code_totalCounterKubernetes API client responses
kn_workqueue_depthGaugeCurrent workqueue depth
kn_workqueue_adds_totalCounterTotal workqueue additions
kn_workqueue_queue_duration_secondsHistogramTime items spend waiting in the workqueue
kn_workqueue_process_duration_secondsHistogramTime spent processing workqueue items
kn_workqueue_retries_totalCounterTotal workqueue retries
kn_workqueue_unfinished_work_secondsGaugeDuration of unfinished workqueue items

There are no client_latency, reconcile_count, or reconcile_latency series with a component prefix.


Setting Up ServiceMonitor

To enable Prometheus scraping for Tekton components, deploy ServiceMonitor resources.

Prerequisites are listed in Prerequisites.

Use the following guidance based on your monitoring stack:

  • If you use Prometheus (Prometheus Operator), labels such as metadata.labels.prometheus: kube-prometheus must match the Prometheus CR spec.serviceMonitorSelector; otherwise, this ServiceMonitor will not be scraped.
  • If you use VictoriaMetrics, you typically do not need labels like prometheus: kube-prometheus; create ServiceMonitor/VMServiceScrape according to your monitoring setup.

When using Prometheus, use the following commands to find and verify the selector. They need read access to Prometheus resources (the first command lists them in all namespaces), usually a cluster administrator:

# 1) Locate Prometheus CRs (resource type: monitoring.coreos.com/v1, Kind=Prometheus)
kubectl get prometheus -A

# 2) Check ServiceMonitor selector on the target Prometheus instance
kubectl get prometheus -n <prometheus-namespace> <prometheus-name> -o jsonpath='{.spec.serviceMonitorSelector}'

# 3) Check the ServiceMonitor namespace selector on the same instance
kubectl get prometheus -n <prometheus-namespace> <prometheus-name> -o jsonpath='{.spec.serviceMonitorNamespaceSelector}'

If the second command prints nothing, the Prometheus CR does not set serviceMonitorSelector. The Prometheus Operator API defines an unset selector as matching no ServiceMonitor and an empty selector ({}) as matching all of them, so in that case this Prometheus instance does not select the ServiceMonitor resources below, whatever their labels.

If the third command prints nothing, the Prometheus CR does not set serviceMonitorNamespaceSelector. The Prometheus Operator API defines an unset namespace selector as matching only the namespace of the Prometheus CR and an empty one ({}) as matching all namespaces, so a Prometheus CR outside tekton-pipelines without this field does not select the ServiceMonitor resources below.

If no Prometheus CR exists in your cluster, monitoring is usually platform-managed (for example, VictoriaMetrics) or implemented differently. In such cases, labels like prometheus: kube-prometheus are usually not required; follow your platform scraping rules.

For more info please refer to Integrating External Metrics.

Pipeline ServiceMonitor

Pipeline ServiceMonitor YAML
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: tekton-pipelines-metrics
  namespace: tekton-pipelines
  labels:
    app.kubernetes.io/name: tekton-pipelines
    # prometheus: kube-prometheus
spec:
  selector:
    matchLabels:
      app.kubernetes.io/part-of: tekton-pipelines
  endpoints:
  - port: http-metrics
    path: /metrics
    interval: 30s
  namespaceSelector:
    matchNames:
    - tekton-pipelines

This ServiceMonitor matches Pipeline services with the label app.kubernetes.io/part-of: tekton-pipelines (including remote-resolvers) and scrapes them in the tekton-pipelines namespace.

Triggers ServiceMonitor

Triggers ServiceMonitor YAML
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: tekton-triggers-metrics
  namespace: tekton-pipelines
  labels:
    app.kubernetes.io/name: tekton-triggers
    # prometheus: kube-prometheus
spec:
  selector:
    matchLabels:
      app.kubernetes.io/part-of: tekton-triggers
      app.kubernetes.io/component: controller
  endpoints:
  - port: http-metrics
    path: /metrics
    interval: 30s
  namespaceSelector:
    matchNames:
    - tekton-pipelines

This ServiceMonitor collects Triggers controller metrics (controller_*) only. It does not include EventListener sink metrics.

EventListener Sink ServiceMonitor

EventListener Sink ServiceMonitor YAML
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: tekton-eventlistener-sink-metrics
  namespace: tekton-pipelines
  labels:
    app.kubernetes.io/name: tekton-eventlistener-sink
    # prometheus: kube-prometheus
spec:
  selector:
    matchExpressions:
    - key: eventlistener
      operator: Exists
    - key: app.kubernetes.io/managed-by
      operator: In
      values:
      - EventListener
  endpoints:
  - port: http-metrics
    path: /metrics
    interval: 30s
  namespaceSelector:
    any: true

EventListener Services usually run in application namespaces, so this example uses namespaceSelector.any: true for cross-namespace scraping. If you need tighter scope, switch to matchNames and list allowed namespaces explicitly.

Results ServiceMonitor

The Results services have both app.kubernetes.io/part-of: tekton-results and app.kubernetes.io/name labels. To precisely target API + Watcher (and exclude Postgres), this example matches app.kubernetes.io/name:

Results ServiceMonitor YAML
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: tekton-results-metrics
  namespace: tekton-pipelines
  labels:
    app.kubernetes.io/name: tekton-results
    # prometheus: kube-prometheus
spec:
  selector:
    matchExpressions:
    - key: app.kubernetes.io/name
      operator: In
      values:
      - tekton-results-api
      - tekton-results-watcher
  endpoints:
  - port: prometheus
    path: /metrics
    interval: 30s
  - port: metrics
    path: /metrics
    interval: 30s
  namespaceSelector:
    matchNames:
    - tekton-pipelines

The Results API server uses port name prometheus (9090) and the Watcher uses port name metrics (9090). Each service only exposes one of these port names, so only the matching endpoint will be scraped.

Chains ServiceMonitor

Chains ServiceMonitor YAML
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: tekton-chains-metrics
  namespace: tekton-pipelines
  labels:
    app.kubernetes.io/name: tekton-chains
    # prometheus: kube-prometheus
spec:
  selector:
    matchLabels:
      app.kubernetes.io/part-of: tekton-chains
  endpoints:
  - port: http-metrics
    path: /metrics
    interval: 30s
  namespaceSelector:
    matchNames:
    - tekton-pipelines

Verification

After deploying the ServiceMonitor resources, verify that Prometheus is scraping the targets.

Check Metrics Endpoints Directly

Each kubectl port-forward command below keeps running in the foreground until you stop it with Ctrl+C. Run it in a second terminal and keep it running, run the curl commands that follow it in your first terminal, then stop the port-forward.

These checks assume one replica of each component: kubectl port-forward svc/<service> connects to one Pod of the Service. When a component runs more replicas, a counter can be absent on the Pod that you reach; then repeat the check with pod/<pod-name> in place of svc/<service> for each Pod, or query Prometheus.

# Pipeline controller
$ kubectl port-forward -n tekton-pipelines svc/tekton-pipelines-controller 9090:9090
$ curl -s http://localhost:9090/metrics | grep tekton_pipelines_controller_

# HELP tekton_pipelines_controller_pipelinerun_total Number of pipelineruns
# TYPE tekton_pipelines_controller_pipelinerun_total counter
tekton_pipelines_controller_pipelinerun_total{otel_scope_name="tekton_pipelines_controller",otel_scope_schema_url="",otel_scope_version="",status="cancelled"} 11
tekton_pipelines_controller_pipelinerun_total{otel_scope_name="tekton_pipelines_controller",otel_scope_schema_url="",otel_scope_version="",status="failed"} 90

# Triggers controller
$ kubectl port-forward -n tekton-pipelines svc/tekton-triggers-controller 9000:9000
$ curl -s http://localhost:9000/metrics | grep controller_

# HELP controller_clusterinterceptor_count number of clusterinterceptor
# TYPE controller_clusterinterceptor_count gauge
controller_clusterinterceptor_count{otel_scope_name="github.com/tektoncd/triggers/pkg/reconciler/metrics",otel_scope_schema_url="",otel_scope_version=""} 5

# EventListener sink metrics (replace namespace/service)
$ kubectl port-forward -n <eventlistener-namespace> svc/<eventlistener-service> 9094:9000
$ curl -s http://localhost:9094/metrics | grep eventlistener_

# HELP eventlistener_event_received_total number of events received by sink
# TYPE eventlistener_event_received_total counter
eventlistener_event_received_total{otel_scope_name="github.com/tektoncd/triggers/pkg/sink",otel_scope_schema_url="",otel_scope_version="",status="succeeded"} 1

# HELP eventlistener_triggered_resources_total Count of the number of triggered eventlistener resources
# TYPE eventlistener_triggered_resources_total counter
eventlistener_triggered_resources_total{kind="TaskRun",otel_scope_name="github.com/tektoncd/triggers/pkg/sink",otel_scope_schema_url="",otel_scope_version=""} 1

# Results watcher
# With the shipped watcher argument -logs_api=true, the watcher records
# watcher_run_storage_latency_seconds only with kind="customrun", and can record
# watcher_runs_not_stored_count_total with kind="CustomRun" (when a CustomRun
# without the results.tekton.dev/stored annotation is deleted after
# store_deadline; see the runs_not_stored_count note); until a CustomRun
# completes, the grep prints nothing. The deletion metrics appear only when
# --completed_run_grace_period is set (see the note in the Tekton Results section).
$ kubectl port-forward -n tekton-pipelines svc/tekton-results-watcher 9091:9090
$ curl -s http://localhost:9091/metrics | grep watcher_

# HELP watcher_run_storage_latency_seconds time from run completion to successful storage
# TYPE watcher_run_storage_latency_seconds histogram
watcher_run_storage_latency_seconds_bucket{kind="customrun",namespace="<namespace>",otel_scope_name="tekton.dev/results",otel_scope_schema_url="",otel_scope_version="",le="1"} 0
watcher_run_storage_latency_seconds_bucket{kind="customrun",namespace="<namespace>",otel_scope_name="tekton.dev/results",otel_scope_schema_url="",otel_scope_version="",le="2"} 3
watcher_run_storage_latency_seconds_bucket{kind="customrun",namespace="<namespace>",otel_scope_name="tekton.dev/results",otel_scope_schema_url="",otel_scope_version="",le="5"} 3

# To tell an empty watcher_ output from a failed port-forward or endpoint, count
# the target_info line, which the endpoint returns with every scrape: the command
# prints 1 when the endpoint answered and 0 when it did not.
$ curl -s http://localhost:9091/metrics | grep -c '^target_info'

# Results API
$ kubectl port-forward -n tekton-pipelines svc/tekton-results-api-service 9092:9090
$ curl -s http://localhost:9092/metrics | grep grpc_server_

# HELP grpc_server_handled_total Total number of RPCs completed on the server, regardless of success or failure.
# TYPE grpc_server_handled_total counter
grpc_server_handled_total{grpc_code="Aborted",grpc_method="Check",grpc_service="grpc.health.v1.Health",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Aborted",grpc_method="CreateRecord",grpc_service="tekton.results.v1alpha2.Results",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Aborted",grpc_method="CreateResult",grpc_service="tekton.results.v1alpha2.Results",grpc_type="unary"} 0

# HELP grpc_server_started_total Total number of RPCs started on the server.
# TYPE grpc_server_started_total counter
grpc_server_started_total{grpc_method="Check",grpc_service="grpc.health.v1.Health",grpc_type="unary"} 337606
grpc_server_started_total{grpc_method="CreateRecord",grpc_service="tekton.results.v1alpha2.Results",grpc_type="unary"} 10301
grpc_server_started_total{grpc_method="CreateResult",grpc_service="tekton.results.v1alpha2.Results",grpc_type="unary"} 832

# Chains controller
$ kubectl port-forward -n tekton-pipelines svc/tekton-chains-metrics 9093:9090
$ curl -s http://localhost:9093/metrics | grep watcher_

# HELP watcher_pipelinerun_marked_signed_total Total number of objects marked as signed for pipelineruns
# TYPE watcher_pipelinerun_marked_signed_total counter
watcher_pipelinerun_marked_signed_total{otel_scope_name="github.com/tektoncd/chains/pkg/pipelinerunmetrics",otel_scope_schema_url="",otel_scope_version=""} 56

# A Chains counter appears only after its first increment, so the grep prints
# nothing until the controller has counted a run since it started. To tell that
# from a failed port-forward or endpoint, count the target_info line, which the
# endpoint returns with every scrape: the command prints 1 when the endpoint
# answered and 0 when it did not.
$ curl -s http://localhost:9093/metrics | grep -c '^target_info'

EventListener sink metrics such as eventlistener_event_received_total and eventlistener_http_duration_seconds are request-driven. Send at least one request to the EventListener before validating these metrics.

Check Prometheus Targets

# Verify ServiceMonitor resources exist
$ kubectl get servicemonitor -n tekton-pipelines

NAME                                AGE
tekton-chains-metrics               10m
tekton-eventlistener-sink-metrics   10m
tekton-pipelines-metrics            10m
tekton-results-metrics              10m
tekton-triggers-metrics             10m

# Check Prometheus targets (via Prometheus UI or API)
# Look for targets whose job label is the name of a scraped Service:
# the samples set no spec.jobLabel, so Prometheus Operator uses the
# Service name as the job label, not the ServiceMonitor name

Example PromQL Queries

# PipelineRun cumulative success rate (avoids misinterpretation in empty completion windows)
100 * sum(tekton_pipelines_controller_pipelinerun_total{status="success"}) / clamp_min(sum(tekton_pipelines_controller_pipelinerun_total), 1)

# Completed PipelineRuns in the last 5 minutes (throughput)
round(sum(increase(tekton_pipelines_controller_pipelinerun_total[5m])))

# PipelineRun duration P95 (histogram mode)
histogram_quantile(0.95,
  rate(tekton_pipelines_controller_pipelinerun_duration_seconds_bucket[5m])
)

# TaskRun duration P95, standalone TaskRuns (histogram mode)
histogram_quantile(0.95,
  sum by (le) (rate(tekton_pipelines_controller_taskrun_duration_seconds_bucket[5m]))
)

# TaskRun duration P95, TaskRuns that run in a PipelineRun (histogram mode)
# Each TaskRun is recorded in only one of the two histograms; adding them with `+`
# returns nothing while either histogram has no series in the window.
histogram_quantile(0.95,
  sum by (le) (rate(tekton_pipelines_controller_pipelinerun_taskrun_duration_seconds_bucket[5m]))
)

# PipelineRun duration (lastvalue mode)
avg_over_time(tekton_pipelines_controller_pipelinerun_duration_seconds[5m])

# Currently running PipelineRuns (single series to avoid duplicate legends)
max(tekton_pipelines_controller_running_pipelineruns)

# TaskRuns throttled by resource quota (no data while no TaskRun is throttled).
# This query and the TaskRuns Throttled dashboard panel assume
# metrics.taskrun.throttle.enable-namespace: "false" (the default). With "true" the
# controller reports one series per namespace and no total, so max() returns the
# largest per-namespace count.
max(tekton_pipelines_controller_running_taskruns_throttled_by_quota)

# Trigger resource counts
controller_eventlistener_count
controller_triggertemplate_count

# Chains signing activity
watcher_taskrun_sign_created_total
watcher_pipelinerun_sign_created_total

When to enable. Enable metrics collection when platform or project teams need to see build throughput and duration, signing activity, and event-driven triggering across the cluster. The config-observability ConfigMap controls the shared Pipeline controller and is managed through TektonConfig, and the control-plane ServiceMonitor resources target the tekton-pipelines namespace.

Recommended values. Deploy the Pipeline, Triggers, and Chains ServiceMonitor resources from Setting Up ServiceMonitor, add the EventListener sink ServiceMonitor for namespaces that run EventListeners, and add the Results ServiceMonitor when Results is installed. In config-observability, use the values from Recommended Production Configuration: metrics.pipelinerun.level: "pipeline", metrics.taskrun.level: "task", metrics.pipelinerun.duration-type: "histogram", metrics.taskrun.duration-type: "histogram", and metrics.count.enable-reason: "false". Keep the histogram duration type, because the P50/P95 duration panels in MonitorDashboard Examples query *_bucket series. Use the pipelinerun and taskrun levels only on a test cluster or for a short debugging window, since they add per-run labels and significantly increase the number of time series; the duration type still follows metrics.*.duration-type. Switching back to pipeline and task does not remove the per-run series that were already created: they stay until the tekton-pipelines-controller Pod restarts. Deploy the dashboards once in the cpaas-system namespace under the tekton folder.

To confirm the result, check that the values reached config-observability as described in config-observability Configuration, that the endpoints answer and Prometheus scrapes the targets as described in Verification, and that the dashboards exist with the kubectl get monitordashboard command in MonitorDashboard Examples. On a default installation, config-observability already has the recommended values, so they need no undo; switch the debug levels back as described in Recommended Production Configuration. To remove the monitoring setup, delete the ServiceMonitor and MonitorDashboard resources that you created, and name only those, for example:

kubectl delete servicemonitor -n tekton-pipelines tekton-pipelines-metrics tekton-triggers-metrics tekton-eventlistener-sink-metrics tekton-results-metrics tekton-chains-metrics
kubectl delete monitordashboard -n cpaas-system tekton-pipeline tekton-triggers tekton-results tekton-chains

Deleting them needs delete rights on these resources in tekton-pipelines and cpaas-system.

Prerequisites. A Prometheus or VictoriaMetrics monitoring stack must be able to discover the ServiceMonitor resources you create; with Prometheus Operator, the ServiceMonitor labels must match the Prometheus spec.serviceMonitorSelector. Scraper pods must reach port 9090 on most control-plane services and port 9000 on the Triggers controller and EventListener sinks. See Prerequisites for the full list, including the permissions to create ServiceMonitor resources and, for the dashboards, MonitorDashboard resources in cpaas-system.


MonitorDashboard Examples

The following MonitorDashboard resources provide ready-to-use dashboards for monitoring Tekton components. Deploy them to the cpaas-system namespace under the tekton folder. After you apply them, check that they exist:

kubectl get monitordashboard -n cpaas-system -l cpaas.io/dashboard.folder=tekton

The output must list the dashboards that you applied: tekton-pipeline, tekton-triggers, tekton-results, and tekton-chains. No resources found in cpaas-system namespace. means that none of them was created.

Important: Each panel must include id (unique integer), datasource: prometheus, and transformations: []. Each target must include datasource: prometheus and refId. Duration P50/P95 panels in this document use *_bucket queries and require metrics.*.duration-type=histogram; if you use lastvalue, replace those queries with LastValue-style expressions such as avg_over_time(...). Quote the y key in gridPos ("y": 0); unquoted y is parsed as a boolean and the panel loses its vertical position.

Tekton Pipeline Dashboard

Tekton Pipeline Dashboard YAML
kind: MonitorDashboard
apiVersion: ait.alauda.io/v1alpha2
metadata:
  labels:
    cpaas.io/dashboard.folder: tekton
    cpaas.io/dashboard.is.home.dashboard: "false"
    cpaas.io/dashboard.tag.tekton: "true"
  name: tekton-pipeline
  namespace: cpaas-system
spec:
  body:
    title: Tekton Pipeline Overview
    titleZh: Tekton Pipeline Overview
    tags:
      - tekton
    time:
      from: now-1h
      to: now
    templating:
      list: []
    panels:
      - id: 1
        title: PipelineRun Total (by status)
        type: timeseries
        datasource: prometheus
        gridPos:
          h: 8
          w: 8
          x: 0
          "y": 0
        targets:
          - datasource: prometheus
            expr: sum by (status) (tekton_pipelines_controller_pipelinerun_total)
            refId: A
        fieldConfig:
          defaults:
            color:
              mode: palette-classic
            custom:
              drawStyle: line
              fillOpacity: 0
              lineWidth: 1
              spanNulls: false
            thresholds:
              mode: absolute
              steps:
                - color: green
                  value: null
          overrides: []
        options:
          legend:
            calcs:
              - latest
            displayMode: list
            placement: bottom
            showLegend: true
          tooltip:
            mode: multi
            sort: desc
        transformations: []
      - id: 2
        title: TaskRun Total (by status)
        type: timeseries
        datasource: prometheus
        gridPos:
          h: 8
          w: 8
          x: 8
          "y": 0
        targets:
          - datasource: prometheus
            expr: sum by (status) (tekton_pipelines_controller_taskrun_total)
            refId: A
        fieldConfig:
          defaults:
            color:
              mode: palette-classic
            custom:
              drawStyle: line
              fillOpacity: 0
              lineWidth: 1
              spanNulls: false
            thresholds:
              mode: absolute
              steps:
                - color: green
                  value: null
          overrides: []
        options:
          legend:
            calcs:
              - latest
            displayMode: list
            placement: bottom
            showLegend: true
          tooltip:
            mode: multi
            sort: desc
        transformations: []
      - id: 3
        title: PipelineRun Success Rate (cumulative)
        type: timeseries
        datasource: prometheus
        gridPos:
          h: 8
          w: 4
          x: 16
          "y": 0
        targets:
          - datasource: prometheus
            expr: "100 * sum(tekton_pipelines_controller_pipelinerun_total{status=\"success\"}) / clamp_min(sum(tekton_pipelines_controller_pipelinerun_total), 1)"
            refId: A
        fieldConfig:
          defaults:
            unit: percent
            color:
              mode: thresholds
            custom:
              drawStyle: line
              fillOpacity: 0
              lineWidth: 1
              spanNulls: false
            thresholds:
              mode: absolute
              steps:
                - color: red
                  value: null
                - color: orange
                  value: 80
                - color: green
                  value: 95
          overrides: []
        options:
          legend:
            calcs:
              - latest
            displayMode: list
            placement: bottom
            showLegend: true
          tooltip:
            mode: multi
            sort: desc
        transformations: []
      - id: 12
        title: Completed PipelineRuns (last 5m)
        type: timeseries
        datasource: prometheus
        gridPos:
          h: 8
          w: 4
          x: 20
          "y": 0
        targets:
          - datasource: prometheus
            expr: "round(sum(increase(tekton_pipelines_controller_pipelinerun_total[5m])))"
            legendFormat: completed
            refId: A
        fieldConfig:
          defaults:
            unit: short
            decimals: 0
            color:
              mode: palette-classic
            custom:
              drawStyle: line
              fillOpacity: 0
              lineWidth: 1
              spanNulls: false
            thresholds:
              mode: absolute
              steps:
                - color: green
                  value: null
          overrides: []
        options:
          legend:
            calcs:
              - latest
            displayMode: list
            placement: bottom
            showLegend: true
          tooltip:
            mode: multi
            sort: desc
        transformations: []
      - id: 4
        title: Running PipelineRuns
        type: timeseries
        datasource: prometheus
        gridPos:
          h: 8
          w: 8
          x: 0
          "y": 8
        targets:
          - datasource: prometheus
            expr: max(tekton_pipelines_controller_running_pipelineruns)
            legendFormat: running
            refId: A
        fieldConfig:
          defaults:
            color:
              mode: palette-classic
            custom:
              drawStyle: line
              fillOpacity: 0
              lineWidth: 1
              spanNulls: false
            thresholds:
              mode: absolute
              steps:
                - color: green
                  value: null
          overrides: []
        options:
          legend:
            calcs:
              - latest
            displayMode: list
            placement: bottom
            showLegend: true
          tooltip:
            mode: multi
            sort: desc
        transformations: []
      - id: 5
        title: Running TaskRuns
        type: timeseries
        datasource: prometheus
        gridPos:
          h: 8
          w: 8
          x: 8
          "y": 8
        targets:
          - datasource: prometheus
            expr: max(tekton_pipelines_controller_running_taskruns)
            legendFormat: running
            refId: A
        fieldConfig:
          defaults:
            color:
              mode: palette-classic
            custom:
              drawStyle: line
              fillOpacity: 0
              lineWidth: 1
              spanNulls: false
            thresholds:
              mode: absolute
              steps:
                - color: green
                  value: null
          overrides: []
        options:
          legend:
            calcs:
              - latest
            displayMode: list
            placement: bottom
            showLegend: true
          tooltip:
            mode: multi
            sort: desc
        transformations: []
      - id: 6
        title: TaskRuns Throttled
        type: timeseries
        datasource: prometheus
        gridPos:
          h: 8
          w: 8
          x: 16
          "y": 8
        targets:
          - datasource: prometheus
            expr: max(tekton_pipelines_controller_running_taskruns_throttled_by_quota)
            legendFormat: by quota
            refId: A
          - datasource: prometheus
            expr: max(tekton_pipelines_controller_running_taskruns_throttled_by_node)
            legendFormat: by node
            refId: B
        fieldConfig:
          defaults:
            color:
              mode: palette-classic
            custom:
              drawStyle: line
              fillOpacity: 0
              lineWidth: 1
              spanNulls: false
            thresholds:
              mode: absolute
              steps:
                - color: orange
                  value: null
          overrides: []
        options:
          legend:
            calcs:
              - latest
            displayMode: list
            placement: bottom
            showLegend: true
          tooltip:
            mode: multi
            sort: desc
        transformations: []
      - id: 7
        title: PipelineRun Duration P50 / P95
        type: timeseries
        datasource: prometheus
        gridPos:
          h: 8
          w: 8
          x: 0
          "y": 16
        targets:
          - datasource: prometheus
            expr: (histogram_quantile(0.5, sum by (le) (rate(tekton_pipelines_controller_pipelinerun_duration_seconds_bucket[5m])))) and on() (sum(rate(tekton_pipelines_controller_pipelinerun_duration_seconds_bucket{le="+Inf"}[5m])) > 0)
            legendFormat: P50
            refId: A
          - datasource: prometheus
            expr: (histogram_quantile(0.95, sum by (le) (rate(tekton_pipelines_controller_pipelinerun_duration_seconds_bucket[5m])))) and on() (sum(rate(tekton_pipelines_controller_pipelinerun_duration_seconds_bucket{le="+Inf"}[5m])) > 0)
            legendFormat: P95
            refId: B
        fieldConfig:
          defaults:
            unit: s
            color:
              mode: palette-classic
            custom:
              drawStyle: line
              fillOpacity: 0
              lineWidth: 1
              spanNulls: false
            thresholds:
              mode: absolute
              steps:
                - color: green
                  value: null
          overrides: []
        options:
          legend:
            calcs:
              - latest
            displayMode: list
            placement: bottom
            showLegend: true
          tooltip:
            mode: multi
            sort: desc
        transformations: []
      - id: 8
        title: TaskRun Duration P50 / P95 (Standalone)
        type: timeseries
        datasource: prometheus
        gridPos:
          h: 8
          w: 8
          x: 8
          "y": 16
        targets:
          - datasource: prometheus
            expr: (histogram_quantile(0.5, sum by (le) (rate(tekton_pipelines_controller_taskrun_duration_seconds_bucket[5m])))) and on() (sum(rate(tekton_pipelines_controller_taskrun_duration_seconds_bucket{le="+Inf"}[5m])) > 0)
            legendFormat: P50
            refId: A
          - datasource: prometheus
            expr: (histogram_quantile(0.95, sum by (le) (rate(tekton_pipelines_controller_taskrun_duration_seconds_bucket[5m])))) and on() (sum(rate(tekton_pipelines_controller_taskrun_duration_seconds_bucket{le="+Inf"}[5m])) > 0)
            legendFormat: P95
            refId: B
        fieldConfig:
          defaults:
            unit: s
            color:
              mode: palette-classic
            custom:
              drawStyle: line
              fillOpacity: 0
              lineWidth: 1
              spanNulls: false
            thresholds:
              mode: absolute
              steps:
                - color: green
                  value: null
          overrides: []
        options:
          legend:
            calcs:
              - latest
            displayMode: list
            placement: bottom
            showLegend: true
          tooltip:
            mode: multi
            sort: desc
        transformations: []
      - id: 13
        title: TaskRun Duration P50 / P95 (In-Pipeline)
        type: timeseries
        datasource: prometheus
        gridPos:
          h: 8
          w: 8
          x: 16
          "y": 16
        targets:
          - datasource: prometheus
            expr: (histogram_quantile(0.5, sum by (le) (rate(tekton_pipelines_controller_pipelinerun_taskrun_duration_seconds_bucket[5m])))) and on() (sum(rate(tekton_pipelines_controller_pipelinerun_taskrun_duration_seconds_bucket{le="+Inf"}[5m])) > 0)
            legendFormat: P50
            refId: A
          - datasource: prometheus
            expr: (histogram_quantile(0.95, sum by (le) (rate(tekton_pipelines_controller_pipelinerun_taskrun_duration_seconds_bucket[5m])))) and on() (sum(rate(tekton_pipelines_controller_pipelinerun_taskrun_duration_seconds_bucket{le="+Inf"}[5m])) > 0)
            legendFormat: P95
            refId: B
        fieldConfig:
          defaults:
            unit: s
            color:
              mode: palette-classic
            custom:
              drawStyle: line
              fillOpacity: 0
              lineWidth: 1
              spanNulls: false
            thresholds:
              mode: absolute
              steps:
                - color: green
                  value: null
          overrides: []
        options:
          legend:
            calcs:
              - latest
            displayMode: list
            placement: bottom
            showLegend: true
          tooltip:
            mode: multi
            sort: desc
        transformations: []
      - id: 9
        title: Workqueue Depth
        type: timeseries
        datasource: prometheus
        gridPos:
          h: 8
          w: 8
          x: 0
          "y": 24
        targets:
          - datasource: prometheus
            expr: max(kn_workqueue_depth{name=~"github.com.tektoncd.pipeline.pkg.reconciler.(pipelinerun|taskrun).Reconciler"})
            legendFormat: depth
            refId: A
        fieldConfig:
          defaults:
            color:
              mode: palette-classic
            custom:
              drawStyle: line
              fillOpacity: 0
              lineWidth: 1
              spanNulls: false
            thresholds:
              mode: absolute
              steps:
                - color: green
                  value: null
          overrides: []
        options:
          legend:
            calcs:
              - latest
            displayMode: list
            placement: bottom
            showLegend: true
          tooltip:
            mode: multi
            sort: desc
        transformations: []
      - id: 10
        title: Workqueue Adds and Retries
        type: timeseries
        datasource: prometheus
        gridPos:
          h: 8
          w: 8
          x: 8
          "y": 24
        targets:
          - datasource: prometheus
            expr: sum(increase(kn_workqueue_adds_total{name=~"github.com.tektoncd.pipeline.pkg.reconciler.(pipelinerun|taskrun).Reconciler"}[5m]))
            legendFormat: adds
            refId: A
          - datasource: prometheus
            expr: sum(increase(kn_workqueue_retries_total{name=~"github.com.tektoncd.pipeline.pkg.reconciler.(pipelinerun|taskrun).Reconciler"}[5m]))
            legendFormat: retries
            refId: B
        fieldConfig:
          defaults:
            color:
              mode: palette-classic
            custom:
              drawStyle: line
              fillOpacity: 0
              lineWidth: 1
              spanNulls: false
            thresholds:
              mode: absolute
              steps:
                - color: green
                  value: null
          overrides: []
        options:
          legend:
            calcs:
              - latest
            displayMode: list
            placement: bottom
            showLegend: true
          tooltip:
            mode: multi
            sort: desc
        transformations: []
      - id: 11
        title: Resolution Waiting
        type: timeseries
        datasource: prometheus
        gridPos:
          h: 8
          w: 8
          x: 16
          "y": 24
        targets:
          - datasource: prometheus
            expr: max(tekton_pipelines_controller_running_pipelineruns_waiting_on_pipeline_resolution)
            legendFormat: PR waiting pipeline
            refId: A
          - datasource: prometheus
            expr: max(tekton_pipelines_controller_running_pipelineruns_waiting_on_task_resolution)
            legendFormat: PR waiting task
            refId: B
          - datasource: prometheus
            expr: max(tekton_pipelines_controller_running_taskruns_waiting_on_task_resolution_count)
            legendFormat: TR waiting task
            refId: C
        fieldConfig:
          defaults:
            color:
              mode: palette-classic
            custom:
              drawStyle: line
              fillOpacity: 0
              lineWidth: 1
              spanNulls: false
            thresholds:
              mode: absolute
              steps:
                - color: orange
                  value: null
          overrides: []
        options:
          legend:
            calcs:
              - latest
            displayMode: list
            placement: bottom
            showLegend: true
          tooltip:
            mode: multi
            sort: desc
        transformations: []

Tekton Pipeline Dashboard Interpretation (Common Questions)

  • PipelineRun Total (by status) is a completion-event counter recorded by the controller, not the total number of PipelineRun objects. PipelineRuns that end with reason Cancelled, including those cancelled with spec.status: Cancelled, are counted with status="cancelled". A PipelineRun cancelled or stopped with spec.status: CancelledRunFinally or StoppedRunFinally in which a task or finally task failed ends with reason Failed and is counted with status="failed".
  • Running PipelineRuns is a real-time snapshot (how many are running now). It can change independently from PipelineRun Total.
  • Completed PipelineRuns (last 5m) is throughput (newly completed runs in the last 5 minutes). Seeing 0 during low traffic or idle periods is expected.
  • PipelineRun Success Rate (cumulative) is cumulative since controller start, not a 5-minute window success rate. A short-term failure does not immediately cause a large shift.
  • Workqueue Adds and Retries counts the items added to and retried in the PipelineRun and TaskRun reconciler work queues in the last 5 minutes, not PipelineRun counts.
  • Status series are shown only for label values that actually have samples in the selected time range. If a status has no samples in the window, its curve/legend will not appear.
  • TaskRun Duration P50 / P95 (Standalone) and TaskRun Duration P50 / P95 (In-Pipeline) are split to avoid mixed-query instability. In environments that only expose one histogram family, the other panel may be empty, which is expected.

Tekton Triggers Dashboard

Tekton Triggers Dashboard YAML
kind: MonitorDashboard
apiVersion: ait.alauda.io/v1alpha2
metadata:
  labels:
    cpaas.io/dashboard.folder: tekton
    cpaas.io/dashboard.is.home.dashboard: "false"
    cpaas.io/dashboard.tag.tekton: "true"
  name: tekton-triggers
  namespace: cpaas-system
spec:
  body:
    title: Tekton Triggers Overview
    titleZh: Tekton Triggers Overview
    tags:
      - tekton
    time:
      from: now-1h
      to: now
    templating:
      list: []
    panels:
      - id: 1
        title: EventListener Count
        type: timeseries
        datasource: prometheus
        gridPos:
          h: 6
          w: 5
          x: 0
          "y": 0
        targets:
          - datasource: prometheus
            expr: controller_eventlistener_count
            legendFormat: EventListener
            refId: A
        fieldConfig:
          defaults:
            color:
              mode: palette-classic
            custom:
              drawStyle: line
              fillOpacity: 0
              lineWidth: 1
              spanNulls: false
            thresholds:
              mode: absolute
              steps:
                - color: green
                  value: null
          overrides: []
        options:
          legend:
            calcs:
              - latest
            displayMode: list
            placement: bottom
            showLegend: true
          tooltip:
            mode: multi
            sort: desc
        transformations: []
      - id: 2
        title: TriggerTemplate Count
        type: timeseries
        datasource: prometheus
        gridPos:
          h: 6
          w: 5
          x: 5
          "y": 0
        targets:
          - datasource: prometheus
            expr: controller_triggertemplate_count
            legendFormat: TriggerTemplate
            refId: A
        fieldConfig:
          defaults:
            color:
              mode: palette-classic
            custom:
              drawStyle: line
              fillOpacity: 0
              lineWidth: 1
              spanNulls: false
            thresholds:
              mode: absolute
              steps:
                - color: green
                  value: null
          overrides: []
        options:
          legend:
            calcs:
              - latest
            displayMode: list
            placement: bottom
            showLegend: true
          tooltip:
            mode: multi
            sort: desc
        transformations: []
      - id: 3
        title: TriggerBinding Count
        type: timeseries
        datasource: prometheus
        gridPos:
          h: 6
          w: 5
          x: 10
          "y": 0
        targets:
          - datasource: prometheus
            expr: controller_triggerbinding_count
            legendFormat: TriggerBinding
            refId: A
        fieldConfig:
          defaults:
            color:
              mode: palette-classic
            custom:
              drawStyle: line
              fillOpacity: 0
              lineWidth: 1
              spanNulls: false
            thresholds:
              mode: absolute
              steps:
                - color: green
                  value: null
          overrides: []
        options:
          legend:
            calcs:
              - latest
            displayMode: list
            placement: bottom
            showLegend: true
          tooltip:
            mode: multi
            sort: desc
        transformations: []
      - id: 4
        title: ClusterTriggerBinding
        type: timeseries
        datasource: prometheus
        gridPos:
          h: 6
          w: 5
          x: 15
          "y": 0
        targets:
          - datasource: prometheus
            expr: controller_clustertriggerbinding_count
            legendFormat: ClusterTriggerBinding
            refId: A
        fieldConfig:
          defaults:
            color:
              mode: palette-classic
            custom:
              drawStyle: line
              fillOpacity: 0
              lineWidth: 1
              spanNulls: false
            thresholds:
              mode: absolute
              steps:
                - color: green
                  value: null
          overrides: []
        options:
          legend:
            calcs:
              - latest
            displayMode: list
            placement: bottom
            showLegend: true
          tooltip:
            mode: multi
            sort: desc
        transformations: []
      - id: 5
        title: ClusterInterceptor
        type: timeseries
        datasource: prometheus
        gridPos:
          h: 6
          w: 4
          x: 20
          "y": 0
        targets:
          - datasource: prometheus
            expr: controller_clusterinterceptor_count
            legendFormat: ClusterInterceptor
            refId: A
        fieldConfig:
          defaults:
            color:
              mode: palette-classic
            custom:
              drawStyle: line
              fillOpacity: 0
              lineWidth: 1
              spanNulls: false
            thresholds:
              mode: absolute
              steps:
                - color: green
                  value: null
          overrides: []
        options:
          legend:
            calcs:
              - latest
            displayMode: list
            placement: bottom
            showLegend: true
          tooltip:
            mode: multi
            sort: desc
        transformations: []
      - id: 6
        title: All Trigger Resource Counts (trend)
        type: timeseries
        datasource: prometheus
        gridPos:
          h: 8
          w: 24
          x: 0
          "y": 6
        targets:
          - datasource: prometheus
            expr: controller_eventlistener_count
            legendFormat: EventListener
            refId: A
          - datasource: prometheus
            expr: controller_triggertemplate_count
            legendFormat: TriggerTemplate
            refId: B
          - datasource: prometheus
            expr: controller_triggerbinding_count
            legendFormat: TriggerBinding
            refId: C
          - datasource: prometheus
            expr: controller_clustertriggerbinding_count
            legendFormat: ClusterTriggerBinding
            refId: D
          - datasource: prometheus
            expr: controller_clusterinterceptor_count
            legendFormat: ClusterInterceptor
            refId: E
        fieldConfig:
          defaults:
            color:
              mode: palette-classic
            custom:
              drawStyle: line
              fillOpacity: 0
              lineWidth: 1
              spanNulls: false
            thresholds:
              mode: absolute
              steps:
                - color: green
                  value: null
          overrides: []
        options:
          legend:
            calcs:
              - latest
            displayMode: list
            placement: bottom
            showLegend: true
          tooltip:
            mode: multi
            sort: desc
        transformations: []

Tekton Triggers Dashboard Interpretation (Common Questions)

  • EventListener Count, TriggerTemplate Count, TriggerBinding Count, ClusterTriggerBinding, and ClusterInterceptor are object-count snapshots, not request volume or event-processing throughput.
  • All Trigger Resource Counts (trend) shows the combined trend for the same resource counts. Short deviations versus the single-resource trend panels within a scrape interval are expected.
  • Showing 0 when no Triggers resources exist is normal and does not indicate a scraping failure.

Tekton Results Dashboard

Tekton Results Dashboard YAML
kind: MonitorDashboard
apiVersion: ait.alauda.io/v1alpha2
metadata:
  labels:
    cpaas.io/dashboard.folder: tekton
    cpaas.io/dashboard.is.home.dashboard: "false"
    cpaas.io/dashboard.tag.tekton: "true"
  name: tekton-results
  namespace: cpaas-system
spec:
  body:
    title: Tekton Results Overview
    titleZh: Tekton Results Overview
    tags:
      - tekton
    time:
      from: now-1h
      to: now
    templating:
      list: []
    panels:
      - id: 1
        title: PipelineRun Reconciles and Retries (last 5m)
        type: timeseries
        datasource: prometheus
        gridPos:
          h: 8
          w: 12
          x: 0
          "y": 0
        targets:
          - datasource: prometheus
            expr: round(sum(increase(kn_workqueue_process_duration_seconds_count{name="github.com.tektoncd.results.pkg.watcher.reconciler.pipelinerun.Reconciler"}[5m])))
            legendFormat: processed
            refId: A
          - datasource: prometheus
            expr: round(sum(increase(kn_workqueue_retries_total{name="github.com.tektoncd.results.pkg.watcher.reconciler.pipelinerun.Reconciler"}[5m])))
            legendFormat: retries
            refId: B
        fieldConfig:
          defaults:
            color:
              mode: palette-classic
            custom:
              drawStyle: line
              fillOpacity: 0
              lineWidth: 1
              spanNulls: false
            thresholds:
              mode: absolute
              steps:
                - color: green
                  value: null
          overrides: []
        options:
          legend:
            calcs:
              - latest
            displayMode: list
            placement: bottom
            showLegend: true
          tooltip:
            mode: multi
            sort: desc
        transformations: []
      - id: 2
        title: TaskRun Reconciles and Retries (last 5m)
        type: timeseries
        datasource: prometheus
        gridPos:
          h: 8
          w: 12
          x: 12
          "y": 0
        targets:
          - datasource: prometheus
            expr: round(sum(increase(kn_workqueue_process_duration_seconds_count{name="github.com.tektoncd.results.pkg.watcher.reconciler.taskrun.Reconciler"}[5m])))
            legendFormat: processed
            refId: A
          - datasource: prometheus
            expr: round(sum(increase(kn_workqueue_retries_total{name="github.com.tektoncd.results.pkg.watcher.reconciler.taskrun.Reconciler"}[5m])))
            legendFormat: retries
            refId: B
        fieldConfig:
          defaults:
            color:
              mode: palette-classic
            custom:
              drawStyle: line
              fillOpacity: 0
              lineWidth: 1
              spanNulls: false
            thresholds:
              mode: absolute
              steps:
                - color: green
                  value: null
          overrides: []
        options:
          legend:
            calcs:
              - latest
            displayMode: list
            placement: bottom
            showLegend: true
          tooltip:
            mode: multi
            sort: desc
        transformations: []
      - id: 3
        title: PipelineRun Reconcile Latency P95
        type: timeseries
        datasource: prometheus
        gridPos:
          h: 8
          w: 12
          x: 0
          "y": 8
        targets:
          - datasource: prometheus
            expr: histogram_quantile(0.95, sum by (le) (rate(kn_workqueue_process_duration_seconds_bucket{name="github.com.tektoncd.results.pkg.watcher.reconciler.pipelinerun.Reconciler"}[5m])))
            legendFormat: P95
            refId: A
        fieldConfig:
          defaults:
            unit: s
            color:
              mode: palette-classic
            custom:
              drawStyle: line
              fillOpacity: 0
              lineWidth: 1
              spanNulls: false
            thresholds:
              mode: absolute
              steps:
                - color: green
                  value: null
          overrides: []
        options:
          legend:
            calcs:
              - latest
            displayMode: list
            placement: bottom
            showLegend: true
          tooltip:
            mode: multi
            sort: desc
        transformations: []
      - id: 4
        title: TaskRun Reconcile Latency P95
        type: timeseries
        datasource: prometheus
        gridPos:
          h: 8
          w: 12
          x: 12
          "y": 8
        targets:
          - datasource: prometheus
            expr: histogram_quantile(0.95, sum by (le) (rate(kn_workqueue_process_duration_seconds_bucket{name="github.com.tektoncd.results.pkg.watcher.reconciler.taskrun.Reconciler"}[5m])))
            legendFormat: P95
            refId: A
        fieldConfig:
          defaults:
            unit: s
            color:
              mode: palette-classic
            custom:
              drawStyle: line
              fillOpacity: 0
              lineWidth: 1
              spanNulls: false
            thresholds:
              mode: absolute
              steps:
                - color: green
                  value: null
          overrides: []
        options:
          legend:
            calcs:
              - latest
            displayMode: list
            placement: bottom
            showLegend: true
          tooltip:
            mode: multi
            sort: desc
        transformations: []
      - id: 5
        title: Workqueue Depth (PipelineRun vs TaskRun)
        type: timeseries
        datasource: prometheus
        gridPos:
          h: 8
          w: 12
          x: 0
          "y": 16
        targets:
          - datasource: prometheus
            expr: sum(kn_workqueue_depth{name="github.com.tektoncd.results.pkg.watcher.reconciler.pipelinerun.Reconciler"})
            legendFormat: pipelinerun
            refId: A
          - datasource: prometheus
            expr: sum(kn_workqueue_depth{name="github.com.tektoncd.results.pkg.watcher.reconciler.taskrun.Reconciler"})
            legendFormat: taskrun
            refId: B
        fieldConfig:
          defaults:
            color:
              mode: palette-classic
            custom:
              drawStyle: line
              fillOpacity: 0
              lineWidth: 1
              spanNulls: false
            thresholds:
              mode: absolute
              steps:
                - color: green
                  value: null
          overrides: []
        options:
          legend:
            calcs:
              - latest
            displayMode: list
            placement: bottom
            showLegend: true
          tooltip:
            mode: multi
            sort: desc
        transformations: []
      - id: 6
        title: Workqueue Adds (last 5m)
        type: timeseries
        datasource: prometheus
        gridPos:
          h: 8
          w: 12
          x: 12
          "y": 16
        targets:
          - datasource: prometheus
            expr: round(sum(increase(kn_workqueue_adds_total{name="github.com.tektoncd.results.pkg.watcher.reconciler.pipelinerun.Reconciler"}[5m])))
            legendFormat: pipelinerun adds
            refId: A
          - datasource: prometheus
            expr: round(sum(increase(kn_workqueue_adds_total{name="github.com.tektoncd.results.pkg.watcher.reconciler.taskrun.Reconciler"}[5m])))
            legendFormat: taskrun adds
            refId: B
        fieldConfig:
          defaults:
            color:
              mode: palette-classic
            custom:
              drawStyle: line
              fillOpacity: 0
              lineWidth: 1
              spanNulls: false
            thresholds:
              mode: absolute
              steps:
                - color: green
                  value: null
          overrides: []
        options:
          legend:
            calcs:
              - latest
            displayMode: list
            placement: bottom
            showLegend: true
          tooltip:
            mode: multi
            sort: desc
        transformations: []
      - id: 7
        title: gRPC Request Rate (Results API)
        type: timeseries
        datasource: prometheus
        gridPos:
          h: 8
          w: 12
          x: 0
          "y": 24
        targets:
          - datasource: prometheus
            expr: "sum(rate(grpc_server_handled_total{grpc_service=~\"tekton.results.*\"}[5m]))"
            legendFormat: requests
            refId: A
        fieldConfig:
          defaults:
            color:
              mode: palette-classic
            custom:
              drawStyle: line
              fillOpacity: 0
              lineWidth: 1
              spanNulls: false
            thresholds:
              mode: absolute
              steps:
                - color: green
                  value: null
          overrides: []
        options:
          legend:
            calcs:
              - latest
            displayMode: list
            placement: bottom
            showLegend: true
          tooltip:
            mode: multi
            sort: desc
        transformations: []
      - id: 8
        title: gRPC Error Percentage (Results API, excl. NotFound/Canceled)
        type: timeseries
        datasource: prometheus
        gridPos:
          h: 8
          w: 12
          x: 12
          "y": 24
        targets:
          - datasource: prometheus
            expr: "100 * ((sum(rate(grpc_server_handled_total{grpc_service=~\"tekton.results.*\",grpc_code!~\"OK|NotFound|Canceled\"}[5m])) or vector(0)) / clamp_min((sum(rate(grpc_server_handled_total{grpc_service=~\"tekton.results.*\"}[5m])) or vector(0)), 0.001))"
            legendFormat: error %
            refId: A
        fieldConfig:
          defaults:
            unit: percent
            color:
              mode: palette-classic
            custom:
              drawStyle: line
              fillOpacity: 0
              lineWidth: 1
              spanNulls: false
            thresholds:
              mode: absolute
              steps:
                - color: red
                  value: null
          overrides: []
        options:
          legend:
            calcs:
              - latest
            displayMode: list
            placement: bottom
            showLegend: true
          tooltip:
            mode: multi
            sort: desc
        transformations: []

Tekton Results Dashboard Interpretation (Common Questions)

  • This dashboard revision is based on the Results Watcher workqueue metrics (kn_workqueue_*, one series per reconciler in the name label) plus Results API gRPC metrics, so it stays populated under common deployments (logs_api=true, automatic deletion disabled).
  • PipelineRun Reconciles and Retries (last 5m) and TaskRun Reconciles and Retries (last 5m) show, per 5-minute window, the number of items the reconciler processed (kn_workqueue_process_duration_seconds_count) and the number of items re-added to the queue for another attempt (kn_workqueue_retries_total).
  • PipelineRun Reconcile Latency P95 and TaskRun Reconcile Latency P95 are calculated from the watcher workqueue processing-time histogram (kn_workqueue_process_duration_seconds, in seconds). Under low traffic, the line can be sparse.
  • Workqueue Depth shows current queue depth, and Workqueue Adds (last 5m) shows enqueue volume over the last 5 minutes.
  • gRPC Error Percentage (Results API, excl. NotFound/Canceled) is the percentage of abnormal errors over total requests, excluding common business return codes (NotFound, Canceled).

Tekton Chains Dashboard

Tekton Chains Dashboard YAML
kind: MonitorDashboard
apiVersion: ait.alauda.io/v1alpha2
metadata:
  labels:
    cpaas.io/dashboard.folder: tekton
    cpaas.io/dashboard.is.home.dashboard: "false"
    cpaas.io/dashboard.tag.tekton: "true"
  name: tekton-chains
  namespace: cpaas-system
spec:
  body:
    title: Tekton Chains Overview
    titleZh: Tekton Chains Overview
    tags:
      - tekton
    time:
      from: now-1h
      to: now
    templating:
      list: []
    panels:
      - id: 1
        title: TaskRun Signatures Created (last 5m)
        type: timeseries
        datasource: prometheus
        gridPos:
          h: 8
          w: 12
          x: 0
          "y": 0
        targets:
          - datasource: prometheus
            expr: round(increase(watcher_taskrun_sign_created_total[5m]))
            legendFormat: sign created
            refId: A
        fieldConfig:
          defaults:
            color:
              mode: palette-classic
            custom:
              drawStyle: line
              fillOpacity: 0
              lineWidth: 1
              spanNulls: false
            thresholds:
              mode: absolute
              steps:
                - color: green
                  value: null
          overrides: []
        options:
          legend:
            calcs:
              - latest
            displayMode: list
            placement: bottom
            showLegend: true
          tooltip:
            mode: multi
            sort: desc
        transformations: []
      - id: 2
        title: PipelineRun Signatures Created (last 5m)
        type: timeseries
        datasource: prometheus
        gridPos:
          h: 8
          w: 12
          x: 12
          "y": 0
        targets:
          - datasource: prometheus
            expr: round(increase(watcher_pipelinerun_sign_created_total[5m]))
            legendFormat: sign created
            refId: A
        fieldConfig:
          defaults:
            color:
              mode: palette-classic
            custom:
              drawStyle: line
              fillOpacity: 0
              lineWidth: 1
              spanNulls: false
            thresholds:
              mode: absolute
              steps:
                - color: green
                  value: null
          overrides: []
        options:
          legend:
            calcs:
              - latest
            displayMode: list
            placement: bottom
            showLegend: true
          tooltip:
            mode: multi
            sort: desc
        transformations: []
      - id: 3
        title: Payloads Stored (last 5m, TaskRun vs PipelineRun)
        type: timeseries
        datasource: prometheus
        gridPos:
          h: 8
          w: 12
          x: 0
          "y": 8
        targets:
          - datasource: prometheus
            expr: round(increase(watcher_taskrun_payload_stored_total[5m]))
            legendFormat: TaskRun
            refId: A
          - datasource: prometheus
            expr: round(increase(watcher_pipelinerun_payload_stored_total[5m]))
            legendFormat: PipelineRun
            refId: B
        fieldConfig:
          defaults:
            color:
              mode: palette-classic
            custom:
              drawStyle: line
              fillOpacity: 0
              lineWidth: 1
              spanNulls: false
            thresholds:
              mode: absolute
              steps:
                - color: green
                  value: null
          overrides: []
        options:
          legend:
            calcs:
              - latest
            displayMode: list
            placement: bottom
            showLegend: true
          tooltip:
            mode: multi
            sort: desc
        transformations: []
      - id: 4
        title: Marked Signed (last 5m, TaskRun vs PipelineRun)
        type: timeseries
        datasource: prometheus
        gridPos:
          h: 8
          w: 12
          x: 12
          "y": 8
        targets:
          - datasource: prometheus
            expr: round(increase(watcher_taskrun_marked_signed_total[5m]))
            legendFormat: TaskRun
            refId: A
          - datasource: prometheus
            expr: round(increase(watcher_pipelinerun_marked_signed_total[5m]))
            legendFormat: PipelineRun
            refId: B
        fieldConfig:
          defaults:
            color:
              mode: palette-classic
            custom:
              drawStyle: line
              fillOpacity: 0
              lineWidth: 1
              spanNulls: false
            thresholds:
              mode: absolute
              steps:
                - color: green
                  value: null
          overrides: []
        options:
          legend:
            calcs:
              - latest
            displayMode: list
            placement: bottom
            showLegend: true
          tooltip:
            mode: multi
            sort: desc
        transformations: []

Tekton Chains Dashboard Interpretation (Common Questions)

  • TaskRun Signatures Created (last 5m), PipelineRun Signatures Created (last 5m), Payloads Stored (last 5m), and Marked Signed (last 5m) use increase(...[5m]) and represent increments in the last five minutes.
  • When there is no new signing or storage activity, these lines drop to 0; this does not imply a component fault. A panel whose counter was never incremented since the Chains controller started shows no data instead, because the counter is not in the metrics output yet.
  • Payloads Stored and Marked Signed represent different processing stages, so their values are not expected to always match. Payloads Stored also counts a payload for which the oci storage backend found no image subject and uploaded nothing to the registry (see the Prerequisites in Signing Key Configuration).
  • If Marked Signed grows while TaskRun Signatures Created and PipelineRun Signatures Created show no data (or stay at 0, when the controller created signatures earlier), and *_signing_failures_total does not grow, Chains marks runs as signed without signing them (see the note under Chains Metrics). This happens when no signer is configured (by default: signing-secrets holds no key), or when signing of TaskRun and PipelineRun artifacts is disabled (for example with artifacts.taskrun.storage: "" and artifacts.pipelinerun.storage: "", see Chains Configuration). No dashboard panel shows *_signing_failures_total; query it in Prometheus or in the Chains metrics output. When it grows in this situation, Chains failed to create, encode or sign a payload (error_type payload_creation, marshal_payload or signing) and still marked the run as signed.