Building a Python pipeline using template

This pipeline is designed to automate a full CI/CD workflow for Python applications. It is reusable and flexible, supporting custom build arguments, multiple workspaces, and conditional execution of steps.

Key Features:

✅ Flexible code source
🔌 Pluggable tasks via params
🛠️ Dynamic image build and push
☁️ Workload image update via kubectl

Specifying it using hub resolvers

The following example pipeline run references pipeline from the catalog:

apiVersion: tekton.dev/v1
kind: PipelineRun
metadata:
  name: python-pipeline-run
spec:
  pipelineRef:
    resolver: hub
    params:
    - name: catalog
      value: catalog
    - name: kind
      value: pipeline
    - name: name
      value: python-image-build-scan-deploy
    - name: version
      value: "0.1"

Parameters

Git Clone

git-url
- type: string
- default: ""
- description: Repository URL to clone from.
git-revision
- type: string
- default: ""
- description: Revision to checkout (branch, tag, sha, ref, etc...).
skip-git-clone
- type: string
- default: "false"
- description: Skip to execute the git clone task. It can only be enabled if the corresponding code repository already exists in the workspace.
git-crt-file-name
- type: string
- default: "ca-bundle.crt"
- description: File name of mounted crt using ssl-ca-directory workspace. default value is ca-bundle.crt.

Pre build

pre-build-script
- type: string
- default: ""
- description: The python script to be executed before build. It can be used to unit test, lint, run script before build, etc. pre-build task will be skipped if this param is not set.
pre-build-args
- type: array
- default: []
- description: The arguments passed to the pre-build script.
pre-build-requirements-file
- type: string
- default: "requirements.txt"
- description: The name of the requirements file inside the source location, with fallback to the requirements file in the root location
pre-build-pip-conf-file
- type: string
- default: "pip.conf"
- description: The name of the custom pip config file.

SonarQube Scanner

sonar-url
- type: string
- default: ""
- description: The URL to your SonarQube Server instance. If not specified, sonarqube scanner task will be skipped.
sonar-project-key
- type: string
- default: ""
- description: Unique key of the Sonarqube project.
sonar-properties
- type: array
- default: ["sonar.sources=."]
- description: Additional properties to pass to SonarQube. You can refer to Configuration of SonarQube properties for more details.

Build Images

images
- type: array
- description: Reference of the images buildah will produce. Can contain multiple image addresses separated by commas. For example:
  - busybox:latest
  - busybox:v1 .30.1
dockerfile-path
- type: string
- default: ./Dockerfile
- description: Path to the Dockerfile to build.
build-extra-args
- type: string
- default: ""
- description: Extra parameters passed for the build command when building images. WARNING - must be sanitized to avoid command injection (e.g. --build-arg key=value --label key=value).
build-args
- type: array
- default: [""]
- description: Specifies a build argument and its value, which will be interpolated in instructions read from the Dockerfile in the same way that environment variables are, but which will not be added to environment variable list in the resulting image's configuration. eg. HTTP_PROXY=http://10.10.10.10:8080
build-context
- type: string
- default: .
- description: Path to the directory to use as context.
push-extra-args
- type: string
- default: ""
- description: Extra parameters passed for the push command when pushing images. WARNING - must be sanitized to avoid command injection (e.g. --creds=username:password ).

Trivy Scanner

skip-trivy-scan
- type: string
- default: "false"
- description: Set to skip trivy image scan after build.
trivy-extra-args
- type: string
- default: ""
- description: Extra parameters passed for the trivy command when scanning images. WARNING - must be sanitized to avoid command injection (e.g. --insecure). You can use --skip-db-update --skip-java-db-update to skip updating vulnerability database.

Deploy or upgrade workload

workload-name
- type: string
- default: ""
- description: The name of the workload to deploy or upgrade. If it is not specified, deploy-or-upgrade task will be skipped.
workload-kind
- type: string
- default: Deployment
- description: The kind of workload to deploy or upgrade, such as Deployment, StatefulSet, etc.
workload-namespace
- type: string
- default: ""
- description: The namespace of the workload to deploy or upgrade. If not specified, will use the namespace of the current TaskRun.
workload-container
- type: string
- default: []
- description: This parameter is used to specify the containers within the workload whose image needs to be updated. By default, all of the container's image in the workload will be updated.
workload-rollout-timeout
- type: string
- default: "0"
- description: The length of time to wait before ending waiting workload ready, 0 means never. Any other values should contain a corresponding time unit (e.g. 1s, 2m, 3h).
workload-manifests-dir
- type: string
- default: ""
- description: The manifest of the workload to deploy. If it is not specified, will only update the image of the workload which is already in cluster. You can use the relative path to the manifest directory in the source workspace. For example: "manifests".

Workspaces

source: Workspace to share information between tasks.
git-basic-auth: An optional workspace containing a .gitconfig and .git-credentials file. These will be copied to the user's home before any git commands are run. Any other files in this Workspace are ignored. It is strongly recommended to use ssh-directory over basic-auth whenever possible and to bind a Secret to this Workspace over other volume types.
git-ssh-directory: An optional workspace with private key, known_hosts, config, etc. Copied to the user's home before git commands are executed. Used to authenticate with the git remote when performing the clone. Binding a Secret to this Workspace is strongly recommended over other volume types.
git-ssl-ca-directory: An optional workspace containing CA certificates, this will be used by Git to verify the peer with when fetching or pushing over HTTPS.
pip-conf: An optional workspace consisting of the custom pip settings.
sonar-settings: An optional workspace where SonarQube properties can be mounted.
sonar-credentials: An optional workspace containing credentials for use within SonarQube.
dockerconfig: An optional workspace for distribution registry configuration files, such as config.json or .dockerconfigjson. This is optional and is used for authentication when pushing images to the registry.
kubeconfig: An Optional workspace containing kubeconfig file. The name of kubeconfig should be kubeconfig. If there is no kubeconfig in this workspace, the workspace will be ignored.

Platforms

The Task can be run on linux/amd64 and linux/arm64 platforms.

Usage

Minimal Setup: Run End-to-End With Minimal Parameters

Optional tasks (sonarqube-scanner, deploy-or-upgrade, etc.) will be gracefully skipped when unset or omitted.

apiVersion: tekton.dev/v1
kind: PipelineRun
metadata:
  generateName: python-build-run-
spec:
  pipelineRef:
    name: python-image-build-scan-deploy
  params:
    - name: git-url
      value: https://github.com/example/python-app
    - name: git-revision
      value: refs/heads/main
    - name: images
      value: ["registry.example.com/python-app:latest"]
  workspaces:
    - name: source
      persistentVolumeClaim:
        claimName: source
    - name: git-basic-auth
      secret:
        secretName: gitconfig
    - name: dockerconfig
      secret:
        secretName: dockerconfig

Configuration of SonarQube properties

You can mount sonar-settings workspace or use sonar-properties param to apply the properties for sonar-project.properties file.

The default properties are: sonar.sources=..

You can find the usage of SonarQube properties from analysis parameters, and the properties for Python

Here is some of the properties you can set:

Properties	Description
sonar.python.version	Python code is analyzed by default as compatible with python 2 and python 3. In order to get a more precise analysis you can specify the Python versions your code supports via the sonar.python.version parameter.
sonar.qualitygate.wait	Forces the analysis step to poll the server instance and wait for the Quality Gate status.
sonar.qualitygate.timeout	The number of seconds that the scanner should wait for a report to be processed.
sonar.projectVersion	The project version. It should be set for branch analysis in case you use the new code definition based on the previous version.
sonar.sources	Comma-separated paths to directories containing main source files
sonar.organization	The organization in SonarQube where the project exists

Deploy or upgrade a workload

You can deploy or upgrade a workload using the image built by the build-image task in this pipeline.

By setting the workload-manifests-dir param, you can apply Kubernetes manifests from the specified directory to create or update the workload.

Both JSON and YAML formats are supported. All manifests within the specified directory will be processed using kubectl apply. Note that kubectl apply does not traverse subdirectories.

If the directory contains a Kustomize configuration file (e.g. kustomization.yaml, kustomization.yml, or Kustomization), it will be processed using kubectl kustomize.

Here is an example structure of the workload-manifests-dir:

manifests
├── deployment.yaml
├── kustomization.yaml
└── service.yaml

You can configure the following parameters:

spec:
  params:
    # ...
    - name: images
      value:
        - foo/bar:latest
    - name: workload-name
      value: bar
    - name: workload-namespace
      value: default
    - name: workload-kind
      value: Deployment
    - name: workload-manifests-dir
      value: manifests
    # ...

If you prefer not to deploy the workload using manifests. Simply leave the workload-manifests-dir param empty. In that case, the task will only update the image of an existing workload in the cluster.

You need to manage image pull secrets for the workload. If the new image is hosted in a different registry, make sure to create and configure the appropriate pull secret for your workload.

Run pytest

You can run pytest using the pre-build task in this pipeline.

Here is an example of how to run pytest:

apiVersion: tekton.dev/v1
kind: PipelineRun
metadata:
  generateName: python-build-run-
spec:
  pipelineRef:
    name: python-image-build-scan-deploy
  params:
    - name: git-url
      value: https://github.com/example/python-app
    - name: git-revision
      value: refs/heads/main
    - name: images
      value: ["registry.example.com/python-app:latest"]
    - name: pre-build-script
      value: |
        pytest
  workspaces:
    - name: source
      persistentVolumeClaim:
        claimName: source
    - name: git-basic-auth
      secret:
        secretName: gitconfig
    - name: dockerconfig
      secret:
        secretName: dockerconfig

Skip tasks with Params

The pipeline supports skipping tasks using params:

Task	Param	Default	Description
git-clone	`skip-git-clone`	`"false"`	If code already exists in `source` workspace, you can set this to `"true"` to skip cloning.
pre-build	`pre-build-script`	`""`	Pre-build Task will be skipped if `pre-build-script` is not set.
sonarqube-scanner	`sonar-url`	`""`	Scan will be skipped if `sonar-url` is not set.
trivy-scanner	`skip-trivy-scan`	`"false"`	Scan will be skipped if `skip-trivy-scan` is set to `"true"`.
deploy-or-upgrade	`workload-name`	`""`	If you don't want to deploy or upgrade the workload, you can left `workload-name` empty.

Full Example with All Tasks

This example demonstrates how to use all tasks in the pipeline. It includes following tasks:

Cloning the source code
Running the unit tests with pre-build task.
Building the image and pushing the image to the registry.
Scanning the image with Trivy
Scanning the code with SonarQube
Deploying or Upgrading the workload

apiVersion: tekton.dev/v1
kind: PipelineRun
metadata:
  generateName: python-build-run-
spec:
  pipelineRef:
    name: python-image-build-scan-deploy
  params:
    - name: git-url
      value: https://github.com/example/python-app
    - name: git-revision
      value: refs/heads/main
    - name: sonar-url
      value: https://sonar.example.com
    - name: sonar-project-key
      value: python-pipeline-example-key
    - name: images
      value: ["registry.example.com/python-app:latest"]
    - name: workload-name
      value: python-app
    - name: workload-namespace
      value: default
    - name: pre-build-script
      value: |
        pytest
  workspaces:
    - name: source
      persistentVolumeClaim:
        claimName: source
    - name: git-basic-auth
      secret:
        secretName: gitconfig
    - name: sonar-credentials
      secret:
        secretName: sonar-credentials
    - name: dockerconfig
      secret:
        secretName: registry-creds
    - name: kubeconfig
      configMap:
        name: kubeconfig

#Building a Python pipeline using template

#TOC

#Specifying it using hub resolvers

#Parameters

#Git Clone

#Pre build

#SonarQube Scanner

#Build Images

#Trivy Scanner

#Deploy or upgrade workload

#Workspaces

#Platforms

#Usage

#Minimal Setup: Run End-to-End With Minimal Parameters

#Configuration of SonarQube properties

#Deploy or upgrade a workload

#Run pytest

#Skip tasks with Params

#Full Example with All Tasks