Failed to create pod due to config error when using custom images in Tekton
TOC
Problem DescriptionError ManifestationRoot Cause AnalysisTroubleshootingSolutionOption 1: Adjust Image Build Configuration to Set the Default User to a Non-root UserPrerequisitesStepsOption 2: Modify TaskRun or PipelineRun Execution ConfigurationPrerequisitesStepsOption 3: Modify Global Tekton ConfigurationPrerequisitesStepsOption 4: Modify Task DefinitionPrerequisitesStepsPrevent ErrorsRelated ContentProblem Description
In the Tekton pipeline, using images provided by the product works correctly, but when using user-defined images, you may encounter TaskRun execution failures.
Error Manifestation
-
TaskRun execution fails with a status of
False, and the reason isCreateContainerConfigError: -
The TaskRun event displays an error message:
-
Relevant pod events show an error message that contains:
Root Cause Analysis
Such issues are typically caused by the following two reasons:
- The image itself has issues.
- The image is incompatible with the Task configuration.
Troubleshooting
If this issue only appears when using custom images, it is recommended to follow these steps for troubleshooting:
-
Verify if the image itself has issues. Run it with a container CLI such as
podman, ordocker, which takes the same arguments, on a workstation that can pull the image. A step'scommandreplaces the ENTRYPOINT of the image, so the check passes the step's command with--entrypoint. Replace<image>with the image and<command>with the first element of the step'scommand. Add the other elements ofcommandand the step'sargsafter<image>:Do not try to reproduce a step with
scriptthis way: Tekton runs the script file with the step'sargs. For such a step, check only that the image has the interpreter of the script's first line, given with its full path. Tekton adds a#!/bin/shline to a script that has no#!line. For/bin/sh, the command below exits with status0when the image has/bin/sh; otherwise the container does not start, and the container CLI prints an error and exits with a non-zero status.-c trueis an option of/bin/sh; other interpreters may not accept it:For a first line
#!/usr/bin/env <name>,/usr/bin/envlooks up<name>in thePATHof the step container (the image'sPATHunless anenvsetting overrides it). If the interpreter accepts--version, check it with:For a step without
commandorscript, leave out--entrypoint <command>and add the step'sargsafter<image>. When the image has an ENTRYPOINT, this runs what the step runs: the ENTRYPOINT with the step'sargs, or with the CMD of the image when the step has noargs. -
Check the compatibility of the Task configuration with the image:
- Check if the Task is configured with
runAsNonRoot: true. - Check whether the default user of the image is root or a non-numeric user ID.
- Check if the Task is configured with
Example Task configuration:
Example Containerfile configuration:
Solution
Option 1: Adjust Image Build Configuration to Set the Default User to a Non-root User
Prerequisites
- Environment and permissions to rebuild the image.
Steps
Refer to Adjust Containerfile for Building Task-Compatible Custom Images to modify the Containerfile configuration for catalog Task usage.
Option 2: Modify TaskRun or PipelineRun Execution Configuration
Prerequisites
- Permissions to modify TaskRun or PipelineRun.
Steps
A Pod-template securityContext replaces the securityContext of the default Pod template as a whole; the fields are not merged. The samples therefore repeat the shipped fsGroup: 65532 and fsGroupChangePolicy: "OnRootMismatch" next to runAsUser; repeat every other field you still need (see the Priority Rule in Configuration Methods Comparison). The default Pod template is the default-pod-template key of the config-defaults ConfigMap in tekton-pipelines. The roles shipped with Tekton Pipelines let every authenticated user read only the pipelines-info ConfigMap in that namespace, so if you cannot read config-defaults, ask your cluster administrator for the securityContext of the default Pod template. tekton-pipelines is the default of TektonConfig spec.targetNamespace; replace it in the commands if your installation uses another namespace.
-
Add configuration when executing TaskRun separately:
-
Add configuration when executing PipelineRun. Use one of the two methods below.
Method 1: Add configuration for all Tasks:
Method 2: Add configuration for specific Tasks:
Option 3: Modify Global Tekton Configuration
Prerequisites
- Cluster operation permissions.
- Permissions to modify the TektonConfig resource.
- Note: This configuration will affect all Tasks.
Steps
-
Modify the TektonConfig resource: Set the following
spec.pipeline.default-pod-templateconfiguration. The value replaces the current default Pod template as a whole. Read the value in effect and the value set inTektonConfigfirst, save both outputs, and keep the default's fields in the sample:On a default installation it is the shipped template (
fsGroup: 65532,fsGroupChangePolicy: "OnRootMismatch"), so the sample keeps those two fields next torunAsUser. It changes the default Pod template of every new TaskRun and PipelineRun on the cluster.Save the sample as
tektonconfig-pod-template.yamland merge it into the existingTektonConfig:To undo the change later: if the saved
TektonConfigoutput was empty, remove the field fromTektonConfig; otherwise set the savedTektonConfigvalue again: -
Verify whether the configuration takes effect. The command prints the whole default Pod template, which must be the template of the sample, including any fields of the default that you kept:
Option 4: Modify Task Definition
Prerequisites
- Permissions to modify the Task.
- Note: This configuration will affect all TaskRuns or PipelineRuns that use this Task.
Steps
-
Method 1: Remove runAsNonRoot configuration. The step then runs as the image user, which is root for an image with
USER root, unless a Pod template setsrunAsUser. A namespace that enforces therestrictedPod Security Standard rejects a Pod in which neither the Pod nor the step setsrunAsNonRoot: true(PodAdmissionFailed,runAsNonRoot != true). Prefer Method 2, an image with a non-rootUSER(Option 1), or Option 3: -
Method 2: Add runAsUser configuration:
Prevent Errors
-
Image Building
- Prioritize using non-root users for building images.
- Use UID 65532 as the non-root user consistently.
- Ensure that the application can run normally with a non-root user.
-
Task Configuration
- Decide whether to enable
runAsNonRootbased on security requirements. - If required, configure
runAsUseraccordingly.
- Decide whether to enable
-
Permission Management
- Follow the principle of least privilege.
- Plan directory permissions in advance.
- Regularly review permission configurations.