Secure Supply Chain - Tekton Chains

by Parth Patel | Monday, Nov 8, 2021 | Secure Supply Chain Open-Source Security

Tekton has been growing in popularity as a go-to CI/CD cloud native pipeline tool. Tekton installs and runs as an extension on a Kubernetes cluster and comprises a set of Kubernetes Custom Resources that define the building blocks you can create and reuse for your pipelines. It provides the ability to create custom pipelines with various tasks, from building images, storing and scanning the images, and deploying them to a kubernetes cluster. It has also made tremendous strides to help organizations achieve the secure supply chain methodology by providing a method to meet the SLSA (Supply-chain Levels for Software Artifacts) security levels. SLSA is a security framework that defines how anyone working with a software can go securely from source to service. Tekton Chains can automatically create a signed SLSA compliant provenance for tasks that are run within the pipeline. This provenance can be used within a secure supply chain to verify that a man-in-the-middle did not compromise your supply chain.

A quick overview of the SLSA Security Levels

Based on the SLSA Levels, Tekton Chains allows organizations to easily meet parts of Level 1 and 2 without much extra work on their ends. Combined with a secure Infrastructure as Code and audibility, organizations can be on the right path to start mitigating those supply chains attacks that have become predominate in the past few years.

Let's create a quick Tekton pipeline where we can demonstrate how Tekton Chains works and how to retrieve the provenance file.

Prerequisites

There's a few things we will need for Tekton to run locally:

• Docker
• A running Kubernetes cluster (Docker Desktop Kubernetes, k3d, minikube)
• jq, a lightweight and flexible command-line JSON processor

Setting up Tekton

To install Tekton and Tekton Pipelines run the following command:

kubectl apply --filename https://storage.googleapis.com/tekton-releases/pipeline/latest/release.yaml

The pods may take a few mins to initialize based on your system. You can check the progress by running:

kubectl get pods --namespace tekton-pipelines --watch

and check that all the pods are in a running state.

Setting up Tekton Chains and Cosign

Once Tekton Pipeline is installed, the next step is to install Tekton Chains. This will be used to create the provenance for each TaskRun within Tekton.

To install the latest version of Chains to your Kubernetes cluster, run:

kubectl apply --filename https://storage.googleapis.com/tekton-releases/chains/latest/release.yaml

Ensure that the Tekton Chains pods are in the running state by running:

kubectl get pods -n tekton-chains --watch

One more tool is needed to meet SLSA Level 2 of signed provenance. Tekton Chains creates the provenance file, but how is signing and verification going to be handled? Sigstore Cosign is the answer. Cosign allows us to generate a keypair that will be used by Chains for signing the provenance that TaskRuns create. This tools can be installed by going to the latest release page of cosign and installing the binary based on your system.

Setting up Tekton Chains

Now that we have Tekton and Tekton Chains both running in our kubernetes environment, we can configure Tekton Chains based on our needs. Currently the configuration of Chains is done via edits to the configmap in the tekton-chains namespace. In future releases, the Tekton CLI will have the ability to change configuration.

The two main configurations that we need to look at are artifacts.taskrun.format and artifacts.taskrun.storage.

TaskRun Configuration

artifacts.taskrun.format: Determines which format the provenance will be created in. Currently, it supports multiple variations, but the official SLSA Provenance has become the industry standard. Creating provenance files with this spec will allow for further use with other tools in the future for verification. Currently within Tekton Chains, the in-toto format is the SLSA format.

Example of SLSA provenance:

{
  // Standard attestation fields:
  "_type": "https://in-toto.io/Statement/v0.1",
  "subject": [{ ... }],

  // Predicate:
  "predicateType": "https://slsa.dev/provenance/v0.2",
  "predicate": {
    "builder": {
      "id": "<URI>"
    },
    "buildType": "<URI>",
    "invocation": {
      "configSource": {
        "uri": "<URI>",
        "digest": { /* DigestSet */ },
        "entryPoint": "<STRING>"
      },
      "parameters": { /* object */ },
      "environment": { /* object */ }
    },
    "buildConfig": { /* object */ },
    "metadata": {
      "buildInvocationId": "<STRING>",
      "buildStartedOn": "<TIMESTAMP>",
      "buildFinishedOn": "<TIMESTAMP>",
      "completeness": {
        "parameters": true/false,
        "environment": true/false,
        "materials": true/false
      },
      "reproducible": true/false
    },
    "materials": [
      {
        "uri": "<URI>",
        "digest": { /* DigestSet */ }
      }
    ]
  }
}

The SLSA Provenance takes in multiple system inputs and external inputs (such as user defined parameters, materials passed in and commands that are run) and combines this information into a provenance file that can be used downstream to verify its integrity and non-repudiation.

artifacts.taskrun.storage: Determines where to store the provenance for the TaskRuns once they are created. Currently the options are within Tekton itself, an OCI registry, google cloud storage (GCS) and docdb. Choosing the Tekton storage option will store the provenance and signature in the annotation (base64-encoded) of the taskrun object itself within kubernetes. The OCI storage will push the provenance and signature into an OCI registry where it can be pulled down and verified. We will explore both options in this blog.

For the sake of the initial test we will change the configmap to use in-toto format and tekton as the storage. This can be done by running:

kubectl patch configmap chains-config -n tekton-chains -p='{"data":{"artifacts.taskrun.format": "in-toto"}}'
kubectl patch configmap chains-config -n tekton-chains -p='{"data":{"artifacts.taskrun.storage": "tekton"}}'

Next, we will scale the deployment to apply these changes:

kubectl scale deployment -n tekton-chains tekton-chains-controller --replicas=0
kubectl scale deployment -n tekton-chains tekton-chains-controller --replicas=1

Finally we need to create our own keypair that will be used for signing the TaskRun and payload. This will allow us to verify that no third party corrupted our provenance by verifying the signature is valid at the end of the run. Cosign will automatically create and store the keypair as a kubernetes secret in the tekton-chains namespace and name the secret signing-secrets

cosign generate-key-pair k8s://tekton-chains/signing-secrets

Run Taskrun and Analyze Provenance

We will run a pre-configured TaskRun for demo purposes, but this can also be done by creating a simple Tekton task of your own. (There will be links to other examples and documentation at the end of the post.)

This simple build-push-run-output-image- will output an image after some modification.

Taskrun can be run via:

kubectl create -f https://raw.githubusercontent.com/tektoncd/chains/main/examples/taskruns/task-output-image.yaml

Give the TaskRun a few seconds to complete. We can check the status by running:

kubectl get taskrun --watch

There should be an output similar to:

NAME                                SUCCEEDED   REASON      STARTTIME   COMPLETIONTIME
build-push-run-output-image-l2fvs   True        Succeeded   95s         68s

Now that the TaskRun has succeeded, wait for a few seconds for Chains to create the payload. We can retrieve the payload and signature from the TaskRun object directly. If we had changed the configuration to use OCI, GCS or docdb as the storage, it would be automatically uploaded based on the authentication. In this case, we used Tekton as the storage so our payload and signature are stored as base64-encoded annotations in the TaskRun.

First we will export the taskrun UID so we can use it to quickly retrieve the information from the TaskRun later:

export TASKRUN=<Name of your TaskRun> # Replace with your TaskRun name

export TASKRUN_UID=$(kubectl get taskrun $TASKRUN -o=json | jq -r '.metadata.uid')

Next we will run the following commands piped to jq to retrieve and decode the payload and signature. This command will also store the payload and signature into a file that we can use later.

kubectl get taskrun $TASKRUN -o=json | jq  -r ".metadata.annotations[\"chains.tekton.dev/payload-taskrun-$TASKRUN_UID\"]" | base64 --decode > payload
kubectl get taskrun $TASKRUN -o=json | jq  -r ".metadata.annotations[\"chains.tekton.dev/signature-taskrun-$TASKRUN_UID\"]" | base64 --decode > signature

Note: If the payload or signature is empty, wait a few seconds and re-try the above commands. Tekton Chains waits for the taskrun to complete before it creates the provenance and signs it.

We can view the created provenance payload using jq:

jq . ./payload

The payload will look similar to this. More updates are being made at the time this post is being written. Thus, future Chains releases will have more information captured based on updates to the SLSA provenance:

{
  "_type": "https://in-toto.io/Statement/v0.1",
  "predicateType": "https://slsa.dev/provenance/v0.1",
  "subject": null,
  "predicate": {
    "builder": {
      "id": "tekton-chains"
    },
    "recipe": {
      "type": "https://tekton.dev/attestations/chains@v1",
      "entryPoint": "build-push-run-output-image-l2fvs"
    },
    "metadata": {
      "buildStartedOn": "2021-11-08T15:44:21Z",
      "buildFinishedOn": "2021-11-08T15:44:48Z",
      "completeness": {
        "arguments": false,
        "environment": false,
        "materials": false
      },
      "reproducible": false
    },
    "materials": [
      {
        "uri": "pkg:docker/busybox@sha256:15e927f78df2cc772b70713543d6b651e3cd8370abf86b2ea4644a9fba21107f",
        "digest": {
          "sha256": "15e927f78df2cc772b70713543d6b651e3cd8370abf86b2ea4644a9fba21107f"
        }
      },
      {
        "uri": "pkg:docker/busybox@sha256:15e927f78df2cc772b70713543d6b651e3cd8370abf86b2ea4644a9fba21107f",
        "digest": {
          "sha256": "15e927f78df2cc772b70713543d6b651e3cd8370abf86b2ea4644a9fba21107f"
        }
      },
      {
        "uri": "pkg:docker/distroless/base@sha256:aa4fd987555ea10e1a4ec8765da8158b5ffdfef1e72da512c7ede509bc9966c4?repository_url=gcr.io",
        "digest": {
          "sha256": "aa4fd987555ea10e1a4ec8765da8158b5ffdfef1e72da512c7ede509bc9966c4"
        }
      },
      {
        "uri": "pkg:docker/tekton-releases/github.com/tektoncd/pipeline/cmd/git-init@sha256:c0b0ed1cd81090ce8eecf60b936e9345089d9dfdb6ebdd2fd7b4a0341ef4f2b9?repository_url=gcr.io",
        "digest": {
          "sha256": "c0b0ed1cd81090ce8eecf60b936e9345089d9dfdb6ebdd2fd7b4a0341ef4f2b9"
        }
      },
      {
        "uri": "pkg:docker/tekton-releases/github.com/tektoncd/pipeline/cmd/imagedigestexporter@sha256:e38dd0d32253fce5aaf1e501c0bc71facc3720564b7e97055921cc5390d612e0?repository_url=gcr.io",
        "digest": {
          "sha256": "e38dd0d32253fce5aaf1e501c0bc71facc3720564b7e97055921cc5390d612e0"
        }
      }
    ]
  }
}

As you can see from the provenance file, much information about what occurred during the taskrun its checksums are captured within the file. Materials such as pods and containers that were created during each step within the taskrun are captured with their corresponding checksums. Further updates to Chains and the SLSA format will allow for capturing more granular step information in each task. This will include the commands that each step executes, arguments and environment details.

Storing in OCI Registry

In the next example, we will store into the OCI registry. First we must change the chains-config

kubectl patch configmap chains-config -n tekton-chains -p='{"data":{"artifacts.taskrun.format": "tekton-provenance"}}'
kubectl patch configmap chains-config -n tekton-chains -p='{"data":{"artifacts.taskrun.storage": "oci"}}'

Note: OCI does not currently work with in-toto format at this time. Future versions will resolve this as the PR to fix this issue have been merged. At the time of this blog post Tekton Chains v0.5.0 was the latest release. If you are using any version newer, then you can keep the in-toto format for this example also!

Next we will scale the deployment to apply these changes:

kubectl scale deployment -n tekton-chains tekton-chains-controller --replicas=0
kubectl scale deployment -n tekton-chains tekton-chains-controller --replicas=1

Create the Kaniko Task:

kubectl create -f https://raw.githubusercontent.com/tektoncd/chains/main/examples/kaniko/kaniko.yaml

To run the Kaniko task run the following TaskRun.

apiVersion: tekton.dev/v1beta1
kind: TaskRun
metadata:
  name: kaniko-run
spec:
  taskRef:
    name: kaniko-chains
  params:
  - name: IMAGE
    value: ttl.sh/kaniko-chains
  workspaces:
  - name: source
    emptyDir: {}

This taskrun will feed in the missing parameters that are needed for the Kaniko task to run. We will be storing to an ephemeral public registry called ttl.sh. This registry is great for quick tests and demos. Copy the taskrun into a yaml file and apply it using kubectl to start the taskrun.

Once the TaskRun has successfully completed, you'll need to wait a few seconds for Chains to generate provenance and sign it. You can check that the TaskRun was signed by viewing the TaskRun and checking that the annotations include one set to chains.tekton.dev/signed: "true"

kubectl get tr kaniko-run -o json | jq -r .metadata.annotations

We can use cosign to verify the signature and the attestation stored in the OCI. Cosign will use our public key to verify that the private key secret we created before is still the same key that was used to sign the taskrun and payload. This can be checked at the end of the pipeline or before the image is deployed in a production environment, to ensure that a man-in-the-middle attack did not take place in the supply chain.

cosign verify --key cosign.pub ttl.sh/kaniko-chains

You should a similar output:

Verification for ttl.sh/kaniko-chains --
The following checks were performed on each of these signatures:
  - The cosign claims were validated
  - The signatures were verified against the specified public key
  - Any certificates were verified against the Fulcio roots.

[{"critical":{"identity":{"docker-reference":"ttl.sh/kaniko-chains"},"image":{"docker-manifest-digest":"sha256:12dd668e021b6803774462d9b177625dab8d3161c5e34dfd28750c148e926521"},"type":"Tekton container signature"},"optional":{}}]

cosign verify-attestation --key cosign.pub ttl.sh/kaniko-chains

You should a similar output:

Verification for ttl.sh/kaniko-chains --
The following checks were performed on each of these signatures:
  - The cosign claims were validated
  - The signatures were verified against the specified public key
  - Any certificates were verified against the Fulcio roots.
{"payloadType":"application/vnd.in-toto+json","payload":

Using a tool such as Crane, you can interact with the remote registry to see the image, signature and attestation that is stored within. Crane can also be used you pull down the signature file and attestation provenance file.

crane ls ttl.sh/kaniko-chains

latest
sha256-12dd668e021b6803774462d9b177625dab8d3161c5e34dfd28750c148e926521.att
sha256-12dd668e021b6803774462d9b177625dab8d3161c5e34dfd28750c148e926521.sig

Conclusion

This quick examples shows us how Tekton Chains can be used with the Tekton CI/CD pipeline to create provenance files that we can use for verification. Further work is still being done to tie other open source projects such as in-toto and SPIFFE SPIRE together to allow us to use the provenance file that is created from Tekton Chains to verify the full pipeline. SPIRE is being used in conjunction with in-toto to provide short-lived attested keys that can be used for signing.

As the tools evolve and further functionality is added, we will continue to add further blog posts on creating a truly SLSA compliant CI/CD pipeline. Stay tuned for future updates!

Here at BoxBoat we are always thinking about security first. Our Engineers are ready to implement DevSecOps in your organization using various tools and industry-leading security practices. Contact us for more information.