BoxBoat Blog

Service updates, customer stories, and tips and tricks for effective DevOps

Building Containers with Kubernetes and Knative

by Cole Kennedy | Friday, Aug 10, 2018 | Kubernetes

Building Containers with Kubernetes and Knative

Developing for Kubernetes can be a daunting task for any developer not familiar with the ecosystem. The developer needs to understand how to create spec files, author CI/CD scripts with a system such as Jenkins or CircleCI, and instrument logging and tracing. Knative aims to solve some of these issues, abstracting the details of building images away from the developer.

Knative helps developers build, deploy, and manage modern serverless workloads on Kubernetes.

Knative has the backing of industry giants and is designed from lessons learned at companies such as Google, Red Hat, IBM, and SAP. It is a set of components that enables modern CI/CD workflows that are native to kubernetes. The current components that are available today are

  • Build - Build containers from source.
  • Serving - Serverless, or “Request-driven compute”
  • Eventing - Eventing

In this article, we will be focusing on the Build component.

Build Components

Build

The Knative Build component extends Kubernetes with custom resource definitions or CRDs this will give us a new object Build we can call from kubectl ex. kubectl get Builds Knative Build includes a few batteries included BuildTemplate that cover most situations.

BuildTemplate

BuildTemplate encapsulates a shareable build process with some limited parameterization capabilities.

BuildTemplates are provided for

  • Bazel - Bazel is a tool that automates software builds and tests.
  • Buildpack - Buildpack is a core link in the chain of the Cloud Foundry deployment process
  • Jib - Jib builds Docker and OCI images for your Java applications and is available as plugins for Maven and Gradle.
  • Kaniko - Kaniko is a tool to build container images from a Dockerfile, inside a container or Kubernetes cluster.

We will be using the Kaniko build template. Kankio is a project that builds containers without elevated privileges solving many of the issues when using the Docker daemon for building containers in a distributed environment.

Kaniko doesn’t depend on a Docker daemon and executes each command within a Dockerfile entirely in userspace. This enables building container images in environments that can’t easily or securely run a Docker daemon, such as a standard Kubernetes cluster.

ServiceAccount

Service accounts are the way Kubernetes can pass secrets to build templates. Each service account is associated with a single account on a service provider such as GitHub, or Docker Hub.

Builder

A builder is an image that executes a step in the build process. For example, you may have a builder that runs unit tests on your code before it is deployed. We will not be using the Builder component today.

Getting Started

We have a simple web application that we use for smoke testing. I have disabled automatic build on docker hub for this example. At the end of the example, Knative will pull the source code from GitHub, build the Docker image, and push it to Docker Hub with a single Kubernetes command.

Install Knative (On GKE)

I have stood up a isolated cluster to experiment with Knative. I do not recommend installing alpha software, such as Knative on a cluster you rely on. Instructions specific to your provider are found in the Knative GitHub repo. I will be using Google Kubernetes Engine version 1.10.5-gke.3. The install instructions below only apply if you are using GKE. Follow instructions specific to your provider. Remember, this is ALPHA software, things may not work as expected.


Initialize kubectl

gcloud container clusters get-credentials knative-builder --zone us-east1-b \
--project knative-playground

Ensure cluster is running

kubectl get nodes

Give the current user cluster admin rights

kubectl create clusterrolebinding cluster-admin-binding \
--clusterrole=cluster-admin --user=$(gcloud config get-value core/account)

Install Knative Build:

kubectl apply -f https://storage.googleapis.com/build-crd/latest/release.yaml

Wait for pods to come online

kubectl -n knative-build get pods -w
NAME                               READY     STATUS    RESTARTS   AGE
build-controller-b76f56b4c-6km8j   1/1       Running   0          1m
build-webhook-6867b944b4-vbfxq     1/1       Running   0          1m

Check the the CRDs

kubectl get crd

NAME                                    CREATED AT
backendconfigs.cloud.google.com         2018-08-10T03:24:50Z
builds.build.knative.dev                2018-08-10T03:32:37Z
buildtemplates.build.knative.dev        2018-08-10T03:32:37Z
scalingpolicies.scalingpolicy.kope.io   2018-08-10T03:25:10Z

Set project up for Kaniko

Kaniko requires a Dockerfile to be present. Below is the multi-stage Dockerfile for our simple web app. Because the application is written in Go it requires no dependencies allowing us to use the SCRATCH base image for a minimal artifact.

# Multistage Dockerfile
# Build stage
FROM golang:alpine AS build-env
ENV APP hello-world-webbapp
ENV GOOS linux
ENV GOARCH amd64
WORKDIR /src
COPY . .
RUN CGO_ENABLED=0 GOOS=${GOOS} GOARCH=${GOARCH} go build -a -o server

# Final
FROM scratch
COPY --from=build-env /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
COPY --from=build-env /src/server .
CMD ["./server"]

Authentication

To push to Docker Hub we need to provide the Build object with a ServiceAccount, a ServiceAccount requires a Secret so let us make that first. Note: StringData is converted to a base64 encoded data object when applied to the cluster.

vim secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: basic-user-pass
  annotations:
    build.knative.dev/docker-0: https://index.docker.io/v1/
type: kubernetes.io/basic-auth
stringData: #NOT Base64 encoded
  username: <docker username>
  password: <docker password>
kubectl apply -f secret.yaml

Now make a ServiceAccount so we can pass this secret to a Build spec.

vim service-account.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: build-bot
secrets:
  - name: basic-user-pass
kubectl apply -f service-account.yaml

Build

Create a Build spec.

vim build.yaml
apiVersion: build.knative.dev/v1alpha1
kind: Build
metadata:
  name: docker-build
spec:
  serviceAccountName: knative-build  #Service account create above
  source:
    git:
      url: https://github.com/boxboat/hello-world-webapp.git
      revision: master
  steps:
  - name: build-and-push
    image: gcr.io/kaniko-project/executor:v0.1.0
    args:  #See kaniko docs for more info about the arguments passed
    - --dockerfile=/workspace/Dockerfile
    - --destination=docker.io/boxboat/hello-world-webapp

Now cross your fingers and kubectl apply -f build.yaml

Verify

The build should have been kicked off. Let us take a look.

kubectl get pods you should see a pod named docker-build with a postfix.

kubectl logs docker-build-XXXXX -c build-step-build-and-push
...
time="2018-08-10T04:58:40Z" level=info msg="Taking snapshot of full filesystem..."
time="2018-08-10T04:58:41Z" level=info msg="cmd: copy [/etc/ssl/certs/ca-certificates.crt]"
time="2018-08-10T04:58:41Z" level=info msg="dest: /etc/ssl/certs/"
time="2018-08-10T04:58:41Z" level=info msg="Copying file /kaniko/0/etc/ssl/certs/ca-certificates.crt to /etc/ssl/certs/ca-certificates.crt"
time="2018-08-10T04:58:41Z" level=info msg="Taking snapshot of files [/etc/ssl/certs/ca-certificates.crt]..."
time="2018-08-10T04:58:41Z" level=info msg="cmd: copy [/src/server]"
time="2018-08-10T04:58:41Z" level=info msg="dest: ."
time="2018-08-10T04:58:41Z" level=info msg="Copying file /kaniko/0/src/server to /server"
time="2018-08-10T04:58:41Z" level=info msg="Taking snapshot of files [/server]..."
time="2018-08-10T04:58:43Z" level=info msg="cmd: CMD"
time="2018-08-10T04:58:43Z" level=info msg="Replacing CMD in config with [./server]"
time="2018-08-10T04:58:43Z" level=info msg="No files changed in this command, skipping snapshotting."
time="2018-08-10T04:58:43Z" level=info msg="No files were changed, appending empty layer to config."
2018/08/10 04:58:44 pushed blob sha256:95c547f3e953339de0631b594c759ce7fa849723a8c7f73a153c4784f2ead2d7
2018/08/10 04:58:44 pushed blob sha256:e16ad3b2385f4045d00eb3e686e8ab708a5cb282a30866607ec818f8156c44c2
2018/08/10 04:58:46 pushed blob sha256:1c9588050b0ef607466397076c3ca79735a0d96805f766164bd271719e82c8c4
index.docker.io/boxboat/hello-world-webapp:latest: digest: sha256:9335c0b0016c9482012ccf51cb818eb09c2a03f10406615ab430cfffd24e036c size: 590

Take a look at the Build objects.

kubectl describe builds
Name:         docker-build
Namespace:    default
Labels:       <none>
Annotations:  kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"build.knative.dev/v1alpha1","kind":"Build","metadata":{"annotations":{},"name":"docker-build","namespace":"default"},"spec":{"serviceAcc...
API Version:  build.knative.dev/v1alpha1
Kind:         Build
Metadata:
  Cluster Name:
  Creation Timestamp:  2018-08-10T04:58:10Z
  Generation:          1
  Resource Version:    11158
  Self Link:           /apis/build.knative.dev/v1alpha1/namespaces/default/builds/docker-build
  UID:                 fa4afb12-9c59-11e8-b076-42010a8e021e
Spec:
  Generation:            1
  Service Account Name:  build-bot
  Source:
    Git:
      Revision:  master
      URL:       https://github.com/boxboat/hello-world-webapp.git
  Steps:
    Args:
      --dockerfile=/workspace/Dockerfile
      --destination=docker.io/boxboat/hello-world-webapp
    Image:  gcr.io/kaniko-project/executor:v0.1.0
    Name:   build-and-push
    Resources:
Status:
  Builder:  Cluster
  Cluster:
    Namespace:      default
    Pod Name:       docker-build-jpnfg
  Completion Time:  2018-08-10T04:58:47Z
  Conditions:
    State:     Succeeded
    Status:    True
  Start Time:  2018-08-10T04:58:10Z
  Step States:
    Terminated:
      Container ID:  docker://570002d349fc9e4309dc22235e94587bcf5685141f771d146fec9ebd9ed9ec44
      Exit Code:     0
      Finished At:   2018-08-10T04:58:10Z
      Reason:        Completed
      Started At:    2018-08-10T04:58:10Z
    Terminated:
      Container ID:  docker://86af6b46d41e53297b6d646fc30bb72a239a052899a62c392e845629e8041a14
      Exit Code:     0
      Finished At:   2018-08-10T04:58:12Z
      Reason:        Completed
      Started At:    2018-08-10T04:58:11Z
    Terminated:
      Container ID:  docker://9c6c952ab653b150d7ad24b50eec0c1040a283f3921b477bf38690c4d927097f
      Exit Code:     0
      Finished At:   2018-08-10T04:58:46Z
      Reason:        Completed
      Started At:    2018-08-10T04:58:12Z
Events:
  Type    Reason  Age                From              Message
  ----    ------  ----               ----              -------
  Normal  Synced  17s (x21 over 8m)  build-controller  Build synced successfully

Hopefully, you have a similar output. Check Docker Hub and make sure everything pushed correctly. Docker Hub screenshot

Caveats

  • Knative is new, beyond bleeding edge. It is likely to move fast breaking all of your work.
  • Every time you want to make a build you need to push a new build spec, this can be automated with Knative Serving, but that will have to wait for another post.
  • Knative requires Istio. Istio adds many complications to your cluster, and it may not be correct for your situation.

Summary

I have shown you how to use Knative to move from source to container only having to define a Dockerfile. While Knative is not ready for production use we are excited about the value it will bring to our customers as it matures.

In the next few weeks, I will be writing follow-on articles that explore more of the Knative eco-system. We will complete the loop by deploying our web app using the Knative Serving component and integrate with GitHub webhooks to automate the build kickoff. Make sure you subscribe to updates (upper right of page).

Don’t hesitate to contact boxboat for any of your containerization, devops, and orchestration needs.

BoxBoat Accelerator

Learn how to best introduce Docker into your organization. Leave your name and email, and we'll get right back to you.

Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.