BoxBoat Blog

Service updates, customer stories, and tips and tricks for effective DevOps

Get Hands-On Experience with BoxBoat's Cloud Native Academy

Kubernetes Ingress Automatic Let's Encrypt Certificates

by Caleb Lloyd | Tuesday, Jun 12, 2018 | Kubernetes

Kubernetes Ingress is a powerful resource that can automate load balancing and SSL/TLS termination. Let's Encrypt is a fantastic service that provides free SSL/TLS certificates. This is a comprehensive guide to provision automated Let's Encrypt certificates for your Kubernetes Ingress using Kubernetes Jobs to generate and Cron Jobs to renew Let's Encrypt certificates.

All of the sample code in this guide can be found at the GitHub repository boxboat/kube-ingress-lets-encrypt.

Prerequisites

You must have an Ingress Controller deployed to your Kubernetes cluster. If you are running on Google Kubernetes Engine, you can easily use the GCE Ingress Controller. Otherwise you can use the NGINX Ingress Controller.

You should have an NFS server available with an NFS path owned by UID/GID 1000:1000 for storage of the Let's Encrypt certificates. You can follow the NFS Example to setup a NFS server in your Kubernetes cluster.

Define Variables

Separating configuration from logic is an important first step to staying organized. First, we'll create a file called vars.env that holds all of the configuration for our resources.

NAMESPACE="bb-system"
CERT_NAME="boxboat.com"
DOMAIN_1="boxboat.com"
DOMAIN_2="*.boxboat.com"
EMAIL="admin@example.com"
KUBECTL_VERSION="1.10.3"
LEGO_VERSION="1.0.1"
NFS_HOSTNAME="nfs"
NFS_PATH="/mnt/data/lego"
TLS_SECRET="ingress-cert"

NAMESPACE: the Kubernetes namespace that the Ingress, Job, and CronJob will be deployed to
CERT_NAME: the primary certificate name. For a normal hostname, this is the same as DOMAIN_1. For a wildcard hostname, replace the asterics with an underscore, i.e. if DOMAIN_1 is *.boxboat.com, then CERT_NAME should be _.boxboat.com
DOMAIN_1: the primary domain name on the certificate
DOMAIN_2: the first Subject Alternate Name (SAN) on the certificate
EMAIL: your email address for Let's Encrypt certificate renewal emails
KUBECTL_VERSION: tag used for the boxboat/kubectl Docker image
LEGO_VERSION: tag used for the boxboat/lego Docker image
NFS_HOSTNAME: hostname or IP address of your NFS server
NFS_PATH: path on your NFS server with read/write permissions for UID/GID 1000:1000 to store Let's Encrypt certificates
TLS_SECRET: name of the TLS Secret that will be used for your Ingress resource

Setup Kubernetes Service Account and NFS Volume Resources

The certificate generation and renewal jobs will need to automatically update the TLS Secret on the Ingress resource with generated Let's Encrypt certificates. We'll create a ServiceAccount, Role, and RoleBinding to update the TLS Secret.

The Let's Encrypt client will need to store generated certificates in a persistent volume. We'll create a NFS PersistentVolume and PersistentVolumeClaim to store the certificates.

Create a file called lego-setup.yml to create these resources:

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: lego
  namespace: "${NAMESPACE}"

---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: lego-secret-update
  namespace: "${NAMESPACE}"
rules:
- apiGroups: [""]
  resources: ["secrets"]
  verbs: ["create"]
- apiGroups: [""]
  resources: ["secrets"]
  resourceNames: ["${TLS_SECRET}"]
  verbs: ["get", "patch"]

---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: lego-secret-update
  namespace: "${NAMESPACE}"
subjects:
- kind: ServiceAccount
  name: lego
  namespace: "${NAMESPACE}"
roleRef:
  kind: Role
  name: lego-secret-update
  apiGroup: rbac.authorization.k8s.io

---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: "${NAMESPACE}-lego-nfs"
  labels:
    volume: "${NAMESPACE}-lego-nfs"
spec:
  capacity:
    storage: 10Mi
  accessModes:
    - ReadWriteMany
  storageClassName: ""
  persistentVolumeReclaimPolicy: Retain
  nfs:
    server: "${NFS_HOSTNAME}"
    path: "${NFS_PATH}"

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: lego-nfs
  namespace: "${NAMESPACE}"
spec:
  selector:
    matchLabels:
      volume: "${NAMESPACE}-lego-nfs"
  resources:
    requests:
      storage: 10Mi
  accessModes:
    - ReadWriteMany
  storageClassName: ""

In order to inject variables from the vars.env file, create a script called lego-setup.sh:

#!/bin/bash

cd $(dirname $0)
set -a
. "vars.env"
set +a

envsubst < lego-setup.yml \
    | kubectl apply -f -

Run the script ./lego-setup.sh. You should now see resources when you run the following:

# should print a "lego" service account
kubectl -n YOUR_NAMESPACE get serviceaccount

# should print a "lego-secret-update" role
kubectl -n YOUR_NAMESPACE get role

# should print a "lego-secret-update" rolebinding
kubectl -n YOUR_NAMESPACE get rolebinding

# should print a "YOUR_NAMESPACE-lego-nfs" pv
kubectl get pv

# should print a "lego-nfs" pvc
kubectl -n YOUR_NAMESPACE get pvc

Setup DNS Challenge Secret

The lego Let's Encrypt client is used to generate certificates. The dns-01 challenge is used to prove that we own the domains in the certificate request. Our example uses the AWS Route 53 provider, however it can be easily customized for any supported DNS provider.

Create file called lego-secret.yml to hold the DNS provider credentials:

---
apiVersion: v1
kind: Secret
metadata:
  name: lego-aws
  namespace: "${NAMESPACE}"
type: Opaque
data:
  access-key-id: base64
  secret-access-key: base64

Put your base64-encoded Access Key ID and Secret Access Key in the file. To base64-encode these values, use printf 'your-key-here' | base64

In order to inject variables from the vars.env file, create a script called lego-secret.sh:

#!/bin/bash

cd $(dirname $0)
set -a
. "vars.env"
set +a

envsubst < lego-secret.yml \
    | kubectl apply -f -

Run the script ./lego-secret.sh. You should now see resources when you run the following:

# should print a "lego-aws" secret
kubectl -n YOUR_NAMESPACE get secret

Setup Certificate Generation Job

The lego Let's Encrypt client must be run initially with the run command to create a new certificate.

Create a file called lego-generate-cert.yml to create the Job that will generate the certificate:

---
apiVersion: batch/v1
kind: Job
metadata:
  name: lego-generate
  namespace: "${NAMESPACE}"
spec:
  backoffLimit: 2
  template:
    spec:
      initContainers:
      - name: lego
        image: boxboat/lego:${LEGO_VERSION}
        args:
        - --accept-tos
        - --email=${EMAIL}
        - --domains=${DOMAIN_1}
        - --domains=${DOMAIN_2}
        - --dns=route53
        - run
        env:
        - name: AWS_REGION
          value: us-east-1
        - name: AWS_ACCESS_KEY_ID
          valueFrom:
            secretKeyRef:
              name: lego-aws
              key: access-key-id
        - name: AWS_SECRET_ACCESS_KEY
          valueFrom:
            secretKeyRef:
              name: lego-aws
              key: secret-access-key
        volumeMounts:
        - name: lego-nfs
          mountPath: /home/alpine/.lego
      containers:
      - name: kubectl
        image: boxboat/kubectl:${KUBECTL_VERSION}
        command:
        - sh
        - -c
        - >
          kubectl create secret tls "${TLS_SECRET}"
          -n "${NAMESPACE}"
          --cert=.lego/certificates/${CERT_NAME}.crt
          --key=.lego/certificates/${CERT_NAME}.key
          --dry-run
          -o yaml | kubectl apply -f -
        volumeMounts:
        - name: lego-nfs
          mountPath: /home/alpine/.lego
      restartPolicy: Never
      serviceAccountName: lego
      volumes:
      - name: lego-nfs
        persistentVolumeClaim:
          claimName: lego-nfs

In order to inject variables from the vars.env file, create a script called lego-generate-cert.sh:

#!/bin/bash

cd $(dirname $0)
set -a
. "vars.env"
set +a

if kubectl -n "${NAMESPACE}" get job lego-generate > /dev/null 2>&1
then
    kubectl -n "${NAMESPACE}" delete job lego-generate
fi

envsubst < lego-generate-cert.yml \
    | kubectl apply -f -

Run the script ./lego-generate-cert.sh. You should now see resources when you run the following:

# should print a "lego-generate" job
kubectl -n YOUR_NAMESPACE get jobs

# should print a "lego-generate-XXXXX" pod
kubectl -n YOUR_NAMESPACE get pods

If the lego-generate-XXXXX pod has errors starting, run the following commands to debug:

# Kubernetes system logs for the pod
kubectl -n YOUR_NAMESPACE describe pod lego-generate-XXXXX

# logs from the Let's Encrypt client container
kubectl -n YOUR_NAMESPACE logs -f -c lego lego-generate-XXXXX

# logs from the Kubernetes TLS secret update kubectl client container
kubectl -n YOUR_NAMESPACE logs -f -c kubectl lego-generate-XXXXX

Any time that you change domains in your certificate or need to force a certificate update, re-run the ./lego-generate-cert.sh job.

Setup Certificate Renewal CronJob

The lego Let's Encrypt client should be run nightly with the renew --days=30 command to renew the certificate.

Create a file called lego-renew-cert.yml to create the CronJob that will renew the certificate:

---
apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: lego-renew
  namespace: "${NAMESPACE}"
spec:
  schedule: "0 0 * * *"
  jobTemplate:
    spec:
      backoffLimit: 2
      template:
        spec:
          initContainers:
          - name: lego
            image: boxboat/lego:${LEGO_VERSION}
            args:
            - --accept-tos
            - --email=${EMAIL}
            - --domains=${DOMAIN_1}
            - --domains=${DOMAIN_2}
            - --dns=route53
            - renew
            - --days=30
            env:
            - name: AWS_REGION
              value: us-east-1
            - name: AWS_ACCESS_KEY_ID
              valueFrom:
                secretKeyRef:
                  name: lego-aws
                  key: access-key-id
            - name: AWS_SECRET_ACCESS_KEY
              valueFrom:
                secretKeyRef:
                  name: lego-aws
                  key: secret-access-key
            volumeMounts:
            - name: lego-nfs
              mountPath: /home/alpine/.lego
          containers:
          - name: kubectl
            image: boxboat/kubectl:${KUBECTL_VERSION}
            command:
            - sh
            - -c
            - >
              kubectl create secret tls "${TLS_SECRET}"
              -n "${NAMESPACE}"
              --cert=.lego/certificates/${CERT_NAME}.crt
              --key=.lego/certificates/${CERT_NAME}.key
              --dry-run
              -o yaml | kubectl apply -f -
            volumeMounts:
            - name: lego-nfs
              mountPath: /home/alpine/.lego
          restartPolicy: Never
          serviceAccountName: lego
          volumes:
          - name: lego-nfs
            persistentVolumeClaim:
              claimName: lego-nfs

In order to inject variables from the vars.env file, create a script called lego-renew-cert.sh:

#!/bin/bash

cd $(dirname $0)
set -a
. "vars.env"
set +a

envsubst < lego-renew-cert.yml \
    | kubectl apply -f -

Run the script ./lego-renew-cert.sh. You should now see resources when you run the following:

# should print a "lego-renew" cron job
kubectl -n YOUR_NAMESPACE get cronjobs

The cron job will run every night at midnight. After the cron job has run, you should see a pod for the renewal job:

# should print a "lego-renew-XXXXX" pod
kubectl -n YOUR_NAMESPACE get pods

If the lego-renew-XXXXX pod has errors starting, follow the same debugging steps listed in the section above fore the lego-generate-XXXXX pod.

Create an Ingress using the TLS Secret

Now that the TLS Secret ingress-cert has been created with a valid Let's Encrypt certificate, you should be able to create a Ingress referencing the secret:

---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ingress
  namespace: "${NAMESPACE}"
spec:
  tls:
  - secretName: "${TLS_SECRET}"
  backend:
    serviceName: your-service-name
    servicePort: 80

In order to inject variables from the vars.env file, create a script called ingress.sh:

#!/bin/bash

cd $(dirname $0)
set -a
. "vars.env"
set +a

envsubst < ingress.yml \
    | kubectl apply -f -

Run the script ./ingress.sh. You should now see resources when you run the following:

# should print a "ingress" Ingress
kubectl -n YOUR_NAMESPACE get ingress

Ensure that your DNS is configured properly and your Let's Encrypt certificate should work, and should automatically update every 30 days.

All of the sample code in this guide can be found at the GitHub repository boxboat/kube-ingress-lets-encrypt. To use a wildcard Let's Encrypt certificate across multiple namespaces, check out our Kubernetes NGINX Ingress TLS Secrets in All Namespaces blog post.