BoxBoat Blog
Service updates, customer stories, and tips and tricks for effective DevOps
Kubernetes Ingress Automatic Let's Encrypt Certificates
by Caleb Lloyd | Tuesday, Jun 12, 2018 | Kubernetes
Kubernetes Ingress is a powerful resource that can automate load balancing and SSL/TLS termination. Let's Encrypt is a fantastic service that provides free SSL/TLS certificates. This is a comprehensive guide to provision automated Let's Encrypt certificates for your Kubernetes Ingress using Kubernetes Jobs to generate and Cron Jobs to renew Let's Encrypt certificates.
All of the sample code in this guide can be found at the GitHub repository boxboat/kube-ingress-lets-encrypt.
Prerequisites
You must have an Ingress Controller deployed to your Kubernetes cluster. If you are running on Google Kubernetes Engine, you can easily use the GCE Ingress Controller. Otherwise you can use the NGINX Ingress Controller.
You should have an NFS server available with an NFS path owned by UID/GID 1000:1000
for storage of the Let's Encrypt certificates. You can follow the NFS Example to setup a NFS server in your Kubernetes cluster.
Define Variables
Separating configuration from logic is an important first step to staying organized. First, we'll create a file called vars.env
that holds all of the configuration for our resources.
NAMESPACE="bb-system"
CERT_NAME="boxboat.com"
DOMAIN_1="boxboat.com"
DOMAIN_2="*.boxboat.com"
EMAIL="admin@example.com"
KUBECTL_VERSION="1.10.3"
LEGO_VERSION="1.0.1"
NFS_HOSTNAME="nfs"
NFS_PATH="/mnt/data/lego"
TLS_SECRET="ingress-cert"
- NAMESPACE: the Kubernetes namespace that the Ingress, Job, and CronJob will be deployed to
- CERT_NAME: the primary certificate name. For a normal hostname, this is the same as
DOMAIN_1
. For a wildcard hostname, replace the asterics with an underscore, i.e. ifDOMAIN_1
is*.boxboat.com
, thenCERT_NAME
should be_.boxboat.com
- DOMAIN_1: the primary domain name on the certificate
- DOMAIN_2: the first Subject Alternate Name (SAN) on the certificate
- EMAIL: your email address for Let's Encrypt certificate renewal emails
- KUBECTL_VERSION: tag used for the boxboat/kubectl Docker image
- LEGO_VERSION: tag used for the boxboat/lego Docker image
- NFS_HOSTNAME: hostname or IP address of your NFS server
- NFS_PATH: path on your NFS server with read/write permissions for UID/GID
1000:1000
to store Let's Encrypt certificates - TLS_SECRET: name of the TLS Secret that will be used for your Ingress resource
Setup Kubernetes Service Account and NFS Volume Resources
The certificate generation and renewal jobs will need to automatically update the TLS Secret on the Ingress resource with generated Let's Encrypt certificates. We'll create a ServiceAccount, Role, and RoleBinding to update the TLS Secret.
The Let's Encrypt client will need to store generated certificates in a persistent volume. We'll create a NFS PersistentVolume and PersistentVolumeClaim to store the certificates.
Create a file called lego-setup.yml
to create these resources:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: lego
namespace: "${NAMESPACE}"
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: lego-secret-update
namespace: "${NAMESPACE}"
rules:
- apiGroups: [""]
resources: ["secrets"]
verbs: ["create"]
- apiGroups: [""]
resources: ["secrets"]
resourceNames: ["${TLS_SECRET}"]
verbs: ["get", "patch"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: lego-secret-update
namespace: "${NAMESPACE}"
subjects:
- kind: ServiceAccount
name: lego
namespace: "${NAMESPACE}"
roleRef:
kind: Role
name: lego-secret-update
apiGroup: rbac.authorization.k8s.io
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: "${NAMESPACE}-lego-nfs"
labels:
volume: "${NAMESPACE}-lego-nfs"
spec:
capacity:
storage: 10Mi
accessModes:
- ReadWriteMany
storageClassName: ""
persistentVolumeReclaimPolicy: Retain
nfs:
server: "${NFS_HOSTNAME}"
path: "${NFS_PATH}"
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: lego-nfs
namespace: "${NAMESPACE}"
spec:
selector:
matchLabels:
volume: "${NAMESPACE}-lego-nfs"
resources:
requests:
storage: 10Mi
accessModes:
- ReadWriteMany
storageClassName: ""
In order to inject variables from the vars.env
file, create a script called lego-setup.sh
:
#!/bin/bash
cd $(dirname $0)
set -a
. "vars.env"
set +a
envsubst < lego-setup.yml \
| kubectl apply -f -
Run the script ./lego-setup.sh
. You should now see resources when you run the following:
# should print a "lego" service account
kubectl -n YOUR_NAMESPACE get serviceaccount
# should print a "lego-secret-update" role
kubectl -n YOUR_NAMESPACE get role
# should print a "lego-secret-update" rolebinding
kubectl -n YOUR_NAMESPACE get rolebinding
# should print a "YOUR_NAMESPACE-lego-nfs" pv
kubectl get pv
# should print a "lego-nfs" pvc
kubectl -n YOUR_NAMESPACE get pvc
Setup DNS Challenge Secret
The lego Let's Encrypt client is used to generate certificates. The dns-01
challenge is used to prove that we own the domains in the certificate request. Our example uses the AWS Route 53 provider, however it can be easily customized for any supported DNS provider.
Create file called lego-secret.yml
to hold the DNS provider credentials:
---
apiVersion: v1
kind: Secret
metadata:
name: lego-aws
namespace: "${NAMESPACE}"
type: Opaque
data:
access-key-id: base64
secret-access-key: base64
Put your base64-encoded Access Key ID and Secret Access Key in the file. To base64-encode these values, use printf 'your-key-here' | base64
In order to inject variables from the vars.env
file, create a script called lego-secret.sh
:
#!/bin/bash
cd $(dirname $0)
set -a
. "vars.env"
set +a
envsubst < lego-secret.yml \
| kubectl apply -f -
Run the script ./lego-secret.sh
. You should now see resources when you run the following:
# should print a "lego-aws" secret
kubectl -n YOUR_NAMESPACE get secret
Setup Certificate Generation Job
The lego Let's Encrypt client must be run initially with the run
command to create a new certificate.
Create a file called lego-generate-cert.yml
to create the Job that will generate the certificate:
---
apiVersion: batch/v1
kind: Job
metadata:
name: lego-generate
namespace: "${NAMESPACE}"
spec:
backoffLimit: 2
template:
spec:
initContainers:
- name: lego
image: boxboat/lego:${LEGO_VERSION}
args:
- --accept-tos
- --email=${EMAIL}
- --domains=${DOMAIN_1}
- --domains=${DOMAIN_2}
- --dns=route53
- run
env:
- name: AWS_REGION
value: us-east-1
- name: AWS_ACCESS_KEY_ID
valueFrom:
secretKeyRef:
name: lego-aws
key: access-key-id
- name: AWS_SECRET_ACCESS_KEY
valueFrom:
secretKeyRef:
name: lego-aws
key: secret-access-key
volumeMounts:
- name: lego-nfs
mountPath: /home/alpine/.lego
containers:
- name: kubectl
image: boxboat/kubectl:${KUBECTL_VERSION}
command:
- sh
- -c
- >
kubectl create secret tls "${TLS_SECRET}"
-n "${NAMESPACE}"
--cert=.lego/certificates/${CERT_NAME}.crt
--key=.lego/certificates/${CERT_NAME}.key
--dry-run
-o yaml | kubectl apply -f -
volumeMounts:
- name: lego-nfs
mountPath: /home/alpine/.lego
restartPolicy: Never
serviceAccountName: lego
volumes:
- name: lego-nfs
persistentVolumeClaim:
claimName: lego-nfs
In order to inject variables from the vars.env
file, create a script called lego-generate-cert.sh
:
#!/bin/bash
cd $(dirname $0)
set -a
. "vars.env"
set +a
if kubectl -n "${NAMESPACE}" get job lego-generate > /dev/null 2>&1
then
kubectl -n "${NAMESPACE}" delete job lego-generate
fi
envsubst < lego-generate-cert.yml \
| kubectl apply -f -
Run the script ./lego-generate-cert.sh
. You should now see resources when you run the following:
# should print a "lego-generate" job
kubectl -n YOUR_NAMESPACE get jobs
# should print a "lego-generate-XXXXX" pod
kubectl -n YOUR_NAMESPACE get pods
If the lego-generate-XXXXX
pod has errors starting, run the following commands to debug:
# Kubernetes system logs for the pod
kubectl -n YOUR_NAMESPACE describe pod lego-generate-XXXXX
# logs from the Let's Encrypt client container
kubectl -n YOUR_NAMESPACE logs -f -c lego lego-generate-XXXXX
# logs from the Kubernetes TLS secret update kubectl client container
kubectl -n YOUR_NAMESPACE logs -f -c kubectl lego-generate-XXXXX
Any time that you change domains in your certificate or need to force a certificate update, re-run the ./lego-generate-cert.sh
job.
Setup Certificate Renewal CronJob
The lego Let's Encrypt client should be run nightly with the renew --days=30
command to renew the certificate.
Create a file called lego-renew-cert.yml
to create the CronJob that will renew the certificate:
---
apiVersion: batch/v1beta1
kind: CronJob
metadata:
name: lego-renew
namespace: "${NAMESPACE}"
spec:
schedule: "0 0 * * *"
jobTemplate:
spec:
backoffLimit: 2
template:
spec:
initContainers:
- name: lego
image: boxboat/lego:${LEGO_VERSION}
args:
- --accept-tos
- --email=${EMAIL}
- --domains=${DOMAIN_1}
- --domains=${DOMAIN_2}
- --dns=route53
- renew
- --days=30
env:
- name: AWS_REGION
value: us-east-1
- name: AWS_ACCESS_KEY_ID
valueFrom:
secretKeyRef:
name: lego-aws
key: access-key-id
- name: AWS_SECRET_ACCESS_KEY
valueFrom:
secretKeyRef:
name: lego-aws
key: secret-access-key
volumeMounts:
- name: lego-nfs
mountPath: /home/alpine/.lego
containers:
- name: kubectl
image: boxboat/kubectl:${KUBECTL_VERSION}
command:
- sh
- -c
- >
kubectl create secret tls "${TLS_SECRET}"
-n "${NAMESPACE}"
--cert=.lego/certificates/${CERT_NAME}.crt
--key=.lego/certificates/${CERT_NAME}.key
--dry-run
-o yaml | kubectl apply -f -
volumeMounts:
- name: lego-nfs
mountPath: /home/alpine/.lego
restartPolicy: Never
serviceAccountName: lego
volumes:
- name: lego-nfs
persistentVolumeClaim:
claimName: lego-nfs
In order to inject variables from the vars.env
file, create a script called lego-renew-cert.sh
:
#!/bin/bash
cd $(dirname $0)
set -a
. "vars.env"
set +a
envsubst < lego-renew-cert.yml \
| kubectl apply -f -
Run the script ./lego-renew-cert.sh
. You should now see resources when you run the following:
# should print a "lego-renew" cron job
kubectl -n YOUR_NAMESPACE get cronjobs
The cron job will run every night at midnight. After the cron job has run, you should see a pod for the renewal job:
# should print a "lego-renew-XXXXX" pod
kubectl -n YOUR_NAMESPACE get pods
If the lego-renew-XXXXX
pod has errors starting, follow the same debugging steps listed in the section above fore the lego-generate-XXXXX
pod.
Create an Ingress using the TLS Secret
Now that the TLS Secret ingress-cert
has been created with a valid Let's Encrypt certificate, you should be able to create a Ingress referencing the secret:
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: ingress
namespace: "${NAMESPACE}"
spec:
tls:
- secretName: "${TLS_SECRET}"
backend:
serviceName: your-service-name
servicePort: 80
In order to inject variables from the vars.env
file, create a script called ingress.sh
:
#!/bin/bash
cd $(dirname $0)
set -a
. "vars.env"
set +a
envsubst < ingress.yml \
| kubectl apply -f -
Run the script ./ingress.sh
. You should now see resources when you run the following:
# should print a "ingress" Ingress
kubectl -n YOUR_NAMESPACE get ingress
Ensure that your DNS is configured properly and your Let's Encrypt certificate should work, and should automatically update every 30 days.
All of the sample code in this guide can be found at the GitHub repository boxboat/kube-ingress-lets-encrypt. To use a wildcard Let's Encrypt certificate across multiple namespaces, check out our Kubernetes NGINX Ingress TLS Secrets in All Namespaces blog post.