BoxBoat Blog
Service updates, customer stories, and tips and tricks for effective DevOps
Kubernetes NGINX Ingress TLS Secrets in All Namespaces
by Caleb Lloyd | Monday, Jul 2, 2018 | Kubernetes
Kubernetes Ingress is a powerful resource that can automate load balancing and SSL/TLS termination. The NGINX Ingress Controller is currently the only supported cloud-agnostic ingress controller for Kubernetes. A single ingress controller can be deployed to the cluster and service requests for all namespaces in a cluster. There is currently an outstanding issue where Ingress resources can only reference TLS secrets within their own namespace:
- Referring to TLS secret from other namespace (i.e. not the namespace in which ingress is created)
- Advice around ingress secret cross-namespace
This can be a problem when a cluster has a wildcard certificate that needs to be used across multiple ingress resources in different namespaces. A prime example is a cluster that is setup to automatically request Let's Encrypt wildcard certificates. In this blog post, we explain an approach for automatically reflecting TLS secrets to every namespace in the cluster.
Define Variables
Separating configuration from logic is an important first step to staying organized. First, we'll create a file called vars.env that holds all of the configuration for our resources.
KUBECTL_VERSION="1.10.3"
TLS_SECRET="ingress-cert"
NAMESPACE="bb-system"
- KUBECTL_VERSION: tag used for the boxboat/kubectl Docker image
- TLS_SECRET: name of the TLS Secret that will be reflected across the cluster
- NAMESPACE: the Kubernetes namespace where the TLS Secret is controlled from. The Ingress Certificate Reflector will watch the TLS Secret in this namespace and copy updates to all other namespaces in the cluster.
Setup Ingress Certificate Reflector
The Ingress Certificate Reflector needs permission to list/watch all Namespaces in the cluster and create/get/update/watch the TLS Secret in all Namespaces in the cluster. We'll create a ServiceAccount, ClusterRole, and ClusterRoleBinding to add these permissions.
The Ingress Cert Reflector has 2 primary functions:
- Watch all Namespaces in the cluster. When a new Namespace is added, copy the TLS Secret to it
- Watch the TLS Secret. When it changes, copy it to every Namespace
To accomplish this, we'll create a Deployment with 2 containers: ns-watch and secret-watch
Create a file called ingress-cert-reflector.yml to create these resources:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: ingress-cert-reflector
namespace: "${NAMESPACE}"
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: ingress-cert-reflector
rules:
- apiGroups: [""]
resources: ["secrets"]
verbs: ["create", "watch"]
- apiGroups: [""]
resources: ["secrets"]
resourceNames: ["${TLS_SECRET}"]
verbs: ["get", "patch"]
- apiGroups: [""]
resources: ["namespaces"]
verbs: ["list", "watch"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: ingress-cert-reflector
subjects:
- kind: ServiceAccount
name: ingress-cert-reflector
namespace: "${NAMESPACE}"
roleRef:
kind: ClusterRole
name: ingress-cert-reflector
apiGroup: rbac.authorization.k8s.io
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: ingress-cert-reflector
namespace: "${NAMESPACE}"
spec:
replicas: 1
selector:
matchLabels:
app: ingress-cert-reflector
template:
metadata:
labels:
app: ingress-cert-reflector
spec:
serviceAccountName: ingress-cert-reflector
containers:
- name: ns-watch
image: boxboat/kubectl:${KUBECTL_VERSION}
command:
- sh
- -c
- |
set -e
while true; do
echo "$(date '+%Y-%m-%d %H:%M:%S') starting watch loop"
kubectl get ns --watch --field-selector="status.phase==Active" --no-headers -o "custom-columns=:metadata.name" | \
while read ns; do
if [ "$ns" != "${NAMESPACE}" ]; then
echo "$(date '+%Y-%m-%d %H:%M:%S') namespace - $ns"
kubectl -n "${NAMESPACE}" get secret "${TLS_SECRET}" -o yaml --export | \
kubectl -n "$ns" apply -f -
fi
done
done
- name: secret-watch
image: boxboat/kubectl:${KUBECTL_VERSION}
command:
- sh
- -c
- |
set -e
while true; do
echo "$(date '+%Y-%m-%d %H:%M:%S') starting watch loop"
kubectl -n "${NAMESPACE}" get secret "${TLS_SECRET}" --watch --no-headers -o "custom-columns=:metadata.name" | \
while read secret; do
export=$(kubectl -n "${NAMESPACE}" get secret "$secret" -o yaml --export)
for ns in $(kubectl get ns --field-selector="status.phase==Active" --no-headers -o "custom-columns=:metadata.name"); do
if [ "$ns" != "${NAMESPACE}" ]; then
echo "$(date '+%Y-%m-%d %H:%M:%S') namespace - $ns"
echo "$export" | kubectl -n "$ns" apply -f -
fi
done
done
done
In order to inject variables from the vars.env file, create a script called ingress-cert-reflector.sh:
#!/bin/bash
cd $(dirname $0)
set -a
. "vars.env"
set +a
envsubst '${KUBECTL_VERSION},${TLS_SECRET},${NAMESPACE}' < ingress-cert-reflector.yml \
| kubectl apply -f -
Run the script ./ingress-cert-reflector.sh. You should now see resources when you run the following:
# should print a "ingress-cert-reflector" service account
kubectl -n YOUR_NAMESPACE get serviceaccount
# should print a "ingress-cert-reflector" clusterrole
kubectl get clusterrole ingress-cert-reflector
# should print a "ingress-cert-reflector" clusterrolebinding
kubectl get clusterrolebinding ingress-cert-reflector
# should print a "ingress-cert-reflector" deployment
kubectl -n YOUR_NAMESPACE get deployments
# should print a "ingress-cert-reflector-xxxxx" pod
kubectl -n YOUR_NAMESPACE get pods
If the ingress-cert-reflector-xxxxx pod has errors starting, run the following commands to debug:
# Kubernetes system logs for the pod
kubectl -n YOUR_NAMESPACE describe pod ingress-cert-reflector-xxxxx
# logs from the namespace watcher container
kubectl -n YOUR_NAMESPACE logs -f -c ns-watch ingress-cert-reflector-xxxxx
# logs from the secret watcher container
kubectl -n YOUR_NAMESPACE logs -f -c secret-watch ingress-cert-reflector-xxxxx
Test the TLS Secret Reflection
- Check to see that the
defaultnamespace now contains the TLS Secret:
# should print the TLS Secret ingress-cert
kubectl -n default get secret ingress-cert -o yaml
- Update the TLS Secret in the namespace listed in
vars.envby adding a “test” key/value to the secrettest: dGVzdA==
# add the key/value "test: dGVzdA==" to the "data" section
kubectl -n YOUR_NAMESPACE edit secret ingress-cert
- Check to see that the “test” key/value has been reflected to the TLS Secret in the
defaultnamespace:
# should see "test: dGVzdA==" in the "data" section
kubectl -n default get secret ingress-cert -o yaml
That's it! You should now be able to reference the TLS Secret from an ingress resource in any namespace:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: ingress
spec:
tls:
- secretName: ingress-cert
To automate wildcard TLS certificates in the cluster with Let's Encrypt, check out our Kubernetes Ingress Automatic Let's Encrypt Certificates blog post.
