BoxBoat Blog

Service updates, customer stories, and tips and tricks for effective DevOps

x ?

Get Hands-On Experience with BoxBoat's Cloud Native Academy

Kubernetes NGINX Ingress TLS Secrets in All Namespaces

by Caleb Lloyd | Monday, Jul 2, 2018 | Kubernetes

featured.png

Kubernetes Ingress is a powerful resource that can automate load balancing and SSL/TLS termination. The NGINX Ingress Controller is currently the only supported cloud-agnostic ingress controller for Kubernetes. A single ingress controller can be deployed to the cluster and service requests for all namespaces in a cluster. There is currently an outstanding issue where Ingress resources can only reference TLS secrets within their own namespace:

This can be a problem when a cluster has a wildcard certificate that needs to be used across multiple ingress resources in different namespaces. A prime example is a cluster that is setup to automatically request Let's Encrypt wildcard certificates. In this blog post, we explain an approach for automatically reflecting TLS secrets to every namespace in the cluster.

Define Variables

Separating configuration from logic is an important first step to staying organized. First, we'll create a file called vars.env that holds all of the configuration for our resources.

KUBECTL_VERSION="1.10.3"
TLS_SECRET="ingress-cert"
NAMESPACE="bb-system"
  • KUBECTL_VERSION: tag used for the boxboat/kubectl Docker image
  • TLS_SECRET: name of the TLS Secret that will be reflected across the cluster
  • NAMESPACE: the Kubernetes namespace where the TLS Secret is controlled from. The Ingress Certificate Reflector will watch the TLS Secret in this namespace and copy updates to all other namespaces in the cluster.

Setup Ingress Certificate Reflector

The Ingress Certificate Reflector needs permission to list/watch all Namespaces in the cluster and create/get/update/watch the TLS Secret in all Namespaces in the cluster. We'll create a ServiceAccount, ClusterRole, and ClusterRoleBinding to add these permissions.

The Ingress Cert Reflector has 2 primary functions:

  1. Watch all Namespaces in the cluster. When a new Namespace is added, copy the TLS Secret to it
  2. Watch the TLS Secret. When it changes, copy it to every Namespace

To accomplish this, we'll create a Deployment with 2 containers: ns-watch and secret-watch

Create a file called ingress-cert-reflector.yml to create these resources:

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: ingress-cert-reflector
  namespace: "${NAMESPACE}"

---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: ingress-cert-reflector
rules:
- apiGroups: [""]
  resources: ["secrets"]
  verbs: ["create", "watch"]
- apiGroups: [""]
  resources: ["secrets"]
  resourceNames: ["${TLS_SECRET}"]
  verbs: ["get", "patch"]
- apiGroups: [""]
  resources: ["namespaces"]
  verbs: ["list", "watch"]

---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: ingress-cert-reflector
subjects:
- kind: ServiceAccount
  name: ingress-cert-reflector
  namespace: "${NAMESPACE}"
roleRef:
  kind: ClusterRole
  name: ingress-cert-reflector
  apiGroup: rbac.authorization.k8s.io

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ingress-cert-reflector
  namespace: "${NAMESPACE}"
spec:
  replicas: 1
  selector:
    matchLabels:
      app: ingress-cert-reflector
  template:
    metadata:
      labels:
        app: ingress-cert-reflector
    spec:
      serviceAccountName: ingress-cert-reflector
      containers:
      - name: ns-watch
        image: boxboat/kubectl:${KUBECTL_VERSION}
        command:
        - sh
        - -c
        - |
          set -e
          while true; do
            echo "$(date '+%Y-%m-%d %H:%M:%S') starting watch loop"
            kubectl get ns --watch --field-selector="status.phase==Active" --no-headers -o "custom-columns=:metadata.name" | \
            while read ns; do
              if [ "$ns" != "${NAMESPACE}" ]; then
                echo "$(date '+%Y-%m-%d %H:%M:%S') namespace - $ns"
                kubectl -n "${NAMESPACE}" get secret "${TLS_SECRET}" -o yaml --export | \
                  kubectl -n "$ns" apply -f -
              fi
            done
          done
      - name: secret-watch
        image: boxboat/kubectl:${KUBECTL_VERSION}
        command:
        - sh
        - -c
        - |
          set -e
          while true; do
            echo "$(date '+%Y-%m-%d %H:%M:%S') starting watch loop"
            kubectl -n "${NAMESPACE}" get secret "${TLS_SECRET}" --watch --no-headers -o "custom-columns=:metadata.name" | \
            while read secret; do
              export=$(kubectl -n "${NAMESPACE}" get secret "$secret" -o yaml --export)
              for ns in $(kubectl get ns --field-selector="status.phase==Active" --no-headers -o "custom-columns=:metadata.name"); do
                if [ "$ns" != "${NAMESPACE}" ]; then
                  echo "$(date '+%Y-%m-%d %H:%M:%S') namespace - $ns"
                  echo "$export" | kubectl -n "$ns" apply -f -
                fi
              done
            done
          done

In order to inject variables from the vars.env file, create a script called ingress-cert-reflector.sh:

#!/bin/bash

cd $(dirname $0)
set -a
. "vars.env"
set +a

envsubst '${KUBECTL_VERSION},${TLS_SECRET},${NAMESPACE}' < ingress-cert-reflector.yml \
    | kubectl apply -f -

Run the script ./ingress-cert-reflector.sh. You should now see resources when you run the following:

# should print a "ingress-cert-reflector" service account
kubectl -n YOUR_NAMESPACE get serviceaccount

# should print a "ingress-cert-reflector" clusterrole
kubectl get clusterrole ingress-cert-reflector

# should print a "ingress-cert-reflector" clusterrolebinding
kubectl get clusterrolebinding ingress-cert-reflector

# should print a "ingress-cert-reflector" deployment
kubectl -n YOUR_NAMESPACE get deployments

# should print a "ingress-cert-reflector-xxxxx" pod
kubectl -n YOUR_NAMESPACE get pods

If the ingress-cert-reflector-xxxxx pod has errors starting, run the following commands to debug:

# Kubernetes system logs for the pod
kubectl -n YOUR_NAMESPACE describe pod ingress-cert-reflector-xxxxx

# logs from the namespace watcher container
kubectl -n YOUR_NAMESPACE logs -f -c ns-watch ingress-cert-reflector-xxxxx

# logs from the secret watcher container
kubectl -n YOUR_NAMESPACE logs -f -c secret-watch ingress-cert-reflector-xxxxx

Test the TLS Secret Reflection

  1. Check to see that the default namespace now contains the TLS Secret:
# should print the TLS Secret ingress-cert
kubectl -n default get secret ingress-cert -o yaml
  1. Update the TLS Secret in the namespace listed in vars.env by adding a “test” key/value to the secret test: dGVzdA==
# add the key/value "test: dGVzdA==" to the "data" section
kubectl -n YOUR_NAMESPACE edit secret ingress-cert
  1. Check to see that the “test” key/value has been reflected to the TLS Secret in the default namespace:
# should see "test: dGVzdA==" in the "data" section
kubectl -n default get secret ingress-cert -o yaml

That's it! You should now be able to reference the TLS Secret from an ingress resource in any namespace:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ingress
spec:
  tls:
  - secretName: ingress-cert

To automate wildcard TLS certificates in the cluster with Let's Encrypt, check out our Kubernetes Ingress Automatic Let's Encrypt Certificates blog post.