GitOps Kubernetes Rolling Update when ConfigMaps and Secrets Change

by Caleb Lloyd | Thursday, Jul 5, 2018 | Kubernetes

The Kubernetes ConfigMap resource is used to mount configuration files into pods. The Kubernetes Secret resource is used to mount secret files into pods. Both of these resources are commonly used when deploying a GitOps Configuration as Code workflow.

ConfigMap and Secret files inside of containers are updated automatically when the underlying ConfigMap or Secret is updated. If an application reads a ConfigMap or Secret value on startup, it may have a stale configuration after a ConfigMap or Secret is updated.

One approach for dealing with this is to add application logic to watch ConfigMap and Secret files for changes, and reconfigure the application on the fly. This can lead to complicated logic since objects using the old configuration will need to be detected and recreated.

Another approach is to trigger a rolling update of the Deployment when it's dependent ConfigMaps and Secrets are updated. This blog post describes a solution that creates ConfigMaps and Secrets alongside a Deployment, and uses a hash of these resources to automatically trigger a rolling update if they have changed.

Configuration as Code with GitOps

GitOps is a method for storing Configuration as Code in a source control repository, and automatically updating deployed state when the repository changes. Essentially, Git becomes the source of truth for your deployment, which ensures that your currently deployed system is documented in source control and makes disaster recovery easier.

When dealing with Configuration as Code, it is nice to be able to stage the configuration for deployment just as it will appear inside of the container. Our sample will stage configuration files and secret files, and use a deployment script to translate these to Kubernetes ConfigMaps and Secrets.

All of the sample code in this guide can be found at the GitHub repository boxboat/kube-configmap-secret-update.

Stage ConfigMap and Secret Files

Let's consider an nginx container that defines 2 hosts - a frontend website that is unrestricted, and an admin website that is protected by HTTP basic auth.

We will stage 2 files for a ConfigMap that will hold the website configuration -frontend.conf and admin.conf. We will stage 1 file for a Secret that will hold the HTTP basic auth credentials - htpasswd.

config/
  conf.d/
    frontend.conf
    admin.conf
  protected/
    .htpasswd

Copy the following contents to config/conf.d/frontend.conf:

server {
    listen 8080 default_server;
    server_name  _ ;
    root /usr/share/nginx/html;

    location / {
      default_type text/plain;
      return 200 'Frontend Website';
    }
}

Copy the following contents to config/conf.d/admin.conf:

server {
    listen 8081 default_server;
    server_name  _ ;
    root /usr/share/nginx/html;

    auth_basic "Admin Login";
    auth_basic_user_file /etc/nginx/protected/htpasswd; 

    location / {
      try_files does_not_exist @admin;
    }

    location @admin {
      default_type text/plain;
      return 200 'Admin Website';
    }
}

Copy the following contents to config/protected/htpasswd to create a user with the username admin and the password password:

admin:$apr1$puyQaI/a$lphikhByCbacJKD9taAxN1

Stage the Deployment and Service Specification

The Deployment mounts the frontend.conf and admin.conf from the ConfigMap to /etc/nginx/conf.d. It mounts the htpasswd from the Secret to /etc/nginx/protected. It exposes websites as NodePort services with the frontend website on port 31228 and the admin website on port 31229.

In order to trigger a rolling update if any of the configuration items change, we have added an environment variable CONFIG_HASH. This is set to a variable - we will make a deployment script that performs variable substitution to insert the hash later.

Create a file called nginx-config-example.yml to create the Deployment and 2 Services for the websites:

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-config-example
spec:
  replicas: 2
  selector:
    matchLabels:
      app: nginx-config-example
  template:
    metadata:
      labels:
        app: nginx-config-example
    spec:
      containers:
      - name: nginx
        image: nginx:alpine
        env:
        - name: CONFIG_HASH
          value: ${CONFIG_HASH}
        ports:
        - name: frontend
          protocol: TCP
          containerPort: 8080
        - name: admin
          protocol: TCP
          containerPort: 8081
        volumeMounts:
        - name: nginx-config-example-configmap
          mountPath: /etc/nginx/conf.d
        - name: nginx-config-example-secret
          mountPath: /etc/nginx/protected
      volumes:
      - name: nginx-config-example-configmap
        configMap:
          name: nginx-config-example-configmap
      - name: nginx-config-example-secret
        secret:
          secretName: nginx-config-example-secret

---
kind: Service
apiVersion: v1
metadata:
  name: nginx-config-example-frontend
spec:
  type: NodePort
  selector:
    app: nginx-config-example
  ports:
  - name: http
    protocol: TCP
    targetPort: frontend
    port: 80
    nodePort: 31228

---
kind: Service
apiVersion: v1
metadata:
  name: nginx-config-example-admin
spec:
  type: NodePort
  selector:
    app: nginx-config-example
  ports:
  - name: http
    protocol: TCP
    targetPort: admin
    port: 80
    nodePort: 31229

Create the Deployment Script

The deployment script must accomplish the following:

1. Create a ConfigMap with admin.conf and frontend.conf
2. Create a Secret with htpasswd
3. Hash all of the configuration-related files and store to the CONFIG_HASH variable
4. Apply the Deployment and Services in nginx-config-example.yml

Kubernetes has helper functions to create ConfigMaps and Secrets from files, however these functions only exist in kubectl create. Since ConfigMaps or Secrets may already exist, we will use the --dry-run option and pipe the resulting configuration to kubectl apply.

In order to generate the CONFIG_HASH variable, we will use the find command to find all configuration files, sort them so that they are in a consistent order, and md5sum the contents.

Create a file called nginx-config-example.sh with the following contents:

#!/bin/bash

cd $(dirname $0)

kubectl create configmap nginx-config-example-configmap \
    --from-file=admin.conf=./config/conf.d/admin.conf \
    --from-file=frontend.conf=./config/conf.d/frontend.conf \
    --dry-run -o yaml \
    | kubectl apply -f -

kubectl create secret generic nginx-config-example-secret \
    --from-file=htpasswd=./config/protected/htpasswd \
    --dry-run -o yaml \
    | kubectl apply -f -

export CONFIG_HASH=$(find ./config -type f | sort | xargs md5sum | md5sum | cut -d' ' -f 1)

envsubst '${CONFIG_HASH}' < nginx-config-example.yml \
    | kubectl apply -f -

Run the Deployment Script

Run the script ./nginx-config-example.sh. You should now see resources when you run the following:

# should print a "nginx-config-example" deployment
kubectl get deployment

# should print two "nginx-config-example-xxxxx" pods
kubectl get pods

# should print two NodePort services
kubectl get services

If the nginx-config-example-xxxxx pod has errors starting, run the following commands to debug:

# Kubernetes system logs for the pod
kubectl describe pod nginx-config-example-xxxxx

# logs from the container
kubectl logs nginx-config-example-xxxxx

Once everything is up and running, you should be able to view the two websites at the following URLs:

• Frontend Website: http://Kubernetes-Worker-IP:31228
• Backend Website: http://Kubernetes-Worker-IP:31229
  • Username: admin
  • Password: password

Change the Config and Redeploy

Change the configuration of the Frontend Website to in config/conf.d/frontend.conf to return the text Updated Frontend Website:

server {
    listen 8080 default_server;
    server_name  _ ;
    root /usr/share/nginx/html;

    location / {
      default_type text/plain;
      return 200 'Updated Frontend Website';
    }
}

Run the script ./nginx-config-example.sh again. You should see 2 new containers:

# should print two new "nginx-config-example-xxxxx" pods
kubectl get pods

Refresh the frontend website at http://Kubernetes-Worker-IP:31228. You should see the new message on the frontend website.

All of the sample code in this guide can be found at the GitHub repository boxboat/kube-configmap-secret-update. Hopefully you will find that this is a helpful pattern for handling ConfigMap and Secret updates when using a GitOps Configuration as Code workflow.