BoxBoat Blog

Service updates, customer stories, and tips and tricks for effective DevOps

Get Hands-On Experience with BoxBoat's Cloud Native Academy

Managing Multiple Microservices with Traefik in Docker Swarm

by Brandon Mitchell | Tuesday, Oct 10, 2017 | Docker

Docker's Swarm Mode is a great way to run web applications in a highly available distributed environment. Docker provides that high availability with a quorum of managers and multiple instances of the application container distributed across the workers. With the application being distributed across the workers, you have a new challenge of how to know which node to contact to reach your application. Docker has solved that with it's ingress network that publishes the port for your application across every node and then automatically routes your request to a container in the swarm providing that service, even if it's on another node.

Publishing the port for your application across all nodes in the swarm makes it easy to connect to your application, but you are left managing which application is published on which port. That's where reverse proxies like Traefik come into play. You can publish a single port that's your reverse proxy, and it automatically forwards the request to the appropriate container. Traefik is popular with Docker's Swarm Mode users because it's lightweight, handles HTTP and HTTPS requests, and most importantly, it can dynamically adjust to changes in your running swarm services.

Traefik is configured in three parts. First is the “entrypoint” which are the ports that it listens on and you publish on all nodes in your swarm. In this example, we will use port 80 for HTTP, and port 443 with HTTPS that will handle TLS termination. The second part is Traefik's “frontend” which are the rules attached to entrypoints that identify which requests to accept and where to send them. The last part is Traefik's backend which are pointers to the containers in Docker and how to connect to them. These backends are the targets attached to each frontend. Traefik allows you to prioritize these backends for failover from one container to another, but we've stuck with the default which is to round robin load balance to every matching swarm service that's available.

Checkout Our Example Code

Instead of copy/pasting from the blog, you can clone this repo on github:

git clone https://github.com/boxboat/traefik-tls.git

Step 1: Build Your TLS Certificates

To keep this example simple, we've used self signed TLS certificates created with OpenSSL. You can replace this step with LetsEncrypt if your server if publicly reachable with a valid DNS entry, or you can use your own certificates. For this simplified example, we're going to create a “tls” directory and then run the following script:

#!/bin/sh

certdir="tls"
host="localhost"

# setup a CA key
if [ ! -f "$certdir/ca-key.pem" ]; then
  openssl genrsa -out "${certdir}/ca-key.pem" 4096
fi

# setup a CA cert
openssl req -new -x509 -days 365 \

  -subj "/CN=Local CA" \

  -key "${certdir}/ca-key.pem" \

  -sha256 -out "${certdir}/ca.pem"

# setup a host key
if [ ! -f "${certdir}/key.pem" ]; then
  openssl genrsa -out "${certdir}/key.pem" 2048
fi

# create a signing request
extfile="${certdir}/extfile"
openssl req -subj "/CN=${host}" -new -key "${certdir}/key.pem" \

   -out "${certdir}/${host}.csr"
echo "subjectAltName = IP:127.0.0.1,DNS:localhost" &gt;${extfile}

# create the host cert
openssl x509 -req -days 365 \

   -in "${certdir}/${host}.csr" -extfile "${certdir}/extfile" \

   -CA "${certdir}/ca.pem" -CAkey "${certdir}/ca-key.pem" -CAcreateserial \

   -out "${certdir}/cert.pem"

# cleanup
if [ -f "${certdir}/${host}.csr" ]; then
        rm -f -- "${certdir}/${host}.csr"
fi
if [ -f "${extfile}" ]; then
        rm -f -- "${extfile}"
fi

Note that the above script is creating a certificate for localhost with the ip 127.0.0.1. You can add additional IP addresses and DNS names, or replace the ones above to match your own environment.

Step 2: Docker container networking

Traefik communicates from container to container to reach the backend containers. This means you do not need to publish the ports of the backend containers, but you do need to have a common docker network for the containers to communicate between each other. There are two approaches to the common network. One is to create a network just for the proxy and connect all the applications to this network. And the second option is to keep each application on its own network and attach the proxy to every applications network. You may have a hybrid of these approaches with something like a common network for a group of containers. That may be done if you have multiple test environments for a development and CI environment running on the same docker swarm. The advantage of a single proxy network is simplicity of the configuration, the proxy does not need to be modified for each new application. Instead the application itself is attached to the proxy network if it needs to be exposed. The disadvantage of a single proxy network is that multiple applications can now communicate with each other over the proxy network that you may want to keep separate. For this example, we're using a single proxy network, named “proxy” that we create in advance with the command:

docker network create -d overlay proxy

The important setting here is to use the overlay driver so that the network is available to containers we run in swarm mode.

Step 3: Traefik configuration

Traefik is configured with a traefik.toml file that tells it what entrypoints to configure, and predefined frontends and backends, and what sources to use for updating its configuration. It's also possible to make these configurations with command line options. We have built the following traefik.toml file (excluding comments):

accessLogsFile = "/dev/stdout"
defaultEntryPoints = ["https"]
[entryPoints]
  [entryPoints.http]
  address = ":80"
  [entryPoints.https]
  address = ":443"
    [entryPoints.https.tls]
      [[entryPoints.https.tls.certificates]]
      CertFile = "/run/secrets/cert.pem"
      KeyFile = "/run/secrets/key.pem"
[web]
address = ":8080"
[docker]
endpoint = "unix:///var/run/docker.sock"
domain = "localhost"
watch = true
swarmmode = true
exposedbydefault = false

The traefik.toml comments give more details on the configuration options. In brief, the above options have the following results:

accessLogsFile: This redirects the access logs to stdout where they appear in docker service logs.
defaultEntryPoints: This defaults containers to https only.
entryPoints: This configures an http listener on port 80 and an https listener on port 443 with the TLS certificate in the secrets folder.
web: This sets up a dashboard on port 8080 to see the current configuration.
docker endpoint: This uses the docker socket to monitor the swarm manager for changes to running swarm services.
docker watch: watches the above endpoint for changes.
docker swarmmode: watches swarm services instead of docker containers
docker exposedbydefault: disables behavior of exposing every service, only exposes those with the label “traefik.enable=true”.

Step 4: Run Traefik in Swarm Mode

With the prerequisites done, it's time to make our traefik containers in swarm mode. We are publishing 3 ports with Traefik:

HTTP 80: This is configured with a redirect to HTTPS.
HTTPS 443: This is the reverse proxy to all of our backend containers.
HTTP 8080: This is a traefik dashboard you can use internally to see how traefik is currently configured and check the health.

Configs and Secrets

Before creating the traefik container, we need to manage a few dependencies. To allow traefik to migrate between nodes in the swarm and still have access to the TLS certificates and traefik.toml file we have created, we are using docker configs and secrets. These import your files into docker's raft based internal key value store, and automatically create files inside our containers regardless of where the container is running. Prior to swarm mode, these files may have been left unencrypted on the filesystem and mounted into the container directly from the docker host. With swarm mode, mounting files from the host would require you to manually synchronize these files across all nodes in the swarm or depend on an external fileserver.

While convenient and secure, there are two downsides to the configs and secrets. The first is that these are immutable, so if you need to change a configuration file stored in a docker config you need to change the config name with a version number so that the change gets pushed. Alternatively you can delete and recreate the config or secret with the same name if it's currently unused. The second disadvantage is that configs and secrets only work with containers running in docker's swarm mode, so if developers are running containers with docker run or docker-compose they won't have access to this feature. We are developing with swarm mode, and have made versioning these part of our workflow, to get all the advantages they provide and mirror the production environment.

Traefik's HTTP to HTTPS Redirect Bug

While creating this example, we encountered our first Traefik issue, #1957, where the path based frontend rules we will be using strip the path off of the request before the redirect is processed. We have worked around this by running an nginx container with a single configuration to redirect every request it receives to HTTPS. Once this issue is resolved, we'll be able to make a small configuration change to the traefik.toml file and remove the nginx redirect container.

Traefik Stack Definition

To deploy the traefik stack, we run docker stack deploy -c docker-compose.traefik.yml traefik with the following docker-compose.traefik.yml file:

version: '3.3'

networks:
  proxy:
    external:
      name: proxy

configs:
  traefik_toml_v2:
    file: ./traefik.toml
  nginx_conf:
    file: ./nginx-redirect.conf

secrets:
  traefik_cert:
    file: ./tls/cert.pem
  traefik_key:
    file: ./tls/key.pem

services:
  traefik:
    image: traefik:1.4
    deploy:
      replicas: 2
      placement:
        constraints:
        - node.role == manager
    volumes:
    - /var/run/docker.sock:/var/run/docker.sock
    networks:
    - proxy
    ports:
    - target: 80
      protocol: tcp
      published: 80
      mode: ingress
    - target: 443
      protocol: tcp
      published: 443
      mode: ingress
    - target: 8080
      protocol: tcp
      published: 8080
      mode: ingress
    configs:
    - source: traefik_toml_v2
      target: /etc/traefik/traefik.toml
      mode: 444
    secrets:
    - source: traefik_cert
      target: cert.pem
      uid: "0"
      mode: 400
    - source: traefik_key
      target: key.pem
      uid: "0"
      mode: 400
  nginx_redirect:
    image: nginx:1.13
    networks:
    - proxy
    deploy:
      replicas: 2
      labels:
      - traefik.frontend.entryPoints=http
      - traefik.frontend.rule=PathPrefix:/
      - traefik.docker.network=proxy
      - traefik.port=80
      - traefik.enable=true
    configs:
    - source: nginx_conf
      target: /etc/nginx/conf.d/default.conf
      mode: 444

The above docker-compose.traefik.yml creates the secrets, configs, and starts the traefik and nginx redirect containers. The traefik container is configured to expose ports 80, 443, and 8080 on the ingress network so they can be reached from any docker node in the swarm. To allow traefik to monitor the swarm services for dynamic configuration changes, we have mounted in the docker socket and have constrained it to run on a manager (the docker socket on workers do not have access to query swarm services). The nginx redirect container has the labels in there to configure traefik to send http requests to it. We will go into the labels in more detail in the backends discussion since nginx is a very simple backend service.

Step 5: Stack Definition for Backends

Our containers being run in the backend for this example are a collection of web servers: nginx, apache, and caddy. For each of these, we configure labels on the swarm service to dynamically configure traefik with the frontend rule and how to communicate with the container (docker network and port the application is listening on). When traefik is configured in swarm mode, these labels must be defined on the service rather than the individual containers, which is achieved by defining them in the deploy section of the compose file. The labels we are using for traefik are:

traefik.frontend.endPoints: This defaults to https in the traefik.toml, but can be overridden, e.g. the nginx http to https redirect seen in the traefik stack definition.
traefik.frontend.rule: This rule must be matched for traefik to send a request to this backend. Traefik has default priorities that may be overridden to handle multiple matching rules. We are using the path in the URL to identify the desired backend with a “PathPrefixStrip” rule. Other rules are described in traefik's documentation.
traefik.docker.network: If your backend service is connected to multiple networks, this is required to be set to the network in common with traefik. Otherwise traefik may configure itself for an IP it is unable to reach.
traefik.port: This is the port inside the container. Note that we did not need to publish or expose this port, traefik connects from container to container directly over the “proxy” network, but needs to know which port to connect to.
traefik.enable: This is needed when we do not expose every service by default. If set to “true”, traefik will proxy for this service.

Deploying this stack looks very similar to deploying the traefik stack: docker stack deploy -c docker-compose.webapps.yml webapps. The following docker-compose.webapps.yml was used:

version: '3.3'

networks:
  proxy:
    external: true

services:
  nginx:
    image: nginx:1.13
    networks:
    - proxy
    deploy:
      replicas: 2
      labels:
      - traefik.frontend.rule=PathPrefixStrip:/nginx
      - traefik.docker.network=proxy
      - traefik.port=80
      - traefik.enable=true
  apache:
    image: httpd:2.4
    networks:
    - proxy
    deploy:
      replicas: 2
      labels:
      - traefik.frontend.rule=PathPrefixStrip:/apache
      - traefik.docker.network=proxy
      - traefik.port=80
      - traefik.enable=true
  caddy:
    image: abiosoft/caddy:0.10.7
    networks:
    - proxy
    deploy:
      replicas: 2
      labels:
      - traefik.frontend.rule=PathPrefixStrip:/caddy
      - traefik.docker.network=proxy
      - traefik.port=2015
      - traefik.enable=true

High Availability

Note that in both of these docker-compose.*.yml files, we configured for 2 replicas of each container. This allows for rolling upgrades of the application. Docker will also make a best effort to place each replica on a different node in the docker swarm, minimizing the risk of a stopped node stopping the entire instance of the service. By placing a load balancer in front of the docker swarm with at least three managers, it's possible to provide an environment that will continue to operate even if a node fails.

Test It Out

If you've been running these commands on your own cluster of docker swarm, you can now try reaching each of the web servers by their path, and even take advantage of the port 80 redirection. Try the following if you deployed on your local development machine:

http://127.0.0.1/nginx

http://127.0.0.1/apache

http://127.0.0.1/caddy

You'll see that you get redirected to the HTTPS entrypoint, and with the self signed TLS certificates you'll get a browser warning where you'll need to approve the certificate.

To view the traefik dashboard on your local development machine, go to the following:

http://127.0.0.1:8080/

With that, you're now ready to add your own services behind traefik by configuring your service labels and connecting it to the proxy network. There's no need to know where in the swarm your service is running or track an ever growing list of published ports.