BoxBoat Blog

Service updates, customer stories, and tips and tricks for effective DevOps

x ?

Get Hands-On Experience with BoxBoat's Cloud Native Academy

What’s New in Docker 1.12 – Part 1

by Brandon Mitchell | Wednesday, Aug 10, 2016 | Docker

featured.jpg

Docker 1.12 adds quite a few new features, the biggest of those being the new Swarm Mode that was announced at DockerCon 2016. However, the release notes includes a long list of other changes, some of which may have been lost in the excitement over Swarm. In this first part, we are taking a deeper dive into the changes introduced to building images, distribution, and logging.

Building Images

The release notes begins with building images using Dockerfiles and one small change to .dockerignore. The first of these changes to the Dockerfile is second only to the Swarm updates in terms of it's popularity, healthcheck!

Healthcheck

Pull request #23218

The healthcheck is a user configurable entry in the Dockerfile that tells Docker how to monitor your container. Before this, the most Docker would say is whether the process was still up. Now you can define your own command that gets run inside of your container which Docker will use to monitor your container's health. The new Dockerfile entry has the following syntax:

HEALTHCHECK  CMD

Possible options include:

  • --interval=DURATION (default: 30s)
  • --timeout=DURATION (default: 30s)
  • --retries=N (default: 1)

Docker looks at the exit value of the cmd to determine the state:

  • 0: success – the container is healthy and ready for use
  • 1: unhealthy – the container is not working correctly
  • 2: starting – the container is not ready for use yet

Only one healthcheck is available per image, with the last definition overwriting all previous definitions, identical to the behavior of entries like CMD and ENTRYPOINT. To disable the healthcheck (e.g. from an upstream image you included in your FROM) you use:

HEALTHCHECK none

A failing healthcheck:

  • Generates an event
  • Shows in docker ps
  • Does not restart container when unhealthy outside of Swarm
  • Does not delay startup of dependent containers with Compose
  • Swarm containers will be stopped and replaced with healthy instances

With Swarm mode, a container also avoids being ready on startup until this check succeeds. One important note is that the command provided will bypass the SHELL, CMD, and ENTRYPOINT.

Healthcheck Example

To simulate this, we used the following Dockerfile:

# Using official python runtime base image
FROM python:2.7-alpine

# Set the application directory
WORKDIR /app

# Install our requirements.txt
ADD requirements.txt /app/requirements.txt
RUN pip install -r requirements.txt
# install curl for healthcheck
RUN apk update && apk add curl

# Copy our code from the current folder to /app inside the container
ADD . /app

# Make port 80 available for links and/or publish
EXPOSE 80

HEALTHCHECK --interval=10s --timeout=5s --retries=1 \
  CMD curl -f http://localhost/status || exit 1
# HEALTHCHECK NONE # disable healthcheck from parent image

# Define our command to be run when launching the container
CMD ["gunicorn", "app:app", "-b", "0.0.0.0:80", "--log-file", "-", \
     "--access-logfile", "-", "--workers", "1", "--keep-alive", "0"]

The application itself simply listens on port 80 for requests to change the “/status” link from healthy to failed with a 500 HTTP status. The important thing to note from the above Dockerfile is the install of curl for the healthcheck since this check runs inside of your container. We then launch the container and monitor the status with the following commands:

$ docker run -d -p 8080:80 --name ex-hc ex-healthcheck

$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
950f934c51be ex-healthcheck "gunicorn app:app -b " 3 seconds ago Up 1 seconds (health: starting) 0.0.0.0:8080->80/tcp ex-hc

$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
950f934c51be ex-healthcheck "gunicorn app:app -b " About a minute ago Up About a minute (healthy) 0.0.0.0:8080->80/tcp ex-hc

# "crash" the application

$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
950f934c51be ex-healthcheck "gunicorn app:app -b " About a minute ago Up About a minute (unhealthy) 0.0.0.0:8080->80/tcp ex-hc


$ docker inspect ex-hc
# ...
        "State": {
            "Status": "running",
            "Running": true,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 24031,
            "ExitCode": 0,
            "Error": "",
            "StartedAt": "2016-08-01T18:00:27.004722687Z",
            "FinishedAt": "0001-01-01T00:00:00Z",
            "Health": {
                "Status": "unhealthy",
                "FailingStreak": 1,
                "Log": [
                    {
                        "Start": "2016-08-01T14:01:07.400120773-04:00",
                        "End": "2016-08-01T14:01:07.524140936-04:00",
                        "ExitCode": 0,
                        "Output": "Status is up"
                    },
                    {
                        "Start": "2016-08-01T14:01:17.524300821-04:00",
                        "End": "2016-08-01T14:01:17.612064573-04:00",
                        "ExitCode": 0,
                        "Output": "Status is up"
                    },
                    {
                        "Start": "2016-08-01T14:01:27.612211348-04:00",
                        "End": "2016-08-01T14:01:27.712360583-04:00",
                        "ExitCode": 0,
                        "Output": "Status is up"
                    },
                    {
                        "Start": "2016-08-01T14:01:37.712530411-04:00",
                        "End": "2016-08-01T14:01:37.80402707-04:00",
                        "ExitCode": 0,
                        "Output": "Status is up"
                    },
                    {
                        "Start": "2016-08-01T14:01:47.804203459-04:00",
                        "End": "2016-08-01T14:01:47.904040542-04:00",
                        "ExitCode": 1,
                        "Output": ""
                    }
                ]
            }
        }
# ...

In addition to monitoring the status in docker ps, the inspect output shows the current healthcheck state along with the results of the past runs. Those results include a timestamp, output, and exit code to make debugging easier.

Healthcheck with Swarm

To take full advantage of the healthcheck, you need a tool listening for the events and recovering from the failures. Docker provides this out of the box as part of their Swarm mode offering. Here's a similar scenario run with 3 replicas in Swarm:

$ docker service create -p 8888:80 --replicas 3 --network hc-net \
  --name ex-hc ex-healthcheck:latest

$ docker service ps ex-hc
ID                         NAME     IMAGE                  NODE              DESIRED STATE  CURRENT STATE           ERROR
bvb62pynvbfyvo1ai7q7szcjc  ex-hc.1  ex-healthcheck:latest  bmitch  Running        Starting 5 seconds ago
6rov5mr7yjw9ehcopjqabx3bk  ex-hc.2  ex-healthcheck:latest  bmitch  Running        Starting 4 seconds ago
5v4w813vwaw5zjq7hmzuwtb8k  ex-hc.3  ex-healthcheck:latest  bmitch  Running        Starting 4 seconds ago

# "crash" one container
# preparing state from container creation

$ docker service ps ex-hc
ID                         NAME         IMAGE                  NODE              DESIRED STATE  CURRENT STATE            ERROR
e27vxrmyjy3u4bfkewar8er87  ex-hc.1      ex-healthcheck:latest  bmitch  Ready          Preparing 1 seconds ago
bvb62pynvbfyvo1ai7q7szcjc   \_ ex-hc.1  ex-healthcheck:latest  bmitch  Shutdown       Complete 1 seconds ago
6rov5mr7yjw9ehcopjqabx3bk  ex-hc.2      ex-healthcheck:latest  bmitch  Running        Running 23 seconds ago
5v4w813vwaw5zjq7hmzuwtb8k  ex-hc.3      ex-healthcheck:latest  bmitch  Running        Running 23 seconds ago

# starting state before health check succeeds
$ docker service ps ex-hc
ID                         NAME         IMAGE                  NODE              DESIRED STATE  CURRENT STATE            ERROR
e27vxrmyjy3u4bfkewar8er87  ex-hc.1      ex-healthcheck:latest  bmitch  Running        Starting 11 seconds ago
bvb62pynvbfyvo1ai7q7szcjc   \_ ex-hc.1  ex-healthcheck:latest  bmitch  Shutdown       Complete 16 seconds ago
6rov5mr7yjw9ehcopjqabx3bk  ex-hc.2      ex-healthcheck:latest  bmitch  Running        Running 38 seconds ago
5v4w813vwaw5zjq7hmzuwtb8k  ex-hc.3      ex-healthcheck:latest  bmitch  Running        Running 38 seconds ago

# running state after healthcheck is successful, running time resets
$ docker service ps ex-hc
ID                         NAME         IMAGE                  NODE              DESIRED STATE  CURRENT STATE            ERROR
e27vxrmyjy3u4bfkewar8er87  ex-hc.1      ex-healthcheck:latest  bmitch  Running        Running 6 seconds ago
bvb62pynvbfyvo1ai7q7szcjc   \_ ex-hc.1  ex-healthcheck:latest  bmitch  Shutdown       Complete 23 seconds ago
6rov5mr7yjw9ehcopjqabx3bk  ex-hc.2      ex-healthcheck:latest  bmitch  Running        Running 45 seconds ago
5v4w813vwaw5zjq7hmzuwtb8k  ex-hc.3      ex-healthcheck:latest  bmitch  Running        Running 45 seconds ago

In this scenario, no user intervention was needed to recover from the outage. Swarm detected the failure, shutdown the faulty container, and started a new one.

Shell

Pull request #22489

Docker introduced a new SHELL command for Dockerfiles. This can be used to modify the shell used in RUN, CMD, and ENTRYPOINT when they are defined as strings. When these values are defined as JSON arrays, they have always bypassed the shell and will continue to do so. If you define the SHELL multiple times, the last value is used. This value is inherited from parent images.

The main use case for this will be Windows images where they are looking at the ability to swap between cmd and powershell with the following entries:

SHELL ["powershell", "-command"]
SHELL ["cmd", "/S"", "/C"]

Here's an example Dockerfile that swaps between multiple shells:

FROM debian:latest

RUN echo "Default shell: $0" >>/shell.log

COPY bmitch-sh /bmitch-sh

SHELL ["/bin/sh", "-c"]
RUN echo "Bourne shell: $0" >>/shell.log

SHELL ["/bin/bash", "-c"]
RUN echo "Bash shell: $0" >>/shell.log

SHELL ["/bmitch-sh"]
RUN echo "BMitch shell: $0" >>/shell.log

ENTRYPOINT ["/bmitch-sh"]
HEALTHCHECK CMD exit 0

The bmitch-sh is just a wrapper around /bin/sh that logs the command it's running to shell.log. The resulting shell.log when the container is started with a tail -f /dev/null looks like:

Default shell: /bin/sh
Bourne shell: /bin/sh
Bash shell: /bin/bash
bmitch-sh running: echo "BMitch shell: $0" >>/shell.log
bmitch-sh running: tail -f /dev/null

Note how the HEALTHCHECK does not write to this log which indicates it's not subject to the SHELL setting.

Escape

Pull request #22268

Another change being added in preparation for Windows images is the ability to change the escape character from \ to `. The backslashes, while well known as an escape in Linux, are unfortunately path separators in Windows. This directive, unlike other Dockerfile commands, is defined as a comment at the top of the Dockerfile in the format # escape=`. It only applies to the current Dockerfile, not to any child images that are built off of the resulting image. This prevents an upstream change from breaking your Dockerfiles.

Note that only the two preselected escape characters are currently allowed, \ and `. This was done to avoid Dockerfiles that changed the escape to unexpected characters like a space.

A Dockerfile for Windows will look like:

# escape=`

# Example Dockerfile with escape character changed

FROM windowsservercore
COPY testfile.txt c:\
RUN dir c:\

Comments in .dockerignore

Docker added the ability to include comments inside of .dockerignore files. These use the # prefix and are only supported at the beginning of the line. If you include a # on the same line as a filename, it is interpreted as part of the filename and the file is no longer ignored. Here's an example of what a .dockerignore file would look like:

# Sample comment
.git
passwords # this file is still included

Other Build Changes

One other change in the build section is the ability to build images without a bridged network configured in the docker engine. This is a pretty unique use case that most users won't encounter. Pull request #22932 has more details.

Distribution

There were only a few changes to distribution in this release, focusing on incremental changes to push, pull, save, and load.

Concurrent Downloads

Pull request #22445

Docker now allows you to configure the number of concurrent downloads and uploads for push and pull operations. The defaults of 3 downloads and 5 uploads remain the same. This change is made on the Docker daemon and affects all pull and pull operations on that host. If you're on a restricted network that doesn't allow more than one connection out per destination concurrently, adjusting these settings to 1 each may resolve issues. Conversely, if you are on a large server and fast network with lots of images to move, increasing these values may improve your throughput.

To adjust these settings you'll need to modify your engine's startup scripts with the following flags:

$ dockerd --help
# ...
  --max-concurrent-downloads=3             Set the max concurrent downloads for each pull
  --max-concurrent-uploads=5               Set the max concurrent uploads for each push
# ...

Show Image Name or ID on Load

Pull request #23377

This next one is small but can help with debugging scripts that move lots of images from the docker save and docker load CLI. Now after loading an image, the client will output the image name (tag) or if it's untagged, it will show the image id. Before, load wouldn't output anything unless there was an error, so if you loaded multiple images, you may not know which threw the error.

$ docker save busybox:latest > busybox.tar

# ... transfer file to a new system

$ docker load

Other Distribution Changes

Docker made a few bug fixes, including a fix for image digest that affected cleaning untagged images in pull request #23996.

There was also additional support made for proxy settings when pushing and pulling images in pull request #22316.

Logging

The logging changes were relatively minor for this release and focused on additions that would help external integrations.

Log Labels

Pull request #21889

Docker extended --log-opt to allow “labels” to be included in the output. This will be useful for log aggregation tools that gather logs from multiple containers and need to filter the results. You pass the name of the label to include with --log-opt labels= where list can be a comma separate list. The labels themselves can be assigned to the individual containers with --label =. The labels will only be shown in the output of docker logs if you include the --details option. Putting that all together looks like:

$ docker run -d --name ex-log-label \
  --label hello=world --label env=dev --label site=us_east \
  --log-opt labels=hello,site \
  busybox echo hi

$ docker logs --details ex-log-label
hello=world,site=us_east hi

Microsecond Support in Syslog Driver

Pull request #21844

The syslog logging driver has a new output format “rfc5424micro” that includes microseconds in the timestamps. This timestamp is otherwise identical to the rfc5424 format that was already available. This will be useful if you're using logs to debug performance problems. Here's an example of using this new format:

$ docker run -d --name ex-log-micro \
  --log-driver=syslog --log-opt syslog-format=rfc5424micro \
  busybox echo hello world

$ sudo tail /var/log/messages | grep hello world
Aug  3 11:18:40 debian <30>1 2016-08-03T11:18:40.881449-04:00 debian docker/80775302d6ab 898 docker/80775302d6ab hello world

Other Logging Changes

Other changes to the logging include:

Stay Tuned

That finishes up part 1. The part 2 will looks at the changes to the networking, a new plugin interface, and updates to the client and API.