The Moby Inception

by Brandon Mitchell | Friday, May 12, 2017 | Docker

[ Note, this is a follow up to our earlier blog post about the Moby Project. Check that one out for a less developer oriented take on the Moby project. ]

The announcement of Moby Project at DockerCon 2017 left many confused. My own take is that it's confusing because it means at least three different things depending on what piece you are talking about:

1. The Moby Project which is Docker's upstream development that's open to the community.
2. The Moby Tool that is used to package together pieces into a VM image.
3. The Moby Repo where the Docker engine was migrated to.

Lets dig a little deeper into what's happening with each one of these.

The Moby Project

This is often considered Docker's equivalent of RedHat's Fedora, the community's development home where changes can be made to the open source tool outside of the Docker company namespace. Docker then uses this as their upstream, and creates releases based off of that for both their community and enterprise editions. From Docker's graphic, you can see these downstream releases on the right side.

The Moby project also serves as a staging ground for projects that Docker is moving out to the community. The Moby Project has their website at mobyproject.org and is on github at github.com/moby.

The Moby Tool

This is the core of Docker's announcement and visualized in the middle of Docker's graphic. This tool is used to create a minimal container based Linux OS machine image. For those that spend a lot of time in Docker for Windows or Docker for Mac, you may have seen the Moby name before running your embedded VM. This is no coincidence since Docker has been using the Moby Tool for a while now to generate these embedded VM images. The DockerCon announcement was simply Docker releasing more of this code out to the community.

The input to the Moby Tool is a yaml file that contains configurations and pointers to the needed containers. Every startup script and service is run as a container using runc and containerd. OS tasks like mounting filesystems, dhcp, and the docker engine itself each run within a container. Even the kernel and init are packaged as containers and unpacked into the root filesystem of the OS image by the Moby Tool. In many ways, containers are Moby Tool's equivalent of an RPM or DEB file.

The LinuxKit project is currently the source for these containers. Browsing through their repo, you will find Makefiles that call docker build. And the included Dockerfiles pull from the various upstream sources like containerd and runc.

The resulting VM is run from an initramfs, in memory. On any failures, restarting the OS returns to the initial state and upgrades are shipped as new VM images. For persistent data, you use an external volume. This is the container model of immutability and versioning, but implemented at the OS level.

The Moby Repo

One of the first ways many developers learned about the Moby Project was finding the docker engine in github.com/docker/docker had been moved to github.com/moby/moby. This is Docker's way of using the Moby Project as a staging area for transitioning projects to the community. It's also a repo that is in rapid flux as the various parts of the transition are made. The Moby Tool, previously residing in the LinuxKit repo, will likely be moved to the moby/moby repo. And the docker engine code that is in moby/moby will soon be shifted to a subdirectory moby/moby/monolith.

If that name catches your attention, that's by design. There's another big change on the horizon…

Breaking Up the Monolith

Up to this point, we've been talking about the Moby Project and virtual machines. But at the heart of docker is their engine for building, shipping, and running containers. And this engine is growing into a monolith from the various components merged into one large binary. Components like containerd, swarmkit, and libnetwork which each have their own upstream repo are get packaged together and tightly integrated, making changes more difficult.

Docker has plans to practice what they preach and break up this monolith into a set of microservices. Each of these components will be integrated using gRPC API's and wired together with a small gRPC multiplexer not unlike the reverse proxies many of us use when building our own microservices architectures. Docker plans to chip away at their monolith until the moby/moby/monolith directory of code is empty. The resulting docker engine would be a small API routing traffic via gRPC to a pluggable set of components that can each be swapped in and out, independently of each other.

This will be another layer of the Moby inception. Containers in the OS running containers for the applications, and microservices for the engine running your microservices. At every level of the abstraction, it's becoming the same pattern of pluggable components.

Help wanted

If this interests you, and you enjoy programming in Go, check out the Moby forums at forums.mobyproject.org and the #moby channel on the Docker community slack, because they are looking for help to make this happen.