BoxBoat Blog
Service updates, customer stories, and tips and tricks for effective DevOps
The Security Benefits of Podman-in-Docker vs Docker-in-Docker in Gitlab (And How To Set That Up)
by Carly Rodriguez | Monday, Feb 7, 2022 | Education Docker Security
Containers have given the tech industry a convenient way to bundle up dependencies and code into a portable image that can run seamlessly across different computing environments. This convenience, however, can sometimes come at a cost. In order to leverage docker building capabilities within a Gitlab environment, the docker-executor must be given host privileges to run a docker-in-docker service that will allow connection to the docker daemon on the host machine. Unfortunately this will also enable all of the capabilities the host can do. Gitlab currently has a solution for that; namely, leverage the official kaniko container from GCR to create builds without Docker-in-Docker enabled in privileged mode. With the release of podman however, we no longer have to rely on a docker daemon to run docker containers, and subsequently, can leverage the buildah capabilities that run behind the scenes during podman builds and run within a rootless namespace on the container. This blog will provide an example build process on how to configure a gitlab-runner to allow an unprivileged podman container to run podman build
commands and podman run
commands.
Runner Configuration
Docker Executor Gitlab Runner:
The following gitlab-runner was registered on an EC2 instance running off of a RHEL 8 AMI provided by AWS:
sudo gitlab-runner register
--non-interactive
--name "docker-runner-configged"
--url "<Your-URL>"
--registration-token "<Your-Token>"
--executor docker
--docker-image alpine:latest
--tag-list "docker-runner-configged"
This configuration uses the default settings to create a docker-executor using docker itself, but running unprivileged. The following section will indicate what is required to satisfy the minimum requirements needed to run podman build
vs podman run
commands
Config.toml Modifications (podman build, podman push, podman pull):
Luckily, if your gitlab-runner will only require you to run podman build
and podman push
or podman pull
commands, the default config.toml that is created at the gitlab-runners inception will suffice! Your config.toml file should look like the following (If you're running your gitlab-runners in sudo mode, it is located in etc/gitlab-runner/config.toml
):
[[runners]]
name = "docker-runner-configged"
url = "<Your-URL>"
token = "<Your-Token">
executor = "docker"
[runners.custom_build_dir]
[runners.cache]
[runners.cache.s3]
[runners.cache.gcs]
[runners.cache.azure]
[runners.docker]
tls_verify = false
image = "alpine:latest"
privileged = false
disable_entrypoint_overwrite = false
oom_kill_disable = false
disable_cache = false
volumes = ["/cache"]
shm_size = 0
BUT WAIT! You're probably getting an error on your podman build
step that looks similar to the following:
creating cgroup directory /sys/fs/cgroup/perf_event/buildah-buildah872758903: No such file or directory
This is because we need to isolate this step to run in chroot mode (podman build
is using buildah under the hood). The following is an example of how to do that with podman:
podman build --isolation=chroot -t $CONTAINER_BUILD_IMAGE .
You should get a successful build now with podman using podman-in-docker unprivileged!
Config.toml Modifications (podman run in rootful podman):
In order to have the docker runner run images with podman without a --privileged
state, we will need to then modify the config.toml file to include a few extra things. For this demo, we are only going to add the most minimally required necessary tools and permissions to allow podman to run images seamlessly within a podman image (Podman-in-Docker). The config.toml will look like the following:
[[runners]]
name = "docker-runner-configged"
url = "<Your-URL>"
token = "<Your-Token>"
executor = "docker"
[runners.custom_build_dir]
[runners.cache]
[runners.cache.s3]
[runners.cache.gcs]
[runners.cache.azure]
[runners.docker]
tls_verify = false
image = "alpine:latest"
privileged = false
disable_entrypoint_overwrite = false
oom_kill_disable = false
disable_cache = false
volumes = ["/cache"]
shm_size = 0
devices = ["/dev/fuse"]
security_opt = ["label=disable", "seccomp=unconfined"]
cap_add = ["sys_admin", "mknod"]
Things to note are:
cap_add = [“sys_admin”, “mknod”]
SYS_ADMIN: While we are giving the container a capability addition of SYS_ADMIN, this is still more secure than granting --privileged
access to the entire podman container, though still dangerous. One example of that is that --privileged
will mount /dev and /sys as read/write, where as SYS_ADMIN mounts them only as read only. This means that a privileged container has full access to devices on the system, whereas SYS_ADMIN does not. Its not generally recommended to give that sort of access to a container, but this is required when running a rootful docker container. You can skip this step when running rootless
MKNOD: This is required to create devices in /dev
By default, the podman container runs with the user root
. Both of these are required when running in rootful mode with the podman container.
THIS STEP CAN BE OMITTED DURING A ROOTLESS PODMAN CONFIGURATION (SEE gitlab-ci.yml SECTION BELOW)
devices = ["/dev/fuse”]
/dev/fuse: Required to be added onto the container so that podman can use fuse-overlayfs inside of the container
security_opt = [“label=disable”, “seccomp=unconfined”]
label=disable: Disables SELinux. This will allow the containerized processes to mount all of the file systems required to run inside a container.
seccomp=unconfined: Since this is Podman-in-Docker, we need to relax the seccomp
security requirement on docker.
gitlab-ci.yml Configuration:
Using the stable version of the podman container provided directly from RedHat is the container we are pulling to showcase the capability of podman-in-docker.
ROOTFUL (Requires dangerous SYS_ADMIN and MKNOD Capability added)
image:
name: quay.io/podman/stable
script:
- podman login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY
- podman --version
- whoami <will display `root` here>
- pwd
- podman build --isolation=chroot -t $CONTAINER_BUILD_IMAGE .
- podman run alpine echo hello
- podman push $CONTAINER_BUILD_IMAGE
the root
user is used by default here.
For even more added security, we can instead change the user to podman
, and omit the cap_add
step above that is providing the dangerous sys_admin and mknod capabilities to the image. You can do that the following way using the entrypoint
keyword, or by building a new container that changes its USER by default while using the quay.io/podman/stable
as its base image:
image:
name: quay.io/podman/stable
entrypoint: ['bash', '-c', 'exec su podman -c bash']
script:
- podman login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY
- podman --version
- whoami <shoud now display `podman` here>
- pwd
- podman build --isolation=chroot -t $CONTAINER_BUILD_IMAGE .
- podman run alpine echo hello
- podman push $CONTAINER_BUILD_IMAGE
With SYS_ADMIN and MKNOD not being needed and running in a rootless state, we now have a secure solution for running podman-in-docker!
Final Thoughts
When running docker containers, it's always recommended that we reduce the privileges granted to it to only leverage the capabilities it needs to do. With podman, we now have a solution that gives us that capability in gitlab that also gives us the similar capabilities that docker has.