The Security Benefits of Podman-in-Docker vs Docker-in-Docker in Gitlab (And How To Set That Up)
Containers have given the tech industry a convenient way to bundle up dependencies and code into a portable image that can run seamlessly across different computing environments. This convenience, however, can sometimes come at a cost. In order to leverage docker building capabilities within a Gitlab environment, the docker-executor must be given host privileges to run a docker-in-docker service that will allow connection to the docker daemon on the host machine. Unfortunately this will also enable all of the capabilities the host can do. Gitlab currently has a solution for that; namely, leverage the official kaniko container from GCR to create builds without Docker-in-Docker enabled in privileged mode. With the release of podman however, we no longer have to rely on a docker daemon to run docker containers, and subsequently, can leverage the buildah capabilities that run behind the scenes during podman builds and run within a rootless namespace on the container. This blog will provide an example build process on how to configure a gitlab-runner to allow an unprivileged podman container to run
podman build commands and
podman run commands.
Docker Executor Gitlab Runner:
The following gitlab-runner was registered on an EC2 instance running off of a RHEL 8 AMI provided by AWS:
sudo gitlab-runner register --non-interactive --name "docker-runner-configged" --url "<Your-URL>" --registration-token "<Your-Token>" --executor docker --docker-image alpine:latest --tag-list "docker-runner-configged"
This configuration uses the default settings to create a docker-executor using docker itself, but running unprivileged. The following section will indicate what is required to satisfy the minimum requirements needed to run
podman build vs
podman run commands
Config.toml Modifications (podman build, podman push, podman pull):
Luckily, if your gitlab-runner will only require you to run
podman build and
podman push or
podman pull commands, the default config.toml that is created at the gitlab-runners inception will suffice! Your config.toml file should look like the following (If you're running your gitlab-runners in sudo mode, it is located in
[[runners]] name = "docker-runner-configged" url = "<Your-URL>" token = "<Your-Token"> executor = "docker" [runners.custom_build_dir] [runners.cache] [runners.cache.s3] [runners.cache.gcs] [runners.cache.azure] [runners.docker] tls_verify = false image = "alpine:latest" privileged = false disable_entrypoint_overwrite = false oom_kill_disable = false disable_cache = false volumes = ["/cache"] shm_size = 0
BUT WAIT! You're probably getting an error on your
podman build step that looks similar to the following:
creating cgroup directory /sys/fs/cgroup/perf_event/buildah-buildah872758903: No such file or directory
This is because we need to isolate this step to run in chroot mode (
podman build is using buildah under the hood). The following is an example of how to do that with podman:
podman build --isolation=chroot -t $CONTAINER_BUILD_IMAGE .
You should get a successful build now with podman using podman-in-docker unprivileged!
Config.toml Modifications (podman run in rootful podman):
In order to have the docker runner run images with podman without a
--privileged state, we will need to then modify the config.toml file to include a few extra things. For this demo, we are only going to add the most minimally required necessary tools and permissions to allow podman to run images seamlessly within a podman image (Podman-in-Docker). The config.toml will look like the following:
[[runners]] name = "docker-runner-configged" url = "<Your-URL>" token = "<Your-Token>" executor = "docker" [runners.custom_build_dir] [runners.cache] [runners.cache.s3] [runners.cache.gcs] [runners.cache.azure] [runners.docker] tls_verify = false image = "alpine:latest" privileged = false disable_entrypoint_overwrite = false oom_kill_disable = false disable_cache = false volumes = ["/cache"] shm_size = 0 devices = ["/dev/fuse"] security_opt = ["label=disable", "seccomp=unconfined"] cap_add = ["sys_admin", "mknod"]
Things to note are:
cap_add = [“sys_admin”, “mknod”]
SYS_ADMIN: While we are giving the container a capability addition of SYS_ADMIN, this is still more secure than granting
--privileged access to the entire podman container, though still dangerous. One example of that is that
--privileged will mount /dev and /sys as read/write, where as SYS_ADMIN mounts them only as read only. This means that a privileged container has full access to devices on the system, whereas SYS_ADMIN does not. Its not generally recommended to give that sort of access to a container, but this is required when running a rootful docker container. You can skip this step when running rootless
MKNOD: This is required to create devices in /dev
By default, the podman container runs with the user
root. Both of these are required when running in rootful mode with the podman container.
THIS STEP CAN BE OMITTED DURING A ROOTLESS PODMAN CONFIGURATION (SEE gitlab-ci.yml SECTION BELOW)
devices = ["/dev/fuse”]
/dev/fuse: Required to be added onto the container so that podman can use fuse-overlayfs inside of the container
security_opt = [“label=disable”, “seccomp=unconfined”]
label=disable: Disables SELinux. This will allow the containerized processes to mount all of the file systems required to run inside a container.
seccomp=unconfined: Since this is Podman-in-Docker, we need to relax the
seccomp security requirement on docker.
Using the stable version of the podman container provided directly from RedHat is the container we are pulling to showcase the capability of podman-in-docker.
ROOTFUL (Requires dangerous SYS_ADMIN and MKNOD Capability added)
image: name: quay.io/podman/stable script: - podman login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY - podman --version - whoami <will display `root` here> - pwd - podman build --isolation=chroot -t $CONTAINER_BUILD_IMAGE . - podman run alpine echo hello - podman push $CONTAINER_BUILD_IMAGE
root user is used by default here.
For even more added security, we can instead change the user to
podman, and omit the
cap_add step above that is providing the dangerous sys_admin and mknod capabilities to the image. You can do that the following way using the
entrypoint keyword, or by building a new container that changes its USER by default while using the
quay.io/podman/stable as its base image:
image: name: quay.io/podman/stable entrypoint: ['bash', '-c', 'exec su podman -c bash'] script: - podman login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY - podman --version - whoami <shoud now display `podman` here> - pwd - podman build --isolation=chroot -t $CONTAINER_BUILD_IMAGE . - podman run alpine echo hello - podman push $CONTAINER_BUILD_IMAGE
With SYS_ADMIN and MKNOD not being needed and running in a rootless state, we now have a secure solution for running podman-in-docker!
When running docker containers, it's always recommended that we reduce the privileges granted to it to only leverage the capabilities it needs to do. With podman, we now have a solution that gives us that capability in gitlab that also gives us the similar capabilities that docker has.