The Security Benefits of Podman-in-Docker vs Docker-in-Docker in Gitlab (And How To Set That Up)

by Carly Rodriguez | Monday, Feb 7, 2022 | Education Docker Security

Containers have given the tech industry a convenient way to bundle up dependencies and code into a portable image that can run seamlessly across different computing environments. This convenience, however, can sometimes come at a cost. In order to leverage docker building capabilities within a Gitlab environment, the docker-executor must be given host privileges to run a docker-in-docker service that will allow connection to the docker daemon on the host machine. Unfortunately this will also enable all of the capabilities the host can do. Gitlab currently has a solution for that; namely, leverage the official kaniko container from GCR to create builds without Docker-in-Docker enabled in privileged mode. With the release of podman however, we no longer have to rely on a docker daemon to run docker containers, and subsequently, can leverage the buildah capabilities that run behind the scenes during podman builds and run within a rootless namespace on the container. This blog will provide an example build process on how to configure a gitlab-runner to allow an unprivileged podman container to run podman build commands and podman run commands.

Runner Configuration

Docker Executor Gitlab Runner:

The following gitlab-runner was registered on an EC2 instance running off of a RHEL 8 AMI provided by AWS:

sudo gitlab-runner register 
--non-interactive 
--name "docker-runner-configged" 
--url "<Your-URL>" 
--registration-token "<Your-Token>" 
--executor docker 
--docker-image alpine:latest 
--tag-list "docker-runner-configged"

This configuration uses the default settings to create a docker-executor using docker itself, but running unprivileged. The following section will indicate what is required to satisfy the minimum requirements needed to run podman build vs podman run commands

Config.toml Modifications (podman build, podman push, podman pull):

Luckily, if your gitlab-runner will only require you to run podman build and podman push or podman pull commands, the default config.toml that is created at the gitlab-runners inception will suffice! Your config.toml file should look like the following (If you're running your gitlab-runners in sudo mode, it is located in etc/gitlab-runner/config.toml):

[[runners]]
  name = "docker-runner-configged"
  url = "<Your-URL>"
  token = "<Your-Token">
  executor = "docker"
  [runners.custom_build_dir]
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]
    [runners.cache.azure]
  [runners.docker]
   tls_verify = false
    image = "alpine:latest"
    privileged = false
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/cache"]
    shm_size = 0

BUT WAIT! You're probably getting an error on your podman build step that looks similar to the following:

creating cgroup directory /sys/fs/cgroup/perf_event/buildah-buildah872758903: No such file or directory

This is because we need to isolate this step to run in chroot mode (podman build is using buildah under the hood). The following is an example of how to do that with podman:

podman build --isolation=chroot -t $CONTAINER_BUILD_IMAGE .

You should get a successful build now with podman using podman-in-docker unprivileged!

Config.toml Modifications (podman run in rootful podman):

In order to have the docker runner run images with podman without a --privileged state, we will need to then modify the config.toml file to include a few extra things. For this demo, we are only going to add the most minimally required necessary tools and permissions to allow podman to run images seamlessly within a podman image (Podman-in-Docker). The config.toml will look like the following:

[[runners]]
  name = "docker-runner-configged"
  url = "<Your-URL>"
  token = "<Your-Token>"
  executor = "docker"
  [runners.custom_build_dir]
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]
    [runners.cache.azure]
  [runners.docker]
    tls_verify = false
    image = "alpine:latest"
    privileged = false
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/cache"]
    shm_size = 0
    devices = ["/dev/fuse"]
    security_opt = ["label=disable", "seccomp=unconfined"]
    cap_add = ["sys_admin", "mknod"]

Things to note are:

cap_add = [“sys_admin”, “mknod”]

SYS_ADMIN: While we are giving the container a capability addition of SYS_ADMIN, this is still more secure than granting --privileged access to the entire podman container, though still dangerous. One example of that is that --privileged will mount /dev and /sys as read/write, where as SYS_ADMIN mounts them only as read only. This means that a privileged container has full access to devices on the system, whereas SYS_ADMIN does not. Its not generally recommended to give that sort of access to a container, but this is required when running a rootful docker container. You can skip this step when running rootless

MKNOD: This is required to create devices in /dev

By default, the podman container runs with the user root. Both of these are required when running in rootful mode with the podman container.

THIS STEP CAN BE OMITTED DURING A ROOTLESS PODMAN CONFIGURATION (SEE gitlab-ci.yml SECTION BELOW)

devices = ["/dev/fuse”]

/dev/fuse: Required to be added onto the container so that podman can use fuse-overlayfs inside of the container

security_opt = [“label=disable”, “seccomp=unconfined”]

label=disable: Disables SELinux. This will allow the containerized processes to mount all of the file systems required to run inside a container.

seccomp=unconfined: Since this is Podman-in-Docker, we need to relax the seccomp security requirement on docker.

gitlab-ci.yml Configuration:

Using the stable version of the podman container provided directly from RedHat is the container we are pulling to showcase the capability of podman-in-docker.

ROOTFUL (Requires dangerous SYS_ADMIN and MKNOD Capability added)

image:
  name: quay.io/podman/stable
script:
  - podman login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY
  - podman --version
  - whoami <will display `root` here>
  - pwd
  - podman build --isolation=chroot -t $CONTAINER_BUILD_IMAGE .
  - podman run alpine echo hello
  - podman push $CONTAINER_BUILD_IMAGE

the root user is used by default here.

For even more added security, we can instead change the user to podman, and omit the cap_add step above that is providing the dangerous sys_admin and mknod capabilities to the image. You can do that the following way using the entrypoint keyword, or by building a new container that changes its USER by default while using the quay.io/podman/stable as its base image:

image:
  name: quay.io/podman/stable
  entrypoint: ['bash', '-c', 'exec su podman -c bash']
script:
  - podman login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY
  - podman --version
  - whoami <shoud now display `podman` here>
  - pwd
  - podman build --isolation=chroot -t $CONTAINER_BUILD_IMAGE .
  - podman run alpine echo hello
  - podman push $CONTAINER_BUILD_IMAGE

With SYS_ADMIN and MKNOD not being needed and running in a rootless state, we now have a secure solution for running podman-in-docker!

Final Thoughts

When running docker containers, it's always recommended that we reduce the privileges granted to it to only leverage the capabilities it needs to do. With podman, we now have a solution that gives us that capability in gitlab that also gives us the similar capabilities that docker has.