Tracing Containerized Processes with Sysdig

by Leon Castellanos | Thursday, Apr 19, 2018 | Education

Everyone is jumping on the bandwagon when it comes to adopting containerization and CI/CD technologies. The power and flexibility provided by containerization is undeniable, but due to the isolated nature of Linux Control Groups, Namespaces, and Security Modules, it becomes difficult to get adequate visibility into what is actually happening within containers.

This is where Sysdig comes in! Think of Sysdig as a Swiss Army Knife for tracing containerized processes. It encompasses many of the well known and beloved diagnosis tools out there, such as: tcpdump, strace, fuser, lsof, iostat, htop, lspci, ethtool, netstat, etc.

Let's Get Started

In this post we’re going to go over some hands-on examples of what you can do with open source Sysdig.

Sysdig is primarily made up of two components:

1. A Kernel Module called “sysdig_probe”.
2. A CLI tool called “sysdig” that can query the data gathered by the Kernel Module.

The Sysdig installer below will take care of compiling a module compatible with the Kernel.

I’ll be using Digital Ocean to spin up an Ubuntu instance where we can quickly get a Docker environment to run our examples. You don't have you use Digital Ocean, just spin up Ubuntu on your cloud provider of choice or your favorite virtualization platform.

Login to Digital Ocean, create a new Ubuntu 17.10 Droplet, and ssh into it.

Install the pre-requisites:

• Install the latest Docker-CE Engine:

curl -fsSL https://get.docker.com | sudo bash

• Install the latest Open Source Sysdig:

curl -fsSL https://s3.amazonaws.com/download.draios.com/stable/install-sysdig | sudo bash

• If the kernel module didn't load automatically run the following command to load it:

lsmod | grep sysdig
modprobe sysdig_probe

Spin up a NGINX container and test it:

• Start the container:

docker run --name test-nginx -d -p 80:80 nginx

• Check that it's running:

docker ps

• Check it's responding:

curl localhost

Finally, browse to the Droplet IP, and you should see the NGINX page:

That's it!

Now, let's play with Sysdig!

Start a new data capture:

• This will generate a file called nginx.scap for you to analyze later:

sysdig -w nginx.scap &

• Let this process run for a few minutes and meanwhile refresh the NGINX page several times to create access logs in the capture
• After a few minutes terminate capture:

fg and ctrl-c

• Search for a specific process in the capture file:

sysdig -r nginx.scap proc.name=nginx

• Search for a specific container name:

sysdig -r nginx.scap container.name=test-nginx

• Search for processes that accessed /etc on the host:

sysdig -pc -r nginx.scap fd.name contains /etc

• Show “open” events for NGINX:

sysdig -r nginx.scap "evt.type=open and evt.dir=< and proc.name=nginx"

• Show “open” events for NGINX and display only event time+directory+filename:

sysdig -r nginx.scap -p "%evt.time %fd.directory %fd.filename" "evt.type=open and evt.dir=< and proc.name=nginx"

• Show I/O per File Descriptor type:

sysdig -r nginx.scap -c fdbytes_by fd.type

• List containers by number of files in use excluding the host:

sysdig -r nginx.scap -c fdcount_by container.name "fd.type=file and container.name!=host

• Show a count of connections by port:

sysdig -r nginx.scap -c fdcount_by fd.sport evt.type=accept

• Show bytes sent by port:

sysdig -r nginx.scap -c fdcount_by fd.sport

• Show top clients by connection:

sysdig -r nginx.scap -c fdcount_by fd.cip evt.type=accept

• Show top bytes by client:

sysdig -r nginx.scap -c fdcount_by fd.cip -pc

• Show HTTP requests:

sysdig -r nginx.scap -pc -c httplog

• Show HTTP GET requests:

sysdig -r nginx.scap -pc -s 2000 -A -c echo_fds fd.port=80 and evt.buffer contains GET

• Show incoming connections not handled by nginx:

sysdig -r nginx.scap -p "%proc.name %fd.name" "evt.type=accept and proc.name!=nginx"

• Show commands executed on a container (make sure a capture is running before executing commands):

docker exec -it test-nginx /bin/sh
ls
ls /etc
whoami
ctrl-c
sysdig -r nginx.scap -pc -c spy_users container.name=test-nginx

That should be enough to show you the power and flexibility of Sysdig. There are literally thousands of little tricks like these to gain an incredible level of visibility into your containerized processes. For more info, check out the Sysdig website.