Self-Hosted Azure DevOps Pipeline Agents in Kubernetes

by Daniel Morrison | Thursday, Oct 21, 2021 | Azure Kubernetes Open-Source

There are many pros to hosting your own Azure DevOps(ADO) Pipeline Agents including: cost savings, increased control, and a cloud-native design. However, self-hosting anything can be tricky, and maintenance of agents can be tiresome. For these reasons a short-lived, ephemeral agent is ideal. One powerful way to achieve this is by running your agents on Kubernetes.

Let’s dive into setting up our own agent's Docker image and self-hosted agent pool.

Prerequisites

There's a few things you'll need ready before we get started:

• Docker
• Microsoft Azure DevOps account
• A running Kubernetes cluster (k3d is used in the example repo for running locally)

Follow along with this post or checkout the example repo.

Setting up Azure

Within Azure DevOps, create a self-hosted Agent Pool for the agents to belong to. This can be done using your favorite flavor of IaC, or in the UI following the screenshot below.

Note the name of your Agent Pool, you'll need it later on.

You'll also need an Azure Personal Access Token(PAT) in order to authenticate with Azure. Simple instructions on creating a PAT can be found in the offical Microsoft documentation.

It's a good practice to use narrowly scoped access tokens, created for a specific purpose. Be sure to store your PAT safely.

Setting up the Dockerfile

Choose your favorite flavored Docker image as a base for the agent, in this example we'll be using the python:3-slim-buster debian image. We'll start the Dockerfile by defining the base image and installing curl.

FROM python:3-slim-buster
RUN apt update && \
  apt install -y curl

Now let's download the Azure Agent tarball in our image, and unzip it's content. Be sure to check the official azure-pipelines-agent release page and use the agent appropriate for your base image's system architecture.

It's important to note: the scripts in the azure-pipelines-agent tarball should not be run as root. For this reason, we'll create the ado-agent user and assign ownership of our unpacked tarball directory to this user.

# create directories for our agent files
RUN mkdir /opt/agent && mkdir /home/ado-agent && \
# download and unpack tarball
    curl -L https://vstsagentpackage.azureedge.net/agent/2.192.0/vsts-agent-linux-x64-2.192.0.tar.gz -o /tmp/agent.tar.gz && tar xzf /tmp/agent.tar.gz -C /opt/agent && \
# create user and assign ownership
    useradd ado-agent && chown -R ado-agent /opt/agent

With the agent unpacked, we can run the scripts included to finish configuring the agent. Be sure to switch users to the ado-agent user we created earlier, otherwise the config.sh script will not execute properly.

When working with an enterprise network, it will be important to review the detailed documentation about running agents with certificates. Certificate setup will introduce additional arguments to the simpler invocation of config.sh shown below.

WORKDIR /opt/agent
RUN /opt/agent/bin/installdependencies.sh

USER ado-agent:ado-agent
ENTRYPOINT ./config.sh --replace --work _work --acceptTeeEula --url $AZ_URL \
  --auth pat --token $AZ_PAT --agent $AZ_AGENT_NAME --pool $AZ_AGENT_POOL && ./run.sh --once

That's all there is to the Dockerfile! Build the image and push it to your favorite repository. If you're using the example repo, run the included build.sh script to build the image and push it to the local k3d registry.

Azure Agents in Kubernetes

To demonstrate how to run the agent image in Kubernetes we'll use a basic Deployment manifest, but feel free to use your favorite flavour of IaC. However, before jumping into the Deployment, we need to store our Azure configuration in a ConfigMap. This can be done through the method of your choosing, we'll use an environment file. If you're using the example repo be sure to replace the kube/vars.env file content with your own values.

Your environment file should look as follows:

AZ_AGENT_POOL=<YOUR-AGENT-POOL-NAME>
AZ_URL=https://dev.azure.com/<YOUR-ORGANIZATION-NAME>
AZ_PAT=<YOUR-AZURE-PERSONAL-ACCESS-TOKEN>

Using the environment file we can create a ConfigMap with the command kubectl create cm azure-agent-config --from-env-file=<YOUR-FILE>.env With a Kubernetes Deployment we can run as many copies of the agent as we need, injecting our ConfigMap values along with an AZ_AGENT_NAME into the pod environments.

Below is an example of what that Deployment manifest may look like:

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: azure-agent-pool
  name: azure-agent-pool
spec:
  replicas: 1
  selector:
    matchLabels:
      app: azure-agent-pool
  template:
    metadata:
      labels:
        app: azure-agent-pool
    spec:
      containers:
      - image: k3d-myregistry.localhost:12345/ado-agent:local
        name: ado-agent-1
        env:
        - name: AZ_AGENT_NAME
          value: agent-1
        envFrom:
        - configMapRef:
            name: azure-agent-config
      - image: k3d-myregistry.localhost:12345/ado-agent:local
        name: ado-agent-2
        env:
        - name: AZ_AGENT_NAME
          value: agent-2
        envFrom:
        - configMapRef:
            name: azure-agent-config
      - image: k3d-myregistry.localhost:12345/ado-agent:local
        name: ado-agent-3
        env:
        - name: AZ_AGENT_NAME
          value: agent-3
        envFrom:
        - configMapRef:
            name: azure-agent-config

Applying this manifest will create 3 ephemeral azure agents in your defined Agent Pool. Checking the Agent Pool in the Azure UI will confirm that we have three agents running and listening for jobs.

Congratulations, you're ready to run Azure DevOps Pipeline jobs on your own self-hosted agents, running in Kubernetes! Hopefully this guide has been helpful to kickstart your journey into self-hosted agents on Microsoft Azure, but we have just scratched the surface.

What about persisting logs? Scaling the number of agents in the pool? Rotating the Azure PAT? Building OCI images inside our agent? Much can still be improved upon to make this a hardened, enterprise-grade solution, but that's better left for another time, in another post.