Self-Hosted Azure DevOps Pipeline Agents in Kubernetes
There are many pros to hosting your own Azure DevOps(ADO) Pipeline Agents including: cost savings, increased control, and a cloud-native design. However, self-hosting anything can be tricky, and maintenance of agents can be tiresome. For these reasons a short-lived, ephemeral agent is ideal. One powerful way to achieve this is by running your agents on Kubernetes.
Let’s dive into setting up our own agent's Docker image and self-hosted agent pool.
There's a few things you'll need ready before we get started:
- Microsoft Azure DevOps account
- A running Kubernetes cluster (k3d is used in the example repo for running locally)
Follow along with this post or checkout the example repo.
Setting up Azure
Within Azure DevOps, create a self-hosted Agent Pool for the agents to belong to. This can be done using your favorite flavor of IaC, or in the UI following the screenshot below.
Note the name of your Agent Pool, you'll need it later on.
You'll also need an Azure Personal Access Token(PAT) in order to authenticate with Azure. Simple instructions on creating a PAT can be found in the offical Microsoft documentation.
It's a good practice to use narrowly scoped access tokens, created for a specific purpose. Be sure to store your PAT safely.
Setting up the Dockerfile
Choose your favorite flavored Docker image as a base for the agent, in this example we'll be using the
python:3-slim-buster debian image.
We'll start the Dockerfile by defining the base image and installing curl.
FROM python:3-slim-buster RUN apt update && \ apt install -y curl
Now let's download the Azure Agent tarball in our image, and unzip it's content. Be sure to check the official azure-pipelines-agent release page and use the agent appropriate for your base image's system architecture.
It's important to note: the scripts in the azure-pipelines-agent tarball should not be run as root.
For this reason, we'll create the
ado-agent user and assign ownership of our unpacked tarball directory to this user.
# create directories for our agent files RUN mkdir /opt/agent && mkdir /home/ado-agent && \ # download and unpack tarball curl -L https://vstsagentpackage.azureedge.net/agent/2.192.0/vsts-agent-linux-x64-2.192.0.tar.gz -o /tmp/agent.tar.gz && tar xzf /tmp/agent.tar.gz -C /opt/agent && \ # create user and assign ownership useradd ado-agent && chown -R ado-agent /opt/agent
With the agent unpacked, we can run the scripts included to finish configuring the agent.
Be sure to switch users to the
ado-agent user we created earlier, otherwise the
config.sh script will not execute properly.
When working with an enterprise network, it will be important to review the detailed documentation about running agents with certificates.
Certificate setup will introduce additional arguments to the simpler invocation of
config.sh shown below.
WORKDIR /opt/agent RUN /opt/agent/bin/installdependencies.sh USER ado-agent:ado-agent ENTRYPOINT ./config.sh --replace --work _work --acceptTeeEula --url $AZ_URL \ --auth pat --token $AZ_PAT --agent $AZ_AGENT_NAME --pool $AZ_AGENT_POOL && ./run.sh --once
That's all there is to the Dockerfile! Build the image and push it to your favorite repository.
If you're using the example repo, run the included
build.sh script to build the image and push it to the local k3d registry.
Azure Agents in Kubernetes
To demonstrate how to run the agent image in Kubernetes we'll use a basic Deployment manifest, but feel free to use your favorite flavour of IaC.
However, before jumping into the Deployment, we need to store our Azure configuration in a ConfigMap.
This can be done through the method of your choosing, we'll use an environment file. If you're using the example repo be sure to replace the
kube/vars.env file content
with your own values.
Your environment file should look as follows:
AZ_AGENT_POOL=<YOUR-AGENT-POOL-NAME> AZ_URL=https://dev.azure.com/<YOUR-ORGANIZATION-NAME> AZ_PAT=<YOUR-AZURE-PERSONAL-ACCESS-TOKEN>
Using the environment file we can create a ConfigMap with the command
kubectl create cm azure-agent-config --from-env-file=<YOUR-FILE>.env
With a Kubernetes Deployment we can run as many copies of the agent as we need, injecting our ConfigMap values along with an AZ_AGENT_NAME into the pod environments.
Below is an example of what that Deployment manifest may look like:
apiVersion: apps/v1 kind: Deployment metadata: labels: app: azure-agent-pool name: azure-agent-pool spec: replicas: 1 selector: matchLabels: app: azure-agent-pool template: metadata: labels: app: azure-agent-pool spec: containers: - image: k3d-myregistry.localhost:12345/ado-agent:local name: ado-agent-1 env: - name: AZ_AGENT_NAME value: agent-1 envFrom: - configMapRef: name: azure-agent-config - image: k3d-myregistry.localhost:12345/ado-agent:local name: ado-agent-2 env: - name: AZ_AGENT_NAME value: agent-2 envFrom: - configMapRef: name: azure-agent-config - image: k3d-myregistry.localhost:12345/ado-agent:local name: ado-agent-3 env: - name: AZ_AGENT_NAME value: agent-3 envFrom: - configMapRef: name: azure-agent-config
Applying this manifest will create 3 ephemeral azure agents in your defined Agent Pool. Checking the Agent Pool in the Azure UI will confirm that we have three agents running and listening for jobs.
Congratulations, you're ready to run Azure DevOps Pipeline jobs on your own self-hosted agents, running in Kubernetes! Hopefully this guide has been helpful to kickstart your journey into self-hosted agents on Microsoft Azure, but we have just scratched the surface.
What about persisting logs? Scaling the number of agents in the pool? Rotating the Azure PAT? Building OCI images inside our agent? Much can still be improved upon to make this a hardened, enterprise-grade solution, but that's better left for another time, in another post.