BoxBoat Blog
Service updates, customer stories, and tips and tricks for effective DevOps
Deployment Methodologies with GitLab -- Blue/Green
by Zach Hackett | Wednesday, Feb 9, 2022 | DevOps CICD GitLab
The world of CICD is home to a plethora of deployment methodologies, or ways we deploy our code into various environments. From canary deployments to manual rollouts, there is a strategy for everyone. With this series of articles, we aim to give you insight into the different options out there, so you and your team can choose the best deployment strategy for you.
This time around, we're diving into the Blue/Green deployment methodology.
What are blue/green deployments?
Blue/Green is a software deployment strategy which leverages two separate, but similar, deployments. We refer to these deployments as “blue” and “green” deployments, but any nomenclature would suffice, as long as we can simply distinguish the two deployments.
Only one of these deployments is ever live at a given time, except during incremental rollouts. For example, the blue deployment can be currently active on production, while the green deployment is “live” for testing, but not deployed to the production environment. In this way, internal developers, QA testers, and product managers can use the inactive deployment as a staging or pre-production environment.
New features can be deployed to the inactive environment without impacting users and, if any issues are found, the inactive deployment can be updated with affecting production. When testers and management are happy with the new features on the inactive deployment, the production environment can be switched to make use of the new features.
This means that the previously active environment (blue in our previous example) becomes inactive and the inactive environment (green in our previous example) becomes active. Our blue environment can be updated to match the state of green, and then any future feature sets can be tested there (as was done when green was our inactive environment).
Why should I use them?
As with any deployment methodology, there is a tradeoff between the benefits and drawbacks of blue/green deployments. Let’s take a look at the pros and cons:
Pros
- reduces downtime as there is no need to take down the production deployment to switch to a different one. Both deployments run in parallel, and can be switched to at any time. (no negative impact on users; rapid releases)
- Simple rollbacks. As quickly as you can switch from blue to green deployments, you can switch back from green to blue should a problem arise.
- Built-in disaster recovery (duplicate production environments means that you have a ready-to-go backup at all times.
- Easier A/B testing. Your idle environment can house experimental features that can be tested in a production environment without impacting users. It also makes splitting traffic for specific features across both environments, should one desire to do so.
Cons
- Resource-intensive. There a second production-level environment to maintain, so naturally there is an increase in the amount infrastructure that needs to be resourced and maintained. As such, it may lead some teams to only leverage Blue/Green deployments for specific, high-value products.
- Extra database management. To facilitate a seamless switch between blue and green environments, both need to keep databases in sync. What happens when a new feature requires a column name to change? Both databases will need this change and could cause breaking changes if not handled carefully.
- Product management overhead. PMs will need access to reliable tools to keep track of both of these production environments. What is live where? What is being tested where? Which environment is currently live? Which services are receiving updates? And more.
How to use them?
Before we jump into an example of how GitLab enables us to easily implement this deployment strategy, let’s look at some of the best practices we need to keep in mind when it comes to blue/green deployments.
Best practices
- automate as much as possible. speed up cutover, remove manual adjustments, guarantee all the necessary steps are executed (no one can forget a step) and keep the process running as smoothly as possible.
- monitor your systems. Despite one environment having an “idle” status, both environments in a blue/green deployment strategy should be monitored at all times.
- careful coding practices. Code should be written such that it is both forward- and backward-compatible. In order to avoid unforeseen issues when swapping between blue and green environments your code. One way to help in this scenario is to segment releases into smaller release components or steps.
- Release less, more often. The DevOps world is all about shortening feedback loops. As developers, product owners, project managers, etc, getting feedback on the success of new features is paramount to success. Deploying smaller releases more frequently shortens that feedback loop and enables future releases to be more robust and less faulty.
- Avoid monolithic applications. Splitting development features into microservices enables these smaller, more frequent releases as one feature isn’t waiting on the completion of another feature before it can be released.
- Feature flags. Whether or not you can leverage microservices, feature flags provide a manageable way to prevent unfinished features from halting the release of a completed one.
How to enable blue/green deployments with GitLab
This example uses a simple Ruby project, but the pipeline setup is agnostic of the project technologies. As with all pipelines in GitLab we will leverage some clever configuration to enable our Blue/Green deployments.
Our pipeline needs to handle a few steps:
- Build
- Separate steps to deploy to either environment
- Separate steps to switch which environment is active
Since the two deployments will essentially be copies of each other (just to different destinations), we’ll make use of some anchors to keep our pipeline config clean.
GitLab doesn’t currently support native Blue/Green deployments (v14.7), so we need to force our pipeline to use two different deployment names. When it’s eventually a native feature of GitLab, we’ll no longer need this piece.
.deploy_template: &deploy_template
stage: deploy
rules:
- if: '$CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH'
when: manual
allow_failure: true
script:
- export HELM_RELEASE_NAME="$BLUE_GREEN_DEPLOY"
The anchor syntax enables use to reference this job elsewhere through our config without needing to completely rewrite the same job more the once. We’ll make use of the $BLUE_GREEN_DEPLOY
environment variable to control which environment we’re deploying to. More on this later.
Before we can deploy our new feature, we need to check if we’re deploying to the environment that is currently our live production environment. So let’s add some logic to our deploy_template
job to do just that.
- >
export BLUE_GREEN_IS_ACTIVE="$(
if VALUE=$(kubectl -n "$KUBE_NAMESPACE" \
get deploy "$HELM_RELEASE_NAME" \
-o=jsonpath='{.metadata.annotations.bluegreen/live}'); then
echo "$VALUE"
else
echo "false"
fi
)"
Here, we’re checking whether our previously set $HELM_RELEASE_NAME
is present on our cluster, and if so what does the annotation say about its liveness. If we can’t find the deployment in the first place, we just return false.
If we find that we are deploying to the currently live environment, we’ll want to make sure we include the live production url.
...
- >
if [ "$BLUE_GREEN_IS_ACTIVE" = "true" ]; then
export ADDITIONAL_HOSTS="$CI_PROJECT_PATH_SLUG.$KUBE_INGRESS_BASE_DOMAIN"
fi
Great work! With those preliminary steps taken care of, we can run our necessary deployment scripts. For this example, we're using a basic Helm chart that creates a deployment to spin up an alpine pod that will echo the live environment. To that end, we need to pull down our chart and deploy it to our cluster in the proper namespace.
...
- helm pull demo-chart --untar
- kubectl get namespace "$KUBE_NAMESPACE" || kubectl create namespace "$KUBE_NAMESPACE"
- >
helm upgrade --install \
--set service.url="$CI_ENVIRONMENT_URL" \
--set service.commonName="$CI_PROJECT_ID.$KUBE_INGRESS_BASE_DOMAIN" \
--set-string image.tag="CI_APPLICATION_REPOSITORY"
--set deploymentType="$BLUE_GREEN_DEPLOY"
demo-chart/
- echo $CI_ENVIRONMENT_URL > environment_url.txt
We’re almost done with the main deployment logic! Lastly, we need to add a few more attributes to our deploy_template
job.
dependencies:
- build
environment:
name: "$BLUE_GREEN_DEPLOY"
url: http://$BLUE_GREEN_DEPLOY-$CI_PROJECT_PATH_SLUG.$KUBE_INGRESS_BASE_DOMAIN
kubernetes:
namespace: production
Before we deploy anything, we want to make sure that we’ve built our project successfully. No sense running a deploy script if we have nothing to deploy! We’re also taking the extra step of linking this job with GitLab Environments.
With GitLab tracking the environment we’re deploying to we get extra details in our metrics on the platform along with the ability to track which deployments where deployed where and when.
In the end our deploy_template
job looks like this:
.deploy_template:
stage: deploy
rules:
- if: '$CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH'
when: manual
allow_failure: true
script:
- export HELM_RELEASE_NAME="$BLUE_GREEN_DEPLOY"
- >
export BLUE_GREEN_IS_ACTIVE="$(
if VALUE=$(kubectl -n "$KUBE_NAMESPACE" \
get deploy "$HELM_RELEASE_NAME" \
-o=jsonpath='{.metadata.annotations.bluegreen/live}'); then
echo "$VALUE"
else
echo "false"
fi
)"
- >
if [ "$BLUE_GREEN_IS_ACTIVE" = "true" ]; then
export ADDITIONAL_HOSTS="$CI_PROJECT_PATH_SLUG.$KUBE_INGRESS_BASE_DOMAIN"
fi
- helm pull demo-chart --untar
- kubectl get namespace "$KUBE_NAMESPACE" || kubectl create namespace "$KUBE_NAMESPACE"
- >
helm upgrade --install \
--set service.url="$CI_ENVIRONMENT_URL" \
--set service.commonName="$CI_PROJECT_ID.$KUBE_INGRESS_BASE_DOMAIN" \
--set-string image.tag="CI_APPLICATION_REPOSITORY"
--set deploymentType="$BLUE_GREEN_DEPLOY"
demo-chart/
- echo $CI_ENVIRONMENT_URL > environment_url.txt
dependencies:
- build
environment:
name: "$BLUE_GREEN_DEPLOY"
url: http://$BLUE_GREEN_DEPLOY-$CI_PROJECT_PATH_SLUG.$KUBE_INGRESS_BASE_DOMAIN
kubernetes:
namespace: production
Time to Deploy
With our deployment logic taken care of, we turn our focus to the jobs which will actually make use of this logic.
deploy to blue:
extends: *deploy_template
variables:
BLUE_GREEN_DEPLOY: blue
deploy to green:
extends: *deploy_template
variables:
BLUE_GREEN_DEPLOY: green
We’ll use a different job for each of our blue and green deployments. Both extend our deploy_template
job and provide a value for the $BLUE_GREEN_DEPLOY
environment variable corresponding with the correct environment.
With these pieces in place, we’re ready to deploy to either our blue or green environment!
Switching Gears
While it’s great that we can deploy to both of our environments, that doesn’t do us much good if we can’t switch which one is our live production environment. We’ll approach this similarly to our deploy logic. One job will house the main switching logic, while two simpler jobs extend this logic and provide reasonable environment variables to adjust its behavior.
To start, we’ll initiate the helm upgrade with our $BLUE_GREEN_ACTIVATE
value. This example uses a simple Helm chart that creates a deployment to spin up an alpine pod that will echo whether it's a blue or green deployment. With that referenced, we need to update the liveness annotation on the deployment to true
.
.switch_template: &switch_template
stage: switch
rules:
- if: '$CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH'
when: manual
allow_failure: true
script:
- helm pull demo-chart --untar
- # pull down helm chart for our project
- kubectl get namespace "$KUBE_NAMESPACE" || kubectl create namespace "$KUBE_NAMESPACE"
# Disable default provisioning of PostgreSQL
- export POSTGRES_ENABLED="false"
# In the future green and blue should be implemented as tracks so that you
# can do incremental rollouts between them. Also note that with this setup,
# each deployment will trigger the GitLab managed cert-manager to issue a
# new certificate. This is not ideal as there should only be one certiface
# issued for the "production" environment, which includes blue and green.
# Blue and green should both be considered the "production" environment.
- export HELM_HOST="localhost:44134"
- >
helm upgrade --reuse-values \
--wait \
--set service.additionalHosts="{$CI_PROJECT_PATH_SLUG.$KUBE_INGRESS_BASE_DOMAIN}" \
--namespace="$KUBE_NAMESPACE" \
"$BLUE_GREEN_ACTIVATE" \
demo-chart/
- >
kubectl -n "$KUBE_NAMESPACE" \
patch deploy "$BLUE_GREEN_ACTIVATE" \
--type json \
-p='[{"op": "replace", "path": "/metadata/annotations/bluegreen~1live", "value": "true"}]'
With our newly active deployment updated, we now need to set our previously active deployment to ‘inactive.’ For posterity’s sake, we’ll first check if our inactive environment was already deployed. Without this check our pipeline would error any time there is only one deployment.
- >
export DEPLOYED="$(
if kubectl -n "$KUBE_NAMESPACE" get deploy "$BLUE_GREEN_DEACTIVATE" >/dev/null 2>&1; then
echo "true"
else
echo "false"
fi
)"
- kubectl -n "$KUBE_NAMESPACE" get deploy "$BLUE_GREEN_DEACTIVATE" || true
- >
if [ "$DEPLOYED" = "true" ]; then
helm upgrade --reuse-values \
--wait \
--set service.additionalHosts="" \
--namespace="$KUBE_NAMESPACE" \
"$BLUE_GREEN_DEACTIVATE" \
chart/
fi
- >
if [ "$DEPLOYED" = "true" ]; then
kubectl -n "$KUBE_NAMESPACE" \
patch deploy "$BLUE_GREEN_DEACTIVATE" \
--type json \
-p='[{"op": "replace", "path": "/metadata/annotations/bluegreen~1live", "value": "false"}]'
fi
environment:
name: production
kubernetes:
namespace: production
Lastly, we associate the switch_template job with our production GitLab Environment. Just as with our deployment process, we’ll add two jobs to facilitate switching between the blue and green environments.
switch to blue:
extends: .switch_template
variables:
BLUE_GREEN_ACTIVATE: blue
BLUE_GREEN_DEACTIVATE: green
switch to green:
extends: .switch_template
variables:
BLUE_GREEN_ACTIVATE: green
BLUE_GREEN_DEACTIVATE: blue
There you have it! We’re all set with Blue/Green Deployments! Our pipeline now supports deploying to both (or either) of our production environments and we can switch on or off either deployment at will as well.
The full .gitlab-ci.yaml
file can be found in GitLab’s public project repository..
Production Ready
Whether you’re leveraging kubernetes or not, Blue/Green deployments provide a straightforward way to test new features in a production-ready environment. Gone are the days of worrying about rollbacks or impacting your users with failed upgrades. And GitLab’s CICD tools enable the use of Blue/Green Deployments in just a few short lines of yaml!