BoxBoat Blog

Service updates, customer stories, and tips and tricks for effective DevOps

Get Hands-On Experience with BoxBoat's Cloud Native Academy

Deployment Methodologies with GitLab -- Blue/Green

by Zach Hackett | Wednesday, Feb 9, 2022 | DevOps CICD GitLab

The world of CICD is home to a plethora of deployment methodologies, or ways we deploy our code into various environments. From canary deployments to manual rollouts, there is a strategy for everyone. With this series of articles, we aim to give you insight into the different options out there, so you and your team can choose the best deployment strategy for you.

This time around, we're diving into the Blue/Green deployment methodology.

What are blue/green deployments?

Blue/Green is a software deployment strategy which leverages two separate, but similar, deployments. We refer to these deployments as “blue” and “green” deployments, but any nomenclature would suffice, as long as we can simply distinguish the two deployments.

Only one of these deployments is ever live at a given time, except during incremental rollouts. For example, the blue deployment can be currently active on production, while the green deployment is “live” for testing, but not deployed to the production environment. In this way, internal developers, QA testers, and product managers can use the inactive deployment as a staging or pre-production environment.

New features can be deployed to the inactive environment without impacting users and, if any issues are found, the inactive deployment can be updated with affecting production. When testers and management are happy with the new features on the inactive deployment, the production environment can be switched to make use of the new features.

This means that the previously active environment (blue in our previous example) becomes inactive and the inactive environment (green in our previous example) becomes active. Our blue environment can be updated to match the state of green, and then any future feature sets can be tested there (as was done when green was our inactive environment).

Why should I use them?

As with any deployment methodology, there is a tradeoff between the benefits and drawbacks of blue/green deployments. Let’s take a look at the pros and cons:

Pros

reduces downtime as there is no need to take down the production deployment to switch to a different one. Both deployments run in parallel, and can be switched to at any time. (no negative impact on users; rapid releases)
Simple rollbacks. As quickly as you can switch from blue to green deployments, you can switch back from green to blue should a problem arise.
Built-in disaster recovery (duplicate production environments means that you have a ready-to-go backup at all times.
Easier A/B testing. Your idle environment can house experimental features that can be tested in a production environment without impacting users. It also makes splitting traffic for specific features across both environments, should one desire to do so.

Cons

Resource-intensive. There a second production-level environment to maintain, so naturally there is an increase in the amount infrastructure that needs to be resourced and maintained. As such, it may lead some teams to only leverage Blue/Green deployments for specific, high-value products.
Extra database management. To facilitate a seamless switch between blue and green environments, both need to keep databases in sync. What happens when a new feature requires a column name to change? Both databases will need this change and could cause breaking changes if not handled carefully.
Product management overhead. PMs will need access to reliable tools to keep track of both of these production environments. What is live where? What is being tested where? Which environment is currently live? Which services are receiving updates? And more.

How to use them?

Before we jump into an example of how GitLab enables us to easily implement this deployment strategy, let’s look at some of the best practices we need to keep in mind when it comes to blue/green deployments.

Best practices

automate as much as possible. speed up cutover, remove manual adjustments, guarantee all the necessary steps are executed (no one can forget a step) and keep the process running as smoothly as possible.
monitor your systems. Despite one environment having an “idle” status, both environments in a blue/green deployment strategy should be monitored at all times.
careful coding practices. Code should be written such that it is both forward- and backward-compatible. In order to avoid unforeseen issues when swapping between blue and green environments your code. One way to help in this scenario is to segment releases into smaller release components or steps.
Release less, more often. The DevOps world is all about shortening feedback loops. As developers, product owners, project managers, etc, getting feedback on the success of new features is paramount to success. Deploying smaller releases more frequently shortens that feedback loop and enables future releases to be more robust and less faulty.
Avoid monolithic applications. Splitting development features into microservices enables these smaller, more frequent releases as one feature isn’t waiting on the completion of another feature before it can be released.
Feature flags. Whether or not you can leverage microservices, feature flags provide a manageable way to prevent unfinished features from halting the release of a completed one.

How to enable blue/green deployments with GitLab

This example uses a simple Ruby project, but the pipeline setup is agnostic of the project technologies. As with all pipelines in GitLab we will leverage some clever configuration to enable our Blue/Green deployments.

Our pipeline needs to handle a few steps:

Build
Separate steps to deploy to either environment
Separate steps to switch which environment is active

Since the two deployments will essentially be copies of each other (just to different destinations), we’ll make use of some anchors to keep our pipeline config clean.

GitLab doesn’t currently support native Blue/Green deployments (v14.7), so we need to force our pipeline to use two different deployment names. When it’s eventually a native feature of GitLab, we’ll no longer need this piece.

.deploy_template: &deploy_template  
	stage: deploy
	rules:
		- if: '$CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH'
	when: manual
	allow_failure: true
	script:
		- export HELM_RELEASE_NAME="$BLUE_GREEN_DEPLOY"

The anchor syntax enables use to reference this job elsewhere through our config without needing to completely rewrite the same job more the once. We’ll make use of the $BLUE_GREEN_DEPLOY environment variable to control which environment we’re deploying to. More on this later.

Before we can deploy our new feature, we need to check if we’re deploying to the environment that is currently our live production environment. So let’s add some logic to our deploy_template job to do just that.

- >
	  export BLUE_GREEN_IS_ACTIVE="$(
	    if VALUE=$(kubectl -n "$KUBE_NAMESPACE" \
	                  get deploy "$HELM_RELEASE_NAME" \
	                  -o=jsonpath='{.metadata.annotations.bluegreen/live}'); then
	      echo "$VALUE"
	    else
	      echo "false"
	    fi
	  )"

Here, we’re checking whether our previously set $HELM_RELEASE_NAME is present on our cluster, and if so what does the annotation say about its liveness. If we can’t find the deployment in the first place, we just return false.

If we find that we are deploying to the currently live environment, we’ll want to make sure we include the live production url.

...
- >
    if [ "$BLUE_GREEN_IS_ACTIVE" = "true" ]; then
      export ADDITIONAL_HOSTS="$CI_PROJECT_PATH_SLUG.$KUBE_INGRESS_BASE_DOMAIN"
    fi

Great work! With those preliminary steps taken care of, we can run our necessary deployment scripts. For this example, we're using a basic Helm chart that creates a deployment to spin up an alpine pod that will echo the live environment. To that end, we need to pull down our chart and deploy it to our cluster in the proper namespace.

...
- helm pull demo-chart --untar
- kubectl get namespace "$KUBE_NAMESPACE" || kubectl create namespace "$KUBE_NAMESPACE"
- > 
    helm upgrade --install \
    --set service.url="$CI_ENVIRONMENT_URL" \
    --set service.commonName="$CI_PROJECT_ID.$KUBE_INGRESS_BASE_DOMAIN" \
    --set-string image.tag="CI_APPLICATION_REPOSITORY"
    --set deploymentType="$BLUE_GREEN_DEPLOY"
    demo-chart/
- echo $CI_ENVIRONMENT_URL > environment_url.txt

We’re almost done with the main deployment logic! Lastly, we need to add a few more attributes to our deploy_template job.

dependencies:
  - build
environment:
  name: "$BLUE_GREEN_DEPLOY"
  url: http://$BLUE_GREEN_DEPLOY-$CI_PROJECT_PATH_SLUG.$KUBE_INGRESS_BASE_DOMAIN
  kubernetes:
    namespace: production

Before we deploy anything, we want to make sure that we’ve built our project successfully. No sense running a deploy script if we have nothing to deploy! We’re also taking the extra step of linking this job with GitLab Environments.

With GitLab tracking the environment we’re deploying to we get extra details in our metrics on the platform along with the ability to track which deployments where deployed where and when.

In the end our deploy_template job looks like this:

.deploy_template:
  stage: deploy
  rules:
    - if: '$CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH'
      when: manual
      allow_failure: true
  script:
    - export HELM_RELEASE_NAME="$BLUE_GREEN_DEPLOY"
    - >
        export BLUE_GREEN_IS_ACTIVE="$(
          if VALUE=$(kubectl -n "$KUBE_NAMESPACE" \
                        get deploy "$HELM_RELEASE_NAME" \
                        -o=jsonpath='{.metadata.annotations.bluegreen/live}'); then
            echo "$VALUE"
          else
            echo "false"
          fi
        )"
    - >
        if [ "$BLUE_GREEN_IS_ACTIVE" = "true" ]; then
          export ADDITIONAL_HOSTS="$CI_PROJECT_PATH_SLUG.$KUBE_INGRESS_BASE_DOMAIN"
        fi
    - helm pull demo-chart --untar
    - kubectl get namespace "$KUBE_NAMESPACE" || kubectl create namespace "$KUBE_NAMESPACE"
    - > 
        helm upgrade --install \
        --set service.url="$CI_ENVIRONMENT_URL" \
        --set service.commonName="$CI_PROJECT_ID.$KUBE_INGRESS_BASE_DOMAIN" \
        --set-string image.tag="CI_APPLICATION_REPOSITORY"
        --set deploymentType="$BLUE_GREEN_DEPLOY"
        demo-chart/
    - echo $CI_ENVIRONMENT_URL > environment_url.txt
  dependencies:
    - build
  environment:
    name: "$BLUE_GREEN_DEPLOY"
    url: http://$BLUE_GREEN_DEPLOY-$CI_PROJECT_PATH_SLUG.$KUBE_INGRESS_BASE_DOMAIN
    kubernetes:
      namespace: production

Time to Deploy

With our deployment logic taken care of, we turn our focus to the jobs which will actually make use of this logic.

deploy to blue:
  extends: *deploy_template
  variables:
    BLUE_GREEN_DEPLOY: blue

deploy to green:
  extends: *deploy_template
  variables:
    BLUE_GREEN_DEPLOY: green

We’ll use a different job for each of our blue and green deployments. Both extend our deploy_template job and provide a value for the $BLUE_GREEN_DEPLOY environment variable corresponding with the correct environment.

With these pieces in place, we’re ready to deploy to either our blue or green environment!

Switching Gears

While it’s great that we can deploy to both of our environments, that doesn’t do us much good if we can’t switch which one is our live production environment. We’ll approach this similarly to our deploy logic. One job will house the main switching logic, while two simpler jobs extend this logic and provide reasonable environment variables to adjust its behavior.

To start, we’ll initiate the helm upgrade with our $BLUE_GREEN_ACTIVATE value. This example uses a simple Helm chart that creates a deployment to spin up an alpine pod that will echo whether it's a blue or green deployment. With that referenced, we need to update the liveness annotation on the deployment to true.

.switch_template: &switch_template
  stage: switch
  rules:
    - if: '$CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH'
      when: manual
      allow_failure: true
  script:
    - helm pull demo-chart --untar
    -  # pull down helm chart for our project
    - kubectl get namespace "$KUBE_NAMESPACE" || kubectl create namespace "$KUBE_NAMESPACE"
    # Disable default provisioning of PostgreSQL
    - export POSTGRES_ENABLED="false"
    # In the future green and blue should be implemented as tracks so that you
    # can do incremental rollouts between them. Also note that with this setup,
    # each deployment will trigger the GitLab managed cert-manager to issue a
    # new certificate. This is not ideal as there should only be one certiface
    # issued for the "production" environment, which includes blue and green.
    # Blue and green should both be considered the "production" environment.
    - export HELM_HOST="localhost:44134"
		- >
		  helm upgrade --reuse-values \
		    --wait \
		    --set service.additionalHosts="{$CI_PROJECT_PATH_SLUG.$KUBE_INGRESS_BASE_DOMAIN}" \
		    --namespace="$KUBE_NAMESPACE" \
		    "$BLUE_GREEN_ACTIVATE" \
		    demo-chart/
		- >
		    kubectl -n "$KUBE_NAMESPACE" \
		      patch deploy "$BLUE_GREEN_ACTIVATE" \
		      --type json \
		      -p='[{"op": "replace", "path": "/metadata/annotations/bluegreen~1live", "value": "true"}]'

With our newly active deployment updated, we now need to set our previously active deployment to ‘inactive.’ For posterity’s sake, we’ll first check if our inactive environment was already deployed. Without this check our pipeline would error any time there is only one deployment.

- >
  export DEPLOYED="$(
    if kubectl -n "$KUBE_NAMESPACE" get deploy "$BLUE_GREEN_DEACTIVATE" >/dev/null 2>&1; then
      echo "true"
    else
      echo "false"
    fi
  )"
- kubectl -n "$KUBE_NAMESPACE" get deploy "$BLUE_GREEN_DEACTIVATE" || true
- >
  if [ "$DEPLOYED" = "true" ]; then
    helm upgrade --reuse-values \
      --wait \
      --set service.additionalHosts="" \
      --namespace="$KUBE_NAMESPACE" \
      "$BLUE_GREEN_DEACTIVATE" \
      chart/
  fi
- >
  if [ "$DEPLOYED" = "true" ]; then
    kubectl -n "$KUBE_NAMESPACE" \
      patch deploy "$BLUE_GREEN_DEACTIVATE" \
      --type json \
      -p='[{"op": "replace", "path": "/metadata/annotations/bluegreen~1live", "value": "false"}]'
  fi
environment:
  name: production
  kubernetes:
    namespace: production

Lastly, we associate the switch_template job with our production GitLab Environment. Just as with our deployment process, we’ll add two jobs to facilitate switching between the blue and green environments.

switch to blue:
  extends: .switch_template
  variables:
    BLUE_GREEN_ACTIVATE: blue
    BLUE_GREEN_DEACTIVATE: green

switch to green:
  extends: .switch_template
  variables:
    BLUE_GREEN_ACTIVATE: green
    BLUE_GREEN_DEACTIVATE: blue

There you have it! We’re all set with Blue/Green Deployments! Our pipeline now supports deploying to both (or either) of our production environments and we can switch on or off either deployment at will as well.

The full .gitlab-ci.yaml file can be found in GitLab’s public project repository..

Production Ready

Whether you’re leveraging kubernetes or not, Blue/Green deployments provide a straightforward way to test new features in a production-ready environment. Gone are the days of worrying about rollbacks or impacting your users with failed upgrades. And GitLab’s CICD tools enable the use of Blue/Green Deployments in just a few short lines of yaml!