Preview environment for Code reviews, part 2: CI/CD pipelines for the Cloud

Read before you start

This blog post is the second part of a series of articles that cover building a separate Preview environment for each Code Review (Merge Request).

If you are not familiar with the concept and assumptions behind this solution, be sure to read previous post on the subject. The information given there and the design decisions made will help you understand the next steps that are included in this and subsequent articles.

Which part of the solution will we address in this article?

We start by recalling the process that leads to the creation of an environment for Preview. We can see below that once a Merge Request is created for a production branch (main), a Preview pipeline is started. This is a pipeline in the GitLab repository that is an integral part of the CI/CD mechanism.

The example company behind the 'Nice-app.com' application already has a Production pipeline implemented, which uploads only the application files (HTML) to the production bucket in the AWS S3 service. In this article, we want to address the implementation and deployment of the Preview pipeline, which does not exist yet.

The new pipeline will differ from the production pipeline in that it will upload the 'Nice-app.com' application files into a subdirectory in the staging bucket (instead of the root folder), which will have a unique name for each Preview. In addition, subdomain records will be added that redirect to the LoadBalancer in the Kubernetes cluster (Google GKE service) with the same name as the subdirectory. This will serve us later to use a Reverse-proxy server, which we will deal with in the next article.

Above, we see the moment when the Preview environment was created and its effects made in the AWS cloud.

Step 1: A new bucket in AWS S3.

Creating a staging bucket

The application uses one production bucket. We want to reproduce the same, but for staging. To do this, we log into the Amazon Web Services cloud console.

1. Go to Amazon S3, select the "Buckets" tab and click on the "Create bucket" button.

2. We fill in the bucket form with data and options just as it is in production. We also select the option "Copy settings from existing bucket - optional", where we indicate the production bucket "nice-app-production". We call the new one "nice-app-staging".

3. We make sure that the bucket will be publicly accessible. We uncheck the option to block access.

4. We create the bucket by pressing the “Create bucket” button at the bottom of the page.
5. Once it has been created, we look for the newly created bucket in the bucket list. Go to its settings, "Properties" tab.

6. We scroll through the page looking for the heading "Static website hosting", where we make sure it is enabled. If it is off, we go into the “Edit” options and make the appropriate changes (turn it on).

In addition, we make sure that this is "Hosting type: Bucket hosting". This is important due to the fact that we want the bucket itself to host these files externally.

At the bottom, we also see the URL we will use as the base URL to which we will redirect traffic in the Reverse-proxy server. Let's save this URL in our notes (nice-app-staging.s3-website.eu-central-1.amazonaws.com), as we will need it later.

7. Let’s go to the “Permissions” tab to add the following permission (if not automatically generated) under the “Bucket policy” heading:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "PublicReadGetObject",
            "Effect": "Allow",
            "Principal": "*",
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::nice-app-staging/*"
        }
    ]
}

This is the permission needed to read bucket information and access it through the pipeline in GitLab CI.

Adding a retention policy (7 days)

In addition (optionally), we can add a retention policy that will remove files older than 7 days from this S3 bucket. To do this, we need to go to the "Management" tab and, in the "Lifecycle rules" section, add a new rule using the "Create lifecycle rule" button.

This is an additional optional step, as the deletion of the files of a given preview from the bucket will happen automatically from within GitLab CI when the Merge Request is closed/merged or 7 days after the Preview environment was created.

Step 2: New environment in GitLab CI.

With the bucket created for the staging, we can proceed to automate the deployment of the Preview environment in response to new Merge Requests or changes within them.

Current configuration of the CI/CD in GitLab

We know that a client has a project in the form of a Git repository hosted on GitLab. Inside this project (in the root directory) is a configuration file ".gitlab-ci.yml", which contains the CI manifests for this project.

It currently only has deployment to production implemented:

stages:
  - production

variables:
  ARTIFACT_DIR_NAME: static-files
  AWS_DEFAULT_REGION: eu-central-1

.build-template: &build-template
  needs: []
  image: nice-company/aws-tools-bundle

build-and-deploy-production:
  <<: *build-template
  stage: production
  variables:
    S3_BUCKET: nice-app-production
  script:
    - aws s3 cp $ARTIFACT_DIR_NAME/ s3://$S3_BUCKET/ --recursive --exclude "*index.html" --cache-control "max-age=315360000, no-transform, public"
    - aws s3 cp $ARTIFACT_DIR_NAME/ s3://$S3_BUCKET/ --recursive --exclude "*" --include "*index.html" --cache-control "public, must-revalidate, proxy-revalidate, max-age=0"
    - echo "Deployment of artifact to AWS S3 completed."
  rules:
    - if: '$CI_PIPELINE_SOURCE == "web"'
  environment:
    name: production
    url: https://nice-app.com/

Let us analyse this file together. It is worth noting that:

• there is only one stage: "production"
• static files with the application are located under the "static-files" subdirectory in the repository, these files will be copied and uploaded to the S3 bucket (the production one is called "nice-app-production"
• the index.html file is not cached, but the rest of the files such as images etc. are cached
• we have set the environment to “production” along with a URL leading to it. This URL will appear as a button on the Merge Request subpage when we access it at the GitLab level. This will make it quick and easy to preview the web page with the application.

In addition, the build template included in the file uses a pre-built Docker image nice-company/aws-tools-bundle, which is available in the private Docker image repository in the GCP cloud (the so-called Google Artifact Registry, or GAR for short).

The contents of this image are ready-made scripts that install, among other things, the tools required by the AWS CLI (along with it) that we use for AWS cloud operations.

The "Dockerfile" file in the above-mentioned image looks like this:

FROM php:7.4-apache
RUN apt-get --allow-releaseinfo-change update
RUN apt-get install -y ssh git libgnutls30 ssh libpng-dev libjpeg-dev zlib1g-dev libwebp-dev libzip-dev webp rsync unzip libonig-dev
RUN gd mysqli opcache zip mbstring
WORKDIR /tmp
RUN curl https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip -o awscliv2.zip \
    && unzip awscliv2.zip \
    && ./aws/install \
        && rm -rf aws*
WORKDIR /var/www/html

Adding a new environment and upload to the AWS S3 staging bucket

What we should do is add a new environment that will respond to changes in Merge Requests. Then cause it to upload static files on AWS S3 to the new staging bucket, as intended.

Below you will find the new lines of code added to the current ".gitlab-ci.yml" file:

stages:
  - staging

build-and-deploy-staging-for-review:
  <<: *build-template
  stage: staging
  variables:
    S3_BUCKET: nice-app-staging
  script:
    - aws s3 cp $ARTIFACT_DIR_NAME/ s3://$S3_BUCKET/$CI_COMMIT_SHA --recursive --cache-control "public, must-revalidate, proxy-revalidate, max-age=0"
    - echo "Deployment of artifact to AWS S3 staging bucket completed."
  rules:
    - if: '$CI_PIPELINE_SOURCE == "merge_request_event" && $CI_MERGE_REQUEST_TARGET_BRANCH_NAME == "main"'
  environment:
    name: review/staging/$CI_COMMIT_SHA

As you can see, a new stage staging has been added, to which a build-and-deploy-staging-for-review job is assigned, which reacts to changes in the MR (which has a target as branch main).

The environment name is review/staging/$CI_COMMIT_SHA where the variable $CI_COMMIT_SHA corresponds to the commit's hash. This is an acceptable path to have the review/staging prefix created with unique hashes (to leave space in the address pool for other types of environments).

In terms of operation, it is similar to the production job, but with the difference that it uploads the files into another bucket, where the files will be in a subdirectory instead of the root dir. The subdirectory has the commit hash contained in the $CI_COMMIT_SHA variable in its name. In addition, nothing is cached (because it does not need to be).

Creating a subdomain in the AWS Route 53

We will now add code to the CI job that will additionally create new subdomains in the staging DNS zone under “Nice-app.com” domain. To do this, we first need to find the public IP address of the LoadBalancer that leads to the Kubernetes cluster that the client company owns.

LoadBalancer IP address retrieval

The company's cluster is located under the GCP (Google Cloud Platform), you need to log into it and find the GKE (Google Kubernetes Engine) service, then click on its name to go to the details.

Once in the details, click on the “Servies & Ingress” tab in the left sidebar menu. A list of services with their endpoints will appear. Next, in the "Type" column, find the entry with the load balancer, i.e. "External load balancer". In this way, we found the IP address of the load balancer assigned to the cluster, i.e. "37.120.15.182".

Script that generates a JSON file for adding a new subdomain

Having the IP address to which the subdomain should redirect, we can return to our GitLab CI configuration.

The AWS Route 53 service, according to the AWS CLI documentation, requires us to use a special form of JSON when making changes to DNS entries (both additions and deletions). Therefore, we will now prepare a Bash script to be used by GitLab CI to create such a JSON.

The file "create_staging_subdomain_route_json.sh" looks like this:

#!/bin/bash

# --- SCRIPT DESCRIPTION ---
# this script creates JSON file, which describes the needed schema for creating a new subdomain for staging reverse proxy server

# --- INPUT ARGS ---
STAGING_DEPLOYMENT_HASH=$1
STAGING_SUBDOMAIN_POSTFIX=$2
REVERSE_PROXY_IP_ADDRESS=$3

cat > $STAGING_DEPLOYMENT_HASH.json <<- EOM
{
  "Comment": "CREATE a record for new staging subdomain (hash: $STAGING_DEPLOYMENT_HASH)",
  "Changes": [
    {
      "Action": "CREATE",
      "ResourceRecordSet": {
        "Name": "$STAGING_DEPLOYMENT_HASH.$STAGING_SUBDOMAIN_POSTFIX",
        "Type": "A",
        "TTL": 300,
        "ResourceRecords": [{ "Value": "$REVERSE_PROXY_IP_ADDRESS"}]
      }
    }
  ]
}
EOM

This script takes three parameters:

• STAGING_DEPLOYMENT_HASH - this is the same identifier as the hash of the commit in Git, e.g. "a6fe012bcca"
• STAGING_SUBDOMAIN_POSTFIX - this is the remainder of the subdomain, in our case it would be "staging.nice-app.com"
• REVERSE_PROXY_IP_ADDRESS - this is the public IP address of the LoadBalancer where our reverse-proxy server will be located, we just obtained this IP address from the GKE cluster dashboard: "37.120.15.182"

The effect of running this will be to create a new file called <STAGING_DEPLOYMENT_HASH>.json, which will then be used by the AWS CLI script to add the subdomain.

Extending the new code with a section to add a subdomain via the AWS CLI

At this point, we already have all the necessary elements in place to be able to add a subdomain.

We are therefore extending the newly added CI code with additional elements:

variables:
  ARTIFACT_DIR_NAME: static-files
  AWS_DEFAULT_REGION: eu-central-1
  STAGING_SUBDOMAIN_POSTFIX: staging.nice-app.com
  STAGING_REVERSE_PROXY_IP_ADDRESS: 37.120.15.182 # LoadBalancer in k8s GKE cluster
  HOSTED_ZONE_ID: Z900633027TAFXZ7CD6SX # 'nice-app.com' zone

build-and-deploy-staging-for-review:
  <<: *build-template
  stage: staging
  variables:
    S3_BUCKET: nice-app-staging
  script:
    - aws s3 cp $ARTIFACT_DIR_NAME/ s3://$S3_BUCKET/$CI_COMMIT_SHA --recursive --cache-control "public, must-revalidate, proxy-revalidate, max-age=0"
    - echo "Deployment of the artifact to AWS S3 staging bucket completed."
    - ./aws-scripts/create_staging_subdomain_route_json.sh $CI_COMMIT_SHA $STAGING_SUBDOMAIN_POSTFIX $STAGING_REVERSE_PROXY_IP_ADDRESS
    - aws route53 change-resource-record-sets --hosted-zone-id $HOSTED_ZONE_ID --change-batch file://$CI_COMMIT_SHA.json # optionally we can add a loop checking the status of change like PENDING/INSYNC then close the job
    - echo "Subdomain in AWS S3 route created (DNS server usually needs ~60 seconds to be in sync)."
    - rm $CI_COMMIT_SHA.json
    - echo "Preview link http://$CI_COMMIT_SHA.$STAGING_SUBDOMAIN_POSTFIX/"
  rules:
    - if: '$CI_PIPELINE_SOURCE == "merge_request_event" && $CI_MERGE_REQUEST_TARGET_BRANCH_NAME == "main"'
  environment:
    name: review/staging/$CI_COMMIT_SHA
    url: http://$CI_COMMIT_SHA.$STAGING_SUBDOMAIN_POSTFIX/

And now let’s explain the changes:

• several new variables have been added:
  a) STAGING_SUBDOMAIN_POSTFIX, as in the previously mentioned Bash script, is the remainder of the subdomain
  b) STAGING_REVERSE_PROXY_IP_ADDRESS, the IP address of the LoadBalancer
  c) The HOSTED_ZONE_ID, is the DNS zone ID contained in the service in AWS Route 53 for the web domain "nice-app.com" (it can be extracted by going into the details of this zone in the AWS console dashboard regarding Route 53)
• run a new script create_staging_subdomain_route_json.sh, which will generate a file with the same name as the environment/commit hash
• using the AWS CLI ( aws route53 change-resource-record-sets) we add a new subdomain using this file and delete it afterward
• we generate a URL in both logs and environment that redirects to the subdomain for that particular build. With this procedure, the Merge Request will from now on show a "View app" button redirecting to this URL for the reviewer:

Automatic and manual cleaning of the environment (retention)

In line with what we assumed at the beginning, we should add additional automation that will allow us to clean up an environment that is no longer needed.

To do this, we will use the two additional options that GitLab provides for environments:

• "on_stop" - in this parameter we indicate the name of the Job that is to be started in case the conditions that were used to trigger the new environment are no longer true
• "auto_stop_in" - automatic triggering of the above Job indicated in "on_stop" after a selected period of time (in our case 7 days)

The modified new code in the “.gitlab-ci.yaml” file will look as follows:

build-and-deploy-staging-for-review:
  <<: *build-template
  stage: staging
  variables:
    S3_BUCKET: nice-app-staging
  script:
    - aws s3 cp $ARTIFACT_DIR_NAME/ s3://$S3_BUCKET/$CI_COMMIT_SHA --recursive --cache-control "public, must-revalidate, proxy-revalidate, max-age=0"
    - echo "Deployment of the artifact to AWS S3 staging bucket completed."
    - ./aws-scripts/create_staging_subdomain_route_json.sh $CI_COMMIT_SHA $STAGING_SUBDOMAIN_POSTFIX $STAGING_REVERSE_PROXY_IP_ADDRESS
    - aws route53 change-resource-record-sets --hosted-zone-id $HOSTED_ZONE_ID --change-batch file://$CI_COMMIT_SHA.json # optionally we can add a loop checking the status of change like PENDING/INSYNC then close the job
    - echo "Subdomain in AWS S3 route created (DNS server usually needs ~60 seconds to be in sync)."
    - rm $CI_COMMIT_SHA.json
    - echo "Preview link http://$CI_COMMIT_SHA.$STAGING_SUBDOMAIN_POSTFIX/"
  rules:
    - if: '$CI_PIPELINE_SOURCE == "merge_request_event" && $CI_MERGE_REQUEST_TARGET_BRANCH_NAME == "main"'
  environment:
    name: review/staging/$CI_COMMIT_SHA
    url: http://$CI_COMMIT_SHA.$STAGING_SUBDOMAIN_POSTFIX/
    on_stop: erase-staging-after-review
    auto_stop_in: 7 days

erase-staging-after-review:
  <<: *build-template
  stage: staging
  variables:
    S3_BUCKET: nice-app-staging
  script:
    - aws s3 rm s3://$S3_BUCKET/ --recursive --exclude "*" --include "$CI_COMMIT_SHA/*"
    - echo "Cleaning the artifact '$CI_COMMIT_SHA' inside AWS S3 staging bucket completed."
    - ./aws-scripts/delete_staging_subdomain_route_json.sh $CI_COMMIT_SHA $STAGING_SUBDOMAIN_POSTFIX $STAGING_REVERSE_PROXY_IP_ADDRESS
    - aws route53 change-resource-record-sets --hosted-zone-id $HOSTED_ZONE_ID --change-batch file://$CI_COMMIT_SHA.json
    - echo "Subdomain in AWS S3 route deleted (DNS server usually needs ~60 seconds to be in sync)."
    - rm $CI_COMMIT_SHA.json
  rules:
    - if: '$CI_PIPELINE_SOURCE == "merge_request_event" && $CI_MERGE_REQUEST_TARGET_BRANCH_NAME == "main"'
      when: manual
  environment:
    name: review/staging/$CI_COMMIT_SHA
    url: https://$CI_COMMIT_SHA.$STAGING_SUBDOMAIN_POSTFIX/
    action: stop

And let's review the changes again:

• the build-and-deploy-staging-for-review job has now gained two new parameters "auto_stop_in" (7 days) and "on_stop", which, as previously described, are used to clean up the environment
• a new job erase-staging-after-review, which is only triggered by closing the preview build job, it first removes the subdirectory with the commit hash name from AWS S3 and then removes the subdomain from AWS Route 53
• deleting a subdomain similarly to adding a subdomain requires a special JSON file in which it is declared what the subdomain to be deleted looks like, a Bash script called delete_staging_subdomain_route_json.sh looks analogous in this case (except that now we have "Action": "DELETE"):

#!/bin/bash

# --- SCRIPT DESCRIPTION ---
# this script deleted the JSON file, which describes the needed schema for deletion of existing subdomain for staging reverse proxy server

# --- INPUT ARGS ---
STAGING_DEPLOYMENT_HASH=$1
STAGING_SUBDOMAIN_POSTFIX=$2
REVERSE_PROXY_IP_ADDRESS=$3

cat > $STAGING_DEPLOYMENT_HASH.json <<- EOM
{
  "Comment": "DELETE the record of existing staging subdomain (hash: $STAGING_DEPLOYMENT_HASH)",
  "Changes": [
    {
      "Action": "DELETE",
      "ResourceRecordSet": {
        "Name": "$STAGING_DEPLOYMENT_HASH.$STAGING_SUBDOMAIN_POSTFIX",
        "Type": "A",
        "TTL": 300,
        "ResourceRecords": [{ "Value": "$REVERSE_PROXY_IP_ADDRESS"}]
      }
    }
  ]
}
EOM

Brief summary

We were able to fully configure the CI environment in GitLab, which adds the subdomain along with the files to the staging bucket and then removes them when the Merge Request is closed or 7 days have passed since the preview was created.

In the next article (part 3), we will already focus mainly on the Reverse-proxy server configuration and implementation to be closer to the finish of needed elements for our main goal. There will be a URL link to the next part here in the future once it's published, so keep an eye out ;)

The full code of the “.gitlab-ci.yaml” file (with production pipeline) now looks as follows:

stages:
  - staging
  - production

variables:
  ARTIFACT_DIR_NAME: static-files
  AWS_DEFAULT_REGION: eu-central-1
  STAGING_SUBDOMAIN_POSTFIX: staging.nice-app.com
  STAGING_REVERSE_PROXY_IP_ADDRESS: 37.120.15.182 # LoadBalancer in k8s GKE cluster
  HOSTED_ZONE_ID: Z900633027TAFXZ7CD6SX # 'nice-app.com' zone

.build-template: &build-template
  needs: []
  image: nice-company/aws-tools-bundle

build-and-deploy-staging-for-review:
  <<: *build-template
  stage: staging
  variables:
    S3_BUCKET: nice-app-staging
  script:
    - aws s3 cp $ARTIFACT_DIR_NAME/ s3://$S3_BUCKET/$CI_COMMIT_SHA --recursive --cache-control "public, must-revalidate, proxy-revalidate, max-age=0"
    - echo "Deployment of the artifact to AWS S3 staging bucket completed."
    - ./aws-scripts/create_staging_subdomain_route_json.sh $CI_COMMIT_SHA $STAGING_SUBDOMAIN_POSTFIX $STAGING_REVERSE_PROXY_IP_ADDRESS
    - aws route53 change-resource-record-sets --hosted-zone-id $HOSTED_ZONE_ID --change-batch file://$CI_COMMIT_SHA.json # optionally we can add loop checking the status of change like PENDING/INSYNC then close the job
    - echo "Subdomain in AWS S3 route created (DNS server usually needs ~60 seconds to be in sync)."
    - rm $CI_COMMIT_SHA.json
    - echo "Preview link http://$CI_COMMIT_SHA.$STAGING_SUBDOMAIN_POSTFIX/"
  rules:
    - if: '$CI_PIPELINE_SOURCE == "merge_request_event" && $CI_MERGE_REQUEST_TARGET_BRANCH_NAME == "main"'
  environment:
    name: review/staging/$CI_COMMIT_SHA
    url: http://$CI_COMMIT_SHA.$STAGING_SUBDOMAIN_POSTFIX/
    on_stop: erase-staging-after-review
    auto_stop_in: 7 days

erase-staging-after-review:
  <<: *build-template
  stage: staging
  variables:
    S3_BUCKET: nice-app-staging
  script:
    - aws s3 rm s3://$S3_BUCKET/ --recursive --exclude "*" --include "$CI_COMMIT_SHA/*"
    - echo "Cleaning the artifact '$CI_COMMIT_SHA' inside AWS S3 staging bucket completed."
    - ./aws-scripts/delete_staging_subdomain_route_json.sh $CI_COMMIT_SHA $STAGING_SUBDOMAIN_POSTFIX $STAGING_REVERSE_PROXY_IP_ADDRESS
    - aws route53 change-resource-record-sets --hosted-zone-id $HOSTED_ZONE_ID --change-batch file://$CI_COMMIT_SHA.json
    - echo "Subdomain in AWS S3 route deleted (DNS server usually needs ~60 seconds to be in sync)."
    - rm $CI_COMMIT_SHA.json
  rules:
    - if: '$CI_PIPELINE_SOURCE == "merge_request_event" && $CI_MERGE_REQUEST_TARGET_BRANCH_NAME == "main"'
      when: manual
  environment:
    name: review/staging/$CI_COMMIT_SHA
    url: http://$CI_COMMIT_SHA.$STAGING_SUBDOMAIN_POSTFIX/
    action: stop

build-and-deploy-production:
  <<: *build-template
  stage: production
  variables:
    S3_BUCKET: nice-app-production
  script:
    - aws s3 cp $ARTIFACT_DIR_NAME/ s3://$S3_BUCKET/ --recursive --exclude "*index.html" --cache-control "max-age=315360000, no-transform, public"
    - aws s3 cp $ARTIFACT_DIR_NAME/ s3://$S3_BUCKET/ --recursive --exclude "*" --include "*index.html" --cache-control "public, must-revalidate, proxy-revalidate, max-age=0"
    - echo "Deployment of artifact to AWS S3 completed."
  rules:
    - if: '$CI_PIPELINE_SOURCE == "web"'
  environment:
    name: production
    url: https://nice-app.com/