Continuous Verification using VMware Code Stream

If you are familiar with the CloudJourney.io team already, then you should have already heard the term Continuous Verification by now. If not, feel free to check out a great article written by Dan Illson and Bill Shetti here. Let’s review here:

Continuous Verification is:

“A process of querying external system(s) and using information from the response to make decision(s) to improve the development and deployment process.”

Put in laymans terms is it the idea of “shifting left” ANY day 2 operations into the CD process. The ideas and practice of Continuous Verification can be brought into just about any pipeline scenario. It is simply adding in additional checks as gates into the CD portion of your pipeline.

A detailed configuration and implementation of the full original version of CV, including the actual code used here.

To illustrate this, today we’re going to be going over how to build governance checks into a deployment pipeline using VMware Code Stream.

VMware Code Stream

There are many CI/CD tools in the ecosystem, however VMware Code Stream is helps provide a more user friendly mechanism to create and deploy pipelines (CD to be specific). Rather than using YAML to do everything, like in Jenkins, you get a very nice UI. This allows you to actually see what the pipeline configuration actually looks like.

VMware Code Stream’s canvas editor (UI) lets you simply click to add stages, click and drag to add stage tasks, fill in some blanks, and use some drop downs to configure the stages and tasks. That’s it.

Compared to other tools in the CI/CD Pipelining space, this is very convenient. Especially for newer users to these types of tools. Other tools, such as Jenkins X, have no UI at all. For an experienced developer, this can be just fine. For someone just tryin to learn the concepts, this can be difficult to pick up. That is why I see Code Stream as a perfect CI/CD pipelining tool for beginners.

Pipeline Configuration

As noted above, we will be covering a simple example of creating a full governance check stage and a deployment into a staging environment. Here is what the resulting configuration is in VMware Code Stream

You will notice that the implementation into Code Stream is really quite simple, due to the nature of the service.

Let’s review what is in these two stages of the pipeline.

1. “Governance Checks” - there are two main checks being performed in governance. Budgetary check against CloudHealth by VMware and a Security check against AWS using VMware Secure State
2. “Deploy - Staging” - If Governance passes then the application (Acme Fitness Shop) is deployed into a AKS Cluster (Kubernetes)

Before we get into the full pipeline configuration, there are somes things we need to have setup before we can get this going. Let’s take a look at the prerequisitie information needed for the pipeline to be successful.

Prerequisies

Three things need to be setup beforehand to make this pipeline work.

• API Tokens
• Endpoints
• Variables

API Tokens

To make this process work, we’re going to need 3 different API tokens:

1. CSP (VMware Cloud Services Platform - Used for Code Stream API)
2. CloudHealth by VMware
3. VMware Secure State

These API tokens will be used by the scripts to make the appropriate API calls into CloudHealth and SecureState to check the Budget and Security, and to make the API call back in to Code Stream to set the Variable for validation.

Endpoints

Endpoints in Code Stream are simply a source or destination of things the pipeline will use. Git repo’s, Kubernetes Clusters, Jenkins servers, etc. We will be using 3 Endpoings for this pipeline:

1. GitLab Repository
2. Agent (Used for the SSH tasks to execute the API scripts)
3. Kubernetes Cluster (Staging)

The GitLab repository used in this configuration holds the Kubernetes manifests that we are deploying the application from. You could use any form of Git endpoint for this, or even use static YAML inside the pipeline if your manifests never change.

The Kubernetes endpoint configuration is one of the easiest ways to configure Kubernetes in any pipelineing tool. You simply need the API URL, and an authentication token. This also allows you to use any type of Kubernetes cluster you want. Managed, or unmanaged.

Variables

For this implementation example, we need 4 variables to be setup in Code Stream:

1. OVERAGE
2. VSS_VIOLATION_FOUND
3. VSS_PRIVATE_TOKEN (VSS API Token)
4. REFRESH_TOKEN (CSP API Token)

These four variables are setup under the Continuous-Verification project we setup in Cloud Assembly. These variables will be used to pass stage status between tasks in the pipeline so that we can validate the status of any of our Governance checks.

The project under Cloud Assembly is simply an organization unit of sorts used by vRealize Automation Cloud 8. This keeps all our project resources together, and makes it easier to find things when you are configuring the pipeline. More on that later.

Governance Checks

The Governance stage has 3 separate task groups, containing 4 total tasks.

1. CH Check
2. VSS Check
3. CH Variable Validation / VSS Variable Validation

The first two are separate, and sequential. The 3rd group contains 2 variable validation tasks that run in parallel.

1. CH Check

This check is an “SSH” Task. We use utilize the Agent endpoint that we have setup. We have to use SSH tasks for these checks, because we were unable to replicate the multi-container environment we setup for GitLab where every stage had it’s own container. For hostname, we’re using the IP of a server we have setup in Azure. This server has all of the resources we need setup so that we can run our validation checks. Things such as Python3. This same server will be used for the VSS Check as well. I have the authentication information setup so that Code Stream can authenticat to the SSH server. Form there, it executes the following commands:

cd /ch
export REFRESH_TOKEN=$REFRESH_TOKEN
sudo python3 cas-budget-test_py.py

This changes directory into the /ch directory that I have the CloudHealth API script that we edited for use with Code Stream. We set the environment variable of REFRESH_TOKEN to the variable we setup in Code Stream. It then executes the python script which checks the budget and writes the variable back into Code Stream.

2. VSS Check

This task is the exact same “SSH” style task that the CH Check tasks is, except it executes different commands:

cd /vss
export REFRESH_TOKEN=$REFRESH_TOKEN
export SHRI_PRIVATE_TOKEN=$SHRI_PRIVATE_TOKEN
python3 vss-findings.py

3. CH Variable Validation / VSS Variable Validation

Both of these stage tasks are “Conditional” tasks. This is a type of task that simply looks for a condition. If that condition is true, it continues. If it is false, it fails. This is where we’re validating our scripts. We are making sure that both the OVERAGE and VSS_VIOLATION FOUND variable are 0. If either is 1, then the script found a problem, and set the variable to 1 and the pipeline needs to fail. The code for the conditional statement looks like this:

${var.OVERAGE} == 0

${var.VSS_VIOLATION_FOUND} == 0

We now have a full Governance stage that SSH’s into our scripting server, runs the approprite Python scripts, then validates if it should pass or fail based on the status of the appropriate variables.

Deploy - Staging

The Staging Deploy stage has 12 different task groups, containing 12 total tasks:

1. Secrets
2. Catalog DB - Init DB
3. UserDB - Init DB
4. Cart Redis Total
5. Catalog DB Total
6. Catalog Total
7. Order DB Total
8. Order Total
9. Payment Total
10. User DB Total
11. User Total
12. Frontend Total

All of these stage tasks are identical, except that they each call a different YAML manifest for Kubernetes. Let’s go over the first one to show you the configuration.

1. Secrets

This is a “Kubernetes” task. We have set the endpoint to our Staging cluster that we setup earlier. We want to do an “Apply” action. From there, we want to use Source Control. You can also optionally set the YAML locally, but in case ours changes, we always want to use the most up-to-date version from the repo. So we then choose the Git endpoint that we setup originally. From there, we input the file path directly to the YAML manifest from the root of the repository:

/app/kubernetes/kubernetes-manifests/02-secrets.yaml

This task-type repeats 11 more times, but applying each aditional YAML manifest to the Kubernetes cluster. Code Stream makes deployment tasks very easy for Kubernetes clusters, but it can be a bit time consuming, as it isn’t as easy as just cutting and pasting code snippits and editing to replicate tasks.

Pipeline Execution

Because we setup this pipeline with a Git endpoint, the Git trigger has already been setup. So whenever there is a PUSH to the corresponding repository, the pipeline will automatically execute. This is standard in many CI/CD Pipeline tools.

Conclusion

As you can see, we were able to replicate the Continuous Verification process pretty easily within a very different CI/CD Pipelining Tool. This just goes to show that the concept is very portable, no matter what kind of tooling you’re using.

-Tim D, Cloud and Developer Advocate