Ron Harnik, our Product Marketing Manager, and Igor Savchenko, Scalr’s CTO, hosted a webinar on building a CI/CD pipeline using Jenkins and Scalr. The concept of continuous deployment (CD) is the cornerstone of the modern web application. You build fast, run test suites, then push to production. By running your tests on a non-developer machine every time someone pushes new code into the source repository, you will be able to quickly detect small errors (missing line breaks) to major bugs (a third party API was upgraded and now nothing works).
Building A CI/CD Pipeline was one of our highest performing webinars - letting us know that developers and administrators are still searching for the right way to build out their deployment pipelines in the cloud.
If you want to watch the webinar in it’s entirety check the following link. http://hub.scalr.com/webinars/building-intelligent-ci-cd-pipelines-with-scalr
Otherwise, read on.
There’s a lot of different tools to handle continuous deployment but Jenkins is the most popular with our customers. Alternatively, you could use CircleCI, Codeship, Bamboo, CodeDeploy with AWS, and many more. They essentially serve the same purpose and Scalr is open to all CD integrations.
Implementing a CI/CD pipeline using Jenkins & Scalr is popular for two reasons. First is the beauty of Jenkins itself - creating a build schedule, easily moving code from staging to production, and easy-to-read steps from the Jenkinsfile. There’s also a big collection of plugins available.
The second is the cornerstone of what we do at Scalr: creating reactive and preventative policies around those pipelines. Here are some examples:
When Jenkins finishes its tests at the development level, you can promote your code to staging environments and terminate development servers.
You can set automatic alerts for your QA teams if they need to test and fix bugs, or roll back changes if anything has failed.
You can create user permissions that tie directly into your role based access control. Not every developer on a team is the same, so their permissions shouldn’t be. Only senior devs should make the final push to production.
Using Scalr you can abstract away from the Jenkins UI and integrate your CI/CD pipeline with the rest of your workflows. The great thing we heard from customers as we developed this feature was that they didn’t have to do anything new to get Jenkins to work with Scalr.
If you’ve never used Jenkins, it’s a way to define your pipeline as code in a list of steps (called stages) via a Jenkinsfile.
Here’s what that looks like:
This is a simple example, but it explains the concept. Using the node.js image, we checkout our code, confirm that node is running, prune any extraneous packages we don’t need to install, then run tests. If it all checks out we build a new Docker image, then deploy the new image. There’s also a cleanup stage and an error handling not shown here.
You can set an automated build routine (i.e. every 24 hours, twice a month), trigger builds manually in the Jenkins UI, or based off a webhook or a Github commit. The Github commit is the most sensible if your master branch is being monitored.
During the webinar, Ron and Igor created this CI/CD pipeline - using a Github commit to automate the deployment of new code through Environments (a grouping of cloud resources) using the Scalr orchestration engine.
Here’s the workflow:
- First, a user commits a change to GitHub.
GitHub then sends a webhook to Jenkins, which initiates the build process and deploys the code to a Development Environment within Scalr.
The owner of the new application stack in the Development Environment is the user who originally committed to GitHub. So If I, Alex the developer, pushed to Github to test, I would be attached to the servers in Scalr for notifications and auditing purposes.
Jenkins then runs the necessary tests. If all is clear, the code is deployed to a Staging Environment, and the dev stack inside the Development Environment terminated.
Whether you want to automate testing inside Staging Environment is up to you, but typically this is where you’d perform any manual testing or necessary QA tasks.
If all is clear and the code is ready for production, we fire a custom event, DeployToProduction, which will deploy the new code to production and terminate the staging servers.
Beyond the workflow of a CI/CD pipeline, I could set policies where only certain users are allowed to start the build process depending on the Github branch. If they don’t have the right permissions at any stage of the pipeline then we can keep events from firing.
First you start with installing and starting a Jenkins server inside Scalr (via a simple script). Then you set up your code repository, the branch to build from, and the build trigger within Jenkins.
Then on build you run this script. We’ll also grab the commit hash to use when naming the servers inside Scalr so we know which Git commit they came from:
In this example, only changes to the Master branch will trigger the move from the Dev Farm to the Staging Farm. If a change is committed to another branch, the test will run on the Dev Farm and be promoted to Staging, but we can prevent code from branches that aren’t the master branch from going to Production. So, every time a commit is made to the master branch, a server with that code is spun up in our CI/CD Development Environment.
Like we mentioned above, we grab the commit hash and attach it to the server. We can also see the owner, and the datetime that it was created. We also have different branches that have been pushed for testing. If tests suites all pass, the development server is cloned to the Staging Environment and terminated.
Now let’s look inside Staging Environment:
Our servers are promoted to staging, and keep their original commit hash. Here we can do manual testing by connecting to the server directly.
To promote from Staging to Production, we have to fire the DeployToProduction event (custom event). The code is then deployed to the Production Environment.
With this event we will take the code and deploy it to Production. And that’s it!
But part of the beauty of Scalr is setting policies around the users that are using this infrastructure. Even if we’ve abstracted away from the Jenkins UI and API to make life simpler, we want to make sure that not everyone can promote the new code to production. The power of Scalr is provisioning, governance, automation, and security all rolled into one tool.
Let’s start with who has access to the different stages in the CI/CD pipeline.
Users are assigned to Teams, which can have different permissions in the environments they have access to. Meaning that my Developers team can have more flexibility and freedom in Development, but less in Production. I can also assign different permissions to the users inside my team, giving senior developers more flexibility than junior devs.
Below I can see the Teams that can access my Production Environment, and what their permissions are inside that environment. We handle permissions through ACLs, IAM policies that work on all cloud resources, regardless of their cloud provider.
An in-depth look into that ACL shows you that this demo team doesn’t have any access to servers inside Production. Of course, we can change these permissions to read-only, or limited access, or full permission.
Regardless of all of these ACLs, the most important in this CI/CD pipeline example is that only a few users should have the permission to push to production via the DeployToProduction custom event. I would give admins permission to fire the script without guardrails, and allow developers to fire the event with approval. Here’s what that looks like:
And that’s it! Building out a CI/CD pipeline using Jenkins and Scalr ensures that you’ll have control over the infrastructure, and ACLs give you control over the user experience managing that infrastructure. This is just an overview on the webinar, so if you’d like to learn more and specifically how Scalr solves the issues we mentioned above, watch our on-demand webinar - Building Intelligent CI/CD Pipelines with Scalr.
Check below for helpful links about Scalr and guides to setting up a CI/CD pipeline.