CI/CD & DevOps Pipelines: An Introduction

Simply put, the CI/CD pipeline is a workflow for DevOps teams to automate the software delivery process. CI/CD stands for continuous integration/continuous delivery, a software delivery method development teams use to deliver code changes more frequently and reliably. CI/CD embodies two sets of complementary practices, each of which rely heavily on automation.

In the continuous integration process, development teams implement small, regular application code changes and merge them into a shared repository. Modern applications are developed using a variety of tools and platforms, requiring a means of integration and validation of every code change to make sure those changes don’t break the application. Continuous integration automates the process of building, packaging and testing code whenever a team member executes version control changes. This makes it easier for teams to commit code changes more frequently, resulting in improved collaboration and app quality.

Continuous delivery is a workflow engine that allows you to kick off processes like testing for bugs and uploading the development team’s code changes to a staging repository.

The CI/CD process is important because it makes deploying processes easier and more predictable. It also brings consistency and reliability to the software development process, resulting in improved collaboration between development teams and operations, lower costs and better applications.

In this article, we’ll provide a small tutorial on the CI/CD pipeline process, discuss its uses and best delivery practices and explore the numerous benefits CI/CD can bring to the development life cycle and ultimately, your entire organization.

The Basics of DevOps

DevOps is an approach to IT delivery that combines people, practices and tools to break down silos between development and operations teams. DevOps bridges the gap between software development (where application code is created) and IT operations (where those applications are put into production, made available to end users and maintained.) DevOps teams accelerate the development of applications and services and, with a more responsive approach to management of the IT infrastructure, can deploy and update IT products to stay competitive with the industry. 

DevOps grew in part from the agile development movement as a way to solve one of its critical issues — while agile developers produced new apps and code updates more frequently, traditional operations teams struggled to test and deploy them, negating the value of rapid development. By extending agile principles across the entire software development life cycle (SDLC), DevOps is able to optimize the entire workflow with a goal of continuous improvement. High-performing DevOps teams not only see faster code iterations and deployments, but overall shorter time to market for new ideas, fewer bugs and more stable infrastructure.

CI/CD is a foundational process in DevOps. Developers regularly incorporate new code — often daily — into the main source code. As this new code is checked into a central, shared repository, an automated build process tests and validates the changes, allowing developers to quickly identify problems and receive immediate feedback so they can make any necessary adjustments.

What is the CI/CD pipeline process?

The CI/CD pipeline is essentially a workflow that provides a pathway through which DevOps teams automate the software delivery process. In the absence of an automated pipeline, teams would have to configure their workflow to be performed manually, which is time-consuming and prone to error. The CI/CD pipeline removes manual errors, standardizes developers’ feedback loops and increases the speed of product iterations.

Predecessors to CI/CD include application release automation (ARA) and automated release management (ARM), both of which fall under the category of work “release automation,” and represent new iterations of evolving automated processes and trends in this space.

There isn’t a standard methodology or implementation for a CI/CD pipeline, and each DevOps team will choose the tools and services they’ll use to build theirs, but a pipeline typically includes the following stages:

• Source stage: Developers check out and locally work on application source codes stored in a central repository, such as GitHub. They then create a new branch for the feature or bug they want to fix, running tests on it locally in their development environment, before committing it back to the source repository.
• Build stage: New code committed to the source repository triggers the build stage, in which the branch codes are gathered and compiled to build a runnable instance of the release. Developers can build and test their releases multiple times a day to detect errors in the code and receive alerts if a build fails, or about other issues so the team can implement a fix as soon as possible. Once the code is error-free, it moves onto the next stage.
• Test stage: The build runs through several tests to validate the code and ensure it is acting as it should. Most commonly, these include unit tests, where the application is broken down into small units of code and tested individually, and user acceptance testing (UAT), in which people use the app to determine if the code requires more changes before it is deployed to production. Load, security and other continuous testing may be conducted at this stage.
• Deployment stage: When the application is ready for production, there are many different deployment strategies from which to choose. The best choice depends on the type of application, how much overhauling it has undergone, the target environment and other factors. The following are modern deployment approaches for cloud applications. 
  • Rolling deployment: This method delivers the updated application incrementally until all targets have the updated version. Rolling deployments pose less downtime risk and are easy to roll back but require services to support both the new and old versions of the app.
  • Blue-green deployment: In this method, developers run two versions of the app in parallel on separate infrastructures. The last stable version of the app is run in the production environment (blue) and the new version is run in a staging environment (green). The green version of the app is tested for functionality and performance, and as it passes, traffic is shifted from the blue environment to the green one, which becomes the new production environment. Blue-green deployments are fast and easy to implement but can be costly if the replicated production environment is particularly complex.
  • Canary deployment: In this deployment model, developers release an application to a small subset of users who use it and provide feedback. Once the updated app is confirmed to be working correctly, it's rolled out to the remaining users. The strategy allows developers to test two versions of an app with real users side-by-side and allows updates and rollbacks without downtime.

At every stage of the pipeline, the development team receives alerts to errors so they can immediately address the issue. The code changes go through the pipeline again, so only error-free code is deployed to production.

Continuous Deployment vs Continuous Delivery

Continuous deployment is the process of deploying by automating the release of code changes into the production environment, which occurs after new code is integrated and approved through continuous integration and continuous delivery. The code changes run through a series of automated tests, and once they pass, are pushed immediately to the software’s users.

Continuous deployment may sound synonymous with continuous delivery — and both are often referred to by the abbreviation “CD” — but it is a separate practice. Continuous delivery is a delivery process that automatically pushes new builds to an automated QA testing environment to look for bugs and other issues, and once it passes, is approved for deployment and automatically pushed to end users. Because it doesn’t require any pause for human mediation, this final stage is known as continuous deployment.

Although continuous deployment extends the agility and velocity of CI/CD, it comes with its own set of risks because there is no manual intervention between final testing and release to users. Because of this issue, it’s not well-suited for software involving security of sensitive data, regulatory compliance or high financial stakes.

How is CI/CD used?

CI/CD is used to streamline and automate software development to deliver apps and services more quickly and frequently. CI/CD accomplishes this largely through automation, which accelerates testing, feedback, code corrections and deployment.

Implementing CI/CD practices offers many benefits:

• Faster and more robust releases: The most obvious advantage of CI/CD is that it speeds up software delivery and makes the process more efficient. With the automation of most testing and bug fixing, developers can increasingly focus on writing code and improving the quality of the application.
• Earlier bug detection: Thanks to automated testing at every stage of the CI/CD pipeline, developers can identify bugs and other problems as they arise — with the majority detected in early stages — eliminating potentially devastating surprises during deployment.
• Greater visibility: A CI/CD pipeline allows development teams to analyze builds, as well as test results and coding errors with great detail. This gives them a better understanding of what build changes led to problems and inefficiencies and prepares them to avoid these issues in the future.
• Faster feedback loops: CI/CD sets up a constant user feedback loop that developers can use to their advantage to experiment with features and improve the app experience.
• Reduced costs: The automated approach of CI/CD reduces the number of errors at every stage of the development process and simplifies the repair of errors that do occur. Developers no longer have to spend time on mundane, manual tasks, which allows them to focus on improving code quality and making app improvements that boost ROI.
• Increased customer satisfaction: More reliable releases and faster turnarounds for updates, bug fixes and new features incentivize customers to choose you over the competition.
  

CI/CD Dipeline Delivery Best Practices

While every CI/CD pipeline orchestration and implementation will be a bit different, below are some basic guidelines to help you address common issues and maintain the most effective pipeline for your organization:

• Use a consistent environment: A reliable CI/CD pipeline will always produce the same output for a particular input. But you can’t have a reliable pipeline if each run modifies the pipeline environment, so start each workflow from the same isolated environment.
• Apply pipeline analytics: Pipeline analytics visualizes telemetry from your CI/DC processes, expanding visibility and increasing measurement capabilities, so you know how well your delivery pipeline is functioning. 
• Commit early and often: Committing changes early and frequently makes it easier to identify bugs simply because there is less code to comb through. This small-batch approach will improve code quality and allow teams to iterate more effectively.
• Build once: Successful CI/CD pipelines typically include the build process as the first step in the CI/CD cycle, and the step is executed only once with the resulting output used through the entire pipeline. Reusing a single build prevents newly introduced  inconsistencies as it moves through the pipeline stages.
• Implement version control: A version control system is essential for storing source code, tracking code changes and reverting to earlier deployments when needed. Be sure to apply and use version control for scripts, documentation, libraries and configurations for your application. 
• Develop incrementally: Working in small iterations — breaking down the feature into smaller sub-features and using unique feature flags for each one — will make it easier to isolate problems and reduce the risk of integration issues when the feature is pushed to production.
• Monitor your pipelines: Correlate your pipeline data with other metrics in order to achieve optimal overall performance of your applications. Learn more about CI/CD DevOps Pipeline Monitoring

How To Implement the CI/CD Process

There are a few things to keep in mind when implementing a CI/CD process:

• Understand that CI/CD is a workflow and deployment process, not a product: Avoid getting caught up in notions of “the right tool.” The outcome you want from the CI/CD process is to transform your software development from a process that is manual, cumbersome and slow to one that is automated, lean and fast. Identify a starting point, then enlist an enthusiastic mix of developers and operations folks to run a proof of concept and continue to iterate until your pipeline is in place.
• Plan for success: There will be plenty of technical planning required from the get-go, including choosing platforms and refactoring apps to make them more suitable for iterative updating. But there will also be cultural considerations as your implementation spreads beyond the initial enthusiasts to the larger organization. Buy-in from IT and business leaders is critical for long-term success.
• Optimize your processes: Before you start introducing CI/CD tools into your existing deployment pipeline, take some time to evaluate your current processes and workflows. If they aren’t effective, automating them won’t be a magic fix, so be sure they’re in order before you integrate them into the new CI/CD pipeline.
• Emphasize continuous improvement: Approach CI/CD with a mindset of learning from mistakes and consistently improving. It’s important, then, to make sure your CI/CD tools provide continuous metrics that can be integrated within a common dashboard so everyone involved can track progress and determine what adjustments need to be made and where.
  

Building CI/CD on GCP and AWS

AWS (Amazon Web Services) and GCP (Google Cloud Platform) are two popular DevOps-as-a-Service (DaaS) options. Both have unique pros and cons.

AWS is typically fast, offers an easy way to migrate your DevOps to the cloud and operates on a convenient pay-as-you-go model. GCP makes a wealth of tools, such as Stackdriver Monitoring, Stackdriver Debugger, Stackdriver Logging, a security scanner service (App Engine), available to use in your application life cycle pipeline.

Both offer quality DevOps solutions, so the differentiators come down to price and operational zones.

Considerations for CI/CD Toolchains

There are a few things to consider when looking for CI/CD tools to add to your development tools chain.

The first is whether you want to use an open-source CI/CD tool versus a commercial one (it delivers the code) or build your own. An open source solution saves money, but there’s the risk that the developers may drastically alter the code or stop developing the product altogether. Also, open source tools often come with minimal support. Commercial tools, on the other hand, usually offer strong support and have more predictable update cycles, but they can be costly and can offer less flexible integration. Building your own solution lets you tailor it to your organization’s unique needs but requires a lot of resources.

Another consideration is the hosting model. If your codebase has to remain on premises, you’ll need a toolchain that you can deploy on your own servers or virtual infrastructure. However, CI/CD tools are predominantly aligned with cloud and cloud-native infrastructure, so CI/CD tools are also offered as part of a SaaS (software-as-a-service) model that runs on the vendor’s cloud infrastructure, allowing you to offload many of the support costs.

However you answer these questions, you’ll also need to consider the associated costs. Despite the upfront savings of an open source tool, you’ll incur costs setting up and supporting the tool on your infrastructure and lose some productivity as developers learn how to use it. With commercial off-the-shelf and SaaS tools, you will generally pay a fee based on your usage level.

Choosing the best option for your organization will come down to determining how many builds you’ll be doing, whether or not you’ll be doing concurrent builds and how many users will access the tool.

The Bottom Line: Take your business to the next level with CI/CD

The speed at which you deliver software is a big market differentiator. Successful companies boast up to 50 to 100 deployments per day, with some giants like Netflix topping 1,000. A functional  CI/CD pipeline cultivates the productivity, reliability and speed required to reach these numbers and deliver the robust applications necessary to meet ever-changing business needs. Simply put, if you’re not following CI/CD practices, you’re running the risk your business will get left behind. Transforming your software delivery isn’t fast or easy, but the effort is worth it and will position your business to thrive for years to come.