This series of articles aims to identify the best practices to implement in order to secure your delivery process and your CI/CD pipelines:
0. DevSecOps & CI/CD Pipelines: understand the fundamentals.
1. Version Control System Security
2. Secure Continuous Integration (CI) Pipelines
3. Secure Continuous Delivery (CD) Pipelines
By Gologic with the collaboration of Alexandre Couëdelo.
A deployment pipeline handles (automatically or semi-automatically) the process of configuring an environment such that a new version of an application is released. They essentially deploy to various environments (e.g., development, staging, production) based on predefined criteria (branches, tags, quality gate, etc.) and may orchestrate post-deployment tests such as:
- UAT
- Load Testing, etc.
This Gologic article focuses on the security of continuous delivery (CD) pipelines. Ensuring the pipeline’s security requires essential security measures. Firstly, implement compliance and audit validation. Secondly, secure integration points and deployment targets to isolate the deployment environment and properly restrict permission. Thirdly, implement rigorous security checks by protecting your artefact store with access to resource controls, encryption, and artefact signing, thereby enhancing overall security.
Secure Integration Points and Deployment Targets
Integration points refer to securing the systems that deploy the application and deployment targets to various environments such as development, staging, and production. A critical component of this process involves ensuring the integrity of the artifacts and configurations deployed.
Key practice in this area includes:
- Imperative deployments script to execute the deployment, for instance, kubectl or Helm command in the case of Kubernetes. In this case, the deployment script is executed by pipelines using tools like Jenkins, Azure Pipeline, GitHub Actions, etc. This also tends to be referred to as a push system, where changes are pushed to the environment
- Declarative deployments known as GitOps rely on changing configuration in the Git Repository to define the desired state. In the case of Kubernetes, an operator such as ArgoCD or FluxCD will apply those changes to the cluster. This also tends to be referred to as a pull system, where the environment pulls changes
These two deployment methods have advantages and use cases, and organizations may choose one over the other based on their preferences.
Imperative deployments, on the other hand, offer more explicit control but suffer from the same security risk as CI pipelines we discussed in the secure continuous integration (CI) pipelines article. Pipelines should only be able to deploy to a given environment by only granting the minimum access required, thereby mitigating potential attack vectors. Audit and compliance checks should be in place and prevent deployment if criteria are unmet.
Declarative deployments require robust access control over the Git Repository to prevent unauthorized changes, similar to what we have seen in the previous article about secure version control systems. In this case, there isn’t much concern about deployment targets since the environment itself is responsible for applying changes. However, a pull request should be required, and an audit and compliance check should be in place. As we will see in the next section.
Compliance and Audit
Automated deployment in the development cycle should follow a clear validation process. It can be mostly automated, but having this process will not only improve security but also reliability. Here are the main rules you should integrate to your deployment pipeline:
- Maintain a record of all deployment changes, including code changes, configuration updates, and version control history. This can be achieved by publishing the release node and having a reviewer sign off on those changes
- Evidence of vulnerability scanning, penetration testing, and/or static code analysis, should exist before the deployment process to address any potential vulnerability
- For major releases, define a change management process, incorporating composition analysis (SCA), that includes change approval and review by relevant stakeholders. Major changes can be identified by using semantic versioning and considering the number of changes
Finally, it may be important to involve/work closely with your organization’s compliance and legal teams to establish specific rules and procedures for environments with legal compliance standards (e.g., HIPAA, GDPR, PCI DSS)
Secure Artifact Repositories
One integration point we didn’t cover in the first section concerns the Artifact registry, which stores the artefact produced during the continuous integration phase and deployed in your production environment. Ideally, you need to make sure that the artefact produced is actually the one used. Hence detecting breaches in your CI system or Artifact Store and preventing tempered artefacts from going live in production.
Code artefacts can be digitally signed as part of the build process. This signature serves as a tamper-evident seal. The deployment pipeline can later verify this signature to ensure that unauthorized changes have not been introduced to the codebase. This is especially important when deploying to production, as it helps guarantee your software’s integrity.
Conclusion
Secure continuous delivery pipelines are crucial for mitigating any active threat and delivering software that is both secure and reliable. After all, this is the last step in the process, and careful review at this stage will ensure better quality software.
In the development process, it is essential to implement measures, including a focus on potential issues, to ensure compliance and validation are included in the process. You also need to secure integration points such as Git Repository (especially in the case of declarative deployment) or Pipelines System (especially in the case of imperative deployment). In both cases, securing the Artefact store and including artefact signing is crucial to protect and detect issues before going live.
This article is the fourth in our series: “DevSecOps: Secure CI/CD Pipelines,” where we have mapped various security vulnerabilities and covered most CI/CD workflows. This includes:
- Version management systems
- Continuous integration pipelines
- Continuous deployment pipelines
Coming up, we will tackle more specific topics, including secret management, artifact management, observability, and security incident management
Don’t miss this opportunity to deepen your knowledge in DevSecOps. Stay up-to-date, enhance your skills, and secure your CI/CD pipelines with our practical guides. Contact us for a personalized consultation and actionable feedback.
By Gologic with the collaboration of Alexandre Couëdelo.
