This series of articles aims to identify the best practices to implement in order to secure your delivery process and your CI/CD pipelines:
0. DevSecOps & CI/CD Pipelines: understand the fundamentals.
1. Version Control System Security
2. Secure Continuous Integration (CI) Pipelines
3. Secure Continuous Delivery (CD) Pipelines
4. Securely Managing Secrets in CI/CD Pipelines
Welcome back to the CI/CD security series, where we continue to unravel the complexities and nuances of securing continuous integration and continuous delivery pipelines. In our journey so far, we’ve explored various principles of CI/CD security, and from now onward, we will tackle more in-depth concepts.
In this instalment, we delve into one of the most pivotal aspects of CI/CD security: the management of secrets. Handling sensitive data such as passwords, tokens, and keys is a cornerstone of secure software development practices, a principle Gologic stands firmly behind.
This article navigates through a three-stage exploration—starting with a basic solution of encrypted secrets, moving to dedicated secret vaults, and culminating in the implementation of short-lived credentials and modern authentication protocols like OpenID Connect—we will guide you through the nuances of each approach, their challenges, and their benefits.
Golden rules
Before we jump into the comparison of the three secret management solutions, let’s lay down the three golden rules about secret management.
- Least Access Principle: Each component (user, application, service) in your system should only have access to the secrets that are necessary for its function. This reduces the impact an attacker can have on your system if a secret is leaked.
- Short Secret Lifecycle: Secrets should be changed (rotated) regularly and automatically. This prevents attackers from exploiting stolen secrets.
- Never share secrets: Treat all secrets as strictly confidential information that should not be shared between components (users, applications, services). This ensures that if a secret is leaked, the impact is limited and rotating the leaked secret is easy and does not impact on other systems.
The ideal secret management enforce those three golden rules.
Level 1: Encrypted Secrets/Sealed secrets
Among others, Ansible Vault, K8s Sealed Secrets, and Spring Cloud Config Ciphers, popularized these approaches in the early days of configuration management and GitOps. The issue they tried to solve is that we can manage all my K8s config in Git, except secret. Thus, to achieve a complete GitOps workflow, they suggested a way to manage secrets as code.
With this approach, secrets are encrypted using an encryption algorithm and key before being stored in source code repositories. Only users or systems with the correct decryption key can access the original, unencrypted data.
For instance, you might encrypt a database password before committing it to your Git repository. During the CI/CD process, your pipeline has access to the decryption key and can decrypt the password to use it for database access or pass it down to the application when performing a deployment.
The major Security Consideration here is key management. The security of encrypted secrets relies heavily on how securely the encryption keys are stored and managed. If the keys are compromised, so are all the secrets. Additionally, this means developers often have direct access to the secret and may have insecurely stored the encryption key locally on their PC.
Level 2: Secret Vault
A Secret Vault is a dedicated tool or service designed to store secrets securely, providing robust security features like centralized secret management, access controls, and audit logs.
With this approach, during the CI/CD process, the pipeline interacts with the secret vault to retrieve necessary secrets. Depending on the vault, the secret can be automatically interpolated, or you can access secrets via APIs or CLI tools provided by the secret vault.
The key here is to limit credential reuse as much as possible. Two pipelines or applications should not share the same secrets. This reduces the impact of a credential compromise.
We can distinguish several secret vault options:
- CI Provider Vaults: nowadays, all CI/CD platforms offer some kind of built-in secret management tools. These are tightly integrated with the CI/CD processes, ensuring ease of use and minimal configuration.
- When using this type of vault, the secret is often automatically obfuscated from the logs.
- Cloud Provider Vaults: cloud services like AWS Secrets Manager, Azure Key Vault, or Google Cloud Secret Manager offer secret management as part of their cloud offerings.
- Those are often ideal for deployments (CD) as the application would retrieve secrets from the vault. Removing the need for secrets inserted in configuration files.
- Dedicated Solutions (e.g., HashiCorp Vault): these are third-party tools dedicated to secret management, not tied to any specific cloud or CI provider and provide a central place to manage all your secrets.
The most convenient option is to use the CI Provider Vaults, often when using Cloud Provider Vaults or Dedicated Solutions, you can opt to sync secrets from your dedicated vault to the CI Provider Vaults.
The major Security considerations here are the secrets themselves. The vault doesn’t force you to rotate secrets frequently, nor does it prevent you from sharing the same secret across multiple pipelines or applications.
Level 3: Short-lived Credentials and Auths Protocols
Short-lived credentials are temporary access tokens valid for a limited time. They are dynamically generated and expire once their purpose is fulfilled.
For instance, your pipeline needs to push artefacts to the artefact repository (such as Nexus or Docker Registry). When the pipeline is initiated, a short-lived access token is created. The pipeline can use this token to authenticate with your artefact registry and perform authorized actions. Once the pipeline is complete, the token expires and cannot be used anymore.
In this way, we completely eliminate the need for secrets altogether. The only requirement is to validate the identity of the pipeline. OpenID Connect (OIDC) is a popular authentication protocol used for this purpose. It allows resources to verify the identity of the end user and obtain basic access control permission for each request.
Azure DevOps, Gitlab CI, and Github Action, offer all that option. When a pipeline starts, it generates an OpenID token containing metadata about the pipeline (project, repositories, committer email. etc.) and signs the token. Then, you can use that token to authenticate again various systems.
More and more system support this protocol and even if the system doesn’t support it yet you always have the option to use an authentication proxy in front of that resource that would deal with the authentication workflow.
Conclusion
In conclusion, securing secrets in CI/CD pipelines is a critical aspect of maintaining a secure software development lifecycle. We explored various approaches to managing secrets, starting with encrypted secrets and sealed secrets, moving on to dedicated secret vaults, and finally discussing the implementation of short-lived credentials.
- Encrypted/Sealed secrets are a simple solution that used to be popular a few years back, but the management of the encryption/decryption key makes it a rather vulnerable solution.
- Secret Vault is the industry standard for managing secrets. They are specifically designed for storing and accessing secrets, but they do not enforce security best practices such as least privilege and secret reuse.
- Short-lived Credentials are the way forward. They ensure that only authorized pipelines can perform actions on resources, instead of relying on passwords that may be leaked.
Don’t miss this opportunity to deepen your knowledge in DevSecOps. Stay up-to-date, enhance your skills, and secure your CI/CD pipelines with our practical guides. Contact us for a personalized consultation and actionable feedback.
By Gologic with the collaboration of Alexandre Couëdelo.
