Slightly on topic but we've been talking about vaults and keys for years now and I work in CI/CD but I still am no wiser to how I'm supposed to practically use this in my pipelines.
Can someone link something that explains it like I have 20 years in IT but I'm clueless.
I can't get past the fact that a key has to exist somewhere, a key that will give you some sort of access to a secret. So how is it any better if the key already exists in the CI/CD pipeline variables?
Another thing I'm curious about is rotation, which on paper is amazing but in practice would require your vault to have sysadmin access to all your systems, in order to do rotation. It just seems like a tall order to integrate.
I'm going to talk in rather vague terms here as I'm not talking about one specific implementation, but the concept here:
The idea is that there is some system that is capable of provisioning access credentials and automatically providing them to resources based on their identity. This could be your CI runner, scping auth credentials into known locations when the system is deployed. This could be your cloud provider, using something like IAM roles and the metadata service. This could be your kubernetes cluster, providing a mount inside the container. Ideally this system is also capable of rotating these credentials too.
Then you have another system which is able to verify those credentials to give access. If you're fancy, you're doing something like mTLS on all your internal network calls so every endpoint on the other hand is verifying every other endpoint, but it could also be something like secrets manager or vault just dispensing out API keys based on your systems credentials provided in the previous step.
There is definitely a tradeoff here that the system that manages the credentials effectively has the keys to the kingdom and could if compromised provide identity for anything. Some companies respond to this by delegating the whole lot to their cloud provider, which has its own risks of course.
The advantages of doing this are basically:
- You only have to manage one set of credentials manually (whatever your system managing the instance/service identity is)
- You can then easily make credentials for things accessing each other short lived if you know they're getting replaced periodically, which helps reduce the impact of breaches/leaks
- Depending on your specific implementation, you can avoid secrets hitting disk entirely, which further helps reduce the impact of breaches.
The basic concept is that the vault will hand out a secret to an app based on some other form of identification. Let’s say the app platform can assert that App A is allowed to use Secret S. then the vault will hand it out. This turns “something you are” into “something you have”
By centralising the secrets storage, you make it easier to change the secrets in one place rather than having to change a bunch of variables everywhere.
As for how you change the secrets? You should have some automated way of pulling the current secret out of the vault, storing a future secret in the vault, performing an upgrade of the secrets, verifying the new secrets are deployed everywhere, and then moving the "future secret" into "current secret" and keeping a historical copy of the "previous secret".
Whatever configures your infrastructure automatically will already need to have root access to everything and so that's a good place to implement the rotation.
The vault doesn’t need to access each system to rotate the keys. It just needs to sign a new token with a short expiry date.
The system reading the token can verify its integrity based on the signature.
Now the vault obviously must have a master key to do the signature. It’s a very powerful privilege, that can impersonate, yet not quite as privileged as being sysadmin to all systems.
The main advantage of this is that tokens that the users use can quickly be revoked. They always need to go back to the vault to get new tokens, here you can add more powerful protections and logs, always MFA.
So what actually should happen here is your pipeline mints a JWT token with some near term expiry per run.
You send that token to vaults JWT endpoint, and it validates that it knows the issuer and the signature matches the provided keys.
When configuring the vault side, you can further validate the signed token data which will include things like the scm org, the repository name, the actor ( user who made the commit ), whether or not it's a PR, what branch it's against, whether the repository is private.
From there you can set roles within vault that allows different policy per risk, i.e. random PRs from the public shouldn't get deployment secrets.
If you use an app role you can define policies that limit the usage of said app role to specific ip addresses, ec2 instance ids etc so that even if someone steals the key it will be useless unless they somehow root the server hosting your application (and if they do that they're getting the secret no matter what)
Even with a regular token the benefit is that if it is leaked (say, by a git commit etc) this by itself doesn't grant you access to the actual secret because it is stored in vault, and you'd need to have connectivity to the vault server(s) to make use of it.
Workload identity federation is the direction the industry is taking. All the hyperscalers have their own version, if you’re on-premise look into SPIFFE.
Can someone link something that explains it like I have 20 years in IT but I'm clueless.
I can't get past the fact that a key has to exist somewhere, a key that will give you some sort of access to a secret. So how is it any better if the key already exists in the CI/CD pipeline variables?
Another thing I'm curious about is rotation, which on paper is amazing but in practice would require your vault to have sysadmin access to all your systems, in order to do rotation. It just seems like a tall order to integrate.