Thanks :). I'm waiting on Terraform support for Secret Manager[0], then I'll update the configurations to use that for storing the initial root token and certificates.
Vault works great on GCP, and it's used by hundreds of enterprises who want brokered identity management across clouds, dynamic secrets, and an open core model that fits their business.
However, we kept hearing that sometimes Vault was complex for some small problems. If you're not using its advanced functionality, Vault is incredibly cost prohibitive. It doesn't run well in a serverless environment, so you have to pay for VMs 24/7 even when secrets aren't being accessed.
We continue to contribute to Vault and build deeper Vault integrations into GCP - that's not changing. Secret Manager provides choice. Just like there's certain scenarios where you'd prefer a NoSQL database over a relational one, there are scenarios where you'd prefer Secret Manager over Vault and vice versa.
Since this question and "Secret Manager vs Cloud KMS" keep coming up as questions, I'm going to work with our team to put together something in the documentation.
I wonder how is GCP Secrets Manager comparing to Hashicorp Vault? Is GCP SM implemented in Hashicorp Vault (or Berglas) behind the scene? If not, what are the differences in terms of feature set?
Secret Manager is a Google Cloud first-party fully-managed service. Vault still works and runs great on GCP for users who want more advanced functionality like dynamic secrets.
Hey Seth, I know there's been _quite_ the effort under the hood to make this product a reality and am curious if you would be open to writing a bit about lessons learned to help others learn about the research & development process.
Sure - in short, we partnered with multiple customers in designing and building this service. Let me check with my team and see what we _can_ talk about :)
For AWS, as with many cloud offerings, read the pricing small print: AWS Secret Manager also charges per API request [1]. It isn't expensive, but you should keep it in mind when you architect your infrastructure. (We actually switched from fetching secrets at runtime to injecting them into containers are deploy time, and this was one reason).
This bit us too. We use goodaddy's external secrets [1] to fetch secrets from AWS secrets manager and make them available to the cluster. It polls the secret every n seconds, but with many services consuming secrets, it can scale up pretty quickly and start to build up cost.
Sorry the pricing is unclear. In Secret Manager, secrets are versioned (a "secret" is a named collection of "secret versions"). We only charge for active secret versions (you can destroy secret versions, usually after a rotation operation).
The secret version contains that actual secret data (i.e. "ABCD1234"), and you can choose the regions in which you want that secret data replicated. Each region you choose is $0.06.
So if you had 1 secret with 12 versions stored in 2 regions, that would be 12 x 2 x $0.06/mo = $1.44. Hope that helps!
EDIT: replaced "*" with "x" in math because it was getting parsed as italics
Seems more comparable to parameter store, which comes in both a paid and "free" version. Secrets Manager does have some of the same capabilities, but it also extends it to things like short lived credentials.
If you have an enterprise, HashiCorp Vault has the best features and integrations that I know of (everyone should be using short lived credentials, but who actually does if they're not using Vault?). If you can't use that, AWS Secrets Manager for its simpler integration of secret rotation, and if that's too expensive, AWS SSM PS. Google Cloud Secrets Manager would be analogous to AWS Secrets Manager.
The benefit of AWS Secrets Manager has been that it integrates directly with other services. In ECS/Fargate, you can create a task definition and list secrets to be injected at run time. It also has features like secret rotation so you don't have to roll your own (well, you kinda do w/Lambda, but they have samples), and you can use things like IAM to lock down access per-secret or per-IAM-ARN-hierarchy.
Like everything in AWS, it'll cost you more, but it'll also eliminate a lot of engineering time. If you're trying to pinch pennies, use Vault or something hokey like pulling a token from AWS SSM PS or S3 and rely on IAM roles/profiles/etc for access control.
I'm using the SSM Parameter Store [0] a lot, because of its easy integration in applications – many services can directly pull secrets from it and provide them as environment variables. The secrets can be KMS encrypted and cost a fraction of the Secrets Manager, which I think is just the parameter store with secrets being rotated by a Lambda.
It also has a pretty generous free tier, so if you're not using very ephemeral creds and you don't need auto rotation, it's a fantastic option (disclaimer though I work at SSM and am biased - glad to hear you also like the product :)
Hashicorp vault is basically industry standard I believe. But there are various cloud APIs too, which have varying degrees of integration with kubernetes.
I wish this wasnt the case. An open core product shouldn't be the industry standard as it's not truly open source (I doubt HashiCorp would allow a PR that brings enterprise functionality to open core) and the open core/give it away for free model stiffes competition (see Vaults pricing for an example, they know they own the market).
How is open core not truly open source? Of course they wouldn't accept a pull request that makes their pay product obsolete. But open source does not dictate that you must accept all pull requests.
There is nothing stopping you from forking the project and adding all the enterprise functionality you want.
We have been trying to simplify Vault + Kubernetes integration in our KubeVault project. https://github.com/kubevault . I would love to hear if you have any thoughts.
Swarm is on its way out. Mirantis bought that part of Docker and they said this:
"The primary orchestrator going forward is Kubernetes. Mirantis is committed to providing an excellent experience to all Docker Enterprise platform customers and currently expects to support Swarm for at least two years, depending on customer input into the roadmap. Mirantis is also evaluating options for making the transition to Kubernetes easier for Swarm users."
Agreed. I’ve been working a lot with Swarm over the past year and while it’s not perfect, it has been an overall great experience. I really enjoy the simplicity and tight integration with Docker, and I feel that the ecosystem around Swarm was just starting to mature. Hopefully it remains an option for a while.
Also, I haven’t needed to integrate something like Vault because the Swarm secrets feature covers my use case perfectly.
I'm also using the secrets feature but it's very brittle. If your swarm gets destroyed or you want to migrate servers, it's a huge pain to get those secrets back up and running. Unless I'm missing a better way to do this, I'd love to know!
I currently have my secrets stored in an encrypted text file in case I need to bring them back up.
I do something similar, using Ansible Vault (not to be confused with Hashicorp Vault!) to store the encrypted secrets in my ops directory (separate from the application source code). They’re stored as YAML so very easy to upload/regenerate with Ansible.
I feel like so much in the cloud space is actually getting worse because of this. Its like all the old server shufflers and netops guys have taken over.
(shameless plug as well) Try out CyberArk DAP (formerly Conjur): https://hub.docker.com/r/cyberark/conjur. OSS, powerful RBAC, has k8s-native workload ident auth, works on OCP and w/ PCF, lots of tooling integrations (https://github.com/cyberark/summon, https://github.com/cyberark/secretless-broker, etc), and there's a relatively straightforward path up to "enterprise"-y stuff if you think you want the bells and whistles (replication, auto-deployable followers, multi-zone failover, etc).
As others have mentioned, also using SSM Parameter Store (if you're in AWS) + KMS. Running most of our micro services using ECS (Fargate) which can populate environment variables directly from secrets in SSM Parameter Store.
AWS Secrets Manager is fine if you're in that cloud. We inject all our secrets into the API containers' environments when they're deployed. For some things that have refresh tokens and access tokens with shorter lifetimes, we use the Secrets Manager API to fetch them when necessary at runtime.
Vault can do a lot more than AWS Secrets Manager -- possibly GCP too but I've not checked. So if you're paying for Vault (bare in mind Hashicorp also do a free version) then odds are you want enterprise features rather than an encrypted key/value store.
For example, with Postgres it can store root creds and issue non-root, time-bound creds for each service. Vault's "backends" allow it to do _a lot_ of things that a secure static credentials manager can't.
Not to say that this isn't good though – any proper secret management system is better than none :D
Yes, GKE would be great. If we could specify the source of an ENV in a Deployment as a Managed Secret that would be incredible and save us so much hassle.
while a bit of a toy still, might be of interest: https://github.com/masonwr/CloudSecret. basic CRD that maps google "managed" secrets to Kubernetes secrets.
How do permissions work? Can I only give access to a specific secret to a specific person/group of people? Playing with Secrets Manager it looks like anyone with access to the project can access all secrets.
> Can I only give access to a specific secret to a specific person/group of people?
yes.
> Playing with Secrets Manager it looks like anyone with access to the project can access all secrets.
just FYI, this is not the case. the "Secret Manager Secret Accessor" role must be applied per-secret for any identity to read the content of that secret.
You can use KMS as part of your own secret manager, but you would have to store the actual secrets somewhere else, like GCS or Datastore. KMS stores encryption keys, and has an API for encrypting/decrypting. For example to retrieve a secret using KMS you would get the encrypted data from where it is stored, like Datastore, and send it to KMS to have KMS decrypt the data. Secret Manager actually stores the secret, so a single API call can retrieve the decrypted value. Secret Manager also has versioning, which is important when rotating secrets. If you were building your own solution around KMS you would need to do versioning in your storage schema, wherever you end up storing the secrets.
You can also use AWS Parameter Store with KMS (SecureString type) to store secrets in Parameter Store. Of course this doesn't offer rotation of the secrets although KMS will handle key rotation.
KMS manages encryption keys. A secrets manager gives you an API for storing and referencing arbitrary named secret values with audit logs, etc. for them, which are encrypted using KMS.
For example, this might be something like “store a secret ‘my app remote API key’ encrypted with a regularly rotated key and allow ‘my app container’ to retrieve the value”.
The title is missing the "Secret" word which makes a significant difference. "Cloud's Manager" vs "Cloud's Secret Manager". With the current title I was thinking about a generic cloud manager for GCloud
We are very excited to bring Secret Manager to the market. Let us know if you have any questions!