Really the collectivization of wisdom that kubernetes enables is just so amazing.
OpenStack feels like the last comparable extensible & open effort we could compare agaonst. OpenStack seemingly became a vendor fest, & vm's were never a truly great developmemt technology. The world of each platform service kind of being it's own separate thing contrasts so markedly versus a consistent configuration paradigm: desired state for everything is recorded on the api server (in k8s).
Things feel so much better this time around. There's really some good solid core policies we can write up like tbis for how to get safe, & we can really tell, semi-regardless of a lot of the infrastructure particulars. The paradigm holds very well across thr mamy quite varying deploemt possibilities.
And the containerization means we're dealing with much thinner workloads, with vastly reduced surface.
This is such strong evidence of winning, in a way a purely vendored world is so ill equipped to compete with.
I think it's neat to see company's like VMWare try to amalgamate containerization into their portfolio. Tanzu is like all the cons of on-prem like inelasticity applied to K8s
One of the most common misconfigurations I've seen is improper secrets handling. I'm glad to see it called out but I wish they would go into a little deeper detail on detection and remediation. Overall looks pretty good! I'm excited to have some baselines for K8s.
Can you elaborate what is improper secret handling in your opinion.
I personally use sealed secrets, but I think generation of them is sometimes tricky (tied to the namespace etc
I don't have an easy answer for you because I'm still struggling to find the "proper" solution myself. That's why I'd kill to have the agencies weigh in.
I'm not a fan of SealedSecrets or managing secrets in-code at all because of this scenario: Secret rotation requires pushing updated code which may take too much time in a compromised situation. Ie, I don't want to hinder secret rotation with CI tests and merge approvals when there's an active incident triage - I want to kill the pwned secret with fire and get a fresh secret in there ASAP.
I've gotten better results by treating secrets as state instead of infrastructure... Like, by coupling your pod to a secret vending service like Vault or Secrets Manager, you can remove potential for programmers to introduce anti-patterns that weaken our posture.
The benefit is that we can keep things encrypted, auto-rotate things behind the scenes, remove the chance of developers to have even an encrypted copy of secrets, and we can get better turn-around on IRs when secrets inevitably get leaked. And, you can still write out the secret as IaC.
To mitigate the potential for single-points of failure when the secrets service goes down is to allow for secret caching client side. Vault and AWS Secrets Manager, which are the only services I can vouch for, both have client-side caching capabilities.
Take that with a grain of salt. I admit, I'm opinionated
Thank you for this post. I agree and feel the struggle as well.
Often the issue is even to communicate that this type of data is, literally, a _secret_ and should be treated accordingly.
I'm looking forward to things like mTLS and solutions with short token lifetimes and automated rotation. This should definitly reduce the amount of encrypted secrets in Git repos and basic auth logins for every 3rd exposed service.
In the end developers rarely care how the secrets gets to the application, or if it's fresh or been in the same namespace for over a year.
Vault is what I would recommend. It's like a password manager for prod. When used correctly almost no one has to know the actual secrets and they can be easily reset and changed as long as you develop with that in mind.
Support with the Kubernetes secrets API is baked in and well documented.
Have you even looked at the guidance? There's not anything nefarious here. Just imminently reasonable stuff like: "By default, many container services run as the privileged root user, and applications execute inside the container as root despite not requiring privileged execution. Preventing root execution by using non-root containers or a rootless container engine limits the impact of a container compromise. Both methods affect the runtime environment significantly, so applications should be thoroughly tested to ensure compatibility."
They then provide an example Dockerfile in an Appendix.
I've looked through NSA hardening guides for AD, Linux, and many other technologies. They're helpful and imminently reasonable.
Do the NSA and CIA do dirty things? Yes. Should you trust everything they do? No. But you probably should at least skim the hardening document before you completely dismiss it because of your distrust.
Or don't even bother to read the document. Just look at the works cited section, which tells you a lot about the content of the article. Works are cited from: Center for Internet Security, Defense Information Security Agency, Linux Foundation, MITRE ATT&CK, Cybersecurity and Infrastructure Security Agency, Kubernetes.
The guide talks about the following:
- Scanning containers for vulnerabilities and misconfigurations
- Running Pods with least privilege
- Use network segmentation to limit blast radius
- Use firewalls
- Use strong authN and authZ
- Capture and monitor logs
- Periodically check your configurations and do vuln scans
It then points to CIS Kubernetes benchmarks, Kubernetes Security Technical Implimentation Guide, and CISA.
When the NSA pays a crypto company $10 million to promote crypto that the NSA had purposefully weakened, it’s completely reasonable to be wary of advice from them even if that advice, on the surface, seems benign.
So, if I'm understanding you correctly, everyone should be wary of advice such as "don't use service accounts with root permissions", "segment your network", and "regularly scan your assets for vulnerabilities" because of a clandestine program dating back to 2006 which was designed to weaken an encryption standard?
Should all my service accounts run as root because NSA thinks that's a bad idea?
What I am saying is that when taking advice from a proven malicious actor, it’s important scrutinize the advice more carefully than one would usually do.
Osama bin Laden could have said “eat your vegetables to be healthy” and that is obviously reasonable advice but it’s also largely useless because we already know it is true. It’s the stuff that isn’t obviously already correct that we should take extra care to verify.
The sibling comments explain why the link contains sound advice, so I'll just add this: you are ultimately responsible for the security of your Kubernetes cluster, whether you get your hardening tips from william1964 on Stack Overflow, or the NSA.
Most of this guidance can be checked over by anyone with basic competency in the field, and cross referenced against documentation - in general a lot of the guidance from government security agencies is focused on solid and robust design principles (isolating things, not reusing things, turning off specific functionality that exposes risky attack surfaces, being aware of unusual or non obvious defaults).
Not running containers as root is accepted by pretty much everyone I'm aware of as a 'good move. Same for building images carefully and preventing filesystem changes at runtime. Same goes for using sandboxing, syscall filtering etc.
This is a bit different from giving you a series of black box parameters and telling you to blindly rely on them. (And even then, DES being hardened against differential cryptography shows that's not always a bad thing, though more recent examples show the opposite can also happen).
Yes. While many agencies have a bad reputation post-Snowden, CISA and NSA have for many years - and will continue to release hardening guides that are invaluable to security engineers.
OpenStack feels like the last comparable extensible & open effort we could compare agaonst. OpenStack seemingly became a vendor fest, & vm's were never a truly great developmemt technology. The world of each platform service kind of being it's own separate thing contrasts so markedly versus a consistent configuration paradigm: desired state for everything is recorded on the api server (in k8s).
Things feel so much better this time around. There's really some good solid core policies we can write up like tbis for how to get safe, & we can really tell, semi-regardless of a lot of the infrastructure particulars. The paradigm holds very well across thr mamy quite varying deploemt possibilities.
And the containerization means we're dealing with much thinner workloads, with vastly reduced surface.
This is such strong evidence of winning, in a way a purely vendored world is so ill equipped to compete with.