PMapper author here: I can't give PMapper the love it deserves, but I fully support the work done in Fennerr's fork at https://github.com/Fennerr/PMapper .
Kinda related: Is there a good tool for managing all these resources and credentials at AWS in an enterprise environment? We started nicely from top to down, splitting the root account into sub accounts and then having a terraform project per sub account. But now these terraform projects split themselves and start sharing their tfstate and i even saw circular references, as they try to share the variables to resources arns and so on. And policies are growing like crazy. It seems more like a mesh now.
Along with the good advice from liveoneggs, I have some questions/advice:
Are you leveraging modules? For example, if you create a new product in your account, can you just have the product call your own module to create its IAM roles to your own standards and specification? Or are you repeating IAM code over and over?
Basically all of your terraform code should be modularized because you’ll likely repeat it for different environments like staging vs. production. So, for example, when I want to build another product VPC in another environment, it’s just another deployment of the same child module with different input variables.
Are you trying to manage multiple products with the same terraform code? I see this a lot and while it’s not a catastrophe, I try to separate terraform repositories by their product or logical function.
I also wonder if you’re overusing AWS sub-accounts. I only use a separate account when the business needs to separate the business unit entirely, or to separate production from other environments. Everything else should be able to be kept separate via roles and policies.
Yes, there is a distinction between product and staging and everything is put into modules, so there is some reusability. But still, there teams working in different accounts, and sometimes they exchange data between accounts, like putting an output into that bucket and reading input from another bucket.
We started out with just one terraform project per account. Now its a terraform project per project within an account, to keep terraform projects compact.
But it somehow keeps growing and growing and each team and each project, they handle it differently. the taste is differently, it seems. and not everyone puts a lot of thought into rhe structure. So on the one hand its wild and free, but on the other i palm my face, when i look into the future.
Hmm, that’s interesting. This sounds more like an organizational structure and ownership problem than a code problem to me.
I’ve been at multiple companies where we’ve decided to build a set of modules that could be used somewhat universally for products, and then we’d basically say “hey if you don’t want to use this, either convince us that your needs are justified or do whatever you want and DevOps/SRE can’t support you.” Some of it we went as far as calling self-service.
If you have a menu of options than people can’t pick things that aren’t on the menu.
I also find that developers really just shouldn’t have almost any cloud infrastructure rights, with few exceptions.
You could also go the embedded SRE/DevOps route where the embedded infrastructure engineer on the product team can steer each separate product into following more standard approaches.
I’m also going to take a wild guess that your ops to developer ratio is too low.
As I'm guessing you're starting to suspect, there is no magic tool here.
It typically requires a dedicated 'platform engineering' team to manage the tooling and onboarding process, some kind of governance process over the services that are enabled, development and enforcement of the use of (in the case of terraform) terraform modules to constrain use to planned patterns and some kind of 'cloud security posture management' tooling to keep an eye on everything.
It increases costs and slows things down in the short term but it does help keep things reasonably controllable over time. I work in a highly regulated environment (five external audits in the last year) so my perspective is a bit skewed. However when we have an acquisition that has been doing things a bit organically, it's an impenetrable Charlie Foxtrot and the teams responsible for trying to get it under control have almost no understanding of where dependencies are, what's critical to the business or even how to start to insert some kind of influence or control over something that has been wholly unmanaged for years.
Thank you. Well, there you have it, this is a non-regulated wild environment and except for data privacy and looking at costs basically everyone does what they need. So i am very jealous about the discipline you seem to be able to push through. :)
Do you mean by data source, export the state to some external data store and the other projects import it from there instead of importing it directly from tfstate?
Assume role, yes, unless the services accessing the resources from outside.
I’ve been pondering on the question of why AWS Inventory CSPM tooling doesn’t back-propagate relationships via IAM, and this post is helpful with some prior art on the other attempts in the space to solve for it.
