With all the ceremony required to get private subnet instances talking to the internet, why not just put all instances on the public subnet but cut off incoming connections using a security group?
Defence in depth? An extra layer between an attacker and your secret data is rarely a bad thing.
The tradeoff against convienence is worth it for some environments. E.g. what if AWS has an issue with applying security groups properly and it fails open?
Arguably, failing open is the right thing to do in some cases as it avoids causing an outage for your users. However it does expose them to a risk they may not have been prepared for.
I know OpenStack had an issue a few years back where, when the component responsible for managing the security group rule implementation was reset, it failed open until all rules were rebuilt (at the time, this was implemented as iptables rules on the hypervisor's). Another bug (well, poor implementation) meant it could take several minutes or more to rebuild the rules.
It happens, and you should be aware of and understand the risks of things like this happening when designing your infrastructure and choosing which tradeoffs are worth it for your particular use case.
The defence in depth concept makes sense. The good news is that if I understand correctly, security groups fail closed - so that makes things a little safer.
Just spoke to an AWS Architect, and the points made were similar - more secure default state, less chance of screwing up - ham fisted attempts at security groups can open things up too much, private subnets are tough to expose unintentionally, adding multiple security groups makes the rules less restrictive, possibly in unintended ways, etc.
