Big fan of rsync.net but the firewall comment caught me a bit off-guard. The benefit of a firewall is that it's an isolated system which - apart from port blocking - guarantees a certain level of traffic logging and known-good state.
If you have everything on one host I'd say your overall setup on that host becomes much more complex because you only need to get hit by one successful exploit chain and all logs on that host cannot be trusted any more.
On a reasonable-size setup, I would expect that the logs are exported to dedicated log storage (log-only machines) as part of an effort to preserve accurate log files even in the case of a successful attack on one of the hosts. It is not especially hard to ensure that, for example, a record of an SSH login attempt gets recorded to an external server before the request is authenticated. So if you have (for example) an SSH account and a local privilege escalation exploit, there is still some evidence in the logs.
In the past, the benefits of a firewall were more clear-cut, but these days I think that it’s reasonable to have “defense in depth” without using a firewall as part of your solution.
Yes, people make mistakes. That doesn’t mean you need a firewall.
In order for someone to accidentally delete a production database like in the linked article, many people have to make mistakes.
> The firewall is still helpful in case they hire a new person who opens a port and forgets to close it one day
Let’s talk about this scenario a bit.
What does it mean for someone to “open a port”? Really, what it means is that someone is running a program on the machine which listens to a port. But, why should anybody be running services on production machines manually, except in an emergency?
Normally, any changes you make to production machines go through some kind of configuration management system. You can’t just SSH into one of the prod servers. It doesn’t matter if you are an intern or if you’re the CTO. You don’t have the credentials. Nobody does.
Instead, if you want to run a service on a production machine, you have to make that change in source control, send the change to somebody else for review and approval, and once it is approved, submit it. Your configuration management system then pushes this change out to production systems according to the script.
Of course, not everyone works this way. Not everyone can work that way. But many companies do have tight controls over the production environment and the decision to forego a firewall isn’t unreasonable.
If you have everything on one host I'd say your overall setup on that host becomes much more complex because you only need to get hit by one successful exploit chain and all logs on that host cannot be trusted any more.