Small teams work very very well if the team is full of competent people. Such a small team will execute much faster. The problem is that if you need to do more work at some point a small team won't be sufficient and you will end up hiring more. And as you hire more you will inevitably have quality dilution. Both communication in large team size and hiring issues make large teams much less effective.
Communication is the productivity killer. Where before you could have three people spend 95% of their time doing and 5% coordinating, as the number of people goes up, the amount of coordination blows up geometrically. you spend most of your time discussing and very little actually gets done.
Oncall health is another good reason to increase team size -- having a 4 person rotation where everyone is oncall once a month is much less sustainable than an 8 person rotation where you're oncall once every two months
