I don't think it's fair to assume without evidence that existing homeowner are "crotchety neighbors". It's quite possible that they have legitimate concerns that, and would support new construction if their concerns were addressed.
Go to public hearings. Their demands are almost never reasonable. New housing should never add any traffic ever, and it should look fantastic (with different people differing on what this means) and it shouldn't have children because there's no way the schools could cope with the "massive increase", and it should have enough parking to handle the entire extended family at Thanksgiving (in a severely land-constrained area) and by the way it should also be actually "affordable" (far below market rates).
I think you're starting to (unintentionally) argue a straw man, or at least are making assumptions that just aren't true.
No one is saying that developers should just be able to build whatever they want, whenever they want, with no oversight. No one is saying that the community should completely shut up and have no input into the process.
But the actual objections brought up at actual planning meetings that happen in SF are universally bonkers. It literally actually is people opposing change just for the sake of opposing change. If there is anyone at these meetings making reasonable objections, they're completely drowned out by the other 20 people shouting that the sky is falling.
It's protectionism, and the "I got mine, so fuck you" mentality, plain and simple.
You’re assuming the process we have resolves their feelings when it does not. If anything, we’ve normalized the idea that new development is bad and as a community member you should be fighting it, leading to more resistance than there would be otherwise.
