(Look for another reply I made in this discussion, not in this thread)
SF has a different culture than the Valley; the peninsula is its own, highly technical world which SF has fewer tech businesses (though more than it used to). Sure SF has some crazy building constraints, but in the larger contexts (good universities, good laws, a much more "anything goes" environment) the whole region is really strong. SF has had a historical tolerance for crazy people since the gold rush (that kook might have $10K worth of gold in his pocket so if you send him away you won't get any of it) that few other places have had.
So cal (both SD and LA) has a different culture, more extractive, and has a different mix (lower university density for example). Remember when the film industry fled the east coast patent monopoly they went to the Bay Area first, only moving to Hollywood because of the sun (they still had to film outdoors in those days as artificial lighting wasn't good enough).
I know of two largish companies based in SF that ended up opening offices in the south bay to get better quality engineers. And by that I would guess they meant younger and willing to work for less $$$ for more hours, but they can't come out and say that, can they?
I'd be surprised if that was the reason. Engineers in the South Bay are more likely to work at large hardcore tech companies, or at worst Facebook, and why would there be younger people in the peninsula?
Why aren't there more companies in SoCal? Weather's nicer and so many more people live there. Then we could use Santa Clara for prune farms again.