They also do best in areas where there is a large amount of bedrock close to the surface. NYC is geologically blessed in this regard. The Bay Area has far less geologically stable ground to build on, and a lot of what there is makes up our gorgeous parks and nature preserves.
There is ample room to build in San Francisco - not much construction even needs to be skyscrapers. More 4/5/6 story construction would help a lot. The problem is entirely political.