The great bargain of America is that you live in a country with insanely high violent crime rates for the developed world but don't complain about it too much because it's usually well contained to gangs or other communities/neighborhoods which are easy to avoid by those with means. What you experienced, even in East Palo Alto, is exactly that. Even in Chicago which has really high rates of violent crime, the north side neighborhoods are perfectly safe.
You're right, but it's not a sustainable way to govern nor has it ever been fair to anyone not of means. In SF I see this anger that people of means can't separate themselves from crime and I think it's time for Americans to finally have the conversation on safety work for everyone and not just the moneyed.
The Bay Area has always been particularly badly segregated. Palo Alto is wealthy and posh, East Palo Alto is a dump. San Jose boasts huge tech companies while East San Jose is full of gang fighting. I grew up in one of these violent areas of the Bay (starts with East but that's all I'll mention publicly) and the gap in resources is huge. Bad schools, poor libraries and community spaces, poorly maintained parks, unsafe neighborhoods, no pedestrian affordances, high speed badly maintained roads, the list is endless.
The downtown part of SF isn't really considered a posh area by most of the old money interests in SF, who live in the Northwestern parts of the City. The old political families of SF all live there in huge, beautiful houses and the neighborhoods are very different than the parts of FiDi and SOMA that new money tends to hang around.
The Wire is a great show that goes into some of the politics and challenges of America's containment style of policing.
In San Francisco, that bargain is unraveling.