Back in the early 90s my wife and I moved to SF because it had a thriving art and music scene and more interesting culture than the 'burbs of palo alto. But as you say, the long commute to SV was a killer and we moved back down. Back then SF was a bedroom community for SV with no tech sector. Businesses up there were banking (Wells Fargo, BofA, Crocker etc), retail, the local stock exchange, and a bunch of manufacturing.
Nowadays there's a bland sameness -- barely any music or other art much less much craziness. You can't imagine anything like the psychedelic scene appearing in SF much less Palo Alto these days, and most of what's left is in Oakland. Sigh.
I lived in San Francisco in the very late 70s through to the 00s. My first apartment share was $50/mo. The late 70s had the dying embers of the Beat Generation and San Francisco was a sleepy town. San Francisco was great in the 80s with a ton of theater, music, dance, art, everything. It was good in the 90s although the late 90s dotcom boom pushed/priced artists out. The 00s became pretty boring and compressed. I moved to Oakland.
I would say that San Francisco is quite nice now, great bones, although too expensive for interesting people to live.
I've seen similar complaints from cities in rich countries around the globe
Basically to have thriving arts scene you need people in the 20s-30s to be able to live on a minimum wage job and do art/music/whatever in their spare time. Even better if it is a part-time job or some sort of government scheme.
If your cost of living is too high then you are restricting yourself to trust-fund 20yo and older people with a bit of spare time. Also high property reduce the number of venues.
Oakland is more affordable for artists now than San Francisco but not even close to what San Francisco was in the 80s. But it's a very different time. I find myself occasionally trying to relive that and having to remind myself it's gone.
Oakland has crime now but 80s San Francisco had the drug wars with actual machine gun battles between the now gone high rise projects on Army (now Caesar Chavez).
Depends on one's interests. It sounds like my preferences would be more in alignment with yours - music and art - and yes, SF is almost completely lacking that today. But if one were an active part of the LGBT community - SF is a buzzing option. They have various festivals and events almost daily.
Oakland music scene isn't particularly inspiring either. Definitely more independent music events in run down houses, but quality and inventiveness is too often of questionable value.
it's true, the techies of the '90s and '00s who lived in sf embraced the culture (and suffered the commute). what happened in the '10s kinda steamrolled it.
Nowadays there's a bland sameness -- barely any music or other art much less much craziness. You can't imagine anything like the psychedelic scene appearing in SF much less Palo Alto these days, and most of what's left is in Oakland. Sigh.