It can matter for future job prospects. You being at least somewhat physically available to make in-person appearances at least sometimes without it being a big deal is appealing to many employers.
This comment reads as if the only tech jobs that exist are in the Bay Area which is obviously not the case. Most tech jobs in the US are somewhere else.
You are absolutely right. Although it does depend on who you want to work for, the biggest tech companies are all in The Bay, with some in alternative US tech hubs like Seattle, NYC, or Austin (haha, Texas isn't for everyone).
All the "prestigious" large tech companies will also have offices in half a dozen (or more) other US cities. And most of them still allow remote work especially for senior folks. The only benefit I see to being in the Bay Area is if you want your entire career to be Employee #1 of newly-VC-funded startups, not because of the VC funding necessarily but everyone who is trying to get that funding spends at least some time in SF.