Agreed 100%. And forget even having a place to be. Just finding a place to answer the call of nature is a problem. America, where if you don't have money, it's a crime to piss anywhere but in your pants.
I think you're missing that most of the homeless presence is in specific areas, like downtown, where there are not many free restrooms.
Private businesses see a lot of abuse of their restrooms — from people using them and not paying for anything, but also because they make good places to do drugs, and it's common for homeless to overstay their welcome and/or damage the facilities. As a result, many businesses put locks on them and require that paying customers ask for a code.
There are a few publicly provided restrooms, but they tend to be few, far between, and in pretty sorry disrepair. In SF for example, you can find restrooms in parks and libraries, but they're often a mess. There are some single use restrooms on the street that were designed to provide autonomous self-cleaning public facilities, but because of abuse, nowadays they're mostly either out-of-order or have a full-time worker to babysit them (also, there's not many of them, and there throughput is very low).