Nope, it's worse. Websites will use it as an authentication method in password-reset workflows, which means hacking your account reduces to intercepting or redirecting text messages (easily accomplished). Rampant cryptocurrency theft over the past year has conclusively demonstrated that SMS-based 2FA is worse than no 2FA at all. The worst offender? Gmail. It uses your phone number for password recovery and is in turn used for password recovery by every other website.
1) Many people have had their Gmail account for a long time, starting before SMS-based 2FA was widely known as a security disaster (this is in fact still not widely-known)
2) Google still actively encourages users to add a recovery phone number
3) Users could have added a phone number years ago then forgotten (this was the case with myself)
4) Users often have many websites using their Gmail account for password-reset workflows (this is definitely the case with myself)
You are correct this is a significant usability issue. Personally I use Authy, which performs automatic encrypted backups in case I lose my phone. I then have to remember my Authy recovery password (not stored in my password manager, which is itself secured with a TOTP - hello circular dependency danger!) I also keep a yubikey authorized with all my accounts as second backup.
All these things are beyond what the average person wants to worry about, as you say, but HN readers will find it simple. Personally I'm hoping U2F (Yubikeys) are the future, since your average person certainly understands the concept of a key.