(I work for a company that makes a password manager that have this feature too)
I used to think that but I changed my mind.
First, you can set TOTP (or other second factors) authentication on your password manager account, which I think is good philosophically at least, because you gotta have access to your second factor to get access to you website TOTP.
Secondly, using a password manager with strong unique passwords that you don't know brings already a lot of benefits that pushes websites and administrator to push using a second factor (it's very often a way to avoid attacks using reused or bad passwords).
You do lose a bit of security (there is now a risk that your TOTP seed get stollen), but the extra convenience (especially when you lose your TOTP device) means you can enable it on more websites without too much annoyances.
I used to think that but I changed my mind.
First, you can set TOTP (or other second factors) authentication on your password manager account, which I think is good philosophically at least, because you gotta have access to your second factor to get access to you website TOTP.
Secondly, using a password manager with strong unique passwords that you don't know brings already a lot of benefits that pushes websites and administrator to push using a second factor (it's very often a way to avoid attacks using reused or bad passwords).
You do lose a bit of security (there is now a risk that your TOTP seed get stollen), but the extra convenience (especially when you lose your TOTP device) means you can enable it on more websites without too much annoyances.