- the password you store on the phone (in cleartext) is not the real account password, it's a string for this device only and you can revoke access at any time
- Someone should not be able to use your phone (via call/text message) as unlock device, unless you lost it
- In that case you should lock the SIM for for a multitude of reasons anyway - and you'll get a new SIM that you can use to recover your account
I think the first one is the most important though: You just don't have to store your real password.