You would also need to enter your username or email address, though, and that's hard to do over the phone... I guess voice-to-text is improving, so that might be an option.
You don't have to actually enter your email address, you just have to prove that you know it - and telephones have the full alphanumeric set of characters on the keypad. Use a * or # for special characters, and something like "pavel@lishin.org" could be entered as "72835#547446#674".
Edit: I guess that may not be sufficient to identify you, but it could verify that you are the account holder for other services.
My cell phone (Pantech 5000) has a qwerty keypad. It doesn't have the old rotary phone style letter-number association. I know many people whose phones don't have this.