If you don't need to ask permission to spend your money, then neither does anyone else need permission to spend your money.
Fraud protection (and protection against the bank itself making mistakes) may not be perfect, and will vary by jurisdiction, but it's more than nothing.
Yes, they can be stolen. I trust myself more to hold my keys than I trust the bank to always allow me to spend my money, though. I've never lost my keys, but I have had my bank not allow me to spend my money.