I switched to fastmail from gmail a few months ago and I'm incredibly happy with it. I even got to keep my *@mydomain wildcard addresses, and the fastmail calendar implementation is not half bad.
Were you using Google apps for your domain (or whatever it’s called now)? And if, were you able to maintain access to your Google docs and other Google-based services (play, etc)?
I was not. I switched from an @gmail.com address. However, I think the real answer to your question is that your email address is your ID for Google services, whether its MX records point to Google or not.
