I moved from Google Apps to FastMail for six months or so. I'm back to Google Apps now. For me, I had a quite opposite experience...FastMail searching was much slower, and the return to folders instead of labels made organizing my mail more difficult. Additionally, GMail's Inbox organizing (slicing in Primary, Social, Promotions, etc) is invaluable when you receive many, many messages a day. Fastmail's interface, along with the increased spam messages which got through, simply didn't provide me with a good user experience and meant I was more likely to miss an email.
The final straw for me was finding a solid android experience. Specifically, a client that didn't eat my phone battery whilst still providing the near instant notifications I've come to expect from Gmail. When using fastmail via imap in the 'gmail' app it didn't seem to get any push notifications and the polling was either too slow or too battery draining depending on the time interval configured. When I tried fastmail's webapp thing I found the searching and offline experience lacking. Maybe it's moved on. All the (free) 3rd party android imap clients I tried were horrible to use.
So I also came crawling back to Gmail. Which was a shame as fastmail's calendar is so much better, almost worth it alone.
YMMV.