That, and I often have multiple projects open at once. There's a note in the Magit manual about this and how it affects performance, but unfortunately, the workaround means that buffers don't get refreshed and you might see stale information unless you perform a manual refresh.
I only work on Emacs in small projects though... so maybe that's why it's fast for me?!