Most firefox UI code is lazy loaded, and the pocket extension was a pretty simple thing that hit an endpoint. I doubt you could speed things up by removing it.
of course you speed things up, it takes a bit of time to go through the process of removing the icon and disabling it on a fresh install. when you have several install to do, it quickly adds up.