You have a solid intuition, in addition to the usual efficiency losses, a large amount (20%-50% would be usual, up to 80% in some cases) of the energy released during annihilation is in the form of... neutrinos. I don't foresee a realistic means of harnessing that energy any time soon, even where "soon" indicates quite a stretch of time.
