I agree, I see "anti-trans" and accusations of being "transphobic" typically used as a silencing tactic, to stymie and derail discussion.
But these issues really do need to be discussed, as there is real-world harm arising from these medical interventions, and from the non-consensual redrawing of the boundaries around single-sex spaces.
I think you're spot on with your comparison to attitudes towards FGM, it really does highlight the double standard.