The "penis and vagina" characterization is too reductive. What's at issue is that males are generally bigger, stronger, and more sexually aggressive than females. These factors in aggregate create increased risk when forcing proximity between men and women while in vulnerable contexts. But none of this needs extra demonstration, this is all common knowledge.
hrt is effectively chemical castration by another name. one of the primary effects of estrogen and anti androgens is erectile dysfunction combined with the shrinking & atrophy of the genitals along with a typically highly diminished sex drive.
on balance, this issue, just like sports, is incredibly optically poor for trans people regardless of the facts at hand. the suggestion that mtf trans create an elevated risk for rapes and assaults is a deeply unfair cultural bias that is probably not going away for at least a generation. similarly putting mtf trans into the male prison system seems likely to subject them to incredibly high risk. there are other nations & cultures that have had a legally recognized 3rd gender for much longer than trans rights have been an area of focus in the west, i imagine we could learn a great deal from them on these issues in particular.
Not all trans-identifying people use HRT. Indeed some deliberately make no bodily interventions whatsoever - see e.g. Alex Drummond, a 'trans woman' with beard, moustache, and his intact male body.
Third gender in the cultures you allude to is typically just a way of othering gay man, of the "you can't be a real man" variety. It's nothing positive to emulate.
