I have no problem with the possibility that men and women are different. What's unjustified is, given the powerful effect of culture on behaviour, someone making offhand claims that they know what differences are natural.
I read "natural difference between men and women that are implicitly understood by people" as saying that what's "implicitly" (i.e. commonly) understood as to be the natural differences is correct. Because what has been "implicitly" understood has differed substantially across time and cultures, that's a highly suspect claim that requires a matching degree of evidence.
Furthermore, just the long and loud history of people spouting off absolute bullshit about what women are and aren't is, by itself, enough to demand significant evidence for any positive statements of what's natural or not.
Strong claims require strong evidence. I did not make any outrageous claim.
It would be incredibly surprising if men and women were identical in every aspect except for their reproductive body parts
Just consider the difference in selection pressure that men and women face.