Babies are born with the ability to distinguish between any two frequencies that are important for a phonetic distinction. Within the first year, they begin to lose that ability and only retain frequency distinctions necessary for whatever language they're exposed to. So, when an adult can't tell the difference between two phonemes or has difficulty articulating it, it's because they weren't exposed to it early enough. So, the difficulty you describe would apply to any lingua franca. In the bullshit, completely impossible scenario where everyone in the world speaks one language, this wouldn't be a problem because there would only be one set of phonemes.
