Phones are pretty consistent, but I think the biggest piece of the puzzle is that different languages don't care about all the differences. A phone exists in IPA if a language cares about the difference.
For example, if you pronounce "sju sjösjuka sjömän" with [ʃ] instead of [ɧ], I'll understand you just fine, but I'll instantly know that you aren't a native Swedish speaker. (Or, you are, and you're speaking a Swedish dialect that has replaced [ɧ] with [ʃ], because of course that's also a thing)
But if you mis-pronounce "skön" (nice) as [ʃøːn] instead of [ɧøːn], you're getting awfully close to [ɕøːn], which is how you pronounce "kön" in Swedish, which means gender. So as a Swedish speaker, the phoneme has changed, I no longer know if you're saying "skön" or "kön", or maybe "schön" in German? Swedish "cares" about this difference. But you, as an English speaker, might not be able to hear the difference.
And if there was a hypothetical language that had even further subdivisions of these sounds, speakers of that language would be upset that neither you nor me could tell the difference between two sounds that speakers of that language care about.
So the mapping of phones to phonemes is highly language (and dialect) dependent, and you should think about it in terms of continuous ranges or tolerances, instead of a discrete 1:n mapping.
