The ramification being that some people my choose to understand the world differently from you. The answer to that worry is to make compelling points in your favour, or maybe try to understand the other side of the argument. Freedom comes at the expense of security, and security in this instance is overstepping its boundaries.
But that seems like an entirely different argument than your idea of "free speech". Related, but not the same.