If someone says racist[1] things (from whichever POV), it’s fair to call out racism. That’s not the crux of the question. Moreover there are other forms of obnoxious speech in addition to racism. Someone could be a vegan and another a omnivore and the both could be offensive. Or it could be about reproductive rights and so on. People should be able to engage in debate even if it’s repugnant to you or to me or anyone.
[1] meaning actual racist things not the colloquial “racist” which can mean someone disagreeing on policy among other things.