There are actual fascists (and not as few as I would like) and they need to be called out, but using the term inaccurately and provocatively on a broad group makes it easier to oppose the usage outright. Optics are important to politics, like it or not.
