Labels, and words in general, while making it possible to communicate concepts to other people, also inherently make gross generalizations of those concepts, which confuses communication. Labels are therefore also used to “label” people, which is considered a bad thing. Therefore, some people really don't like labels.
http://egscomics.com/comic/2015-05-23