Ah. Fair enough. These people are more along the lines of: "I'm not racist, I have black employees". "I'm not a homophobe, we have a rainbow sidewalk". Meanwhile they exploit and are shitty to everyone not in their group. And all the "locals" are conveniently white while the visible minority transplants never have "local" status.
People in that position tend to try and protect that position through whatever means are available to them, and it wouldn't surprise me that there would be a racial disparity intentional or not. The more isolated and historically homogeneous you get, the more this probably occurs. You probably don't get too many racist people growing up in more heterogeneous communities. I get creepy vibes from most smaller towns anyway, and this is just another facet of that I'd imagine.