What about sunglasses? Why doesn't that feel threatening when you see someone slip on a pair of sunglasses, and a hat? After all sunglasses are a form of a mask for the upper part of the face.
Wearing sunglasses is often viewed as impolite too. I am not arguing against masks I am just saying that there are some cultural obstacles that make people uncomfortable wearing them. It would take some leadership to get around those.