I'd say your comment was rightly downvoted because it was inflammatory, unsubstantive, and off topic. One needn't disagree with your views to see how that breaks the site guidelines.
Even when I'm not "inflammatory", I often get downvoted when I express viewpoints which are not exactly popular here. OTOH, I see comments, sometimes in reply to me, which are pure trolling, but nothing happens.
It's clear as day that the downvote is not used merely for moderating, many people who have the right use it to express disagreement. Certainly, nothing can stop them, can it?
