Up to you, but I'd say let that kind of thing slide. The ideal is that if someone disagrees, they'll explain themselves. Downvoting is supposed to be a mechanism for voting on whether a point adds to the discussion or not.
So either (a) your point did not add to the discussion, in which case further discussion about your point just adds to the noise without contributing signal, or (b) the person or persons downvoting disagree with you but are too lazy to contribute to the discussion by explaining their perspective. In which case, why worry about them?
Responses--whether positive or negative--are discussion gold. Downvotes aren't worth the worry.
