I think simply adding "Agree" and "Disagree" buttons and separate "off topic" or "downvote" button would make a big difference. What "up" and "down" means is a bit vague and everyone has a different opinion on the matter. I will readily upvote comments I agree with but I usually only downvote comments that add no value to the discussion rather than being something I disagree with. So even my own voting pattern on HN is asymmetric.
