You can 'flag' stories with 500 karma, which is basically a super-downvote in the sense that a flag suggests the story doesn't belong on HN at all. My sense is that it is a much stronger signal than a comment downvote.
I am not complaining. I don't trust myself with the responsibility anyway but I have 500+ and I cannot down vote stories.
I used to avoid voting on comments on my phone for a while because I was afraid I'd end up downing with my fat fingers but they've fixed it with an undown link when I down vote. Maybe they've relieved me from the responsibility of down voting stories?
I don't know how I feel about down voting because there is always a desire to down vote dissenting opinion.