Votes are a far bigger problem than flagging because it takes a pavlovian approach to what ideas are allowed. Flagging is important to remove spam and wholly inappropriate content. And if you revoke the ability of users who flag content based on a personal disagreement, that user simply loses that ability.
I suggested to add the ability of users to have their view of HN unaffected by these flags. Without that, HN simply has no ability to defend itself against abuse of users who take it upon themselves to play censor.