It's completely harmless and it seems like a nice way to acknowledgement that the comment was appreciated. How is that possibly a bad thing? -4 means that 5 people decided that that comment was offensive to them. The only thing offensive about it is the lack of an apostrophe. Like, the person is just being nice. That they're at -4 for acknowledging their appreciation for another user's joke (5 people disliked this?!), and I'm only at 0 for an obviously negative, off-topic message where I call everyone heartless bastards (only 1 person disliked this) is absurd.
No, minus four means that five people decided that the comment did not contribute. To indicate that you appreciate a post on this site, you upvote the post.
