If you're the one being accused, you have to worry about the accusation coloring other people's judgement of you. Most people really aren't logical creatures. People tend to not remember the source or veracity of things they know. So even if you're innocent, others may automatically assume you're a jerk.
