I don't wear a helmet and I ride in downtown Toronto. Helmets save lives, but they also make motorist treat you differently. Also, they don't really save that many lives. Cycling without a helmet is safer than walking (kilometer for kilometer) and safer than being in the passenger seat while a new driver is driving.
Not only yours, in my too. That's why I always try to make eye contact with drivers, and when that doesn't work yelling always does the trick. Another thing that improved my safety was installing lights on my road bikes.