How often are the cashiers checking the back of the card? Many of the card readers I see are on the customers' side of the counter, and I swipe it myself, often because I'll enter a pin. Also there are self-checkout kiosks with little or no supervision. I don't think that the lack of a signature line is the _biggest_ downside.
