One of the problems with adjusting inter-letter spacing is that it completely breaks certain types of font: think of calligraphic scripts, or non-western writing systems such as Devanagari where the rendered glyphs touch to give the continuous appearance of the text.
Another problem is ligatures. Where adjusting inter-letter spacing is possible, it usually needs to be applied uniformly across at least a whole word to work well. If you are writing the word "difficult", and the "ffi" is rendered using a ligature to avoid awkward clashes, then how do you adjust the spacing for that? You could keep the ligature, but then the inter-letter spacing isn't uniform across the whole word. Alternatively, you could break the ligature, but that could restore the undesirable clashes between the "f", "f" and "i" that the ligature was designed to prevent. To avoid that, you would have to open up the inter-letter spacing so much that the letters no longer clash, which is probably far too obvious in most fonts to be a useful effect.
From what I read, the latter. Like you said, overdoing it quickly makes a text look bad, and it also ruins the (presumably) carefully created kerning tables that are supplied by the font. I assume you have read Hàn Thế Thành's paper on microtypographic extensions to TeX (http://www.pragma-ade.com/pdftex/thesis.pdf)?
I’m not sure too many people especially object to adjustments of 2-3% to inter-letter space or letter widths.