Ten years ago you were an elite web developer if your site worked well in browsers other than IE. Today you're an elite web developer if you can make your site work well even in IE.
Getting the vast majority of sites working in IE9 is trivial (no changes required). IE8 some changes, but not a major hassle. I don't support IE7 or before. If you use a lot of HTML5 features it is hit or miss on all browsers.
You're right. My recent encounters with IE9 have been pleasantly surprising (and I bet IE10 will be quite competitive), but for us the numbers are so low that, at least for now, a broad sweep feels OK.
