Perhaps Google could introduce an "I'm Feeling Desperate" button for the times you just know your result lies within one of your previously blocked sites.
I've found answers there, too, but I've yet to encounter a situation where they're the only place an answer is found. Now that StackOverflow is around I suspect that's even less likely.
My problem usually, contrasting other comments here, is that I don't look at the URL of the link I'm clicking on but primarily at the title. Since a lot of questions on EE are very specific, I often click by accident since the question sounds a lot like what I'm looking to find an answer for, only to grunt at having to scroll all the down ("Ok, I'll give them one more try..") where no answer usually is found.