This system by itself can be frustrating... as the problem is that they tend to challenge-response everyone.
I would love to see a system that does automatic spam classification (like gmail), but if you get auto-classified as spam, then you get the challenge-response, as a last ditch effort to get through the spam filter.
That would make it too easy to game the spam system, because now the spammer knows they've been classified as spam, and can keep changing the message until it gets through.
This is why it's generally bad practice to reply to an email telling the sender it is spam.
Well, possibly, though spam is a pretty high volume game, and my guess is it's not worth the time for the mass spammers to analyze a few rejects (even if those rejects even do go to their actual address)
It depends on what you prioritize. I prioritize avoiding false-positives, so my spam filter of choice reflects that.
My current spam system (spamstopshere.com) does return bounce/reject messages depending on what filter the email triggers.
You're wrong. Spammers actively try to monitor their success rates and game the filters.
As for the original idea, backscatter is to be avoided at all costs. Nobody likes bounce messages and so forth for mail they never sent. Much better to reject mail at SMTP time, than generate backscatter to the alleged envelope sender after the fact.
Graylisting is more or less this, but instead of a human receiving a challenge, the outbound server receives a "please try again later" (which is enough to stop a lot of spam already).
I would love to see a system that does automatic spam classification (like gmail), but if you get auto-classified as spam, then you get the challenge-response, as a last ditch effort to get through the spam filter.