The Theory
Good mail comes from the senders you know — your friends, your family, the publisher of an email newsletter you have subscribed to. For all other mail, it's safe to assume that it is spam.
Accordingly, challenge/response filters do not try to filter out the spam but look for mail from trusted senders (senders on your so-called "white list") and let it through. Everything else is thought to be spam and quarantined. This makes for a fantastic spam detection rate.
But what about the occasional message from somebody not yet on your white list who is not a spammer? What about the mail from an old friend, from a newsletter you have just signed up for, what about if somebody changes their email address?
The Challenge
Challenge/response filters take care of these situations automatically, too. Every new sender is mailed a challenge. If they respond to the challenge, the sender is automatically put on your white list, the original message is recovered and you can communicate with the now trusted sender unhampered.
Usually, the challenge consists in solving a captcha. Captchas are tasks that are trivial for humans but highly complex and expensive to solve for computers.
Since spammers deliver their emails to millions of email addresses, they can't solve all the captchas "by hand", but they can't have computers answer them either. Senders of legit mail can respond to the challenge easily, however. Thus, their mail gets through while the spam is trashed.
A Set of Serious Problems
Not only does this sound simple enough and sufficiently elegant, it also works reasonably well. Yet, challenge/response systems of filtering spam also introduce their own and very serious set of problem.
