Greylisting and its derivative Techniques

We currently believe that Greylisting (and its derivatives) together with SPF are the most effective techniques to fight the ever rising tide of SPAM.

Please take the time from a busy day to read this explanation carefully. Some of its consequences will affect you directly - it is important you understand their impact.

It is estimated that over 15 billion SPAM messages are sent every day. Some days it used to feel like they all arrived in our mailboxes.

As the volume of spam rises the anti-spam tools and content filters are becoming increasingly aggressive such that the number of false positives is becoming perilously high. We probably all know of at least one incident where our genuine email either got stuck in a spam folder or was probably discarded completely.

Classic Solutions

The problem in fighting spam is finding a cure that is not worse than the disease.

We have reviewed and rejected some potential solutions:

1. Black lists: We refuse to implement a Black List because we feel it can too easily penalise legitimate mail while doing very little to stop SPAM - your SPAM clogged mailboxes are witness to the total lack of effectiveness of Black lists. Having been the unwitting victim of a blacklisting which took less than 2 hours to fix when brought to our notice but took two years for all the effects to finally disappear we feel the implementation on average is not production quality. On its own it is a fatally flawed technique. In combination with other techniques it can add value.

2. Incoming Mail SPAM Filters: It is not up to us, nor should it be, to decide what constitutes SPAM and what does not. One person's legitimate mail may be another persons SPAM and vice versa. While not doing anything to demean the quality of spam-filtering software, the technology relies on inspection of the mail content. This a very subjective matter and will inevitably lead to false positives which is why most such systems place suspected spam in a special folder. You still have to check this material - much of it profoundly offensive. How effective is that.

So now let's get positive and look at what can be done.

Hurt the Bad Guys

The economics of SPAM are at best marginal. So any attempt to make those economics worse is bound to work against spammers. This seemingly trivial insight has profound consequences in fighting spam and had led to a whole new battery of techniques including Greylisting (credited to Evan Harris in this 2003 article).

In practical terms this insight means doing anything which causes the use of additional resources will disproportionately affect spammers. And have the happy side-effect of reducing their capacity to send out any spam. Greylisting was the first of a family of techniques that have the following broad characteristics:

1. Cause more effort to be expended by the spammer.
2. Require tighter compliance with the specifications.
3. Limit the rate that spammers can send email.
4. Take more time to send replies when it is from a known - or even suspected - spam-source. The so called tar-pit techniques.

Grey Listing

Grey listing is currently the most highly developed, and resource light, of these techniques and is implemented on our server (using postgrey) where it has had a dramatic effect. Currently over 90% of the SPAM load has gone. Period.

Greylisting sounds absolutely terrifying at first glance and works like this:

1. Every time the mail server sees an email it constructs a unique triplet consisting of the senders email address, the recipients email address and the sending mail servers source IP address. If the mail server has never seen this triplet before it stores the information in a database - and then discards the email with a temporary failure message. Yes. It throws the email away, without looking at its content, and will not allow it to be retransmitted for a small period of time (a blackout period) that is normally determined by the software implementor or the mail server operator.

2. The mail RFC's specify that compliant mail servers MUST retry under these conditions. Legitimate mail servers will retry, normally in 5 to 15 minutes, automatically - the sender of the email is not involved with the process at all and sees no impact. Spammers may also retry but typically do so immediately and get caught in the blackout period. In any case spammers have no real incentive to retry because it consumes more resources. A marginal business just got more marginal.

3. Once the re-tried mail has been received the mail server marks the mail source as valid and will not throw away anymore email for a period of time - defined by the email operators policy. The whole process is self-regulating.

4. It all sounds too good to be true. And unfortunately it is. Problems can arise for legitimate email in two areas:

   • Some mail servers can take a long time (multiple hours) to retransmit - even though, with normal mail servers, this will typically be 5 to 15 minutes.

   • Some mail servers use a unique sending address with each retry - thus defeating the triplet mechanism.

5. There are a variety of implementation techniques that can both ameliorate the initial delays and solve the problems identified above.

   Whitelists can be built to bypass checks from known good sources or domain names.

   Many greylist implementations allow operators to set policies that will permanently whitelist senders after receipt of a number of emails.

   The number of servers that have very long retries or use unique sending addressing is gradually being discovered and global whitelists are emerging. However since spammers could just use faked addresses from a whitelisted source it is important this technique is used in conjuction with SPF which can then catch this abuse. A classic 1-2 punch. No third strike required here.

Anti-spam is increasingly becoming not a single technique but rather a battery of techniques. Serious work is being done in the area of email authentication (DKIM from Yahoo and others) and other techniques are at the idea stage. Sure the spammers will fight back, but if the problem can be made manageable then we have made progress. Maybe.

---

Problems, comments, suggestions, corrections (including broken links) or something to add? Please take the time from a busy life to 'mail us' (at top of screen), the webmaster (below) or info-support at zytrax. You will have a warm inner glow for the rest of the day.