Stopping spam sent by botnets effectively: combining Passive OS Fingerprinting, Greylisting and Spamassassin

At work, it is of my tasks to make sure we keep the mailboxes of our clients free of spam. Some weeks ago the number of spam mails went up massively and we worked hard to update the filters to keep unwanted mails out. In this blog I’ll describe a few of the things we did.

Using the famous SpamAssassin tool it is possible to score e-mails. One can score on the contents of subject, body, headers etcetera. A lot of good rules are already supplied and it’s possible to write your own. When a new spam run comes in, we used to create new rules for the spam mails that slipped trough. That works, and afterwards the mails are tagged as spam.

As you can imagine, this procedure is both time consuming and a bit late: only after we see mails slipping through we can create rules to catch them. Of course this procedure will always be some sort of a last resort if all else fails but I wanted to setup something more proactive.

To start from the beginning, how is all this spam sent?
It can’t be send from one or a few locations only, because then it’d be easy to block. Instead, most spam is sent by botnets these days. Botnets usually have hundreds of thousands pc’s under control and one of the main things they do is sending spam. For example to advertise online casino’s, fake banking sites or other scams. Because there are so many infected PC’s, it’s not easy to block them all. Or is it?

When thinking about this, I realized most (if not all) members of these botnets are infected Windows pc’s. Also, these mails are often directly sent from the PC to the final destination mailserver (instead of using the SMTP server of their ISP).

If we could detect the OS of the client that connects to our mailserver, we could then apply certain actions based on the OS. The idea here is that most ISP’s use Linux, Unix or Mac servers. And if they are using Windows, it is likely to be some Server version instead of ‘Windows Vista’, ‘Windows XP’, etcetera. Interesting!

What we want to do here is known as Passive OS Fingerprinting. A tool that implements this is for example P0f. You run P0f as a deamon on the mailserver that accepts the incoming connections. Based on the traffic that flows by, P0f is able to guess the OS of the client that connects. It is passive, so the client never knows we’re doing this. Nothing is in between the client and the mailserver, P0f is just analyzing the traffic. Now that P0f knows the OS of the client, we can decide what to do with this information. In our setup it works like this:

1. When the OS is Windows, but not Windows Server, activate Greylisting. When another or unknown OS is detected, start mail delivery immediately;

The idea behind this is that mail sent from infected Windows PC’s is usually poorly written. They cannot handle the fact the mailserver sends a 400 series temporary error message and most give up after just one attempt. This technique is called Greylisting and it alone reduces the number of spam mails significantly. But, Greylisting has drawbacks as well. The biggest drawback in my opinion is that it can delay mail up to 30 minutes or more. Most customers we serve find this annoying.

2. At the time the connection is accepted and the mail is delivered, we set a ‘X-P0f-OS:’-header with the detected OS;

Combining Greylisting and P0f creates a more ideal solution: Windows PC’s should not send mail directly to the recipients mailserver, but use the Provider SMTP server instead. One could say that when such a PC is sending mails directly, it is at least suspicious. That is, in my opinion, enough reason to Greylist them. There must be some spam software that understands Greylisting (now or in the future) and that will eventually connect again after some time and deliver the mail. That’s why there is an action #3:

3. This header is consulted later on in the delivery process, and when Windows appears (again, not Windows Server), Spamassassin assigns some points to the spam-score.

Because mail that is sent directly from a Windows PC is suspicious to me. The OS score helps reaching the score needed for Spamassassin to tag it. The interesting thing is that this is proactive: you just don’t know what new mail spammers will send, but what you do know is that the next mail is probably send by an infected Windows PC.

This setup is now up and running, so I’ll let you all know what my experiences are after some time. When I find the time, I will also write some blogs in more detail on how to setup such a system.

Any other methods you use to stop spam effectively?