Postfix, dovecot, bogofilter and dovecot antispam

Updated guide to getting seamless server side spam filtering running on Ubuntu

Background

When I started using my mobile phone for email I quickly hit a problem - I needed server side spam filtering that could be trained on the go to keep email volumes reasonable.  The solution described here provides just that: spam is automatically filtered out and marked as read on the server.  Moving email from my inbox to spam folder or vice versa trains the spam filter automatically.

History

I originally solved the problem of per-user baysean spam filtering using dspam, writing up a short guide back in 2011.  Time has marched on and dspam is no longer supported on Ubuntu, meaning that it's time to move to another solution.

This document describes setting up Postfix and Dovecot to integrate into Bogofilter through the Dovecot Antispam plugin.  This arrangement means:

1. Only users that are configured to use Bogofilter have it running.
2. Moving emails in / out of the Junk folder triggers re-training of Bogofilter.
3. Spam filtering happens server side, making email workable on mobile devices.

The four step process

Step 1: Get postfix and dovecot working

There are lots of good tutorials on the web on how to do this part, so I'll not attempt to replicate them here.  Do make sure that inbound and outbound email are fully working before attempting the rest of the steps, and make sure you have a backup of your working configuration!

Step 2: Configure postfix to identify spam with bogofilter

Install bogofilter and postfix-pcre with apt-get install bogofilter postfix-pcre.

Edit /etc/postfix/master.cf and add a new configuration line for bogofilter:

bogofilter     unix  -       n       n       -       10      pipe
  flags=Ru user=vmail argv=/usr/local/bin/bogofilter-scan.sh /var/spool/bogofilter/${recipient} ${sender} ${recipient}

Some explanation:

• user=vmail means that bogofilter will run as the vmail user.  In my setup this is the unix user handling all mail and that owns the mailboxes.
• /var/spool/bogofilter/$recipient is the per-user database for Bogofilter.  If the directory doesn't exist, Bogofilter is not used for that user.

Setup a filter that will trigger bogofilter.  To do this create a new file /etc/postfix/bogofilter_access:

/./   FILTER bogofilter:bogofilter

Now configure postfix to use this filter and so pass mail to bogofilter.  Edit /etc/postfix/main.cf:

bogofilter_destination_recipient_limit = 1
smtpd_client_restrictions = permit_sasl_authenticated reject_rbl_client zen.spamhaus.org check_client_access pcre:/etc/postfix/bogofilter_access

Some explanation:

• bogofilter_destination_recipient_limit = 1 passes emails one at a time to bogofilter.
• permit_sasl_authenticated means that any authenticated user will be allowed to send email through this server, and will not pass outbound email through the spam filter.  You must have some mechanism that means outbound email is not filtered for spam.
• pcre:/etc/postfix/bogofilter_access should be last so that bogofilter gets a look at the mail before delivering it.

Add the script that runs Bogofilter for mail delivery - /usr/local/bin/bogofilter-scan.sh:

#!/bin/bash
#Arg 1 - directory of the user's Bogofilter DB
#Arg 2 - sender
#Arg 3 - Recipient
if [ ! -d $1 ]; then
  tee | /usr/sbin/sendmail -i -f $2 $3
  exit 0;
fi
/usr/bin/bogofilter -e -u -p -l -d "$1" | /usr/sbin/sendmail -i -f $2 $3
exit 0

Remember to make the script executable (chmod +x) then test.  If everything is working correctly you should now have an  X-Bogosity header in incoming email.

Step 3: Configure postfix and dovecot to send spam to the Spam folder

Now that postfix and bogofilter can identify and flag an email as spam, I want to move it to a dedicated Spam folder per user.  To do this I can use the sieve functionality in dovecot's LDA (Local Delivery Agent).

To kick things off we need to tell postfix how to deliver email through dovecot.  Add the following to master.cf:

# Dovecot local delivery agent - allows us to use sieve filters for spam
dovecot   unix  -       n       n       -       -       pipe
  flags=DRhu user=vmail:vmail argv=/usr/lib/dovecot/deliver -f ${sender} -d ${recipient}

Note that I'm using the vmail user again so that all mail files are kept under a common ownership.

To make postfix use the dovecot delivery agent add the following to main.cf:

# Dovecot for local delivery
dovecot_destination_recipient_limit = 1
virtual_transport = dovecot

These two changes gets email as far as dovecot, now we need to configure dovecot to look for the spam headers and redirect it to the Spam folder if present:

In /etc/dovecot/conf.d/15-lda.conf enable the sieve plugin for the LDA:

protocol lda {
  ...
  mail_plugins = sieve
  ...

In /etc/dovecot/conf.d/90-plugin.conf configure a location for storing the global sieve filter files:

plugin {
  ...
  sieve_before = /etc/sieve/conf.d/before
  sieve_after = /etc/sieve/conf.d/after
  ...

Now we need to create an appropriate filter that will check for the spam headers.  I did this in /etc/sieve/conf.d/before/spam-folder.sieve (found here):

require ["regex", "fileinto", "imap4flags"];

# Catch mail tagged as Spam, except Spam retrained and delivered to the mailbox
if header :regex "X-Bogosity" "^(Spam)" {

  # Mark as read
  setflag "\\Seen";

  # Move into the Junk folder
  fileinto "Spam";

  # Stop processing here
  stop;
}

The final step is to compile the script ready for dovecot to use (discovered here):

sievec /etc/sieve/conf.d/before/

To test this part of the configuration out ensure that the user has an IMAP folder called "Spam" at the top level of their account.  I'm not sure what happens if this doesn't exist, but I doubt it works.

Step 4: Configure dovecot_antispam to manage learning spam / ham

I've tested this setp on Ubuntu 16.04 with apt-get install dovecot-antispam.  Earlier versions of Ubuntu were problematic (see the old guide from 2011).

Start by creating the script to run when moving email from spam to ham or vice versa - /usr/local/bin/bogofilter-mark-mail.sh:

#!/bin/bash
#Arg 1 - directory of the user's Bogofilter DB
#Arg 2 - spam or ham
if [ ! -d "$1" ]; then
  exit 0;
fi

if [ "$2" == "ham" ]; then
  /usr/bin/bogofilter -e -p -l -Sn -d "$1" 
  exit 0;
fi;

if [ "$2" == "spam" ]; then
  /usr/bin/bogofilter -e -p -l -Ns -d "$1" 
  exit 0;
fi;

Remember to make the script executable (chmod +x).

Enable the antispam plugin in /etc/dovecot/conf.d/20-imap.conf:

protocol imap {
  ...
  mail_plugins = antispam
  ...

And then configure it in /etc/dovecot/conf.d/90-plugin.conf

plugin {
  ...
  antispam_backend = pipe
  antispam_debug_target = syslog
  antispam_signature = X-Bogosity
  antispam_signature_missing = move
  antispam_spam = Spam
  antispam_trash = trash;Trash;Deleted Items; Deleted Messages
  antispam_verbose_debug = 1

  antispam_pipe_program = /usr/local/bin/bogofilter-mark-mail.sh
  antispam_pipe_program_args = /var/spool/bogofilter/%u
  antispam_pipe_program_spam_arg = spam
  antispam_pipe_program_notspam_arg = ham
  antispam_pipe_tmpdir = /tmp
  ...

Now moving an email from the user's inbox to the Spam folder will trigger bogofilter to learn this email as spam, and vice versa.

Migration

Migrating from dspam to bogofilter means giving up all of the existing learning.  To initialise the bogofilter with my existing server side mail I ran some bulk imports:

bogofilter -s -d /var/spool/bogofilter/colin@owlfish.com -B /var/spool/vmail/owlfish.com/colin/.Spam
bogofilter -n -d /var/spool/bogofilter/colin@owlfish.com -B /var/spool/vmail/owlfish.com/colin

Conclusion

I've been using dspam to do server side spam filtering for 6 years, making managing email much more pleasant.  I need to see how well Bogofilter works in its place, but I'm hoping for similarly good results.

Please let me know if this page is of any help - and do send corrections and edits!