path: root/contrib/multidrop

From mlievaart@orion.nl  Mon Jan 10 10:46:33 2000
From: Martijn Lievaart <mlievaart@orion.nl>
To: Eric S. Raymond <esr@thyrsus.com>
Date: zondag 9 januari 2000 0:38
Subject: Re: Thanks for fetchmail and a solution to the multidrop problem (I
Status: O
Content-Length: 8086
Lines: 226

think)

Hello Eric,

Let me first state that I'm no sendmail nor unix guru, so although this
seems to work, I certainly would not say this is the "best" solution. In
fact I would welcome all comments to make this better. In particular, it
seems that that the mailertable feature was made just for this, but I'm
still studying that.

Also, This mail will have lines wrapped. I will put up this on a website
asap, so people can download the relevant portions. In the meantime, I'm
using (stuck on) Outlook, so I won't even attempt to format this mail.
Accept my apoligies and try to mentally reconnect the lines.

Finally, this mail is a bit lengthy, but I guess it is better to get all
information in, so please bear with me.

After some very frustrating attempts to get multidrop to work reliably, it
suddenly hit me. When sendmail has translated the recipient to the mailbox,
the recipient is gone (in the cases we're talking about). So the solution is
not to let sendmail do this translation (completely).

The trick is to let a custom MDA be called with both the mailbox and the
full recipient name. This MDA then just stuffs it in the correct mailbox
after adding the appropriate headers. Luckily I hit on the formail utility.
It reformats a mailmessage and does just what I wanted. Specifically my
script uses it to:
- add a custom header (default: "Delivered-To:") with the recipient
- rewrite the message-ID, so fetchmail will download the same message
multiple times.
- add another header, just for fun.

The rewriting of the message-ID is needed because fetchmail will suppress
multiple messages with the same ID, normally a good idea, but now it gets in
the way. A switch on fetchmail to suppress this behaviour would be great.

At first I hardcoded the domains in the sendmail.cf, but I quickly set out
to do one better and came up with the following solution. In sendmail.cf,
add the following line somewhere at the top.

Kmultidroptable hash -o /etc/mail/multidroptable

this defines a table for all domains we want to use multidrop for. The
format of this file is multiple lines of the format:
<domain>    <mailbox>

e.g:
mailtest.orion.nl       mailtest
mailtest2.orion.nl      mailtest
mailtest3.orion.nl      mailtest
bvh-communicatie.nl     b.bvh
krakatau.nl             b.bvh
personeelzaak.nl        b.bvh
maslowassociates.nl     b.bvh
rtij.nl                 rtij

Of course, create a .db file with makemap. Also, the domains must be added
to class w, so they should be added to your sendmail.cw or RelayTo file, or
whatever you use.

Now add to sendmail.cf:

R$+ < @ $* . >                          $: <MULTIDROP> $(multidroptable $2
$: <NO> $) <?> $1 < @ $2 . >
R<MULTIDROP> <NO> <?> $*                $: $1
R<MULTIDROP> $+ <?> $+ < @ $* . >       $#drop $@ $2 @ $3 $: $1

These lines should be above the existing lines that read:

# short circuit local delivery so forwarded email works
R$=L < @ $=w . >        $#local $: @ $1         special local names
R$+ < @ $=w . >         $#local $: $1                   regular local name

This works as follows (in fact these comments are above my modification in
our sendmail.cf).
#
# MLI. Any drop host gets passed to the drop script
#
# The first rule looks up the domain in the multidrop table.
# The input at this point is always:
#       user@<dom.ain.>
#  If found, the resulting line looks like this:
#       <MULTIDROP> mailbox <?> user@<dom.ain.>
# if not found, the resulting line will be:
#       <MULTIDROP> <NO> <?> user@<dom.ain.>
# The second line restores the "not found" case back to user@<dom.ain.>
# So if this domain was found in the multidroptable, we still have a line
starting with <MULTIDROP>
# as shown above. The third line hands this to the drop script.
#
# Note that the user ($:) is the mailbox this message should be stuffed in,
the host ($@) is the full
# user@<dom.ain>. This is how the dropscript expects it.
#

I guess sendmail guru's are now laughing their pants off, and I hope someone
will show me a better way to achieve this. For now, it works.

Next, we need to define mailer drop (somewhere in the sendmail.cf)

#
# multidrop pop3 support.
#

Mdrop,          P=/usr/local/bin/dropmail, F=lFS,
                T=X-Unix,
                A=dropmail $u $h

The S flag here is crucial, otherwise the dropmail script won't run as root,
and under linux (==bash) suid scripts are not permited. I gather most unices
now disalow suid scripts, so this would be necessary on most unices. There
probably are other flags that would make this better, but this works, so I
decided to divert my attention to other tasks at hand (busy, busy, busy....
;^>).

Now we only need the dropmail script, /usr/local/bin/dropmail, mode 700. It
looks big, but effectively one pipeline does the real work. The rest is
configuration, error checking and locking the mailbox.

#!/bin/bash

#
# Script to force a mail message in a format that fetchmail will recognise.
# use as a MDA from sendmail. Must be executed with F=S.
#

#
# Configuration:
#
maildir=/var/spool/mail
envelope=Delivered-To:

#
# set PATH to a known value to avoid some security issues
#
export PATH=/bin:/usr/bin

#
#
#
to=$2
user=$1
mbox=$maildir/$user

#
# If the mailbox does not exist, create it. Note that we act pretty
paranoid, this is hopefully
# resistant to symlink attacks
#
if [ ! -f $mbox ]
then
        oldumask=`umask`
        umask 077
        touch $mbox
        chmod 660 $mbox || exit 1
        chown $user $mbox || exit 1
        chgrp mail $mbox || exit 1
        umask $oldumask
fi

# First lock the mailbox, if this doesn't succeed in 64 seconds, give up and
send
# mail to postmaster.
# If this period is to short, increase the retries (-r flag to lockfile)
#
# Then run the message through formail to get it into the right mailbox
format with the
# right headers added.
#
# Delivered-To will make fetchmail propagate this mail to the correct user
when
# run with '-E "Delivered-To"'. Set this in the advanced settings of the
TeamInternet f.i.
# (if you changed the envelope at the start of this script, adapt this
accordingly)
#
# We also muck up the messageid, so fetchmail will never skip a message on
the basis of
# duplicate messageIDs. The -i "Message-ID" will rename the old message ID,
the -a will
# add a new one.
#
# Lastly, we add a header indicating which host did the rewriting.
#

if lockfile -r 8 $mbox.lock >/dev/null 2>&1
then
        cat - | formail -i "$envelope <$to>" -i "Message-ID:" -a
"Message-ID:" -i "X-Multidrop-Processing: <`hostname`>" >>$mbox
        rm -f $mbox.lock
else
        (echo "Subject: Cannot lock mailbox for $user" & cat -) |
/usr/lib/sendmail postmaster
fi

#
# EOF
#

This obviously is very Linux (even RedHat?) dependant, locking mailboxes,
creating mailboxes with the right permissions, probably even bash dependent.
I would say that it should be fairly easy to port to other systems, but
alas, my unix knowledge is lacking for that. I'll also rewrite it someday,
a.o. that umask handling can be done much better and the location of the
sendmail binairy should not be fixed.

Now the only thing left to do is to retrieve the mail with fetchmail, using
'envelope "Delivered-To:"' in the poll line. The above script has added this
line, so this is all that fetchmail needs.

All parts of this solution need carefull examination. In particular I think
the new rule lines may not catch all cases, although they worked for
everything I threw at them and work satisfactorily in production. I'm also
wondering if there is a more standard way to drop something in a mailbox. I
yet have to investigate procmail, but all other MDA's mucked with the
message and effectively undid my carefully added header. I'll experiment
some more and rethink it all as I learn more.

I'm still wondering, if I can get formail to include another received
line.... "Received from localhost by dropmail for <user>...." to make it
work without the envelope flag. Well I'll have to experiment. Do you know if
there is a header I can add so fetchmail works out-of-the-box?

Regards,
Martijn Lievaart