fetchmail - [fork] set imap idle timeout to 600s, some servers don't follow rfc recommendations

	Commit message (Expand)	Author	Age	Files	Lines
*	getopt_long is back for Solaris and HP-UX.	Eric S. Raymond	2001-09-30	1	-53/+19
*	Initial revision	Eric S. Raymond	2001-09-30	1	-0/+70

From mlievaart@orion.nl Mon Jan 10 10:46:33 2000 From: Martijn Lievaart <mlievaart@orion.nl> To: Eric S. Raymond <esr@thyrsus.com> Date: zondag 9 januari 2000 0:38 Subject: Re: Thanks for fetchmail and a solution to the multidrop problem (I Status: O Content-Length: 8086 Lines: 226 think) Hello Eric, Let me first state that I'm no sendmail nor unix guru, so although this seems to work, I certainly would not say this is the "best" solution. In fact I would welcome all comments to make this better. In particular, it seems that that the mailertable feature was made just for this, but I'm still studying that. Also, This mail will have lines wrapped. I will put up this on a website asap, so people can download the relevant portions. In the meantime, I'm using (stuck on) Outlook, so I won't even attempt to format this mail. Accept my apoligies and try to mentally reconnect the lines. Finally, this mail is a bit lengthy, but I guess it is better to get all information in, so please bear with me. After some very frustrating attempts to get multidrop to work reliably, it suddenly hit me. When sendmail has translated the recipient to the mailbox, the recipient is gone (in the cases we're talking about). So the solution is not to let sendmail do this translation (completely). The trick is to let a custom MDA be called with both the mailbox and the full recipient name. This MDA then just stuffs it in the correct mailbox after adding the appropriate headers. Luckily I hit on the formail utility. It reformats a mailmessage and does just what I wanted. Specifically my script uses it to: - add a custom header (default: "Delivered-To:") with the recipient - rewrite the message-ID, so fetchmail will download the same message multiple times. - add another header, just for fun. The rewriting of the message-ID is needed because fetchmail will suppress multiple messages with the same ID, normally a good idea, but now it gets in the way. A switch on fetchmail to suppress this behaviour would be great. At first I hardcoded the domains in the sendmail.cf, but I quickly set out to do one better and came up with the following solution. In sendmail.cf, add the following line somewhere at the top. Kmultidroptable hash -o /etc/mail/multidroptable this defines a table for all domains we want to use multidrop for. The format of this file is multiple lines of the format: <domain> <mailbox> e.g: mailtest.orion.nl mailtest mailtest2.orion.nl mailtest mailtest3.orion.nl mailtest bvh-communicatie.nl b.bvh krakatau.nl b.bvh personeelzaak.nl b.bvh maslowassociates.nl b.bvh rtij.nl rtij Of course, create a .db file with makemap. Also, the domains must be added to class w, so they should be added to your sendmail.cw or RelayTo file, or whatever you use. Now add to sendmail.cf: R$+ < @ $* . > $: <MULTIDROP> $(multidroptable $2 $: <NO> $) <?> $1 < @ $2 . > R<MULTIDROP> <NO> <?> $* $: $1 R<MULTIDROP> $+ <?> $+ < @ $* . > $#drop $@ $2 @ $3 $: $1 These lines should be above the existing lines that read: # short circuit local delivery so forwarded email works R$=L < @ $=w . > $#local $: @ $1 special local names R$+ < @ $=w . > $#local $: $1 regular local name This works as follows (in fact these comments are above my modification in our sendmail.cf). # # MLI. Any drop host gets passed to the drop script # # The first rule looks up the domain in the multidrop table. # The input at this point is always: # user@<dom.ain.> # If found, the resulting line looks like this: # <MULTIDROP> mailbox <?> user@<dom.ain.> # if not found, the resulting line will be: # <MULTIDROP> <NO> <?> user@<dom.ain.> # The second line restores the "not found" case back to user@<dom.ain.> # So if this domain was found in the multidroptable, we still have a line starting with <MULTIDROP> # as shown above. The third line hands this to the drop script. # # Note that the user ($:) is the mailbox this message should be stuffed in, the host ($@) is the full # user@<dom.ain>. This is how the dropscript expects it. # I guess sendmail guru's are now laughing their pants off, and I hope someone will show me a better way to achieve this. For now, it works. Next, we need to define mailer drop (somewhere in the sendmail.cf) # # multidrop pop3 support. # Mdrop, P=/usr/local/bin/dropmail, F=lFS, T=X-Unix, A=dropmail $u $h The S flag here is crucial, otherwise the dropmail script won't run as root, and under linux (==bash) suid scripts are not permited. I gather most unices now disalow suid scripts, so this would be necessary on most unices. There probably are other flags that would make this better, but this works, so I decided to divert my attention to other tasks at hand (busy, busy, busy.... ;^>). Now we only need the dropmail script, /usr/local/bin/dropmail, mode 700. It looks big, but effectively one pipeline does the real work. The rest is configuration, error checking and locking the mailbox. #!/bin/bash # # Script to force a mail message in a format that fetchmail will recognise. # use as a MDA from sendmail. Must be executed with F=S. # # # Configuration: # maildir=/var/spool/mail envelope=Delivered-To: # # set PATH to a known value to avoid some security issues # export PATH=/bin:/usr/bin # # # to=$2 user=$1 mbox=$maildir/$user # # If the mailbox does not exist, create it. Note that we act pretty paranoid, this is hopefully # resistant to symlink attacks # if [ ! -f $mbox ] then oldumask=`umask` umask 077 touch $mbox chmod 660 $mbox || exit 1 chown $user $mbox || exit 1 chgrp mail $mbox || exit 1 umask $oldumask fi # First lock the mailbox, if this doesn't succeed in 64 seconds, give up and send # mail to postmaster. # If this period is to short, increase the retries (-r flag to lockfile) # # Then run the message through formail to get it into the right mailbox format with the # right headers added. # # Delivered-To will make fetchmail propagate this mail to the correct user when # run with '-E "Delivered-To"'. Set this in the advanced settings of the TeamInternet f.i. # (if you changed the envelope at the start of this script, adapt this accordingly) # # We also muck up the messageid, so fetchmail will never skip a message on the basis of # duplicate messageIDs. The -i "Message-ID" will rename the old message ID, the -a will # add a new one. # # Lastly, we add a header indicating which host did the rewriting. # if lockfile -r 8 $mbox.lock >/dev/null 2>&1 then cat - | formail -i "$envelope <$to>" -i "Message-ID:" -a "Message-ID:" -i "X-Multidrop-Processing: <`hostname`>" >>$mbox rm -f $mbox.lock else (echo "Subject: Cannot lock mailbox for $user" & cat -) | /usr/lib/sendmail postmaster fi # # EOF # This obviously is very Linux (even RedHat?) dependant, locking mailboxes, creating mailboxes with the right permissions, probably even bash dependent. I would say that it should be fairly easy to port to other systems, but alas, my unix knowledge is lacking for that. I'll also rewrite it someday, a.o. that umask handling can be done much better and the location of the sendmail binairy should not be fixed. Now the only thing left to do is to retrieve the mail with fetchmail, using 'envelope "Delivered-To:"' in the poll line. The above script has added this line, so this is all that fetchmail needs. All parts of this solution need carefull examination. In particular I think the new rule lines may not catch all cases, although they worked for everything I threw at them and work satisfactorily in production. I'm also wondering if there is a more standard way to drop something in a mailbox. I yet have to investigate procmail, but all other MDA's mucked with the message and effectively undid my carefully added header. I'll experiment some more and rethink it all as I learn more. I'm still wondering, if I can get formail to include another received line.... "Received from localhost by dropmail for <user>...." to make it work without the envelope flag. Well I'll have to experiment. Do you know if there is a header I can add so fetchmail works out-of-the-box? Regards, Martijn Lievaart