Back to Fetchmail Home Page To Site Map $Date: 1997/09/29 15:36:26 $

Design Notes On Fetchmail

These notes are for the benefit of future hackers and maintainers. The following sections are both functional and narrative, read from beginning to end.

History

A direct ancestor of the fetchmail program was originally authored (under the name popclient) by Carl Harris . I took over development in June 1996 and subsequently renamed the program `fetchmail' to reflect the addition of IMAP support. In early November 1996 Carl officially ended support for the last popclient versions.

Before accepting responsibility for the popclient sources from Carl, I had investigated and used and tinkered with every other UNIX remote-mail forwarder I could find, including fetchpop1.9, PopTart-0.9.3, get-mail, gwpop, pimp-1.0, pop-perl5-1.2, popc, popmail-1.6 and upop. My major goal was to get a header-rewrite feature like fetchmail's working so I wouldn't have reply problems anymore.

Despite having done a good bit of work on fetchpop1.9, when I found popclient I quickly concluded that it offered the solidest base for future development. I was convinced of this primarily by the presence of multiple-protocol support. The competition didn't do POP2/RPOP/APOP, and I was already having vague thoughts of maybe adding IMAP. (This would advance two other goals: learn IMAP and get comfortable writing TCP/IP client software.)

Until popclient 3.05 I was simply following out the implications of Carl's basic design. He already had daemon.c in the distribution, and I wanted daemon mode almost as badly as I wanted the header rewrite feature. The other things I added were bug fixes or minor extensions.

After 3.1, when I put in SMTP-forwarding support (more about this below) the nature of the project changed -- it became a carefully-thought-out attempt to render obsolete every other program in its class. The name change quickly followed.

The rewrite option

RFC 1123 stipulates that MTAs ought to canonicalize the addresses of outgoing mail so that From:, To:, Cc:, Bcc: and other address headers contain only fully qualified domain names. Failure to do so can break the reply function on many mailers.

This problem only becomes obvious when a reply is generated on a machine different from where the message was delivered. The two machines will have different local username spaces, potentially leading to misrouted mail.

Most MTAs (and sendmail in particular) do not canonicalize address headers in this way (violating RFC 1123). Fetchmail therefore has to do it. This is the first feature I added to the ancestral popclient.

Reorganization

The second thing I did reorganize and simplify popclient a lot. Carl Harris's implementation was very sound, but exhibited a kind of unnecessary complexity common to many C programmers. He treated the code as central and the data structures as support for the code. As a result, the code was beautiful but the data structure design ad-hoc and rather ugly (at least to this old LISP hacker).

I was able to improve matters significantly by reorganizing most of the program around the `query' data structure and eliminating a bunch of global context. This especially simplified the main sequence in fetchmail.c and was critical in enabling the daemon mode changes.

IMAP support and the method table

The next step was IMAP support. I initially wrote the IMAP code as a generic query driver and a method table. The idea was to have all the protocol-independent setup logic and flow of control in the driver, and the protocol-specific stuff in the method table.

Once this worked, I rewrote the POP3 code to use the same organization. The POP2 code kept its own driver for a couple more releases, until I found sources of a POP2 server to test against (the breed seems to be nearly extinct).

The purpose of this reorganization, of course, is to trivialize the development of support for future protocols as much as possible. All mail-retrieval protocols have to have pretty similar logical design by the nature of the task. By abstracting out that common logic and its interface to the rest of the program, both the common and protocol-specific parts become easier to understand.

Furthermore, many kinds of new features can instantly be supported across all protocols by modifying the one driver module.

Implications of smtp forwarding

The direction of the project changed radically when Harry Hochheiser sent me his scratch code for forwarding fetched mail to the SMTP port. I realized almost immediately that a reliable implementation of this feature would make all the other delivery modes obsolete.

Why mess with all the complexity of configuring an MDA or setting up lock-and-append on a mailbox when port 25 is guaranteed to be there on any platform with TCP/IP support in the first place? Especially when this means retrieved mail is guaranteed to look like normal sender- initiated SMTP mail, which is really what we want anyway.

Clearly, the right thing to do was (1) hack SMTP forwarding support into the generic driver, (2) make it the default mode, and (3) eventually throw out all the other delivery modes.

I hesitated over step 3 for some time, fearing to upset long-time popclient users dependent on the alternate delivery mechanisms. In theory, they could immediately switch to .forward files or their non-sendmail equivalents to get the same effects. In practice the transition might have been messy.

But when I did it (see the NEWS note on the great options massacre) the benefits proved huge. The cruftiest parts of the driver code vanished. Configuration got radically simpler -- no more grovelling around for the system MDA and user's mailbox, no more worries about whether the underlying OS supports file locking.

Also, the only way to lose mail vanished. If you specified localfolder and the disk got full, your mail got lost. This can't happen with SMTP forwarding because your SMTP listener won't return OK unless the message can be spooled or processed.

Also, performance improved (though not so you'd notice it in a single run). Another not insignificant benefit of this change was that the manual page got a lot simpler.

Later, I had to bring --mda back in order to allow handling of some obscure situations involving dynamic SLIP. But I found a much simpler way to do it.

The moral? Don't hesitate to throw away superannuated features when you can do it without loss of effectiveness. I tanked a couple I'd added myself and have no regrets at all. As Saint-Exupery said, "Perfection [in design] is achieved not when there is nothing more to add, but rather when t

#!/bin/sh

MSG() {
cat << EOF

# Fetchsetup is a shell script for creating a .fetchmailrc file, that will be
# used by the program "fetchmail" to connect to your mail domain and retrieve
# your mail.
# This script is linux specific, so it may not work on another system.
# Kent Robotti <krobot@erols.com> (3-31-99)

EOF
}

if [ "$(id -ur)" != "0" ]; then
    echo >&2 "$0: You need to be root [found $(id -un)] to run this script."
    echo >&2 "You could login as root"
    echo >&2 "You could also try one of these: # sudo fetchsetup"
    echo >&2 "                                 # su root -c fetchsetup"
    exit 1
fi

MSG
echo -n "Continue? (Y/n) : "
read ans
if [ "$ans" = "n" -o "$ans" = "N" ]; then
    echo "Cancelled."
    exit 0
fi

stty erase "^?" 2>/dev/null

echo
echo "Remote mail site?: pop.boo.com   <Your service providers mail domain name>"
echo -n "Remote mail site?: "
read SITE
echo
echo "Protocol?: pop3   <My service provider uses the 'pop3' mail protocol>"
echo "Protocol?: auto   <If not sure put: auto>"
echo "Choices: apop auto etrn imap imap-gss imap-k4 kpop pop2 pop3 rpop sdps"
echo -n "Protocol?: "
read PROTO
echo
echo "Remote username?: jerry   <My username or login is jerry>"
echo -n "Remote username?: "
read USR
echo
echo "Remote password?: ?       <What's the password for?: $USR>"
echo -n "Remote password?: "
read PASS

echo
echo -n "Create $HOME/.fetchmailrc file? (Y/n) : "
read ans
if [ "$ans" =