NEWS


1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207

				Release Notes:

(The `lines' figures total .c, .h, .l, and .y files under version control.)

* Sunil Shetye's fix to force fetchsizelimit to 1 for APOP and RPOP.
* PopDel.py removed from contrib at author's request.
* Matthias Andree's fix for Sunil Shetye's fetch-split patch
* Include James Stone's moldremover.py script.
* Enable .fetchmailrc permissions checking under Cygwin.
* Nalin Dahyabai's fix for POP3 strong authentication.
* Revised Nalin Dahyabai's fix for POP3 strong authentication (Matthias
  Andree, the original version would go into an infinite loop when CAPA
  failed; found by David Greaves.)
* HOME_ETC patch for PLD Linux.
* Sunil Shetye's fix for SSL configuration.
* Simon Josefsson's patch for GSS library support.
* Added Andrey Lelikov's recupe for Hotmail and Lycos Webmail.
* Remove blank between MAIL FROM: and <, which causes Cyrus to complain.
  Patch by Phil Endecott. (Rob Funk)
* Switched to automake. (Matthias Andree)
* Build fixes for HESIOD and resolv.h trouble on FreeBSD. (Matthias Andree)
* Fabrice Bellet's fix for Red Hat bug #113492, fetchmail hangs in IMAP
  mode after EXPUNGE when the server (Dovecot 0.99.10) doesn't update
  RECENT and EXISTS counts. (Matthias Andree)
* Holger Mauermann's bounce patch, to use a NULL envelope from, not
  write a Return-Path header (both to meet RFC-2821), changed From,
  added Subject header, rewording the human readable part. (Matthias Andree)
* Merge Sunil Shetye's time.h handling fix. (Matthias Andree)
* Merge Gerd von Egidy's patch to avoid a segfault in multidrop/received
  mode when the Received: headers are malformatted. (Matthias Andree)
* MIME-encode bodies and Subject headers of warning messages (Matthias
  Andree), limiting the header to 7 bits.
* Normalize most locale codesets to IANA codesets, based on
  norm_charmap.c by Markus Kuhn. (Matthias Andree)
* Remove sleep(3) after POP3 login, patch by Brian Candler. (Matthias
  Andree)
* Fix option parsing bug that trashes the showdots setting when more
  than one server is configured. Patch by Brian Candler. (Matthias
  Andree)
* Honor sslcertpath setting even if sslcertck is unset. Patch by Brian
  Candler. (Matthias Andree)
* SSL certificate checking fixes, don't display same error message twice
  in succession, make sure that Common Name and fingerprint checking are
  only done once. Print all validation warnings/errors even if not in
  verbose mode. Patch by Brian Candler. (Matthias Andree)
* Import Bjorn Reese and Daniel Stenberg's MIT-licensed Trio 1.10 from
  http://daniel.haxx.se/projects/trio/ for systems that do not support
  snprintf or vsnprintf. (Matthias Andree)
* Clean up the horrible #ifdef HAVE_[V]SNPRINTF that made the code
  unreadable. Use Trio where [v]snprintf is/are missing. (Matthias Andree)
* Default to Linux 2.2 /proc/net/dev format, and use uname(2) to determine the
  kernel version instead of calling uname(1). Thanks to Paul Slootman.
  (Matthias Andree)
* Be more careful when swapping UID lists or writing the .fetchids file,
  requested by Manfred Weihs. (Matthias Andree)
* Print a warning if multidrop configuration is attempted without
  envelope option. (Matthias Andree)
* Split information on fetchmail versions before 6.0.0 to a separate
  OLDNEWS file. (Matthias Andree)
* Merge SuSE patches: (sent by Stanislav Brabec, merged by Matthias Andree)
  - fetchmail-6.2.5-declaration.patch (double sigint_handler decl/getpass.c)
  - fetchmail-6.2.5-implicit-declaration.patch (missing #include)
  - fetchmail-6.2.5-random-result.patch (uninitialized variable/opie.c)

fetchmail-6.2.5 (Wed Oct 15 18:39:22 EDT 2003), 23079 lines:

* Updated Spanish, Turkish, and German translation files.
* Matthew Gregan's patch to handle garbage lengths from dbmail;
  closes Debian bug #207919.
* Fix IMAP query so new-message count doesn't include deleted messages.
* Man page typo fix, closes Debian bug #205892.
* OpenSSL cleanup patches from levinedl@acm.org.
* Benjamin Drieu's patch to fix Debian bug #212240, no oversized-message
  flushing if both "flush" and "limit" were specified.
* Benjamin Drieu's patch for Debian bug #156592, incorrect handing of 
  host/port option.
* Smash all NULs out of headers right after the socket read.
* Dup-killer code now keys on an MD5 hash of the raw headers.
* Sunil Shetye's patches to break up fetching of sizes and UIDLs.

There are 599 people on fetchmail-friends and 748 on fetchmail-announce.

fetchmail-6.2.4 (Wed Aug 13 04:27:35 EDT 2003), 22625 lines:

* Updated German, Spanish, Catalan, and Turkish translations.
* IDLE is now supported using no-ops even if the server doesn't support
  the IMAP IDLE extension.
* Sunil Shetye's patch to do better password shrouding.
* Sunil Shetye's bug-fix rollup patch.
* Introduce a translation item for the word "seen".
* Back out the hack to deal with lack of byte stuffing on some POP3 servers.
* Thomas Steudten's patch to improve SMTP handling of 550 errors.

There are 585 people on fetchmail-friends and 745 on fetchmail-announce.

fetchmail-6.2.3 (Thu Jul 17 14:53:00 EDT 2003), 22490 lines:

* French, German, Danish, Spanish, and Turkish translations updated.
* Brian Sammon's patch to deal with malformed message lines containing NULs.
* Fai's patch to ignore all but the first Return-Path (some spams have
  more than one of these).
* Benjamin Drieu's patch to properly byte-stuff when talking to BSNTP.
  Fixes Debian bug #184469.
* Benjamin Drieu's patch to enable auth=cram-md5.
  Fixes Debian bug #185232.
* Sunil Shetye's configure.in patch to avoid spurious search order messages
  from GCC.
* Header-reading code now copes better with lines ending in \n only.
* Elias Israel's patches for POP3 NTLM support and dealing with byte-
  stuffing failures at socket level.

There are 580 people on fetchmail-friends and 750 on fetchmail-announce.

fetchmail-6.2.2 (Fri Feb 28 21:34:26 EST 2003), 22345 lines:

* Sunil Shetye's patch to improve behavior on empty messages.
* Conform to RFC2595; reissue capability probes after successful 
  STARTTLS negotiation.
* Sunil's patch to make handling of failed STARTTLS more graceful.
* Sunil's JF2 fix patch for .fetchmailrc security.
* Christophe GIAUME <christophe@giaume.com> finished the implementation
  of RFC2177 IDLE.
* Jason Tishler's fix patch for Cygwin.
* Support ssh-style authentication in POP3
* Fix for Debian bug #108977, clean up config file evaluation,
  by Benjamin Drieu.

There are 554 people on fetchmail-friends and 727 on fetchmail-announce.

fetchmail-6.2.1 (Tue Jan 14 08:17:19 EST 2003), 22219 lines:

* Updated German, Turkish, Spanish, and Danish translation files.
* Integrated Sunil Shetye's patch to make mark_seen an explicit method.
* Removed FAQ warning about GMX and associated fetchmailconf check, 
  we have a report that its servers are conformant now.
* Another Sunil patch to fix a minor bug in bouncemail generation.

There are 536 people on fetchmail-friends and 716 on fetchmail-announce.

fetchmail-6.2.0 (Fri Dec 13 00:10:07 EST 2002), 22235 lines:

* Applied Steffen Esser's fix for a buffer-overflow bug in rfc822.c
* Updated Danish, German, and Turkish translation files.
* Sunil Shetye's SMTP timeout patch.

There are 538 people on fetchmail-friends and 701 on fetchmail-announce.

fetchmail-6.1.3 (Thu Nov 28 05:35:15 EST 2002), 22203 lines:

* Updated Turkish, Danish, German, Spanish, Catalan po files.
* Added Slovak support.
* Configure.in update for autoconf 2.5 (Art Haas). 
* Be case-insensitive when looking for IMAP responses.
* Fix logout-after-idle-delivery bug (Sunil Shetye).
* Sunil Shetye's patch to bulletproof end-of-header detection.
* Sunil's fix for the STARTTLS problem -- repoll if TLS nabdshake
  fails.  The attempt to set up STARTTLS can be suppressed with 'sslproto ""'.

There are 540 people on fetchmail-friends and 701 on fetchmail-announce.

fetchmail-6.1.2 (Thu Oct 31 11:41:02 EST 2002), 22135 lines:

* Jan Klaverstijn's verbosity-lowering patch.
* Updated Turkish, German, Catalan, and Danish translation files.
* Fix processing of POP3 messages with missing bodies.
* Minor fixes by Sunil Shetye: fix generation of auth fail note, handle
  unexpected SIGALRM, plug memory leak, handle lines beginning with '\0',
  try to bulletproof error handling against read failures.

There are 535 people on fetchmail-friends and 696 on fetchmail-announce.

fetchmail-6.1.1 (Fri Oct 18 14:53:51 EDT 2002), 22087 lines:

* OTP fix patches from Stanislav Brabec <utx@penguin.cz>
* fix patch for writing antispam capability correctly in conf.c.
* Fix patches for Debian bugs #162571, #156592.
* Correction to manpage re -b and qmail.
* Patch to disable use of STLS if auth passwd is specified.
* Fix specfile generation to handle SSL correctly.
* New Danish, Turkish, and Catalan translation files.
* Improved ODMR debug messages.
* IMAP efficiency hack; don't fetch sizes unless needed.
* Detect and rewrite invalid return paths beginning with @.
* Fix for subtle freeing bug that suppressed information in some bounce msgs.
* Newline fix patches for internationalization files.
* Fix reversed test guarding authentication-failure warnings.
* Fix POP3 breakage starting at 5.9.14.

There are 529 people on fetchmail-friends and 693 on fetchmail-announce.

fetchmail-6.1.0 (Sun Sep 22 18:31:23 EDT 2002), 21999 lines:

* Updated French translation.
* Stefan Esser's fix for potential remote vulnerability in multidrop mode.
  This is an important security fix!

There are 519 people on fetchmail-friends and 680 on fetchmail-announce.

fetchmail-6.0.0 (Tue Sep 17 19:48:25 EDT 2002), 21972 lines:

* Applied Matt Kraai's fix for minor Debian bug #144539.
* Nerijus Baliunas's patch to support STARTTLS over IMAP.
* More cleanups and minor bugfixes from Sunil Shetye.
* Default antispam-response list is now empty.
* Updated de and po translations,

There are 520 people on fetchmail-friends and 683 on fetchmail-announce.
<?xml version="1.0" encoding="ISO-8859-1"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
    "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
    <title>Eric S. Raymond's former Design notes on fetchmail</title>
<link rev="made" href="mailto:esr@snark.thyrsus.com" />
<meta name="description" content="Design notes on fetchmail." />
<meta name="keywords" content="fetchmail, POP, POP2, POP3, IMAP, remote mail" />
<style type="text/css">
/*<![CDATA[*/
 h1.c1 {text-align: center}
/*]]>*/
</style>
</head>
<body>
<table width="100%" cellpadding="0" summary="Canned page header">
<tr>
<td width="30%">Forward to <a href="design-notes.html">Updated design
	notes</a></td>
<td width="30%">Back to <a href="index.html">Fetchmail Home Page</a></td>
<td width="30%" align="right">$Date$</td>
</tr>
</table>

<hr />
<h1 class="c1">Eric S. Raymond's former Design Notes On Fetchmail</h1>

<p>These notes are for the benefit of future hackers and
maintainers. The following sections are both functional and
narrative, read from beginning to end.</p>

<h1>History</h1>

<p>A direct ancestor of the fetchmail program was originally
authored (under the name popclient) by Carl Harris
&lt;ceharris@mal.com&gt;. I took over development in June 1996 and
subsequently renamed the program `fetchmail' to reflect the
addition of IMAP support and SMTP delivery. In early November 1996
Carl officially ended support for the last popclient versions.</p>

<p>Before accepting responsibility for the popclient sources from
Carl, I had investigated and used and tinkered with every other
UNIX remote-mail forwarder I could find, including fetchpop1.9,
PopTart-0.9.3, get-mail, gwpop, pimp-1.0, pop-perl5-1.2, popc,
popmail-1.6 and upop. My major goal was to get a header-rewrite
feature like fetchmail's working so I wouldn't have reply problems
anymore.</p>

<p>Despite having done a good bit of work on fetchpop1.9, when I
found popclient I quickly concluded that it offered the solidest
base for future development. I was convinced of this primarily by
the presence of multiple-protocol support. The competition didn't
do POP2/RPOP/APOP, and I was already having vague thoughts of maybe
adding IMAP. (This would advance two other goals: learn IMAP and
get comfortable writing TCP/IP client software.)</p>

<p>Until popclient 3.05 I was simply following out the implications
of Carl's basic design. He already had daemon.c in the
distribution, and I wanted daemon mode almost as badly as I wanted
the header rewrite feature. The other things I added were bug fixes
or minor extensions.</p>

<p>After 3.1, when I put in SMTP-forwarding support (more about
this below) the nature of the project changed -- it became a
carefully-thought-out attempt to render obsolete every other
program in its class. The name change quickly followed.</p>

<h1>The rewrite option</h1>

<p>MTAs ought to canonicalize the addresses of outgoing non-local
mail so that From:, To:, Cc:, Bcc: and other address headers
contain only fully qualified domain names. Failure to do so can
break the reply function on many mailers. (Sendmail has an option
to do this.)</p>

<p>This problem only becomes obvious when a reply is generated on a
machine different from where the message was delivered. The two
machines will have different local username spaces, potentially
leading to misrouted mail.</p>

<p>Most MTAs (and sendmail in particular) do not canonicalize
address headers in this way (violating RFC 1123). Fetchmail
therefore has to do it. This is the first feature I added to the
ancestral popclient.</p>

<h1>Reorganization</h1>

<p>The second thing I did reorganize and simplify popclient a lot.
Carl Harris's implementation was very sound, but exhibited a kind
of unnecessary complexity common to many C programmers. He treated
the code as central and the data structures as support for the
code. As a result, the code was beautiful but the data structure
design ad-hoc and rather ugly (at least to this old LISP
hacker).</p>

<p>I was able to improve matters significantly by reorganizing most
of the program around the `query' data structure and eliminating a
bunch of global context. This especially simplified the main
sequence in fetchmail.c and was critical in enabling the daemon
mode changes.</p>

<h1>IMAP support and the method table</h1>

<p>The next step was IMAP support. I initially wrote the IMAP code
as a generic query driver and a method table. The idea was to have
all the protocol-independent setup logic and flow of control in the
driver, and the protocol-specific stuff in the method table.</p>

<p>Once this worked, I rewrote the POP3 code to use the same
organization. The POP2 code kept its own driver for a couple more
releases, until I found sources of a POP2 server to test against
(the breed seems to be nearly extinct).</p>

<p>The purpose of this reorganization, of course, is to trivialize
the development of support for future protocols as much as
possible. All mail-retrieval protocols have to have pretty similar
logical design by the nature of the task. By abstracting out that
common logic and its interface to the rest of the program, both the
common and protocol-specific parts become easier to understand.</p>

<p>Furthermore, many kinds of new features can instantly be
supported across all protocols by modifying the one driver
module.</p>

<h1>Implications of smtp forwarding</h1>

<p>The direction of the project changed radically when Harry
Hochheiser sent me his scratch code for forwarding fetched mail to
the SMTP port. I realized almost immediately that a reliable
implementation of this feature would make all the other delivery
modes obsolete.</p>

<p>Why mess with all the complexity of configuring an MDA or
setting up lock-and-append on a mailbox when port 25 is guaranteed
to be there on any platform with TCP/IP support in the first place?
Especially when this means retrieved mail is guaranteed to look
like normal sender- initiated SMTP mail, which is really what we
want anyway.</p>

<p>Clearly, the right thing to do was (1) hack SMTP forwarding
support into the generic driver, (2) make it the default mode, and
(3) eventually throw out all the other delivery modes.</p>

<p>I hesitated over step 3 for some time, fearing to upset
long-time popclient users dependent on the alternate delivery
mechanisms. In theory, they could immediately switch to .forward
files or their non-sendmail equivalents to get the same effects. In
practice the transition might have been messy.</p>

<p>But when I did it (see the NEWS note on the great options
massacre) the benefits proved huge. The cruftiest parts of the
driver code vanished. Configuration got radically simpler -- no
more grovelling around for the system MDA and user's mailbox, no
more worries about whether the underlying OS supports file
locking.</p>

<p>Also, the only way to lose mail vanished. If you specified
localfolder and the disk got full, your mail got lost. This can't
happen with SMTP forwarding because your SMTP listener won't return
OK unless the message can be spooled or processed.</p>

<p>Also, performance improved (though not so you'd notice it in a
single run). Another not insignificant benefit of this change was
that the manual page got a lot simpler.</p>

<p>Later, I had to bring --mda back in order to allow handling of
some obscure situations involving dynamic SLIP. But I found a much
simpler way to do it.</p>

<p>The moral? Don't hesitate to throw away superannuated features
when you can do it without loss of effectiveness. I tanked a couple
I'd added myself and have no regrets at all. As Saint-Exupery said,
"Perfection [in design] is achieved not when there is nothing more
to add, but rather when there is nothing more to take away." This
program isn't perfect, but it's trying.</p>

<h1>The most-requested features that I will never add, and why
not:</h1>

<h2>Password encryption in .fetchmailrc</h2>

<p>The reason there's no facility to store passwords encrypted in
the .fetchmailrc file is because this doesn't actually add
protection.</p>

<p>Anyone who's acquired the 0600 permissions needed to read your
.fetchmailrc file will be able to run fetchmail as you anyway --
and if it's your password they're after, they'd be able to rip the
necessary decoder out of the fetchmail code itself to get it.</p>

<p>All .fetchmailrc encryption would do is give a false sense of
security to people who don't think very hard.</p>

<h2>Truly concurrent queries to multiple hosts</h2>

<p>Occasionally I get a request for this on "efficiency" grounds.
These people aren't thinking either. True concurrency would do
nothing to lessen fetchmail's total IP volume. The best it could
possibly do is change the usage profile to shorten the duration of
the active part of a poll cycle at the cost of increasing its
demand on IP volume per unit time.</p>

<p>If one could thread the protocol code so that fetchmail didn't
block on waiting for a protocol response, but rather switched to
trying to process another host query, one might get an efficiency
gain (close to constant loading at the single-host level).</p>

<p>Fortunately, I've only seldom seen a server that incurred
significant wait time on an individual response. I judge the gain
from this not worth the hideous complexity increase it would
require in the code.</p>

<h2>Multiple concurrent instances of fetchmail</h2>

<p>Fetchmail locking is on a per-invoking-user because
finer-grained locks would be really hard to implement in a portable
way. The problem is that you don't want two fetchmails querying the
same site for the same remote user at the same time.</p>

<p>To handle this optimally, multiple fetchmails would have to
associate a system-wide semaphore with each active pair of a remote
user and host canonical address. A fetchmail would have to block
until getting this semaphore at the start of a query, and release
it at the end of a query.</p>

<p>This would be way too complicated to do just for an "it might be
nice" feature. Instead, you can run a single root fetchmail polling
for multiple users in either single-drop or multidrop mode.</p>

<p>The fundamental problem here is how an instance of fetchmail
polling host foo can assert that it's doing so in a way visible to
all other fetchmails. System V semaphores would be ideal for this
purpose, but they're not portable.</p>

<p>I've thought about this a lot and roughed up several designs.
All are complicated and fragile, with a bunch of the standard
problems (what happens if a fetchmail aborts before clearing its
semaphore, and how do we recover reliably?).</p>

<p>I'm just not satisfied that there's enough functional gain here
to pay for the large increase in complexity that adding these
semaphores would entail.</p>

<h1>Multidrop and alias handling</h1>

<p>I decided to add the multidrop support partly because some users
were clamoring for it, but mostly because I thought it would shake
bugs out of the single-drop code by forcing me to deal with
addressing in full generality. And so it proved.</p>

<p>There are two important aspects of the features for handling
multiple-drop aliases and mailing lists which future hackers should
be careful to preserve.</p>

<ol>
<li>
<p>The logic path for single-recipient mailboxes doesn't involve
header parsing or DNS lookups at all. This is important -- it means
the code for the most common case can be much simpler and more
robust.</p>
</li>

<li>
<p>The multidrop handing does <em>not</em> rely on doing the
equivalent of passing the message to sendmail -t. Instead, it
explicitly mines members of a specified set of local usernames out
of the header.</p>
</li>

<li>
<p>We do <em>not</em> attempt delivery to multidrop mailboxes in
the presence of DNS errors. Before each multidrop poll we probe DNS
to see if we have a nameserver handy. If not, the poll is skipped.
If DNS crashes during a poll, the error return from the next
nameserver lookup aborts message delivery and ends the poll. The
daemon mode will then quietly spin until DNS comes up again, at
which point it will resume delivering mail.</p>
</li>
</ol>

<p>When I designed this support, I was terrified of doing anything
that could conceivably cause a mail loop (you should be too).
That's why the code as written can only append <em>local</em> names
(never @-addresses) to the recipients list.</p>

<p>The code in mxget.c is nasty, no two ways about it. But it's
utterly necessary, there are a lot of MX pointers out there. It
really ought to be a (documented!) entry point in the bind
library.</p>

<h1>DNS error handling</h1>

<p>Fetchmail's behavior on DNS errors is to suppress forwarding and
deletion of the individual message that each occurs in, leaving it
queued on the server for retrieval on a subsequent poll. The
assumption is that DNS errors are transient, due to temporary
server outages.</p>

<p>Unfortunately this means that if a DNS error is permanent a
message can be perpetually stuck in the server mailbox. We've had a
couple bug reports of this kind due to subtle RFC822 parsing errors
in the fetchmail code that resulted in impossible things getting
passed to the DNS lookup routines.</p>

<p>Alternative ways to handle the problem: ignore DNS errors
(treating them as a non-match on the mailserver domain), or forward
messages with errors to fetchmail's invoking user in addition to
any other recipients. These would fit an assumption that DNS lookup
errors are likely to be permanent problems associated with an
address.</p>

<h1>IPv6 and IPSEC</h1>

<p>The IPv6 support patches are really more protocol-family
independence patches. Because of this, in most places, "ports"
(numbers) have been replaced with "services" (strings, that may be
digits). This allows us to run with certain protocols that use
strings as "service names" where we in the IP world think of port
numbers. Someday we'll plumb strings all over and then, if inet6 is
not enabled, do a getservbyname() down in SocketOpen. The IPv6
support patches use getaddrinfo(), which is a POSIX p1003.1g
mandated function. So, in the not too distant future, we'll zap the
ifdefs and just let autoconf check for getaddrinfo. IPv6 support
comes pretty much automatically once you have protocol family
independence.</p>

<h1>Internationalization</h1>

<p>Internationalization is handled using GNU gettext (see the file
ABOUT_NLS in the source distribution). This places some minor
constraints on the code.</p>

<p>Strings that must be subject to translation should be wrapped
with GT_() or N_() -- the former in function arguments, the latter
in static initializers and other non-function-argument
contexts.</p>

<h1>Checklist for Adding Options</h1>

<p>Adding a control option is not complicated in principle, but
there are a lot of fiddly details in the process. You'll need to do
the following minimum steps.</p>

<ul>
<li>Add a field to represent the control in <code>struct
run</code>, <code>struct query</code>, or <code>struct
hostdata</code>.</li>

<li>Go to <code>rcfile_y.y</code>. Add the token to the grammar.
Don't forget the <code>%token</code> declaration.</li>

<li>Pick an actual string to declare the option in the .fetchmailrc
file. Add the token to <code>rcfile_l</code>.</li>

<li>Pick a long-form option name, and a one-letter short option if
any are left. Go to <code>options.c</code>. Pick a new
<code>LA_</code> value. Hack the <code>longoptions</code> table to
set up the association. Hack the big switch statement to set the
option. Hack the `?' message to describe it.</li>

<li>If the default is nonzero, set it in <code>def_opts</code> near
the top of <code>load_params</code> in
<code>fetchmail.c</code>.</li>

<li>Add code to dump the option value in
<code>fetchmail.c:dump_params</code>.</li>

<li>For a per-site or per-user option, add proper
<code>FLAG_MERGE</code> actions in fetchmail.c's optmerge()
function. For a global option, add an override at the end of
load_params; this will involve copying a "cmd_run." field to a
corresponding "run." field, see the existing code for models.</li>

<li>Document the option in fetchmail.man. This will require at
least two changes; one to the collected table of options, and one
full text description of the option.</li>

<li>Hack fetchmailconf to configure it. Bump the fetchmailconf
version.</li>

<li>Hack conf.c to dump the option so we won't have a version-skew
problem.</li>

<li>Add an entry to NEWS.</li>

<li>If the option implements a new feature, add a note to the
feature list.</li>
</ul>

<p>There may be other things you have to do in the way of logic, of
course.</p>

<p>Before you implement an option, though, think hard. Is there any
way to make fetchmail automatically detect the circumstances under
which it should change its behavior? If so, don't write an option.
Just do the check!</p>

<h1>Lessons learned</h1>

<h3>1. Server-side state is essential</h3>

<p>The person(s) responsible for removing LAST from POP3 deserve to
suffer. Without it, a client has no way to know which messages in a
box have been read by other means, such as an MUA running on the
server.</p>

<p>The POP3 UID feature described in RFC1725 to replace LAST is
insufficient. The only problem it solves is tracking which messages
have been read <em>by this client</em> -- and even that requires
tricky, fragile implementation.</p>

<p>The underlying lesson is that maintaining accessible server-side
`seen' state bits associated with Status headers is indispensible
in a Unix/RFC822 mail server protocol. IMAP gets this right.</p>

<h3>2. Readable text protocol transactions are a Good Thing</h3>

<p>A nice thing about the general class of text-based protocols
that SMTP, POP2, POP3, and IMAP belongs to is that client/server
transactions are easy to watch and transaction code correspondingly
easy to debug. Given a decent layer of socket utility functions
(which Carl provided) it's easy to write protocol engines and not
hard to show that they're working correctly.</p>

<p>This is an advantage not to be despised! Because of it, this
project has been interesting and fun -- no serious or persistent
bugs, no long hours spent looking for subtle pathologies.</p>

<h3>3. IMAP is a Good Thing.</h3>

<p>Now that there is a standard IMAP equivalent of the POP3 APOP
validation in CRAM-MD5, POP3 is completely obsolete.</p>

<h3>4. SMTP is the Right Thing</h3>

<p>In retrospect it seems clear that this program (and others like
it) should have been designed to forward via SMTP from the
beginning. This lesson may be applicable to other Unix programs
that now call the local MDA/MTA as a program.</p>

<h3>5. Syntactic noise can be your friend</h3>

<p>The optional `noise' keywords in the rc file syntax started out
as a late-night experiment. The English-like syntax they allow is
considerably more readable than the traditional terse keyword-value
pairs you get when you strip them all out. I think there may be a
wider lesson here.</p>

<h1>Motivation and validation</h1>

<p>It is truly written: the best hacks start out as personal
solutions to the author's everyday problems, and spread because the
problem turns out to be typical for a large class of users. So it
was with Carl Harris and the ancestral popclient, and so with me
and fetchmail.</p>

<p>It's gratifying that fetchmail has become so popular. Until just
before 1.9 I was designing strictly to my own taste. The multi-drop
mailbox support and the new --limit option were the first features
to go in that I didn't need myself.</p>

<p>By 1.9, four months after I started hacking on popclient and a
month after the first fetchmail release, there were literally a
hundred people on the fetchmail-friends contact list. That's pretty
powerful motivation. And they were a good crowd, too, sending fixes
and intelligent bug reports in volume. A user population like that
is a gift from the gods, and this is my expression of
gratitude.</p>

<p>The beta testers didn't know it at the time, but they were also
the subjects of a sociological experiment. The results are
described in my paper, <a
href="//www.catb.org/~esr/writings/cathedral-bazaar/">The
Cathedral And The Bazaar</a>.</p>

<h1>Credits</h1>

<p>Special thanks go to Carl Harris, who built a good solid code
base and then tolerated me hacking it out of recognition. And to
Harry Hochheiser, who gave me the idea of the SMTP-forwarding
delivery mode.</p>

<p>Other significant contributors to the code have included Dave
Bodenstab (error.c code and --syslog), George Sipe (--monitor and
--interface), Gordon Matzigkeit (netrc.c), Al Longyear (UIDL
support), Chris Hanson (Kerberos V4 support), and Craig Metz (OPIE,
IPv6, IPSEC).</p>

<h1>Conclusion</h1>

<p>At this point, the fetchmail code appears to be pretty stable.
It will probably undergo substantial change only if and when
support for a new retrieval protocol or authentication method is
added.</p>

<h1>Relevant RFCS</h1>

<p>Not all of these describe standards explicitly used in
fetchmail, but they all shaped the design in one way or
another.</p>

<dl>
<dt><a href="ftp://ftp.isi.edu/in-notes/rfc821.txt">RFC821</a></dt>

<dd>SMTP protocol</dd>

<dt><a href="ftp://ftp.isi.edu/in-notes/rfc822.txt">RFC822</a></dt>

<dd>Mail header format</dd>

<dt><a href="ftp://ftp.isi.edu/in-notes/rfc937.txt">RFC937</a></dt>

<dd>Post Office Protocol - Version 2</dd>

<dt><a href="ftp://ftp.isi.edu/in-notes/rfc974.txt">RFC974</a></dt>

<dd>MX routing</dd>

<dt><a href="ftp://ftp.isi.edu/in-notes/rfc976.txt">RFC976</a></dt>

<dd>UUCP mail format</dd>

<dt><a
href="ftp://ftp.isi.edu/in-notes/rfc1081.txt">RFC1081</a></dt>

<dd>Post Office Protocol - Version 3</dd>

<dt><a
href="ftp://ftp.isi.edu/in-notes/rfc1123.txt">RFC1123</a></dt>

<dd>Host requirements (modifies 821, 822, and 974)</dd>

<dt><a
href="ftp://ftp.isi.edu/in-notes/rfc1176.txt">RFC1176</a></dt>

<dd>Interactive Mail Access Protocol - Version 2</dd>

<dt><a
href="ftp://ftp.isi.edu/in-notes/rfc1203.txt">RFC1203</a></dt>

<dd>Interactive Mail Access Protocol - Version 3</dd>

<dt><a
href="ftp://ftp.isi.edu/in-notes/rfc1225.txt">RFC1225</a></dt>

<dd>Post Office Protocol - Version 3</dd>

<dt><a
href="ftp://ftp.isi.edu/in-notes/rfc1344.txt">RFC1344</a></dt>

<dd>Implications of MIME for Internet Mail Gateways</dd>

<dt><a
href="ftp://ftp.isi.edu/in-notes/rfc1413.txt">RFC1413</a></dt>

<dd>Identification server</dd>

<dt><a
href="ftp://ftp.isi.edu/in-notes/rfc1428.txt">RFC1428</a></dt>

<dd>Transition of Internet Mail from Just-Send-8 to 8-bit
SMTP/MIME</dd>

<dt><a
href="ftp://ftp.isi.edu/in-notes/rfc1460.txt">RFC1460</a></dt>

<dd>Post Office Protocol - Version 3</dd>

<dt><a
href="ftp://ftp.isi.edu/in-notes/rfc1508.txt">RFC1508</a></dt>

<dd>Generic Security Service Application Program Interface</dd>

<dt><a
href="ftp://ftp.isi.edu/in-notes/rfc1521.txt">RFC1521</a></dt>

<dd>MIME: Multipurpose Internet Mail Extensions</dd>

<dt><a
href="ftp://ftp.isi.edu/in-notes/rfc1869.txt">RFC1869</a></dt>

<dd>SMTP Service Extensions (ESMTP spec)</dd>

<dt><a
href="ftp://ftp.isi.edu/in-notes/rfc1652.txt">RFC1652</a></dt>

<dd>SMTP Service Extension for 8bit-MIMEtransport</dd>

<dt><a
href="ftp://ftp.isi.edu/in-notes/rfc1725.txt">RFC1725</a></dt>

<dd>Post Office Protocol - Version 3</dd>

<dt><a
href="ftp://ftp.isi.edu/in-notes/rfc1730.txt">RFC1730</a></dt>

<dd>Interactive Mail Access Protocol - Version 4</dd>

<dt><a
href="ftp://ftp.isi.edu/in-notes/rfc1731.txt">RFC1731</a></dt>

<dd>IMAP4 Authentication Mechanisms</dd>

<dt><a
href="ftp://ftp.isi.edu/in-notes/rfc1732.txt">RFC1732</a></dt>

<dd>IMAP4 Compatibility With IMAP2 And IMAP2bis</dd>

<dt><a
href="ftp://ftp.isi.edu/in-notes/rfc1734.txt">RFC1734</a></dt>

<dd>POP3 AUTHentication command</dd>

<dt><a
href="ftp://ftp.isi.edu/in-notes/rfc1870.txt">RFC1870</a></dt>

<dd>SMTP Service Extension for Message Size Declaration</dd>

<dt><a
href="ftp://ftp.isi.edu/in-notes/rfc1891.txt">RFC1891</a></dt>

<dd>SMTP Service Extension for Delivery Status Notifications</dd>

<dt><a
href="ftp://ftp.isi.edu/in-notes/rfc1892.txt">RFC1892</a></dt>

<dd>The Multipart/Report Content Type for the Reporting of Mail
System Administrative Messages</dd>

<dt><a
href="ftp://ftp.isi.edu/in-notes/rfc1894.txt">RFC1894</a></dt>

<dd>An Extensible Message Format for Delivery Status
Notifications</dd>

<dt><a
href="ftp://ftp.isi.edu/in-notes/rfc1893.txt">RFC1893</a></dt>

<dd>Enhanced Mail System Status Codes</dd>

<dt><a
href="ftp://ftp.isi.edu/in-notes/rfc1894.txt">RFC1894</a></dt>

<dd>An Extensible Message Format for Delivery Status
Notifications</dd>

<dt><a
href="ftp://ftp.isi.edu/in-notes/rfc1938.txt">RFC1938</a></dt>

<dd>A One-Time Password System</dd>

<dt><a
href="ftp://ftp.isi.edu/in-notes/rfc1939.txt">RFC1939</a></dt>

<dd>Post Office Protocol - Version 3</dd>

<dt><a
href="ftp://ftp.isi.edu/in-notes/rfc1957.txt">RFC1957</a></dt>

<dd>Some Observations on Implementations of the Post Office
Protocol (POP3)</dd>

<dt><a
href="ftp://ftp.isi.edu/in-notes/rfc1985.txt">RFC1985</a></dt>

<dd>SMTP Service Extension for Remote Message Queue Starting</dd>

<dt><a
href="ftp://ftp.isi.edu/in-notes/rfc2033.txt">RFC2033</a></dt>

<dd>Local Mail Transfer Protocol</dd>

<dt><a
href="ftp://ftp.isi.edu/in-notes/rfc2060.txt">RFC2060</a></dt>

<dd>Internet Message Access Protocol - Version 4rev1</dd>

<dt><a
href="ftp://ftp.isi.edu/in-notes/rfc2061.txt">RFC2061</a></dt>

<dd>IMAP4 Compatibility With IMAP2bis</dd>

<dt><a
href="ftp://ftp.isi.edu/in-notes/rfc2062.txt">RFC2062</a></dt>

<dd>Internet Message Access Protocol - Obsolete Syntax</dd>

<dt><a
href="ftp://ftp.isi.edu/in-notes/rfc2195.txt">RFC2195</a></dt>

<dd>IMAP/POP AUTHorize Extension for Simple Challenge/Response</dd>

<dt><a
href="ftp://ftp.isi.edu/in-notes/rfc2177.txt">RFC2177</a></dt>

<dd>IMAP IDLE command</dd>

<dt><a
href="ftp://ftp.isi.edu/in-notes/rfc2449.txt">RFC2449</a></dt>

<dd>POP3 Extension Mechanism</dd>

<dt><a
href="ftp://ftp.isi.edu/in-notes/rfc2554.txt">RFC2554</a></dt>

<dd>SMTP Service Extension for Authentication</dd>

<dt><a
href="ftp://ftp.isi.edu/in-notes/rfc2595.txt">RFC2595</a></dt>

<dd>Using TLS with IMAP, POP3 and ACAP</dd>

<dt><a
href="ftp://ftp.isi.edu/in-notes/rfc2645.txt">RFC2645</a></dt>

<dd>On-Demand Mail Relay: SMTP with Dynamic IP Addresses</dd>

<dt><a
href="ftp://ftp.isi.edu/in-notes/rfc2683.txt">RFC2683</a></dt>

<dd>IMAP4 Implementation Recommendations</dd>

<dt><a
href="ftp://ftp.isi.edu/in-notes/rfc2821.txt">RFC2821</a></dt>

<dd>Simple Mail Transfer Protocol</dd>

<dt><a
href="ftp://ftp.isi.edu/in-notes/rfc2822.txt">RFC2822</a></dt>

<dd>Internet Message Format</dd>
</dl>

<!--
RFC2192 IMAP URL Scheme
RFC2193 IMAP4 Mailbox Referrals
RFC2221 IMAP4 Login Referrals
-->

<h1>Other useful documents</h1>

<dl>
<dt><a
href="http://www.faqs.org/faqs/LANs/mail-protocols/">http://www.faqs.org/faqs/LANs/mail-protocols/</a></dt>

<dd>LAN Mail Protocols Summary</dd>
</dl>

<hr />
<table width="100%" cellpadding="0" summary="Canned page footer">
<tr>
<td width="30%">Back to <a href="index.html">Fetchmail Home Page</a></td>
<td width="30%" align="right">$Date$</td>
</tr>
</table>

<br clear="left" />
<address>Eric S. Raymond <a
href="mailto:esr@thyrsus.com">&lt;esr@snark.thyrsus.com&gt;</a></address>
</body>
</html>