Fetchmail Release Notes ======================= This file is in Unicode charset with UTF-8 encoding. All dates are in Universal Time unless otherwise noted. (The `lines' figures total .c, .h, .l, and .y files under version control. Abbreviations in parentheses are the maintainers who committed the respective change. MA = Matthias Andree, ESR = Eric S. Raymond, RF = Rob Funk.) # ADVANCE WARNING OF FEATURES TO BE REMOVED OR CHANGED IN FUTURE VERSIONS (There are no plans to remove features from a 6.3.X release, but they may be removed from a 6.4.0 or newer release.) * The MX and host alias DNS lookups that fetchmail performs in multidrop mode are based on assumptions that are rarely met in practice, somewhat defective, deprecated and may be removed from a future fetchmail version. They have never supported IPv6 (including IPv6-mapped IPv4). Non-DNS based alias keywords such as "aka" will remain in fetchmail. * The monitor and interface options may be removed from a future fetchmail version as they are not reasonably portable across operating systems. * POP2 is obsolete, support will be removed from a future fetchmail version. * IMAP2 and IMAP4 (not IMAP4r1) are obsolete, support may be removed from a future fetchmail version. * RPOP is obsolete, support will be removed from a future fetchmail release. * --sslcertck will become a default setting in a future fetchmail version. * The multidrop To/Cc guessing code along with the fragile duplicate suppressor is deprecated and may be removed from a future release. * The "envelope Received" option may be removed from a future release, because the Received header was never meant to be machine-readable, the format varies widely, and various other differences in behavior make parsing Received an unreliable undertaking. The envelope option as such will remain though, in order to support Delivered-To, X-Envelope-To, X-Original-To and similar. See also . * The --enable-fallback (fall back to MDA if MTA unavailable) will be removed from a future fetchmail release, because it makes fetchmail's behavior inconsistent and confusing. * The "protocol auto" default inside fetchmail may be removed from a future fetchmail release. Explicit configuration of the protocol is recommended. * Kerberos IV support may be removed from a future fetchmail release. * Kerberos 5 support may be removed from a future fetchmail release. * The --principal option may be removed from a future fetchmail release. * SIGHUP wakeup support may be removed from a future fetchmail release and cause fetchmail to terminate - it was broken for many years. * Support for operating systems that are not sufficiently POSIX compliant may be removed or operation on such systems may be suboptimal for future releases. This means that fetchmail may only continue to work on C99 and POSIX 2001 based systems. * The maintainer may migrate fetchmail to C++ with STL or C#, and impose further requirements (dependencies), such as Boost or other class libraries. * The softbounce option default will change to "false" in the next release. * The --bsmtp - mode of operation may be removed in a future release. * Given that OpenSSL is severely underdocumented, and needs license exceptions, fetchmail may switch to a different SSL library. * SSLv2 support will be removed from a future fetchmail release. It has been obsolete for more than a decade. -------------------------------------------------------------------------------- fetchmail-6.3.20 (not yet released): # CHANGES * fetchmail no longer supports SSL v2, nor the corresponding SSL2 option to --sslproto. SSLv2 is insecure and had been deprecated 15 years ago. fetchmail will actively forbid SSLv2 negotiation by means of SSL_OP_NO_SSLv2. To fix Debian Bug#622054. * fetchmail now always uses its own MD5 implementation. The library and header variants are too diverse, and we've been bitten before -- and configure complains noisily on Cyrus-SASL's RFC1321 md5.h. * fetchmail now supports an environment variable to suppress marking deleted messages as seen at the same time, FETCHMAIL_IMAP_DELETED_REMAINS_UNSEEN. See the manual page for details. Requested by Jonathan Buschmann. * fetchmail sets Internet domain sockets to "keepalive" mode now. Note that there is no portable way to configure actual timeouts for this mode, and some systems only support a system-wide timeout setting. # BUG FIXES * Call strlen() only once when removing CRLF from a line. (Sunil Shetye) * Do not search for UNSEEN messages in ranges. Usually, there are very few new messages and most of the range searches result in nothing. Instead, split the long response to make the IMAP driver think that there are multiple lines of response. (Sunil Shetye) * Do not print "skipping message" for old messages even in verbose mode. If there are too many old messages, the logs just get filled without any real activity. (Sunil Shetye) (suggested by Yunfan Jiang) # TRANSLATION UPDATES [ja] Japanese (Takeshi Hamasaki) fetchmail-6.3.19 (released 2010-12-10, 25945 LoC): # ERRATUM NOTICE ISSUED * fetchmail 6.3.18 contains several bug fixes that were considered sufficiently grave to warrant the issue of an erratum notice, fetchmail-EN-2010-03.txt. # BUG FIXES * When specifying multiple local multidrop lists, do not lose wildcard flag. (Affects "user foo is bar baz * is joe here") * In multidrop configurations, an asterisk can now appear anywhere in the list of local users, not just at the end. * In multidrop mode, header parsing is now more verbose in -vv mode, so that it becomes possible to see which header is used. * Make --antispam work from command line (these used to work in rcfiles). Reported by Kees Bakker, BerliOS Bug #17599. (Sunil Shetye) * Smoke test XHTML 1.1 validation, and if it fails, skip validating HTML documents. Skip validating Mailbox-Names-UTF7.html. Several systems have broken XHTML 1.1 DTD installations that jeopardize the build. Reported by Mihail Nechkin against FreeBSD port. Workaround for 6.3.18: build in a separate directory, i. e: mkdir build && cd build && ../configure --options-go-here * Send a NOOP only after a failed STARTTLS in IMAP. (Sunil Shetye) * Demote GSSAPI verbose/debug syslog to INFO severity. Requested by Carlos E. R. and Derek Simkowiak via the fetchmail-users@ mailing list. * Do STARTTLS/STLS negotiation in IMAP/POP3 if it is mandatory even if the server capabilities do not show support for upgradation to TLS. To use this, configure --sslproto tls1. (Sunil Shetye) * IMAP: Understand empty strings as FETCH response, seen on Yahoo. Reported by Yasin Malli to fetchmail-users@ 2010-12-10. Note that fetchmail continues to expect literals as FETCH response for now. # DOCUMENTATION * The manual page now links to IANA for GSSAPI service names. # TRANSLATION UPDATES [cs] Czech (Petr Pisar) [fr] French (Frédéric Marchal) [de] German [it] Italian (Vincenzo Campanella) [pl] Polish (Jakub Bogusz) # KNOWN BUGS AND WORKAROUNDS (this section floats upwards through the NEWS file so it stays with the current release information - however, it was stuck with 6.3.8 for a while) * fetchmail does not handle messages without Message-ID header well (See sourceforge.net bug #780933) * BSMTP is mostly untested and errors can cause corrupt output. * Sun Workshop 6 (SPARC) is known to miscompile the configuration file lexer in 64-bit mode. Either compile 32-bit code or use GCC to compile 64-bit fetchmail. Note that fetchmail doesn't take advantage of 64-bit code, so compiling 32-bit SPARC code should not cause any difficulties. * fetchmail does not track pending deletes over crashes. * the command line interface is sometimes a bit stubborn, for instance, fetchmail -s doesn't work with a daemon running. * Linux systems may return duplicates of an IP address in some circumstances if no or no global IPv6 addresses are configured. (No workaround. Ubuntu Bug#582585, Novell Bug#606980.) * Kerberos 5 may be broken, particularly on Heimdal, and provide bogus error messages. This will not be fixed, because the maintainer has no Kerberos 5 server to test against. Use GSSAPI. fetchmail-6.3.18 (released 2010-10-09, 25936 LoC): # SECURITY IMPROVEMENTS TO DEFANG X.509 CERTIFICATE ABUSE * Fetchmail now only accepts wildcard certificate common names and subject alternative names if they start with "*.". Previous versions would accept wildcards even if no period followed immediately. * Fetchmail now disallows wildcards in certificates to match domain literals (such as 10.9.8.7), or wildcards in domain literals ("*.168.23.23"). The test is overly picky and triggers if the pattern (after skipping the initial wildcard "*") or domain consists solely of digits and dots, and thus matches more than needed. * Fetchmail now disallows wildcarding top-level domains. # CRITICAL BUG FIXES AND REGRESSION FIXES * Fetchmail 6.3.15, 6.3.16, and 6.3.17 would pick up libmd5 to obtain MD5* functions, as an effect of an undocumented Solaris MD5 fix. This caused all MD5-related functions to malfunction if, for instance, libmd5.so was installed on other operating systems as part of libwww on machines where long isn't 32-bits, i. e. usually on 64-bit computers. Fixes Gentoo Bug #319283, reported, including libwww hint, by Karl Hakimian. Side effect: fetchmail will now use -lmd on Solaris rather than -lmd5. * Fetchmail 6.3.17 warned about insecure SSL/TLS connections even if a matching --sslfingerprint was specified. This is an omission from an SSL usability change made in 6.3.17. Fixes Debian Bug#580796 reported by Roland Stigge. * Fetchmail will now apply timeouts to the authentication stage. This stage encompasses STARTTLS/STLS negotiation in IMAP/POP3. Reported missing by Thomas Jarosch. * Fetchmail now cancels GSSAPI authentication properly when encountering GSS errors, such as no or unsuitable credentials. It now sends an asterisk on a line by its own, as required in SASL. This fixes protocol synchronization issues that cause Authentication failures, often observed with kerberized MS Exchange servers. Fixes Debian Bug #568455 reported by Patrick Rynhart, and Alan Murrell, to the fetchmail-users list. Fix verified by Thomas Voigtmann and Patrick Rynhart. # BUG FIXES * Fetchmail will no longer print connection attempts and errors for one host in "silent" and "normal" logging modes, unless all connections fail. This should reduce irritation around refused-connection logging if services are only on an IPv4 socket if the host also supports IPv6. Often observed as connections refused to ::1/25 when the subsequent connection to 127.0.0.1/25 then - silently - succeeds. Fetchmail, unless in verbose mode, will collect all connect errors and only report them if all of them fail. * Fetchmail will not try GSSAPI authentication automatically, unless it has GSS credentials. However, if GSSAPI authentication is requested explicitly, fetchmail will always try it. * Fetchmail now parses response to "FETCH n:m RFC822.SIZE" and "FETCH n RFC822.HEADER" in a more flexible manner. (Sunil Shetye) * The manual page clearly states that --principal is for Kerberos 4 only, not for Kerberos 5 or GSSAPI. Found by Thomas Voigtmann. # CHANGES * When encountering incorrect headers, fetchmail will refer to the bad-header option in the manpage. Fixes BerliOS Bug #17272, change suggested by Björn Voigt. * Fetchmail now decodes and reports GSSAPI status codes upon errors. * Fetchmail now autoprobes NTLM also for POP3. * The Fetchmail FAQ has a new item #R15 on authentication failures. # INTERNAL CHANGES * The common NTLM authentication code was factored out from pop3.c and imap.c. # TRANSLATION UPDATES [zh_CN] Chinese/simplified (Ji Zheng-Yu) [cs] Czech (Petr Pisar) [nl] Dutch (Erwin Poeze) [fr] French (Frédéric Marchal) [de] German [it] Italian (Vincenzo Campanella) [ja] Japanese (Takeshi Hamasaki) [pl] Polish (Jakub Bogusz) [sk] Slovak (Marcel Telka) fetchmail-6.3.17 (released 2010-05-06, 25767 LoC): # SECURITY FIX * CVE-2010-1167: Fetchmail before release 6.3.17 did not properly sanitize external input (mail headers and UID). When a multi-character locale (such as UTF-8) was in use, this could cause memory exhaustion and thus a denial of service, because fetchmail's report.c functions assumed that non-success of [v]snprintf was due to insufficient buffer size allocation. It would then repeatedly reallocate a larger buffer and fail formatting again. See fetchmail-SA-2010-02.txt. # FEATURES * Fetchmail now supports a --sslcertfile option to specify a "CA bundle" file (a file that contains trusted CA certificates). Since these bundled CA files do not require c_rehash to be run, they are easier to use and immune to OpenSSL library updates that affect the hash function. * Fetchmail now supports a FETCHMAIL_INCLUDE_DEFAULT_X509_CA_CERTS environment variable to force loading the default SSL CA certificate locations even if --sslcertfile or --sslcertpath is used. If neither option is in effect, fetchmail loads the default locations. # REGRESSION FIX * Fix string handling in rcfile scanner, which caused fetchmail to misparse a run control file in certain circumstances. Fixes BerliOS bug #14257. Patch by Michael Banack. This fixes a regression introduced before 6.3.0. # BUG FIXES * Plug memory leak when using a "defaults" entry in the run control file. * Do not print SSL certificate mismatches unless verbose or --sslcertck is enabled. * Do not lose "set invisible" in fetchmailconf. (Michael Barnack) # CHANGES * Usability: SSL certificate chains are fully printed in -v -v mode, and there are now helpful pointers to --sslcertpath and c_rehash for "unable to get local issuer certificate" and self-signed certificates -- these usually hint to missing root signing CAs in the certs directory. * Several fixes for compiler (GCC, Intel C++, CLang) and autotools warnings * Memory allocation failures will now cause abnormal program abort (SIGABRT), no longer an exit with unspecified code. * Print a warning if certificate verification failed and the user did not specify --sslcertck. # DOCUMENTATION * Fix table of global option to read "set softbounce" where there used to be a 2nd copy of "set spambounce". Patch by Michael Banack, BerliOS Bug #17067. * In the --sslcertpath description, mention that OpenSSL upgrade (and a 0.9.X to 1.0.0 upgrade in particular) may require running c_rehash. # TRANSLATION UPDATES [zh_CN] Chinese/simplified (Ji Zheng-Yu) [cs] Czech (Petr Pisar) [nl] Dutch (Erwin Poeze) [fr] French (Frédéric Marchal) [de] German [id] Indonesian (Andhika Padmawan) [it] Italian (Vincenzo Campanella) [ja] Japanese (Takeshi Hamasaki) [pl] Polish (Jakub Bogusz) [sk] Slovak (Marcel Telka) [vi] Vietnamese (Clytie Siddall) fetchmail-6.3.16 (released 2010-04-06, 25574 LoC): # BUG FIX * Fix --interface option, broken in 6.3.15. Reported by Vladmimir Stavrinov. Fixes Debian Bug #576717. # CHANGE * Call OpenSSL_add_all_algorithms(). This is needed to support non-mandatory and non-standard algorithms in certificates. Sjoerd Simons, to fix Debian Bug #576430. OpenSSL 0.9.8* does not load - for instance - the SHA256 digest by default. Reported as OpenSSL RT#2224. fetchmail-6.3.15 (released 2010-03-28, 25572 LoC): # FEATURE * Fetchmail now supports a bad-header command line or rcfile option that takes exactly one argument, accept or reject (default). This specifies how messages with bad headers retrieved from the current server are to be treated. # BUG FIXES * In the rcfile, recognize "local" as abbreviation for "localdomains", as documented. The short form has not ever worked since this feature was added in January 1997. Reported by Frédéric Marchal. * Do not close stdout when using mda and "bsmtp -" at the same time. * Log operating system errors when BSMTP writes fail. * Fix verbose mode progress formatting regression from 6.3.10; SMTP trace lines were no longer on a line of their own. Reported by Melchior Franz. * Check seteuid() return value and abort running MDA if switch fails. * Set global flags in a consistent manner. Make --nosoftbounce and --nobounce work from command line (these used to work in rcfiles). Reported and fix confirmed working by N.J. Mann. (Sunil Shetye) * Properly import h_errno declarations, even on systems where h_errno isn't a macro. (Adds ./configure check, fixes Cygwin dllimport warnings.) # CHANGES * The repository has been converted and moved from the Subversion (SVN) format kindly hosted by Graham Wilson over the past years to Git format hosted on Gitorious.org. My deepest thanks to Graham Wilson for this service that kept us going when BerliOS's Subversion service was faulty in its early days. * This opportunity was used to convert BRANCH_6-2 and BRANCH_1-9-9 to GnuPG-signed tags, as a sign that these are now closed. * The outdated SVN trunk is now called "oldtrunk" in Git just to save the work for future reference. All development in the past few years was on BRANCH_6-3. * master was branched from BRANCH_6-3. BRANCH_6-3 is now obsolete (and in fact was also converted to a tag to record where the conversion from SVN to Git took place). * "make check" now skips HTML validation if xmllint or XHTML DTD are missing. # DOCUMENTATION * Web site and documentation were adjusted to reflect the SVN->Git move. * The fetchmail manual page is now much clearer on the user id switching (seteuid) when using --mda while running as the super user. # TRANSLATION UPDATES, by language name * [zh_CN] Chinese (Simplified), by Ji Zheng-Yu * [cs] Czech, by Petr Pisar * [nl] Dutch, by Erwin Poeze * [fr] French, by Frédéric Marchal * [de] German * [id] Indonesian, by Andhika Padmawan * [it] Italian, by Vincenzo Campanella * [ja] Japanese, by Takeshi Hamasaki * [pl] Polish, by Jakub Bogusz * [vi] Vietnamese, by Clytie Siddall fetchmail 6.3.14 (released 2010-02-05, 25487 LoC): # SECURITY FIXES * CVE-2010-0562: SSL/TLS certificate information is now also reported properly on computers that consider the "char" type signed. Fixes malloc() buffer overrun. Workaround for older versions: do not use verbose mode. See fetchmail-SA-2010-01.txt for details, including a minimal patch. # BUG FIXES * The IMAP client no longer skips messages from several IMAP servers including Dovecot if fetchmail's "idle" is in use. Causes were that fetchmail (a) ignored some untagged responses when it should not (b) relied on EXISTS messages in response to EXPUNGE, which aren't mandated by RFC-3501 (the IMAP standard) and aren't sent by Dovecot either. Fix by Sunil Shetye (the fix also consolidates IMAP response handling, improving overall robustness of the IMAP client), bug report and testing by Matt Doran, with further hints from Timo Sirainen. * The SMTP client now recovers from errors (such as servers dropping the connection after errors) when sending an RSET command. Fix by Sunil Shetye. Report by James Moe. * The IMAP client now uses "SEARCH UNSEEN" rather than "SEARCH UNSEEN NOT DELETED" again on IMAP2, to fix a regression in fetchmail 6.2.5 reported by Will Stringer in June 2004. (Sunil Shetye) * The IMAP client now uses "SEARCH UNSEEN UNDELETED" on IMAP4 and IMAP4r1 servers (Sunil Shetye). * Workaround: The IMAP client now falls back to "FETCH n:m FLAGS" if the server does not support "SEARCH". (Sunil Shetye) * The IMAP client now requests message numbers in batches of 1,000 to avoid problems if there are more than 1860 unseen messages. (Sunil Shetye) Note that this wasn't security relevant because fetchmail would only read up to the maximum buffer size and leave the remainder of the string unread, going out of synch afterwards. * Stricter validation of IMAP responses containing byte or message counts. # CHANGES * Only include gssapi.h if we're not including gssapi/gssapi.h, to fix a FreeBSD compiler warning about gssapi.h being obsolete. # DOCUMENTATION * The README.SSL document was revised for grammar, spelling, and clarity. Courtesy of Robert Mullin. # TRANSLATION UPDATES * [it] Italian, by Vincenzo Campanella fetchmail 6.3.13 (released 2009-10-30, 25333 LoC): # REGRESSION FIXES * The multiline SMTP error fix in release 6.3.12 caused fetchmail to lose message codes 400..599 and treat all of these as temporary error. This would cause messages to be left on the server even if softbounce was turned off. Reported by Thomas Jarosch. # TRANSLATION UPDATES * [cs] Czech, by Petr Pisar * [zh_CN] Chinese (simplified), by Ji ZhengYu * [nl] Dutch, by Erwin Poeze * [id] Indonesian, by Andhika Padmawan * [ja] Japanese, by Takeshi Hamasaki * [pl] Polish, by Jakub Bogusz * [es] Spanish (Castilian), by Franciso Molinero * [vi] Vietnamese, by Clytie Siddall fetchmail 6.3.12 (released 2009-10-05): # REGRESSION FIXES * The CVE-2009-2666 fix in fetchmail release 6.3.11 caused a free() of unallocated memory on SSL connections, which caused crashes or program aborts on some systems (depending on how initialization and free() of unallocated memory is handled in compiler and libc). Workaround for older versions: run in verbose mode. Patch courtesy of Thomas Heinz, fixes Gentoo Bug #280760. This regression affected only the 6.3.11 release, but not the patch that was part of the security announcement fetchmail-SA-2009-01. # BUG FIXES * Fix error reporting for GSSAPI on Heimdal (h5l) Kerberos. * Look for MD5_Init in libcrypto rather than libssl, fixes Gentoo Kerberos builds; fixes upstream parts of Gentoo Bugs #231400 and #185652, and fixes BerliOS Bug #16134. * Report multiline SMTP errors properly, reported by Earl Chew; fixes Debian Bug #529899, reported by Akihiro Terasaki. Note: This fix introduced a regression, fixed in 6.3.13. * Replace control characters in SMTP replies by '?'. * Fetchmailconf: Fix descriptions for smtpaddress and smtpname options; smtpaddress is for RCPT TO, not MAIL FROM. Found by Gerard Seibert. # TRANSLATION UPDATES AND ADDITIONS (ordered by language name): * [ca] Catalan (Ernest Adrogué Calveras) * [zh_CN] Chinese/Simplified (Ji ZhengYu) * [cs] Czech (Petr Pisar) * [ja] Japanese (Takeshi Hamasaki) * [pl] Polish (Jakub Bogusz) * [es] Spanish/Castilian (Francisco Molinero) * [vi] Vietnamese (Clytie Siddall) fetchmail 6.3.11 (released 2009-08-06): # SECURITY BUGFIXES * CVE-2009-2666: SSL NUL prefix impersonation attack through NULs in a part of a X.509 certificate's CommonName and subjectAltName fields. These fields use opaque strings with a separate length field, so that the NUL character isn't a special character inside the certificate. Fetchmail, being written in the C language, used to treat these strings as C strings nonetheless, so that the domain comparison would end at the first embedded NUL character, rather than at the real end of the string. Fetchmail will now abort certificate verification as failed if NULs are encountered inside either of these fields regardless of their position, and drop the connection even if --sslcertck is not used, because NUL is not a valid character in legitimate DNS names. See fetchmail-SA-2009-01.txt for details, including a minimal patch. # BUGFIXES * Remove the spurious message "message delimiter found while scanning headers". RFC-5322 syntax states that the delimiter is part of the body, and the body is optional. * Convert all non-printable characters in certificate Subject/Issuer Common Name or Subject Alternative Name fields to ANSI-C hex escapes (\xnn, where nn are hex digits). Note that this change introduces a regression, fixed in 6.3.12. See the 6.3.12 documentation above for details and a workaround. # TRANSLATION UPDATES AND ADDITIONS (ordered by language name): * [zh_CN] Chinese/Simplified (Ji ZhengYu) * [es] Spanish/Castilian (Francisco Molinero) fetchmail 6.3.10 (released 2009-07-02): # INCOMPATIBLE BUGFIXES AND CHANGES * Fetchmail no longer drops permanently undelivered messages by default, to match historic documentation. It does this by adding a new "softbounce" option, see below. Fixes Debian Bug#471283, demotes Debian Bug#494418 to wishlist. * There is a new "softbounce" global option that prevents the deletion of messages that have not been forwarded. It defaults to "true" for fetchmail 6.3.X in order to match historic documentation. This may change its default in the next major release. # BUGFIXES * Fix misuse of canonical autoconf target as _TARGET when it should have been _HOST. Report and patch courtesy of Diego E. "Flameeyes" Pettenò. Details: http://blog.flameeyes.eu/2009/01/01/the-canonical-target * Do not lose PS_MAXFETCH (13) exit status when hitting maxpoll. Reported by Michelle Konzack, Debian Bug#508667. * Do not overlap source and destination fields in snprintf() in interface.c. Courtesy of Nico Golde, Debian. * When a pre- or post-connect command fails, now report the exit status or termination signal properly through sys/wait.h macros. * When acquiring a body, understand NIL ("no such data item"), as returned by some MS Exchange versions. Fixes BerliOS Bug #11980 by KB Sriram. * Make progress tickers (-v/--showdots) consistent, and update documentation accordingly ("." for each 1024 octets read, "#" for a header written, and "*" for each body line written.) The conditions under which these had been printed were inconsistent, illogical, and documentation hadn't matched real behaviour for long. * For NTLM authentication, use dynamically allocated buffers. Fixes Debian Bug#449179, reported by Stepan Golosunov. * Non-delivery notice ("bounce mail") now mentions the original reason again, before the address list. This fixes a regression introduced in 6.3.0. * Several compiler warnings were fixed. * The minimum recommended SMTP (RFC-5321) timeouts are enforced to leave sufficient time for the listener to respond. Some synchronous listeners, particularly when used with spam filtering and other policy enforcement services, take extended amounts of time to process messages after the sender, recipient, or data block and EOM line. This can cause fetchmail to not wait long enough for the "250 Ok" and make fetchmail believe the message wasn't properly delivered when in fact it was; fetchmail would then retry the download next time and never make progress. Fixes Berlios Bug #10972, reported by Viktor Binzberger. * The ESMTP/LMTP client will now apply an application-specific timeout while waiting for the EHLO/LHLO response, rather than wait for the server or TCP connection timeout. * Treat 530 errors as temporary, so as not to delete messages on configuration errors. Partially taken from Petr Cerny's patch in Novell Bugzilla #246829. The 501 part of said patch was not added, as the maintainer is not convinced 501 is a temporary condition, and softbounce takes care of this anyways. # CHANGES * Make the comparison of the SSL fingerprints case insensitive, to ease its use. Suggested by Daniel Richard G. * Proper precedence ordering for the syslog and logfile options. If the logfile option is effective (i. e. we're in daemon mode and nodetach isn't used), reset the syslog option. If logfile is ineffective (we're not in daemon mode, or nodetach is set), syslog takes precedence. * The sleeping at/awakened at messages appear in logfiles and syslog only if verbose mode is enabled. On the console, they will still appear without verbose mode. Fixes Debian Bug#282259. * fetchmail only requests IPv6 addresses via name service if at least one is configured on the local host, likewise for IPv4. (AI_ADDRCONFIG flag to getaddrinfo()) Extended version of Redhat's patch. * If the server name contains "yahoo.com", offers the "ID" capability, and we're polling via IMAP, send an ID ("guid" "1") transaction first, ignoring its result. This appears needed to be able to log into Yahoo's Zimbra servers, but there are open issues (such as being only able to download one message and server certificate mismatches). # CHANGES TO CONTRIB * Fix bashism in contrib/fetchsetup. Fixes Debian Bug#530081. # DOCUMENTATION * Some parts of the the manual page were
<?xml version="1.0" encoding="ISO-8859-1"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
    "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
    <title>Eric S. Raymond's former Design notes on fetchmail</title>
<link rev="made" href="mailto:esr@snark.thyrsus.com" />
<meta name="description" content="Design notes on fetchmail." />
<meta name="keywords" content="fetchmail, POP, POP2, POP3, IMAP, remote mail" />
<style type="text/css">
/*<![CDATA[*/
 h1.c1 {text-align: center}
/*]]>*/
</style>
</head>
<body>
<table width="100%" cellpadding="0" summary="Canned page header">
<tr>
<td width="30%">Forward to <a href="design-notes.html">Updated design
	notes</a></td>
<td width="30%">Back to <a href="index.html">Fetchmail Home Page</a></td>
<td width="30%" align="right">$Date$</td>
</tr>
</table>

<hr />
<h1 class="c1">Eric S. Raymond's former Design Notes On Fetchmail</h1>

<p>These notes are for the benefit of future hackers and
maintainers. The following sections are both functional and
narrative, read from beginning to end.</p>

<h1>History</h1>

<p>A direct ancestor of the fetchmail program was originally
authored (under the name popclient) by Carl Harris
&lt;ceharris@mal.com&gt;. I took over development in June 1996 and
subsequently renamed the program `fetchmail' to reflect the
addition of IMAP support and SMTP delivery. In early November 1996
Carl officially ended support for the last popclient versions.</p>

<p>Before accepting responsibility for the popclient sources from
Carl, I had investigated and used and tinkered with every other
UNIX remote-mail forwarder I could find, including fetchpop1.9,
PopTart-0.9.3, get-mail, gwpop, pimp-1.0, pop-perl5-1.2, popc,
popmail-1.6 and upop. My major goal was to get a header-rewrite
feature like fetchmail's working so I wouldn't have reply problems
anymore.</p>

<p>Despite having done a good bit of work on fetchpop1.9, when I
found popclient I quickly concluded that it offered the solidest
base for future development. I was convinced of this primarily by
the presence of multiple-protocol support. The competition didn't
do POP2/RPOP/APOP, and I was already having vague thoughts of maybe
adding IMAP. (This would advance two other goals: learn IMAP and
get comfortable writing TCP/IP client software.)</p>

<p>Until popclient 3.05 I was simply following out the implications
of Carl's basic design. He already had daemon.c in the
distribution, and I wanted daemon mode almost as badly as I wanted
the header rewrite feature. The other things I added were bug fixes
or minor extensions.</p>

<p>After 3.1, when I put in SMTP-forwarding support (more about
this below) the nature of the project changed -- it became a
carefully-thought-out attempt to render obsolete every other
program in its class. The name change quickly followed.</p>

<h1>The rewrite option</h1>

<p>MTAs ought to canonicalize the addresses of outgoing non-local
mail so that From:, To:, Cc:, Bcc: and other address headers
contain only fully qualified domain names. Failure to do so can
break the reply function on many mailers. (Sendmail has an option
to do this.)</p>

<p>This problem only becomes obvious when a reply is generated on a
machine different from where the message was delivered. The two
machines will have different local username spaces, potentially
leading to misrouted mail.</p>

<p>Most MTAs (and sendmail in particular) do not canonicalize
address headers in this way (violating RFC 1123). Fetchmail
therefore has to do it. This is the first feature I added to the
ancestral popclient.</p>

<h1>Reorganization</h1>

<p>The second thing I did reorganize and simplify popclient a lot.
Carl Harris's implementation was very sound, but exhibited a kind
of unnecessary complexity common to many C programmers. He treated
the code as central and the data structures as support for the
code. As a result, the code was beautiful but the data structure
design ad-hoc and rather ugly (at least to this old LISP
hacker).</p>

<p>I was able to improve matters significantly by reorganizing most
of the program around the `query' data structure and eliminating a
bunch of global context. This especially simplified the main
sequence in fetchmail.c and was critical in enabling the daemon
mode changes.</p>

<h1>IMAP support and the method table</h1>

<p>The next step was IMAP support. I initially wrote the IMAP code
as a generic query driver and a method table. The idea was to have
all the protocol-independent setup logic and flow of control in the
driver, and the protocol-specific stuff in the method table.</p>

<p>Once this worked, I rewrote the POP3 code to use the same
organization. The POP2 code kept its own driver for a couple more
releases, until I found sources of a POP2 server to test against
(the breed seems to be nearly extinct).</p>

<p>The purpose of this reorganization, of course, is to trivialize
the development of support for future protocols as much as
possible. All mail-retrieval protocols have to have pretty similar
logical design by the nature of the task. By abstracting out that
common logic and its interface to the rest of the program, both the
common and protocol-specific parts become easier to understand.</p>

<p>Furthermore, many kinds of new features can instantly be
supported across all protocols by modifying the one driver
module.</p>

<h1>Implications of smtp forwarding</h1>

<p>The direction of the project changed radically when Harry
Hochheiser sent me his scratch code for forwarding fetched mail to
the SMTP port. I realized almost immediately that a reliable
implementation of this feature would make all the other delivery
modes obsolete.</p>

<p>Why mess with all the complexity of configuring an MDA or
setting up lock-and-append on a mailbox when port 25 is guaranteed
to be there on any platform with TCP/IP support in the first place?
Especially when this means retrieved mail is guaranteed to look
like normal sender- initiated SMTP mail, which is really what we
want anyway.</p>

<p>Clearly, the right thing to do was (1) hack SMTP forwarding
support into the generic driver, (2) make it the default mode, and
(3) eventually throw out all the other delivery modes.</p>

<p>I hesitated over step 3 for some time, fearing to upset
long-time popclient users dependent on the alternate delivery
mechanisms. In theory, they could immediately switch to .forward
files or their non-sendmail equivalents to get the same effects. In
practice the transition might have been messy.</p>

<p>But when I did it (see the NEWS note on the great options
massacre) the benefits proved huge. The cruftiest parts of the
driver code vanished. Configuration got radically simpler -- no
more grovelling around for the system MDA and user's mailbox, no
more worries about whether the underlying OS supports file
locking.</p>

<p>Also, the only way to lose mail vanished. If you specified
localfolder and the disk got full, your mail got lost. This can't
happen with SMTP forwarding because your SMTP listener won't return
OK unless the message can be spooled or processed.</p>

<p>Also, performance improved (though not so you'd notice it in a
single run). Another not insignificant benefit of this change was
that the manual page got a lot simpler.</p>

<p>Later, I had to bring --mda back in order to allow handling of
some obscure situations involving dynamic SLIP. But I found a much
simpler way to do it.</p>

<p>The moral? Don't hesitate to throw away superannuated features
when you can do it without loss of effectiveness. I tanked a couple
I'd added myself and have no regrets at all. As Saint-Exupery said,
"Perfection [in design] is achieved not when there is nothing more
to add, but rather when there is nothing more to take away." This
program isn't perfect, but it's trying.</p>

<h1>The most-requested features that I will never add, and why
not:</h1>

<h2>Password encryption in .fetchmailrc</h2>

<p>The reason there's no facility to store passwords encrypted in
the .fetchmailrc file is because this doesn't actually add
protection.</p>

<p>Anyone who's acquired the 0600 permissions needed to read your
.fetchmailrc file will be able to run fetchmail as you anyway --
and if it's your password they're after, they'd be able to rip the
necessary decoder out of the fetchmail code itself to get it.</p>

<p>All .fetchmailrc encryption would do is give a false sense of
security to people who don't think very hard.</p>

<h2>Truly concurrent queries to multiple hosts</h2>

<p>Occasionally I get a request for this on "efficiency" grounds.
These people aren't thinking either. True concurrency would do
nothing to lessen fetchmail's total IP volume. The best it could
possibly do is change the usage profile to shorten the duration of
the active part of a poll cycle at the cost of increasing its
demand on IP volume per unit time.</p>

<p>If one could thread the protocol code so that fetchmail didn't
block on waiting for a protocol response, but rather switched to
trying to process another host query, one might get an efficiency
gain (close to constant loading at the single-host level).</p>

<p>Fortunately, I've only seldom seen a server that incurred
significant wait time on an individual response. I judge the gain
from this not worth the hideous complexity increase it would
require in the code.</p>

<h2>Multiple concurrent instances of fetchmail</h2>

<p>Fetchmail locking is on a per-invoking-user because
finer-grained locks would be really hard to implement in a portable
way. The problem is that you don't want two fetchmails querying the
same site for the same remote user at the same time.</p>

<p>To handle this optimally, multiple fetchmails would have to
associate a system-wide semaphore with each active pair of a remote
user and host canonical address. A fetchmail would have to block
until getting this semaphore at the start of a query, and release
it at the end of a query.</p>

<p>This would be way too complicated to do just for an "it might be
nice" feature. Instead, you can run a single root fetchmail polling
for multiple users in either single-drop or multidrop mode.</p>

<p>The fundamental problem here is how an instance of fetchmail
polling host foo can assert that it's doing so in a way visible to
all other fetchmails. System V semaphores would be ideal for this
purpose, but they're not portable.</p>

<p>I've thought about this a lot and roughed up several designs.
All are complicated and fragile, with a bunch of the standard
problems (what happens if a fetchmail aborts before clearing its
semaphore, and how do we recover reliably?).</p>

<p>I'm just not satisfied that there's enough functional gain here
to pay for the large increase in complexity that adding these
semaphores would entail.</p>

<h1>Multidrop and alias handling</h1>

<p>I decided to add the multidrop support partly because some users
were clamoring for it, but mostly because I thought it would shake
bugs out of the single-drop code by forcing me to deal with
addressing in full generality. And so it proved.</p>

<p>There are two important aspects of the features for handling
multiple-drop aliases and mailing lists which future hackers should
be careful to preserve.</p>

<ol>
<li>
<p>The logic path for single-recipient mailboxes doesn't involve
header parsing or DNS lookups at all. This is important -- it means
the code for the most common case can be much simpler and more
robust.</p>
</li>

<li>
<p>The multidrop handing does <em>not</em> rely on doing the
equivalent of passing the message to sendmail -t. Instead, it
explicitly mines members of a specified set of local usernames out
of the header.</p>
</li>

<li>
<p>We do <em>not</em> attempt delivery to multidrop mailboxes in
the presence of DNS errors. Before each multidrop poll we probe DNS
to see if we have a nameserver handy. If not, the poll is skipped.
If DNS crashes during a poll, the error return from the next
nameserver lookup aborts message delivery and ends the poll. The
daemon mode will then quietly spin until DNS comes up again, at
which point it will resume delivering mail.</p>
</li>
</ol>

<p>When I designed this support, I was terrified of doing anything
that could conceivably cause a mail loop (you should be too).
That's why the code as written can only append <em>local</em> names
(never @-addresses) to the recipients list.</p>

<p>The code in mxget.c is nasty, no two ways about it. But it's
utterly necessary, there are a lot of MX pointers out there. It
really ought to be a (documented!) entry point in the bind
library.</p>

<h1>DNS error handling</h1>

<p>Fetchmail's behavior on DNS errors is to suppress forwarding and
deletion of the individual message that each occurs in, leaving it
queued on the server for retrieval on a subsequent poll. The
assumption is that DNS errors are transient, due to temporary
server outages.</p>

<p>Unfortunately this means that if a DNS error is permanent a
message can be perpetually stuck in the server mailbox. We've had a
couple bug reports of this kind due to subtle RFC822 parsing errors
in the fetchmail code that resulted in impossible things getting
passed to the DNS lookup routines.</p>

<p>Alternative ways to handle the problem: ignore DNS errors
(treating them as a non-match on the mailserver domain), or forward
messages with errors to fetchmail's invoking user in addition to
any other recipients. These would fit an assumption that DNS lookup
errors are likely to be permanent problems associated with an
address.</p>

<h1>IPv6 and IPSEC</h1>

<p>The IPv6 support patches are really more protocol-family
independence patches. Because of this, in most places, "ports"
(numbers) have been replaced with "services" (strings, that may be
digits). This allows us to run with certain protocols that use
strings as "service names" where we in the IP world think of port
numbers. Someday we'll plumb strings all over and then, if inet6 is
not enabled, do a getservbyname() down in SocketOpen. The IPv6
support patches use getaddrinfo(), which is a POSIX p1003.1g
mandated function. So, in the not too distant future, we'll zap the
ifdefs and just let autoconf check for getaddrinfo. IPv6 support
comes pretty much automatically once you have protocol family
independence.</p>

<h1>Internationalization</h1>

<p>Internationalization is handled using GNU gettext (see the file
ABOUT_NLS in the source distribution). This places some minor
constraints on the code.</p>

<p>Strings that must be subject to translation should be wrapped
with GT_() or N_() -- the former in function arguments, the latter
in static initializers and other non-function-argument
contexts.</p>

<h1>Checklist for Adding Options</h1>

<p>Adding a control option is not complicated in principle, but
there are a lot of fiddly details in the process. You'll need to do
the following minimum steps.</p>

<ul>
<li>Add a field to represent the control in <code>struct
run</code>, <code>struct query</code>, or <code>struct
hostdata</code>.</li>

<li>Go to <code>rcfile_y.y</code>. Add the token to the grammar.
Don't forget the <code>%token</code> declaration.</li>

<li>Pick an actual string to declare the option in the .fetchmailrc
file. Add the token to <code>rcfile_l</code>.</li>

<li>Pick a long-form option name, and a one-letter short option if
any are left. Go to <code>options.c</code>. Pick a new
<code>LA_</code> value. Hack the <code>longoptions</code> table to
set up the association. Hack the big switch statement to set the
option. Hack the `?' message to describe it.</li>

<li>If the default is nonzero, set it in <code>def_opts</code> near
the top of <code>load_params</code> in
<code>fetchmail.c</code>.</li>

<li>Add code to dump the option value in
<code>fetchmail.c:dump_params</code>.</li>

<li>For a per-site or per-user option, add proper
<code>FLAG_MERGE</code> actions in fetchmail.c's optmerge()
function. For a global option, add an override at the end of
load_params; this will involve copying a "cmd_run." field to a
corresponding "run." field, see the existing code for models.</li>

<li>Document the option in fetchmail.man. This will require at
least two changes; one to the collected table of options, and one
full text description of the option.</li>

<li>Hack fetchmailconf to configure it. Bump the fetchmailconf
version.</li>

<li>Hack conf.c to dump the option so we won't have a version-skew
problem.</li>

<li>Add an entry to NEWS.</li>

<li>If the option implements a new feature, add a note to the
feature list.</li>
</ul>

<p>There may be other things you have to do in the way of logic, of
course.</p>

<p>Before you implement an option, though, think hard. Is there any
way to make fetchmail automatically detect the circumstances under
which it should change its behavior? If so, don't write an option.
Just do the check!</p>

<h1>Lessons learned</h1>

<h3>1. Server-side state is essential</h3>

<p>The person(s) responsible for removing LAST from POP3 deserve to
suffer. Without it, a client has no way to know which messages in a
box have been read by other means, such as an MUA running on the
server.</p>

<p>The POP3 UID feature described in RFC1725 to replace LAST is
insufficient. The only problem it solves is tracking which messages
have been read <em>by this client</em> -- and even that requires
tricky, fragile implementation.</p>

<p>The underlying lesson is that maintaining accessible server-side
`seen' state bits associated with Status headers is indispensible
in a Unix/RFC822 mail server protocol. IMAP gets this right.</p>

<h3>2. Readable text protocol transactions are a Good Thing</h3>

<p>A nice thing about the general class of text-based protocols
that SMTP, POP2, POP3, and IMAP belongs to is that client/server
transactions are easy to watch and transaction code correspondingly
easy to debug. Given a decent layer of socket utility functions
(which Carl provided) it's easy to write protocol engines and not
hard to show that they're working correctly.</p>

<p>This is an advantage not to be despised! Because of it, this
project has been interesting and fun -- no serious or persistent
bugs, no long hours spent looking for subtle pathologies.</p>

<h3>3. IMAP is a Good Thing.</h3>

<p>Now that there is a standard IMAP equivalent of the POP3 APOP
validation in CRAM-MD5, POP3 is completely obsolete.</p>

<h3>4. SMTP is the Right Thing</h3>

<p>In retrospect it seems clear that this program (and others like
it) should have been designed to forward via SMTP from the
beginning. This lesson may be applicable to other Unix programs
that now call the local MDA/MTA as a program.</p>

<h3>5. Syntactic noise can be your friend</h3>

<p>The optional `noise' keywords in the rc file syntax started out
as a late-night experiment. The English-like syntax they allow is
considerably more readable than the traditional terse keyword-value
pairs you get when you strip them all out. I think there may be a
wider lesson here.</p>

<h1>Motivation and validation</h1>

<p>It is truly written: the best hacks start out as personal
solutions to the author's everyday problems, and spread because the
problem turns out to be typical for a large class of users. So it
was with Carl Harris and the ancestral popclient, and so with me
and fetchmail.</p>

<p>It's gratifying that fetchmail has become so popular. Until just
before 1.9 I was designing strictly to my own taste. The multi-drop
mailbox support and the new --limit option were the first features
to go in that I didn't need myself.</p>

<p>By 1.9, four months after I started hacking on popclient and a
month after the first fetchmail release, there were literally a
hundred people on the fetchmail-friends contact list. That's pretty
powerful motivation. And they were a good crowd, too, sending fixes
and intelligent bug reports in volume. A user population like that
is a gift from the gods, and this is my expression of
gratitude.</p>

<p>The beta testers didn't know it at the time, but they were also
the subjects of a sociological experiment. The results are
described in my paper, <a
href="//www.catb.org/~esr/writings/cathedral-bazaar/">The
Cathedral And The Bazaar</a>.</p>

<h1>Credits</h1>

<p>Special thanks go to Carl Harris, who built a good solid code
base and then tolerated me hacking it out of recognition. And to
Harry Hochheiser, who gave me the idea of the SMTP-forwarding
delivery mode.</p>

<p>Other significant contributors to the code have included Dave
Bodenstab (error.c code and --syslog), George Sipe (--monitor and
--interface), Gordon Matzigkeit (netrc.c), Al Longyear (UIDL
support), Chris Hanson (Kerberos V4 support), and Craig Metz (OPIE,
IPv6, IPSEC).</p>

<h1>Conclusion</h1>

<p>At this point, the fetchmail code appears to be pretty stable.
It will probably undergo substantial change only if and when
support for a new retrieval protocol or authentication method is
added.</p>

<h1>Relevant RFCS</h1>

<p>Not all of these describe standards explicitly used in
fetchmail, but they all shaped the design in one way or
another.</p>

<dl>
<dt><a href="ftp://ftp.isi.edu/in-notes/rfc821.txt">RFC821</a></dt>

<dd>SMTP protocol</dd>

<dt><a href="ftp://ftp.isi.edu/in-notes/rfc822.txt">RFC822</a></dt>

<dd>Mail header format</dd>

<dt><a href="ftp://ftp.isi.edu/in-notes/rfc937.txt">RFC937</a></dt>

<dd>Post Office Protocol - Version 2</dd>

<dt><a href="ftp://ftp.isi.edu/in-notes/rfc974.txt">RFC974</a></dt>

<dd>MX routing</dd>

<dt><a href="ftp://ftp.isi.edu/in-notes/rfc976.txt">RFC976</a></dt>

<dd>UUCP mail format</dd>

<dt><a
href="ftp://ftp.isi.edu/in-notes/rfc1081.txt">RFC1081</a></dt>

<dd>Post Office Protocol - Version 3</dd>

<dt><a
href="ftp://ftp.isi.edu/in-notes/rfc1123.txt">RFC1123</a></dt>

<dd>Host requirements (modifies 821, 822, and 974)</dd>

<dt><a
href="ftp://ftp.isi.edu/in-notes/rfc1176.txt">RFC1176</a></dt>

<dd>Interactive Mail Access Protocol - Version 2</dd>

<dt><a
href="ftp://ftp.isi.edu/in-notes/rfc1203.txt">RFC1203</a></dt>

<dd>Interactive Mail Access Protocol - Version 3</dd>

<dt><a
href="ftp://ftp.isi.edu/in-notes/rfc1225.txt">RFC1225</a></dt>

<dd>Post Office Protocol - Version 3</dd>

<dt><a
href="ftp://ftp.isi.edu/in-notes/rfc1344.txt">RFC1344</a></dt>

<dd>Implications of MIME for Internet Mail Gateways</dd>

<dt><a
href="ftp://ftp.isi.edu/in-notes/rfc1413.txt">RFC1413</a></dt>

<dd>Identification server</dd>

<dt