At FastMail.FM we use the Cyrus IMAP server very intensively, with hundreds of thousands of users on single servers.
We build our production cyrus servers from the 'fastmail' branches on github at the moment. You can find them here:
You can also download them all as a zip file, or individually below
0001-Ignore-files-in-git-housekeeping.patch.gitignore | 66 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 files changed, 66 insertions(+), 0 deletions(-) create mode 100644 .gitignore |
0002-Detected-deleted-heirarchy-even-if-turned-off.patchEven if you don't have delayed delete enabled, this namespace is reserved. This way is much safer, we know it's a "deleted" namespace regardless.imap/mboxname.c | 2 -- 1 files changed, 0 insertions(+), 2 deletions(-) |
0003-Don-t-process-expunged-if-uidvalidity-changed.patchWe had a process crash during index_check, and further examination showed that "oldexists" was -1 due to a UID validity change (the user had been deleted and then another user renamed into place within about 30 seconds, while an authenticated imap connection was held open) This patch skips the "expunged" check if the uidvalidity has changed, since expunged doesn't make sense if you don't have the same uidvalidity.imap/index.c | 54 ++++++++++++++++++++++++++++-------------------------- 1 files changed, 28 insertions(+), 26 deletions(-) |
0004-Rewrite-mailbox_cache_size-to-populate-a-pointer-str.patchThe code is littered with CACHE_ITEM_NEXT calls. Very annoyingly, these can wind up jumping to random addresses outside the mmaped space, and there's no way of knowing if they're valid or not. This patch changes this apart from some sync code, by creating a struct cacheitem and a datastructure of 10 of those: cacherecord. You call cache_parserecord (or the mailbox_* functions that do the same for a specific msgno/offset) to get the cache items into the data structure, then use them by enum name. Also - cache_parserecord returns the length of the entire cache record, for ease of copying it somewhere else!imap/index.c | 461 ++++++++++++++++++++++------------------------------ imap/index.h | 27 --- imap/mailbox.c | 86 ++++++---- imap/mailbox.h | 54 ++++++- imap/make_md5.c | 2 +- imap/make_sha1.c | 2 +- imap/mbexamine.c | 55 +++---- imap/sync_server.c | 2 +- imap/unexpunge.c | 28 ++-- 9 files changed, 333 insertions(+), 384 deletions(-) |
0005-skiplist-nested-transactions.patchrather than silently corrupting as we used to do, or aborting with an assertion failure as we do now, it is actually quite trivial to make the skiplist database just stack "virtual transactions" on top of the current one. On the plus side, you do actually get fine-grained rollback. You can rollback a sub transaction and then proceed with the original transaction cleanly. On the negative side, if you forget to shut down the original transaction you could keep the file locked forever. This is implemented with a simple stack of transactions. You need to commit or rollback the topmost transaction BEFORE doing anything with the nested transactions - but on the nice side, the nested code doesn't even need to know it's within a transaction - so you can write transactional code without having to care if your caller is already transactional. It if that makes sense. EXAMPLE IN PSEUDOCODE (as in, I haven't checked that I'm using the APIs sanely of filled in all the details!) int foo (struct db* d) { struct txn *t; ... OURDB->set(d, key, keylen, val, vallen, &t); r = mumble(db); if (r) { OURDB->rollback(d, &t); } else { OURDB->commit(d, &t); } } int mumble (struct db* d) { struct txn *t; ... OURDB->fetch(d, key, keylen, &val, &vallen, &t); if (vallen) { char *newval = frob(val, vallen); OURDB->set(d, key, keylen, newval, strlen(newval), &t); OURDB->commit(d, t); } else { OURDB->rollback(d, t); } return r; } See how mumble() could be called without caring about foo(), and yet can enjoy the use of transactions. This is the API simplification this patch can offer!lib/cyrusdb_skiplist.c | 105 +++++++++++++++++++++++++++++++----------------- 1 files changed, 68 insertions(+), 37 deletions(-) |
0006-skiplist-tuning.patchSkiplist tuning Not a candiate for upstream without additional debate With random changes to a mailboxes.db file, it could be nearly 100% random seeks before it recompressed. A seen file would need to reach 16kb before even considering re-compressing, with a real data length of just a couple of hundred bytes. This patch reduces the limits to: 4kb overhead 120% rather than 200% of current "sorted" size. It's been running happily on our servers, but hey - tuning is a black art! I don't really care too much either way. Would prefer a way to force a "checkpoint", since ctl_cyrusdb -c doesn't actually do it...lib/cyrusdb_skiplist.c | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) |
0007-hammer-skiplist.patchHammer skiplist This adds a debugging tool handy for hammering away at a skiplist file.imap/Makefile.in | 6 +- imap/hammer_skiplist.c | 218 ++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 223 insertions(+), 1 deletions(-) create mode 100644 imap/hammer_skiplist.c |
0008-statistics-for-pop3-connections.patchLog statistics on the number of retr, top and dele commands run during a pop3 session (XXX: config option?)imap/pop3d.c | 19 +++++++++++++++++++ 1 files changed, 19 insertions(+), 0 deletions(-) |
0009-traffic-count.patchAdd traffic counters in prot_read/prot_write to allow accurate byte counts to be logged per session.imap/imapd.c | 15 ++++++++++++--- imap/pop3d.c | 15 ++++++++++++++- lib/prot.c | 12 ++++++++++++ lib/prot.h | 15 ++++++++++++--- 4 files changed, 50 insertions(+), 7 deletions(-) |
0010-calculate-a-global-sessionid-for-every-connection.patchThis patch exists to support our 'auditlog' infrastructure, which allows tracing every action back to the login that caused it. The sessionid is returned in the login response or lmtp delivery response, and is logged all the way back through the proxy/filter chain, so we can follow every message end-to-end.imap/global.c | 29 +++++++++++++++++++++++++++++ imap/global.h | 4 ++++ imap/imapd.c | 7 ++++++- imap/lmtpd.c | 2 ++ imap/lmtpengine.c | 6 +++--- imap/pop3d.c | 12 ++++++++---- 6 files changed, 52 insertions(+), 8 deletions(-) |
0011-add-auditlog-syslog-statements.patchAll operations which change the UID's, messages or folders within a user's imap storage are logged with a prefix auditlog: and in a standardised format ( key=imap/append.c | 43 +++++++++++++++++++++++++++++++++++++++---- imap/append.h | 1 + imap/duplicate.c | 3 +++ imap/imapd.c | 6 ++++-- imap/index.c | 16 +++++++++++++++- imap/lmtp_sieve.c | 14 +++++++++++++- imap/mailbox.c | 38 +++++++++++++++++++++++++------------- imap/mboxlist.c | 34 +++++++++++++++++++++++++++++++++- imap/message.c | 7 ++++++- imap/message.h | 3 ++- imap/pop3d.c | 6 ++++-- imap/reconstruct.c | 33 +++++++++++++++++++++++++++++++-- imap/sync_commit.c | 17 +++++++++++++++++ imap/sync_support.c | 2 +- lib/imapoptions | 5 +++++ lib/libconfig.c | 4 ++++ lib/libconfig.h | 1 + 17 files changed, 204 insertions(+), 29 deletions(-) |
0012-Clean-Shutdown.patchThe instructions for Cyrus say to send a signal to master to shut the system down. This generally means a SIGTERM. When master gets a SIGTERM is sends SIGTERM to all children, who don't trap it and exit instantly. Fine if you're never in a "critical section" Unfortunately both replication and fast rename create critical sections. Fast rename thanks to split meta (two renames) and replication because if you have successfully committed a message to the index file but not yet logged the replication event, then it could fail to be copied. In a system as big as fastmail.fm's, this means almost every shutdown causes a couple of messages to fail to sync. Then we bring up the replica, get two messages with the same UID, hilarity ensues. This patch adds a SIGQUIT handler to master, which sends SIGQUIT to children then doesn't exit until all children have. The children already had a SIGQUIT handler which waited for signals_poll() to be checked. The patch also adds a few more signals_poll() in likely places. Some documentation has been added on using this, though I feel it could be made a bit more prominent since clean shutdown is much safer for data integrity. In particular, distribution maintainers should be making their system init scripts use it.doc/install-configure.html | 36 +++++++++++++ imap/Makefile.in | 2 +- imap/sync_client.c | 2 + lib/Makefile.in | 8 ++-- lib/prot.c | 15 +++--- lib/signals.c | 120 ++++++++++++++++++++++++++++++++++++++++++++ lib/signals.h | 54 ++++++++++++++++++++ master/master.c | 120 +++++++++++++++++++++++++++++-------------- master/service-thread.c | 4 +- master/service.c | 5 +- 10 files changed, 312 insertions(+), 54 deletions(-) create mode 100644 lib/signals.c create mode 100644 lib/signals.h |
0013-SIGQUIT-patches-from-Paul-Turgyan.patchAdds SIGQUIT support to cyr_expire and idled as well.imap/cyr_expire.c | 31 +++++++++++++++++++++++++++++++ imap/idled.c | 22 +++++++++++++++++++++- 2 files changed, 52 insertions(+), 1 deletions(-) |
0014-Command-timer.patchWe want to be able to track which commands take a long time to run so we can figure out which parts of our interface to optimise or move "problem" users to less loaded servers. This patch tracks how long each command takes to run and puts an entry into the log if it's longer than the time configured by commandmintimer.imap/imapd.c | 36 ++++++++++++++++++++++++++++++------ lib/imapoptions | 4 ++++ lib/prot.c | 4 ++++ lib/util.c | 40 ++++++++++++++++++++++++++++++++++++++++ lib/util.h | 8 ++++++++ 5 files changed, 86 insertions(+), 6 deletions(-) |
0015-Accept-From-header-from-IMAP-clients.patchHotmail used to create a 'From ' header and export it via IMAP. Outlook would send that header right up to the IMAP server. Yet another workaround for misbehaving clients. The bane of our existance they are, yes indeed. This patch causes any line starting with 'From ' in the header to be ignored rather than triggering a MESSAGE_BADHEADER error.imap/message.c | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) |
0016-Mailwasher-bug-workaround.patchThe program MailWasher (http://www.mailwasher.net/) has a bug where it can't handle getting the CAPABILITY response when it didn't ask for it (i.e. in response to a successful login) This patch removes that information. Clients will fall back to asking for it if they want it anyway.imap/imapd.c | 4 +--- 1 files changed, 1 insertions(+), 3 deletions(-) |
0017-Parse-Received-headers-for-internaldate.patchWe had a drive failure and had to recover a bunch of messages from backup. We lost an index and had to run reconstruct on a replica. Eudora (amongst others?) incorrectly handles the INTERNALDATE field when copying messages between IMAP accounts. A couple of things where the heuristic: "file mtime is close to internaldate" turns out to be somewhat bogus. Something that IS close to actual internaldate - the top Received header of any message delivered via LMTP. This patch adds a new option "internaldate_heuristic". If set to "receivedheader" then it will attempt to parse the first Received: header for the date. Failing that it will fall back to either the passed INTERNALDATE (append) or the current time (lmtp). It also uses the utime() call to update the file modified time to match the INTERNALDATE exactly for safer reconstructs. This pairs well with our other patch (syncmtime) which updates the modified time to the INTERNALDATE value on the replica during replication.imap/append.c | 14 ++++++++++- imap/message.c | 65 +++++++++++++++++++++++++++++++++++++++++++++++++++- imap/message.h | 1 + imap/reconstruct.c | 21 ++++++++++++++-- lib/imapoptions | 6 ++++ 5 files changed, 102 insertions(+), 5 deletions(-) |
0018-Warn-on-mismatched-GUID-instead-of-overwriting.patchThis is part of our "make replication safe" initiative which you can read more about here: http://lists.andrew.cmu.edu/pipermail/cyrus-devel/2007-October/000525.html In particular, this one addresses point (a): a) MUST never lose a message that's been accepted for delivery except in the case of total drive failure. By making sure we don't overwrite a message delivered when the now replica machine was the master but not yet replicated.imap/sync_client.c | 36 ++++++++++++++++++++++++------------ 1 files changed, 24 insertions(+), 12 deletions(-) |
0019-GUID-IMAP-COMMANDS.patchThis patch factors out stuff we used to have in the old MD5UUIDs patch, the following FETCH responses in imapd: FETCH DIGEST.SHA1 => 40 character hex string (message sha1) FETCH RFC822.SHA1 => 40 character hex string (message sha1, calculated) FETCH RFC822.FILESIZE => size of actual file on disk (via stat or mmap) It also adds a capability string item: "DIGEST=SHA1" Totally non-standard of course, but way useful for our replication checking scripts. Embrace and extend 'r' us. Anyone feel like writing an RFC for fetching the digest of a message via IMAP? If the server calculated it on delivery and cached it then you'd have a great way to clean up after a UIDVALIDITY change or other destabilising event without having to fetch every message again. (that would be me - I'm going to RFC the "DIGEST.SHA1" bit)imap/imapd.c | 14 ++++++++++++++ imap/imapd.h | 5 ++++- imap/index.c | 34 +++++++++++++++++++++++++++++++++- imap/version.h | 2 +- 4 files changed, 52 insertions(+), 3 deletions(-) |
0020-Hash-folders-by-userid-only.patchCyrus tries to map IMAP folder names to directory paths on disk fairly directly, and this usually works OK, but has the property that when intermediate paths are missing there are some strange workarounds. E.g.
imap/mailbox.c | 30 ++- lib/imapoptions | 8 +- lib/libconfig.c | 4 +- lib/libconfig.h | 2 +- tools/dohash | 288 -------------- tools/rehash | 1160 +++++++++++++++++++++++++++++++------------------------ tools/undohash | 154 -------- 7 files changed, 691 insertions(+), 955 deletions(-) delete mode 100755 tools/dohash delete mode 100755 tools/undohash |
0021-Complete-rewrite-of-charset-handling-using-Perl.patchimap/imapd.c | 72 +-- lib/Makefile.in | 11 +- lib/charset.c | 1904 +++++++++++++++++++++------------------------ lib/charset.h | 29 +- lib/charset/iso-2022-jp.t | 33 +- lib/charset/iso-2022-kr.t | 12 +- lib/chartable.h | 27 +- lib/mkchartable.c | 975 ----------------------- lib/mkchartable.pl | 531 +++++++++++++ 9 files changed, 1487 insertions(+), 2107 deletions(-) delete mode 100644 lib/mkchartable.c create mode 100644 lib/mkchartable.pl |
0022-Pass-a-pre-utf-8-encoded-body-to-sieve-for-tests.patchimap/message.c | 134 ++++++++++++++++++++++++++++++++++---------------------- imap/message.h | 4 +- 2 files changed, 83 insertions(+), 55 deletions(-) |
0023-Add-iso-8859-10-11-13-14-16-charset-support.patchlib/charset/iso-8859-10.t | 297 +++++++++++++++++++++++++++++++++++++++++++++ lib/charset/iso-8859-11.t | 290 +++++++++++++++++++++++++++++++++++++++++++ lib/charset/iso-8859-13.t | 297 +++++++++++++++++++++++++++++++++++++++++++++ lib/charset/iso-8859-14.t | 297 +++++++++++++++++++++++++++++++++++++++++++++ lib/charset/iso-8859-16.t | 297 +++++++++++++++++++++++++++++++++++++++++++++ lib/mkchartable.pl | 5 + 6 files changed, 1483 insertions(+), 0 deletions(-) create mode 100644 lib/charset/iso-8859-10.t create mode 100644 lib/charset/iso-8859-11.t create mode 100644 lib/charset/iso-8859-13.t create mode 100644 lib/charset/iso-8859-14.t create mode 100644 lib/charset/iso-8859-16.t |
0024-Fix-iso-2202-kr-and-support-euc-kr-as-well.patchlib/charset.c | 45 +- lib/charset/iso-2022-kr.t |25414 +++++++++++++++------------------------------ lib/chartable.h | 6 +- lib/mkchartable.pl | 63 +- 4 files changed, 8433 insertions(+), 17095 deletions(-) |
0025-Convert-to-unicode-5.1.patchlib/Makefile.in | 6 +- lib/charset/unidata2.txt | 6629 --------------- lib/charset/unidata5_1.txt |19336 ++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 19339 insertions(+), 6632 deletions(-) delete mode 100644 lib/charset/unidata2.txt create mode 100644 lib/charset/unidata5_1.txt |
0026-Charset-tool.patchimap/.cvsignore | 1 + imap/Makefile.in | 6 ++- imap/cyr_charset.c | 160 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 166 insertions(+), 1 deletions(-) create mode 100644 imap/cyr_charset.c |
0027-Rewrite-zlib-inflate-handling.patchUse a single fixed-size buffer, and return to "inflate" again for further data rather than resizing the output buffer.lib/prot.c | 119 +++++++++++++++++++++++++++++------------------------------- lib/prot.h | 1 - 2 files changed, 57 insertions(+), 63 deletions(-) |
0028-Add-x-option-to-cyr_expire-to-disable-expunge.patchDelayed expunge is great and all, but it causes a LOT of IO because you need to stat every mailbox's meta files, and you need to read all of every cyrus.expunge file. That's not something you might want to do every time you want to clean out the duplicate delivery database. Change cyr_expire to add a "-x" option. Ideally I'd like to just have not specifying a "-X" mean don't do it, but I think it might be too late for that. It would make disks fill up with junk at too many sites.imap/cyr_expire.c | 79 +++++++++++++++++++++++++++++----------------------- 1 files changed, 44 insertions(+), 35 deletions(-) |
0029-Add-mboxlist_count_inferiors-interface.patchimap/mboxlist.c | 33 +++++++++++++++++++++++++++++++++ imap/mboxlist.h | 5 +++++ 2 files changed, 38 insertions(+), 0 deletions(-) |
0030-Limit-User-Folders.patchA user managed to synchronise our server with their UWash IMAP server's view of a very busy home directory, creating tens of thousands of folders and showing that there's a DOS risk against mailboxes.db here. This patch creates a new config variable user_folder_limit which, if set, returns IMAP_PERMISSION_DENIED to any further attempts to create a folder if the user has too many folders already.imap/mboxlist.c | 20 +++++++++++++++++++- lib/imapoptions | 4 ++++ 2 files changed, 23 insertions(+), 1 deletions(-) |
0031-CRC32-functions.patchAdd support for calculating the CRC32 of a buffer or an ioveclib/Makefile.in | 8 ++++---- lib/crc32.c | 43 +++++++++++++++++++++++++++++++++++++++++++ lib/crc32.h | 12 ++++++++++++ 3 files changed, 59 insertions(+), 4 deletions(-) create mode 100644 lib/crc32.c create mode 100644 lib/crc32.h |
0032-Add-version-11-mailbox-header-with-crc32-fields.patchAdds CRC32 support to index header and record structures, and supporting code to actually calculate the values!imap/append.c | 2 + imap/mailbox.c | 118 +++++++++++++++++++++++++++++++++++++++----------- imap/mailbox.h | 17 +++++-- imap/mbexamine.c | 4 +- imap/message.c | 17 ++++++- imap/reconstruct.c | 22 ++++++++- imap/sync_commit.c | 77 +++++++++++++++++++++++++++------ imap/sync_server.c | 1 + imap/sync_support.c | 2 + imap/sync_support.h | 1 + imap/unexpunge.c | 11 +++++ 11 files changed, 220 insertions(+), 52 deletions(-) |
0001-Remove-sieve-action-string.patchWe use sieve to forward messages to a notification daemon which can send SMSes, and we don't want them filled with information about the actions which Sieve took. This patch stops the action_string being appended to the message sent to the notify daemon.script.c | 6 +++--- 1 files changed, 3 insertions(+), 3 deletions(-) |
0002-Autoreply-to-Envelope-FROM.patchThis patch is from the mailing list, and allows auto-reply to the value of the Envelope 'FROM'.script.c | 4 ++++ 1 files changed, 4 insertions(+), 0 deletions(-) |
0003-Use-UTF-8-decoding-for-searches-and-notify-messages.patchFixes Bug #1721, Bug #1722bc_eval.c | 82 ++++++++++++++++++---------------------------------- script.c | 75 +++++++++++++----------------------------------- sieve_interface.h | 4 +-- 3 files changed, 51 insertions(+), 110 deletions(-) |
0004-Use-UTF8-comparison-functions-if-PCRE-is-enabled.patchbc_eval.c | 6 +++++- sieve.y | 9 +++++++-- 2 files changed, 12 insertions(+), 3 deletions(-) |
Updated at: Tue Aug 25 04:23:05 2009