FastMail.FM patchset updates

[Bron Gondwana]

I get the feeling that it's time to spruik the FastMail patchset
on this august (February?) forum again.

http://cyrus.brong.fastmail.fm/

(warning, I don't have a good method of keeping track of what I've
told you about already, so there might be some repeated mentions
here - but I'm skipping the skiplist stuff because I know that's
gone upstream already)

(warning2, some of the patches have a bit of fuzz against 2.3.11
now, because I've been doing a lot of little bugfixing, not to mention
new feature writing, and fixing fuzz all the time gets boring.  They
still apply fine)

Patches of note:

* More logging of DB errors

  http://cyrus.brong.fastmail.fm/patches/cyrus-error-location-2.3.11.diff

  I'd have more to say about this if we had a single quota DB error
  since I wrote it.  *sigh*.  We got a spate of them a while back,
  and there wasn't enough information to be sure what was going wrong.
  Leaving this in just in case it happens again, I'll be able to see
  the exact cause.

  POTENTIAL CANDIDATE FOR UPSTREAM (doesn't do much, but it doesn't hurt
  either.  On the other hand, it could be buggy - the code path doesn't
  get much stress)


* Open QuotaDB in ctl_cyrusdb

  http://cyrus.brong.fastmail.fm/patches/cyrus-ctlcyrusdb-quota-2.3.11.diff

  So you're half way through renaming a mailbox and some nasty FastMail
  written script nukes your imapd because there was a file missing.
  Oops.  Next think you know there's a folder in "2" mode, half created.
  Next time you restart, ctl_cyrusdb will clean it up.  Except it has a
  quotaroot, and the quota DB isn't open, so actually ctl_cyrusdb will
  segfault.

  CANDIDATE FOR UPSTREAM (nobody likes segfaults, and these 4 lines make
  this set go away)


* Stop sync_client segfaulting if a message file is missing

  http://cyrus.brong.fastmail.fm/patches/cyrus-syncserver-missingmessage-2.3.11.diff

  Speaking of segfaults.  This patch fixes clearly bogus copy and pasted
  syslog code that refers to variables that haven't been set yet.

  CANDIATE FOR UPSTREAM (see above justification about segfaults, even
  worse if they take out the sync at random times)


* Fix sync of non-user mailboxes

  http://cyrus.brong.fastmail.fm/patches/cyrus-sync-nonusermailbox-2.3.11.diff

  Third time lucky.  Now that I get a personal email every time a cyrus
  process segfaults, they get special attention and get fixed up.  This
  is sort of a side effect of making DELETED.* mailboxes no longer a
  isusermailbox().  Still, it's not my code that randomly xstrdup'ed a
  function that's specified to return NULL at times, so I don't feel all
  that guilty.  It still would have broken for shared mailboxes
  everywhere.

  CANDIDATE FOR UPSTREAM (segfaults, bah)


* Sessionid

  http://cyrus.brong.fastmail.fm/patches/cyrus-sessionid-2.3.11.diff

  Ok, we're out of bugfixes and firmly into new features now.  

  This patch adds a couple of functions to global.h session_id_new()
  and "const char *session_id()".  It also adds the following string
  to the response codes on IMAP, POP and LMTP sessions.  IMAP and
  POP after a successful login, LMTP after accepting a message.

  Why?  To allow guaranteed reliable log joins.  We're moving towards
  logging everything important into a nice indexed database table, and
  we want to be able to trace every single message from start to finish
  as it comes through our system.

  the code added is sprintf("SESSIONID=<%s>", session_id()) in all
  cases.

  the format string is sprintf("%s-%d-%d-%d",
    config_getstring(IMAPOPT_SYSLOG_NAME), 
    getpid(),
    time(NULL),
    count++
  );

  where "count" starts at one and is incremented while time(NULL) is
  unchanged.

  NOT YET A CANDIDATE FOR UPSTREAM (we like the idea, though it's a
  little global.  I think this might want more discussion.
  Alternatively, if you really like it, feel free to grab as is!)

* Auditlog

  http://cyrus.brong.fastmail.fm/patches/cyrus-auditlog-2.3.11.diff

  The payoff for the earlier sessionid stuff.  I already posted this
  before, but it's been extended so every audit record contains the
  sessionid of the responsible connection, along with full details
  of the mailbox, uniqueid, uid, guid, message-id and other fun
  details as appropriate.  It also logs all expunges, all mailbox
  creates, renames, deletes, etc.  Lots of stuff about what goes
  on during reconstruct, sieve events including discard, redirect,
  reject, and duplicate tagging.

  CANDIDATE FOR UPSTREAM IF SESSIONID IS TAKEN (all auditlog: records
  are in an identical and easily parseable format:
    my @list = ($logentry =~ m{([^=]+)=<([^>]*)>}g); # perl

  and you need to turn on a config option to make it fill up
  your syslog, so it doesn't hurt to have it sitting there.


=====

Ok, that looks like everything that's new.  Enjoy.

Any questions about any of this, let me know.

Regards,

Bron.