Major changes to cmu/master git branch

[Bron Gondwana]

I just wanted to write about some of the changes that have gone into
cmu/master in the past few days, and will continue to trickle in during
this week.

Greg and I are working on dealing with a pile of our "speculative"
branches and getting them into stable form to go into the eventual 2.5.

Greg has fixed the shutdown process to run more cleanly, so IDLE imapds
will shut down immediately rather than waiting (this bug crept in
somewhere a while back)

I've changed struct index_map to only store 48 bytes per message rather
than 256 bytes.  This means that selecting a massive mailbox now uses
less than a quarter of the memory that it used to, which should help
people on resource constrained machines.

A much more important change is the change to release the namelock
between commands in imapd.  This includes the time sitting in idle, and
it fixes two long-standing issues with 2.4:

1) unlinks can be delayed indefinitely, meaning that mailboxes never
   repack because a user keeps the mailbox open.

2) you can't delete and recreate a folder while it's held open by
   any client.

The delete and recreate is still a theoretical issue - if the user is
doing a long running fetch command on a slow connection, you could hit
it.  We can fix that by making the create lock into a blocking lock, so
you wait around until the fetch is completed rather than getting an
error.  But the important part is, it works!  It's been quite a long
journey to get there.

There is one other side effect - if you run in the (not recommended)
immediate expunge mode, you can fail to fetch data for messages which
were concurrently expunged by another session.  Not a big deal, but a
slight user-visible change.


Upcoming changes include some sync-protocol fixes to make shutdown
nicer, hopefully some progress on the dump/restore format, and some work
on fixing our handling of list-extended so that we pass imaptest and can
start adding those tests to our Jenkins server.


Sorry about my long absence from visible Cyrus community involvement
over the past few months.  I am still working on it, just lots of other
things to do as well.


Regards,

Bron.
-- 
  Bron Gondwana
  brong at fastmail.fm