A couple of goodies

[Bron Gondwana]

On Fri, Jul 13, 2007 at 10:03:06AM +0100, David Carter wrote:
> On Thu, 12 Jul 2007, Wesley Craig wrote:
>
>> > 1) Mailbox renames are rare
>>
>> Accept on the first of the month.
>
> We've been running with my original fast rename patch for about 3 years now 
> and the race condition has never been a problem. Admittedly its been about 
> 4 years since our machine room UPS last decided to spot weld itself to the 
> floor, but an electrician did manage to kill power to 5 racks of my 
> equipment including the active mailstores without adverse side effects.

The fast rename patch doesn't handle split-meta at all by my reading,
which is a reason it can't work as-is in our configuration.  I realise
that all the "hacked in afterwards" stuff in Cyrus is a pain to support
in patches, but anything that doesn't either work safely with any
combination of settings or detect the ones it doesn't support and bail
out is pretty dangerous!  *sigh*.  It's getting really complex to hack
on Cyrus with all the bolted-on stuff!

> It is an imapd.conf option (fast_rename: yes), and is off by default.

Our imapd.conf is 53 lines and growing, most of which is turning on
options that would be sane defaults IMHO but aren't for backwards
compatibility.

>>> 3) The sys admin can always fix the problem by hand (and it would
>>>   be possible to log all renames to make it easier to spot problems).

I don't like anything that requires sysadmins to fix problems by hand.
I have a whole rant queued up in my head about software that assumes the
only thing that the person using it has to do with their life is keep
track of the exact version they're using and its own unique little
quirks and config file incompatibilities, but that's for another day.
I don't have a software blog for spewing such things to.

Much better is if it's logged somewhere that the system reads on
startup and finishes the action - that's heaps more robust.  Harder
to write initially, but a whole lot better for safety.  That said,
the pressure to write our own IMAP server rather than keep hacking
new stuff into Cyrus does build up occasionally.  So far it hasn't
quite reached boiling point, but the number of "the sysadmin can 
just fix it" by hand responses to things like replication fixup
after a system failure and now this rename thing keep pushing me.
This sysadmin doesn't like doing stuff by hand - if it's something
the computer can do automatically then it should be done by the
computer - even if this means contacting your "replica" and checking
if it has any unfinished sync logs before you start blindly overwriting
messages on it.

It's the biggest reason why I want md5 or sha1 stored in the
cyrus.index file too - the computer can go check its integrity
automatically this way, meaning I don't have to do anything 
manual.  Along with logging to ensure that the entire lifecycle
of every single message is logged somewhere we're getting close
to an actual robust and auditable system.

>> Perhaps we should add the new name to the mboxlist first, and then remove 
>> the old name.  That way, even in a power loss case, the user would have 
>> access to their mail, either with the old name or the new name, depending 
>> on whether the rename succeeded.
>
> The new name is already added to the mboxlist (as a reserved item). Two 
> mailbox entries should certainly work, but there isn't (AFAIK) a tool to 
> remove single entries from the mboxlist: you have to dump and restore. I 
> suppose that it wouldn't be hard to write one.

cyr_dbtool (in 2.3.8) can do it:

sudo -u cyrus /usr/cyrus/bin/cyr_dbtool \
   /var/imap/mailboxes.db skiplist delete user.foo.mailbox

> Two other benefits of fast rename:
>   Expunged messages and index files are also moved.

Also done by the patch Ken put in CVS the other day.  This is
worthwhile, whichever way it's done..  The only thing missing 
on my list now for Expunged messages is that the user.seen 
file gets rewritten and expunged messages don't retain their 
seen state.  This is a bit of a pain for seamless "undo style"
restores.

Bron.