Cyrus Patches used at FastMail.FM

[Robert Mueller]

> Cool, some of the patches look really interesting and I'm considering to
> include one or the other into my rpm packages. For example the statuscache
> patch seems very nice. Just to be sure, are there any license restrictions
> on the patches?

No, no license restrictions. We'd love these to get back into cyrus itself, 
so happy to contribute them back to whatever licence cyrus is under.

The statuscache is actually very useful, more so than Bron's comment on the 
page immediately suggests. Originally I thought it was our webmail client 
that was doing most of the status calls, so I actually put caching into our 
web client. It turned out that in fact lots of email clients also blast the 
IMAP server with status calls, so putting it in the server itself has made a 
noticeable difference to load on our servers.

The other one that I've just finished today is the "fastindex" one, which 
I'm also excited about. I discovered that on our brand new server with only 
a couple of hundred users on it, cmdtimer was still showing that some status 
calls were taking > 2 seconds for large mailboxes (for status calls that 
weren't cached that is) on a virtually unloaded server (load < 0.05). 
Checking it out, it became pretty obvious that the problem was the core 
status loop which calls index_insequence() on every message to see if it's 
seen/unseen. When the seen state sequence list becomes quite large, you're 
doing an awful lot of repeated scans of the seen list.

With a bit of slightly messy hacking (static variables + extra parameter) 
instead of cleaner but more major code surgery, I've added a "caching" mode 
when scanning an assumed sequence list of messages into a sequential seen 
list. This has reduced the status call to basically instant even on 40,000+ 
message folders with lots of random seen/unseen messages. I've put the 
latest version of the patch up now:

http://cyrus.brong.fastmail.fm/patches/cyrus-fastindex-2.3.3.diff

> I'd love to see some of the patches integrated upstream, it makes my own
> work easier :)

So would we, the less we have to maintain, the better, and for large 
installations with many IMAP users, the performance impact of the status and 
fastindex patches should be helpful.

Rob