cyrus imap slow on Fedora 13&14

[Bron Gondwana]

On Tue, Dec 28, 2010 at 12:40:31PM +0000, Andre Felipe Machado wrote:
> Hello,
> We actually used pidstat and iwatch to log disk activity of a given webmail
> client  upon our cyrus murder/aggregator.
> 
> The access pattern was ...err...  "unwise", when compared with thunderbird, for
> example.
> 
> The webmail client open account, then opens all folders, then lists all
> messages, for all folders, then reads entire content from all of them, one by
> one, then javascript order the entire list, then split the list in pages to
> render at screen...
> It can at least be configured to not read entire content from all of them, only
> headers, but still one by one...
> All these transactions caused disk writes (a "heavy" fs operation) to cache,
> index and header cyrus account files, and webservers and imap frontends load.
> When you have MANY msg into an inbox, this could be tragic. When you have many
> such simultaneous users, becomes a performance disaster.
> I suspect that other webmail clients have similar "unwise" access patterns.

Now that's just insane.  I'm pretty sure Squirrelmail doesn't do that - or at
least it didn't back when I was working on improving its IMAP client library.
 
> So, one should *aggressively* fine tune filesystem for small files writes/updates.

Why are reads causing a lot of writes?  The only thing that they should be
triggering writes of is \Seen flags IFF the messages were not already seen.
 
> The webmail clients could be redesigned to use wise imap command patterns. Could
> cyrus project publish some hints at site?

It might be worth publishing a "here's the most efficient way to do stuff with
Cyrus" document.  The only thing which has really changed since Cyrus 2.3 and
earlier is that "EXPUNGE" is now a very cheap operation.  We used to set a flag
"medeleted" on messages deleted through our web interface, because a very common
action was walking through the list of messages clicking "Delete and Next" in the
web interface - and expunging every single time caused a lot of rewrite IO.

These days with 2.4, that was actually LESS efficient, because you got a record
rewrite (plus modseq bump) for the flag, and then another one for the expunge
later.  While with a straight expunge, you just got a single record rewrite and
modseq bump.

----

But there's nothing Cyrus can do about a client as insane as the one you're
talking about above.  Slurping the ENTIRE state of the IMAP server at every
connection.  Well, that's one way to make sure you can do all the sorting
yourself with perfect information... I guess.

Bron.