Slow LIST command

[Mika Iisakkila]

Richard Gilbert wrote:
> Having watched the activity, the thing which seems to consume most of the
> resources are the LIST "" * and LIST "" % commands.  These return with a
> response like "OK Completed (1.480 secs 28 calls)" but can take a very
> long real time.  Is there anything which can be done to speed up the
> performance of the LIST command?  My imapd.conf is appended.

If you can go to skiplist, do that; it ought to be much more efficient
for the types of operations that Cyrus mostly does. If you examine the
system load more carefully, you'll probably note that the file system
containing /var/imap is 100% busy at peak times. It's the imapd processes
competing for access to the mailbox database. The longer it takes for an
operation to complete due to I/O wait, the longer the processes have to hold
their database locks and other processes wanting to access any nearby bunch
of mailboxes queue up, being able to do nothing useful.

Nevertheless, you can also tweak the Berkeley backend if you want to or
have to stick with it. Cyrus doesn't do anything to increase the BDB cache
size, and the default (256 kB) is way too small for any reasonably large site.
With some 50000 mailboxes and random operations, I found the hit rate for
default BDB cache to be 70-80%. After growing the cache size to 2M, hit rate
approached 99% and disk traffic was greatly reduced since most of the
operations are reads anyway. Therefore processes could complete their work
and release their locks much more quickly, and the dreaded "DBERROR: xxx
lockers" messages stayed at a comfortable level.

You can modify the source ( /lib/cyrusdb_db3.c, the setting is commented out)
or you can put a DB_CONFIG file under /var/imap/db with the appropriate
setting. Read more about this in the Berkeley docs before trying it, typos
and incorrect settings can cause havoc.

Of course, /var/imap filesystem might still remain the worst bottleneck,
in which case you'll have to do something about that at either OS or
hardware level (tune the kernel/fs, move other stuff elsewhere from that
disk subsystem, get faster disk(s), stripe it, ...).

--mika