LIST is slow for 35K mailboxes

[Ian G Batten]

We have clients which issue LIST "" * when they start up.   We think  
we need them to do this, because we are making quite heavy use of  
shared mailboxes so a mailbox may arrive in a hierarchy other than  
INBOX.* to which the user should subscribe.

We have ~35K mailboxes (as reported by ctl_mboxlist -d | wc -l), and  
the LIST takes upwards of 5 minutes.   The imapd spins as much CPU as  
it can get hold of, too.

We assumed this was down to our elderly hardware (a 4x650MHz E450,  
albeit front-ending fast NAS store) but a freshly installed 8-core  
T2000 with 16GB of RAM and only one imapd executing takes a similar  
amount of time.

The production mailserver is now perfectly usable with a load average  
of >100 thanks to the wonders of Solaris 10 FSS, but the five minute  
responses to LIST "" * cause clients some distress.  We had hoped the  
move to the T2000 would solve the problem, but today's testing shows  
the new machine doesn't make a significant difference.

The old machine is running 2.2.8, the new machine is running 2.3.9.   
The old machine has configdir (/var/imap) striped over four spindles  
and then mirrored into four spindles, the new machine has /var/imap  
in ZFS filesystem of four disks, again mirrored in pairs.   The imap  
daemon that is spinning servicing the LIST "" * doesn't appear to do  
any I/O (other than regularly stating mailboxes.db), so I don't think  
disk performance is at issue anyway.

foolstupidclients: 1 would obviously help the performance, but would  
break the shared mailboxes.  I could modify tghe code to provide a  
limited form of foolstupidclients, which would turn on the option for  
users who don't need shared mailboxes but leave it off for those that  
do.  But at root I don't understand why LIST "" * should take any  
longer than, for example, ctl_mboxlist -d.

Before I wade into the code, can anyone make any helpful suggestions?

ian