extremely poor performance with many maiboxes for 'list "" *'

[Greg A. Woods]

At Thu, 19 Apr 2007 13:50:36 -0400, Wesley Craig wrote:
Subject: Re: extremely poor performance with many maiboxes for 'list "" *'
> 
> I've seen problem like that, but I identified issues in the  
> auth_canonify() code for kerberos.  You don't appear to be using  
> kerberos, so your situation is likely to be different.

Nope, no kerberos here -- and this is happening well after
authentication anyway.

>  On the other  
> hand, I can speculate that LIST is much slower than grep because  
> imapd needs to determine whether every mailbox is readable by the  
> user.  That sort of calculation is significantly more intensive than  
> simple pattern matching.  Also, rather than replicating your  
> environment in test and doing a full profile, I'd probably start by  
> strace-ing a process that I was talking to with imtest.  I suspect  
> you'll find that imapd is accessing the filesystem in a most  
> inefficient way.

ktruss only shows a bunch of stat("/var/imap/mailboxes.db") calls being
made, presumably one for every record in the database.  I've no idea why
this would be useful or necessary, but regardless this doesn't explain
the massive CPU usage.

"Top" and similar system monitors show all the CPU being burned at
user-level, not much at all by the kernel.

Profiling the code would be far more useful than strace/ktrace or
similar, though those may show a pattern recognizable enough to help
identify the wasteful code.

Also I would certainly hope that the matching processing done by the
"list" command could be _far_ more efficient with CPU cycles than a
regex match over the whole mailboxes.db line would be for the same
operation.  The only difference should be in the parsing that would be
necessary to change a flat mailboxes.db record into some more
efficiently stored and compared internal representation.  One thought
I've had is that perhaps the whole mailboxes.db file is being re-parsed
for every record, instead of reading it all in just once and caching the
result before beginning any matching.

-- 
						Greg A. Woods

H:+1 416 218-0098 W:+1 416 489-5852 x122 VE3TCP RoboHack <woods at robohack.ca>
Planix, Inc. <woods at planix.com>       Secrets of the Weird <woods at weird.com>