performance issue (imap spool on san)

[Daniel Eckl]

Hi Michael!

> Thunderbird is NOT an IMAP client.

I don't want to start a flamewar, so I tell you that I write this email
with a very polite temper in mind. Just to not set you up. But you have
to admit, your saying is extremely provocative.

Please can you explain your saying? Pine and Mutt both are console only
applications and I think you simply cannot compare them to a graphical
client like Thunderbird.
An unix-unexperienced user will even simply fail to use them at all,
because the text interface is so limited by console possibilities, that
it cannot be intuitive in any way.

In a graphical client you want to be able to scroll through your whole
message list without any delay. So I think there is no other chance but
caching all header information of a folder. On low bandwith situations
it seems to be impossible for me to do this on demand only.

I was searching for a full featured graphical IMAP client for a long
time and tried everything and I have to say: Not only is thunderbird an
IMAP client, it is the best graphical IMAP client I could find in the
whole Linux and Windows world. It has the most complete feature set and
the most intuitive interface for both Linux and Windows users. And it is
way faster than every IMAP client with a comparable feature set. Kmail
is extremely near to my wishes, it only lacks IMAP IDLE support.

The first time you open a large IMAP folder is not very fast, I have to
admit, but I didn't find any other comparable IMAP client without this
problem. Perhaps there are some, but I didn't try them because of the
lack of other basic email features.

Anyway: I'd happily listen to other suggestions for full featured
graphical IMAP clients which could be better than thunderbird. There
surely are things in thunderbird which could be a lot better! I just
need an alternative which I was not able to find yet.

So regarding the performance problem of the original poster:
When I'm on my LAN, I can initially open a folder with 6000 mails in it
in about 30 seconds. I have a network throughput of about 900 KB/s. So
network just isn't the bottleneck. (see below)
After Thunderbird has this initial header download done, the folder
opens without any delay. It just downloads headers of new emails, so if
you don't get thousands of mails between two folder checks, there is no
problem at all.

I think that your storage backend might get into iowait issues. I had a
much worse issue as you are describing. I had that with an extraordinary
hardware (Dual Xeon and SCSI RAID-5 direct attached with an 64-bit ICP
host raid adapter) but the wrong file system. With ext3 I had incredible
iowait whenever a user's client downloaded all headers of a big folder.
And after some seconds of high iowait, the machine even freezed up for
several (up to 30!) seconds while having a lot harddisk write access.
Load in this periods raised up to over 35!!

Then with a much smaller temporary machine (single P4 with a SATA
Hardware Raid-1 mirror) and reiserfs I had no lockups anymore, but still
high iowait when a user downloaded all headers of a big folder and load
increases to nearly 20. The download started rather fast but as iowait
(and load) increased, it slowed down and all users got slow performance.
But luckily the machine did not freeze anymore... Way better than before
even though this hardware was a not nearly as good as the one before.

Now I switched back to the first hardware platform (SCSI RAID-5) but now
running on XFS. Now I have no iowait and performance issues anymore.
Load is ridiculous low, most time below 0,1. IOwait is mostly zero and
sometimes increases up to 3% and peaks to 5% for only one or two seconds.

So I'd say you should check the load and iowait of your mail server when
you try to open a big folder with thunderbird. I'd bet both is
increasing fast and far until the machine nearly stalls and so you
cannot get the whole folder.

It's very good that you can prevent the problem with kmail. This shows
that Thunderbird really has problems here, but you have to think of
normal users. On the user's side there is a lot of Outlook, Outlook
Express and Thunderbird which all will trigger the problem.
I think it will be better to eliminate this problem instead of working
around. You might find yourself in a situation where the server is in
high usage by your users and it's very hard to replace and migrate
mailboxes without long downtimes.

Best,
Daniel

Michael Loftis schrieb:
> 
> 
> --On July 26, 2006 12:02:41 PM +0200 Rudy Gevaert
> <Rudy.Gevaert at UGent.be> wrote:
> 
>> Hi,
>>
>> I've installed the latest cyrus release and I'm having trouble with large
>> mailboxes.
>>
>> I was going to try if the 4Gig limit is gone and I'm filling up a mailbox
>> with mails.
>>
>> If I open the mailbox trough mutt it gets loaded at a acceptable
>> (lighting fast) speed.  However when using thunderbird it gets very slow,
>> I haven't been able to open the mailbox of 294M.
>>
> 
> 
> This is a thunderbird problem, not a Cyrus problem.  Thunderbird is NOT
> an IMAP client.  It's a POP3/NNTP reader, that's been taught to badly
> parrot IMAP.  It attempts to download and locally index *all* headers. 
> This means that for large mail stores, it'll take a LONG time to figure
> out whats going on, and lots and lots of RAM too.  And this is all on
> the Client.
> 
> <...>
> 
>> The above logs are only an extract of the output.  These messages are
>> going in slow bursts.
>>
>> The imap spool and configdirectory are on a SAN.  How can I further debug
>> this?
> 
> Your SAN is fine, your client is the problem.  Unfortunately since
> Cyrusoft went under (they made Mulberry) aside from Mutt and Pine I
> don't really have any suggestions for loading large mailboxes.
> ----
> Cyrus Home Page: http://asg.web.cmu.edu/cyrus
> Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
> List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html