very slow syncing, any ideas?

[Marten Lehmann]

Hello,

I'm about to migrate several thousand mailboxes from Maildir to Cyrus 
using the tool imapsync. It does its job very well and when I tested the 
migration on a small development server it was very fast.

But now on the production system the synchronisation is very slow with a 
maximum of one message per second (and we have gigabytes of messages in 
the storage, partically > 10,000 messages per mailbox!). The general 
load of the system isn't very high, maybe a load average of 30. I 
disabled the duplicate message suppression. The mailboxes.db is about 8 
megabytes big with approx. 13,000 mailboxes and 4 default folders each 
(Drafts, Junk, Sent, Trash).

I have the following entries in my configuration which should provide a 
better hierarchie and balance of directories than if they were all in 
one main directory:

altnamespace: true
hashimapspool: true
unixhierarchysep: true
virtdomains: userid

I also tried to move the old Maildirs to a different server, so that 
getting messages from the old mailbox and putting it to the new mailbox 
through IMAP doesn't come up with reads and writes on the same server. 
But the performance benefit was minimal.

But in the end, syncing is still really slow. It would take weeks to 
sync all mailboxes that way.

How else could we move them to the new storage if doing it through IMAP 
is too slow? On the other hand we would like to keep all flags so I 
guess syncing it with IMAP is the only choice?

What could be the reason to be that slow? Is it the big mailboxes.db?

Regards
Marten