very slow syncing, any ideas?

[Zachariah Mully]

On Thu, 2006-10-19 at 20:10 +0200, Marten Lehmann wrote:
> Hello,
> 
> I'm about to migrate several thousand mailboxes from Maildir to Cyrus 
> using the tool imapsync. It does its job very well and when I tested the 
> migration on a small development server it was very fast.

> But now on the production system the synchronisation is very slow with a 
> maximum of one message per second (and we have gigabytes of messages in 
> the storage, partically > 10,000 messages per mailbox!). The general 
> load of the system isn't very high, maybe a load average of 30. I 
> disabled the duplicate message suppression. The mailboxes.db is about 8 
> megabytes big with approx. 13,000 mailboxes and 4 default folders each 
> (Drafts, Junk, Sent, Trash).

What is I/O load on the new server? on the old server? Was your test set
representative of your production system? 

You could be running into any number of bottlenecks that you've not
described:
1) disk I/O
2) network I/O
3) message size
4) file system format
etc.
etc.

Find out what those are, then find out which you're hitting, then come
back.

> I also tried to move the old Maildirs to a different server, so that 
> getting messages from the old mailbox and putting it to the new mailbox 
> through IMAP doesn't come up with reads and writes on the same server. 
> But the performance benefit was minimal.

So both the maildir server and the Cyrus server are on the same box? Do
their spools share the disks? Controllers? How many users? How many
connections? How much mail?

You need to better understand the limits of your system. Yes your
development system was fast, but it sounds like you didn't try to
replicate the load of doing everything at once on it, so your
expectations have been set artifically.

Z