Cyrus bulk-loading with APPEND (was: Re: imclient >4k literal error)

[Jure Pecar]

On Mon,  7 Apr 2003 11:40:17 -0700
dkaiser at som.llu.edu wrote:

> Is anyone else attempting to do a large-scale bulk migration via the imap
> APPEND command?  If so, have you done any speed testing/measuring, and/or
> attempts at increasing the server speed?  I would be interested in hearing
> from anyone, success or failure.

Yes, it is just about to happen here ...

We'll have to do an online migration for ~180k users with ~9 milion mails,
total size about 270gb. Old box is a cyrus 2.0.17 installation, with 10 u160
scsi disks, new box is cyrus 2.2a with 30 2gbit fc-al disks. We'll also use
php scripts for the job. While at it, we'll also migrate auth from ldap to
mysql.

I did some tests during peak hours and during the night. The fastest copy i
got was about 1.5MB/s with tar|nc ... showing that the old box is really i/o
starvated. Php script was copying mails at a rate of 1200 per minute during
peak hours and about 1500 during night. So copying users one at a time would
take quite some time ...
However, we found out that copying about 25 users in paralell gives almost
25 times better numbers without putting too much load on the machines;
loadavg stays under 6 on the receiving box. So our goal is about 5-7 days
for full migration.


-- 

Jure Pecar