very slow syncing, any ideas?

[Zachariah Mully]

On Thu, 2006-10-19 at 21:59 +0200, Marten Lehmann wrote:
> Hi,
> 
> > What is I/O load on the new server? on the old server? Was your test set
> > representative of your production system? 
> 
> the test system has much older hardware than the pretty new production 
> server (Xeon 3.4 with RAID5 and SCSI HDs). Old and new server are the 
> same from the hardware side, we have just changed the software.

So your old mailserver (maildir) and your new cyrus server are on two
different machines? Was the test server also running RAID5 for the
spool?

> Even syncing from an IMAP server in the internet to my local server was 
> faster.

To which local server, your Cyrus server or your Maildir server?

> I guess its none of them. uptime shows
> 
> 21:04:59 up 1 day,  5:46,  7 users,  load average: 29.10, 43.29, 41.94

What is the load a result of? Are you doing av checks on the new server?
You've still not answered the fundmental question, what is causing this
load?

> I can easily copy large amounts of files within the system, so it is 
> definetely not a problem of the filesystem. Syncing is always slow, of 
> course large mails take a bit longer, but that isn't the bottleneck.

Large being *greater* than the fs cache? What fs are you using? Is the
same as the one on your old Maildir system?

> The question is: What is Cyrus doing in the background when doing an 
> APPEND in IMAP (I guess thats the way new messages are added)? When I 
> can copy 20 files within a second in the filesystem, but only 1 in three 
> seconds through IMAP, whats wrong then?

You've not told us whether or not the load is a result of Cyrus
processes. Is it? Which Cyrus process? Authenication? IMAP? what? 

> We tried both: syncing from the same box or from one to another. It was 
> slow in both cases.

I don't understand what you mean, do you mean you tried running imapsync
on the old mailserver, then tried running off the new mailserver?

> At this moment it is evening here, so there aren't many deliveries 
> (about 90 per minute).

Argh. How much mail do you need to move, # messages and GB?

> I guess that the bottleneck is somewhere in Cyrus. We are using 
> cyrus-imapd-2.2.12 from RHEL4. Is there a problem with berkeley db with 
> large mailboxes.db? What is Cyrus doing during an IMAP APPEND? Is it 
> always looking for the folder in mailboxes.db? Is it sync'ing the 
> harddisk after each APPEND?

You need to pinpoint the bottleneck. What does vmstat look like while
trying to sync? What filesystem are you running and with what options is
it mounted? Are those options the same/different than your test/old
servers? I have 60GB of email in 860,000 messages and my mailboxes.db is
323KB. 

> Our old way to deliver mails sort of broke down, so we urgently needed a 
> replacement. We worked on Cyrus for some weeks so we knew enough to 
> build the right configuration, but we didn't had the time to do a 
> stresstest.

I can understand that, but you're shooting in the dark here, and you've
given ZERO information that might help anyone on the list help you.
Start by finding the bottleneck, don't assume that it's Cyrus.
Especially since you didn't see this behavior on your old slower test
system, I can't understand why you'd immediately assume it was Cyrus.
Don't arbitrarily rule out possible causes for your problem (as it looks
like you have). Sometimes it is something as stupid as a dns issue. Or a
bad hardware. Or a bad nic. Or a ... 

Z