LARGE single-system Cyrus installs?

[Ian G Batten]

On 17 Nov 07, at 0909, Rob Mueller wrote:
>
> This shouldn't really be a problem. Yes the whole file is locked  
> for the
> duration of the write, however there should be only 1 fsync per
> "transaction", which is what would introduce any latency. The  
> actual writes
> to the db file itself should be basically instant as the OS should  
> just
> cache them.

One thing that's worth noting for ZFS-ites is that on ZFS, you can  
have multiple writer threads in a file simultaneously, which UFS can  
only do for directio under certain conditions I can't recall.  That's  
a win for overlapping transactions into a file-based database.    
We're not hitting mailboxes.db remotely rapidly enough for this to be  
an issue, but I can imagine it being so for big shops.

In production releases of ZFS fsync() essentially triggers sync()  
(fixed in Solaris Next).  So if you anticipate a lot of writes (and  
hence fsync()s) to mailboxes.db then you don't want mailboxes.db in  
the same ZFS filesystem as things with lots of un-sync'd writes going  
on.    I've broken up /var/imap for ease of taking and rolling back  
snapshots, but it has the handy side-effect of isolating delivery.db  
and mailboxes.db from all the metadata partitions.

In my darker moments, by the way, I'm tempted to put deliver.db into  
tmpfs.  For planned reboot I could copy it somewhere stable, and I  
could periodically dump it out to disk.  But if I lost it, the  
consequences aren't serious, and it's most of the write load through  
that particular filesystem.

ian

mailhost-new# zfs list -t filesystem | grep imap; df /var/imap/proc
pool1/mailhost-space/imap              1.34G  24.6G   346M  /var/imap
pool1/mailhost-space/imap-seen          105M  24.6G  22.4M  /var/imap/ 
user
pool1/mailhost-space/meta-partition-1  2.48G  24.6G   972M  /var/imap/ 
meta-partition-1
pool1/mailhost-space/meta-partition-2  12.4G  24.6G  4.82G  /var/imap/ 
meta-partition-2
pool1/mailhost-space/meta-partition-3  4.86G  24.6G  1.60G  /var/imap/ 
meta-partition-3
pool1/mailhost-space/meta-partition-7  5.60G  24.6G  1.41G  /var/imap/ 
meta-partition-7
pool1/mailhost-space/meta-partition-8  14.0G  24.6G  5.39G  /var/imap/ 
meta-partition-8
pool1/mailhost-space/meta-partition-9  1.08G  24.6G   415M  /var/imap/ 
meta-partition-9
pool1/mailhost-space/sieve             5.26M  24.6G  1.62M  /var/imap/ 
sieve
/var/imap/proc     (swap              ):  514496 blocks  2356285 files
mailhost-new#



>
> Still, you have a point that mailboxes.db is a global point of  
> contention,
> and it is access a lot, so blocking all processes on it for a write  
> could be
> an issue.



>
> Which makes me even more glad that we've split up our servers into  
> lots of
> small cyrus instances, even less points of contention...
>
> Rob
>
> ----
> Cyrus Home Page: http://cyrusimap.web.cmu.edu/
> Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
> List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20071119/a4b89759/attachment.html