storing mail across several cyrus partitions

[Bron Gondwana]

On Mon, Sep 03, 2007 at 07:54:25PM +0700, Artem Bokhan wrote:
> 
> > Cyrus puts one folder on one partition. you can put subfolders on 
> > other partitions, but you cant have two partitions connected with one 
> > folder.
> > Cyrus has no way to decide where it would store the email on disk and 
> > it would cost more time to merge these partitions in memory if a usere 
> > access his mailfolder. This structure is used in all parts of cyrus. 
> > To implement
> > something like this would be fundamental work like writing a new 
> > mailserver.
> >
> Hm, does cyrus ever scan the content of filesystem (files,subfolders)?
> Or it uses direct access to filesytem objects like files and folders
> based on information from inner database?
> 
> What is about  creating subpartitions like:
> 
> /partition1/sub1/u/user/user1/*messages*
> /partition1/sub2/u/user/user1/*messages*
> /partition1/sub3/u/user/user1/*messages ?

You would have to patch the code to do something like this.  Besides,
how are you planning to decide which messages go on which partition?
Some sort of hash of the UID?

It's distinctly possible, and dare I say it even reasonably trivial
to implement something like this (I wrote a new hashing scheme for
on-disk folders in about 1 hour last week - but you'd sure want to have
a decent use-case up front.  Is there any reason why you're using such
a big single mailbox that it doesn't even fit on a filesystem?  Ouch.

You'll need split-meta of course and a single location for all the
meta data (cyrus.index, cyrus.cache - assume about 5% the total data
size works for us)

As a datapoint, we use 300Gb data partitions with 15Gb meta partitions
across the board now.  Big enough for a few mailboxes, small enough
that fsck doesn't take days.  (reiserfs, it's the only Linux filesystem
with anywhere _near_ the performance profile under heavy loads with
small files that we need.  Ext3 choked and crawled when we tried it
on one machine.  Unusable)

> > There are more easy ways to do what you want.
> > 1. you can move user or subfolder to an other partition.
> >    In murdersetup you can do this even accros mailserver, without 
> > downtime,

Or you can do godawful unsupported evil stuff like we do with
sync_client and custom config files that sync the mailbox across
then a short changeover period where we disable the user at
Nginx/LmtpProxy level and update the database record.  Not
recommended for the sane.

> >    if you run short in memory, cputime or bandwidth.
> Moving of accounts will cause downtime of the current account and will
> cause high loads on hardware when accounts are large...
> Also I want to be able to extend size of user's accounts without pain
> when my boss will ask to extend them x10 or x100 .....

Buy big disks.  Set up new partition, move users to it.

(or will your boss ask for this without actually providing said space?
If so, maybe upgrade the boss first, THEN the disks)

> > 2. you can put the partition on a filesystem/lvm-volume which supports
> >    resizing, so you can add a new dist to the volume and resize it.
> >
> May somebody recommend reliable/safe filesystem that support resizing?
> I'm afraid to use anything except ext3 in production enviroment...

No, can't do that - but disk is getting so big/cheap these days that
attaching a huge SATA array unit and moving the mailboxes one by one
isn't actually that hard, then just decommission the old drives (or
use them for meta data in RAID1 for the nice speedup)

Bron.