Adding archiving to an existing Cyrus installation

[Michael Menge]

Hi Nic,

Quoting Nic Bernstein <nic at onlight.com>:

> Friends,
> I have a client with Cyrus 2.5.10 installed.  Last year we migrated  
> their old 2.3.18 system to 2.5.10, with an eye towards an eventual  
> move to 3.0.x.  Based on Bron's most excellent email of last year,  
> ([Subject: Cyrus database and file usage data] from Cyrus Devel of 8  
> January 2016) we used a tiered layout for the storage:
>
> The main categories are:
>
>  * Config directory (ssd) [/var/lib/imap]
>      o sieve
>      o seen
>      o sub
>      o quota
>      o mailboxes.db
>      o annotations.db
>  * Ephemeral [/var/run/cyrus -- in tmpfs]
>      o tls_sessions.db
>      o deliver.db
>      o statuscache.db
>      o proc (directory)
>      o lock (directory)
>  * Mailbox data [typical 2.5.X usage]
>      o Meta-data (ssd)
>          + header
>          + index
>          + cache
>          + expunge
>          + squat (search index)
>          + annotations
>      o Spool data (disk: raidX)
>          + messages (rfc822 blobs)
>
> We sized the Fast SSD pool (this is three-drive mirrors on ZFS) to  
> be extra large, so it could eventually handle "Hot" data, and left  
> about 300GB free there.  Data, on spinning media, is currently  
> 5.74TB with 4.8TB free (RAID10).  Metadata is 35GB and /var/lib/imap  
> is 8GB, all of which is in the Fast pool.
>
> Now the client is ready to take the dive into v3.0, and I'm trying  
> to figure out how to put "archive" operation in effect.
>
> I have read the documentation (hell, I wrote most of it) and  
> understand the settings, but what I cannot quite wrap my brain  
> around is this: There is already all of this data sitting in all of  
> these data partitions (we use a total of 34 separate partitions each  
> for data & metadata) so how do I make the transition to separate  
> archive partitions, since all that data is on the "slow" drives? Can  
> I just reassign all of the current data partitions to archivedata  
> partitions, define the new set of "Hot" data partitions on the Fast  
> pool, and let 'er rip, or what?
>
> I promise, if you tell me, I'll write it up as real documentation. :-)
>

We are interested in such a migration too. Our fallback plan, if we  
don't find a
better way to do it is, do use the same method as we introduced the  
ssd meta-data
partition.


1. We created a new partition in our cyrus configuration,
2. we moved moved the accounts from one partition to the other one by one.
3. (this will be new for the archive partition) run cyrus expire to move
the old mails back to the slow disks.

This method will have two downsides.
1. we have to copy all mails to the fast storage, and move the old mails
    back to the slow storage. So we have to move most of the mails twice.
2. the path of the old mail will change so they will be stored again in
    our file based backup

so a method without these downsides will be appreciated

Regards

    Michael

--------------------------------------------------------------------------------
M.Menge                                Tel.: (49) 7071/29-70316
Universität Tübingen                   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung          mail:  
michael.menge at zdv.uni-tuebingen.de
Wächterstraße 76
72074 Tübingen