Cyrus Replication (example) [was Re: restore from cyrdump]

[Nic Bernstein]

On 12/19/2014 01:31 PM, Patrick Goetz wrote:
> Super helpful -- thanks!
>
> I only have one additional question:
>
> On 12/19/2014 09:31 AM, Nic Bernstein wrote:
>>> My current plan is to use imapsync for the migration and then
>>> replication to another dummy server for backup, assuming I can figure
>>> out how to set up replication.
>> I strongly recommend against this course of action.  If you're migrating
>> between two boxes, which it sounds like you are, then you're much better
>> off rsyncing the spool data between them (once you've halted cyrus) and
>> then allowing cyrus to perform the necessary DB updates.
>>
>
> It wasn't clear to me why you strongly recommend against this course of
> action (other than your recommended course of action is vastly simpler
> than messing around with imapsync for multiple users).

Imapsync is slow and finicky. I have used it before and it's great for 
what it is, but it doesn't always preserve everything you might want, 
and can take several tries to work through that.  Given that you may not 
immediately recognize a deficient migration (missing message flags, for 
example, or seen state) you may find yourself in a predicament if you've 
already switched to using the new server. Those sort of things.

What I think imapsync is /really/ good at is migrating between two 
wholly different IMAP servers, or between a native IMAP server, like 
cyrus or uw-imap, and something with an IMAP-like protocol interface, 
like Exchange or Groupwise.  It's also great where you need to remap the 
user mail store space, as it lets one apply regexes and rearrange the 
mail store structure if you'd like.

I believe in using the best tools for the job.  Migrating between 
different systems requires agnostic tools like imapsync.  Upgrading 
within a single system, like you're proposing, is best handled by that 
system's self-provided tools, in this case ctl_cyrusdb and cvt_cyrusdb.

> Also, for the purposes of clarity and documentation:
>
> The only 2 files I need to run cvt_cyrusdb on are:
>
>       - mailboxes.db
>       - annotations.db
>
> the contents of the {configdirectory}/db are generated dynamically, and
> the other db files (e.g. deliver.db and tls_sessions.db) are only
> relevant to the cyrus instance on the current machine?  (I'm not sure
> about deliver.db -- this would seem to need to be converted as well.)

Deliver DB is needed if you're using duplicate suppression.  If you've 
followed this mailing list over the past few years (or read back through 
it) you'll find that temporal DBs like tls_sessions.db, can best be 
stored on ephemeral storage (RAM-based filesystems) such as /var/tmp, 
/var/run or /run (depending upon your flavor).  For example, we often 
use this in our imapd.conf (adujst your paths as needed):

    # These are temporal, thus are stored on tmpfs
    mboxname_lockpath: /var/run/cyrus/lock
    proc_path: /var/run/cyrus/proc
    duplicate_db_path: /var/run/cyrus/deliver.db
    statuscache_db_path: /var/run/cyrus/statuscache.db
    tlscache_db_path: /var/run/cyrus/tls_sessions.db

Your init script may need tinkering with to ensure that /var/run/cyrus 
exists before the daemon is started, however, so check that.  In the 
Debian/Ubuntu world this involves the dpkg-statoverride command, and 
modern packages include that in the init script.

> Then one can just copy the contents of
>
>       - {configdirectory}/user
>       - {configdirectory}/quota
>       - {configdirectory}/sieve
>
> to the new machine.  If that works, this is vastly simpler than running
> imapsync for each individual user.

Yes, but again, check the list in install-upgrade.html for the changes 
made between the version you're coming from and going to. You may need, 
for example, to perform some upgrade step on the seen files (in 
{configdirectory}/user) or something like that.

Also, when going from 2.3.X to 2.4.X there is a new index file structure 
which will require each mailbox be reindexed.  This can be handled in 
one go, by forcing the reindexing, or by letting it happen as each user 
logs in.  To some extent this will depend, too, on which jobs you have 
in the START section of cyrus.conf.  Consult install-upgrade.html for 
more details.

Cheers,
     -nic

>
> ----
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

-- 
Nic Bernstein                             nic at onlight.com
Onlight llc.                              www.onlight.com
219 N. Milwaukee St., Ste. 2A	          v. 414.272.4477
Milwaukee, Wisconsin  53202		  f. 414.290.0335

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20141219/15d9875f/attachment.html