<!DOCTYPE html>
<html>
<head>
<title></title>
</head>
<body><div style="font-family:Arial;">On Fri, 10 Nov 2017, at 00:02, Nic Bernstein wrote:<br></div>
<blockquote type="cite"><div style="font-family:Arial;">Bron,<br></div>
<div style="font-family:Arial;"> Thanks for the response. Your solution has pointed us towards the
proper approach. We are using ZFS, so would be performing ZFS
send/receive replication rather than "mv", to move the filesystems.
Our typical approach for this sort of thing, then, would be:<br></div>
<div style="font-family:Arial;"> <br></div>
<ul><li>Create a filesystem snapshot<br></li><ul><li>zfs snapshot $ropt ${SOURCE_FS}@$newsnap<br></li></ul><li>Perform ZFS send/receive, something like this:<br></li><ul><li>zfs send -p $SENDVERBOSE $cloneorigin@$sourcestartsnap | zfs
recv -u $RECVVERBOSE -F "$destfs"<br></li></ul><li>Then, once we've completed a replication, we need to quiesce
the filesystem and do the same, again, to catch-up to current
state<br></li></ul></blockquote><div style="font-family:Arial;"><br></div>
<div style="font-family:Arial;">That's basically exactly how FastMail's user moves work, and the sync-protocol based XFER that Ken built based on it. Except inside Cyrus rather than at the FS level.<br></div>
<div style="font-family:Arial;"><br></div>
<blockquote type="cite"><ul><li>Finally, replace the existing filesystem with the new replica,
and discard the original<br></li></ul><div style="font-family:Arial;">To quiesce the filesystem, we would normally like to tell whatever
applications are using it to temporarily freeze operations, so the
underlying filesystem is in a consistent state. When I was in
Melbourne, back in April, we (you, Ellie & I) discussed what
would be needed to introduce such a feature to Cyrus. I'm curious
if there's been any further discussion or work on this? Should I
open a feature request? <br></div>
</blockquote><div style="font-family:Arial;"><br></div>
<div style="font-family:Arial;">There has been discussion, not yet work.<br></div>
<div style="font-family:Arial;"><br></div>
<div style="font-family:Arial;"><a href="https://github.com/cyrusimap/cyrus-imapd/issues/1763">https://github.com/cyrusimap/cyrus-imapd/issues/1763</a><br></div>
<div style="font-family:Arial;"><br></div>
<blockquote type="cite"><div style="font-family:Arial;"> It would be nice to be able to complete consistent snapshots for
filesystem operations like replication or backup, and this is a
feature of many large applications, such as mail and DB servers. <br></div>
</blockquote><div style="font-family:Arial;"><br></div>
<div style="font-family:Arial;">Yep :) It's on our radar. There's a whole lot of work planned - Ken is going to write up the notes on that at some point. We had a planning meeting while he was in Australia.<br></div>
<div style="font-family:Arial;"><br></div>
<blockquote type="cite"><div style="font-family:Arial;"> Thanks again for shining a welcome light on how to achieve the goal
of adding archive to an existing system.<br></div>
<div style="font-family:Arial;"> <br></div>
<div style="font-family:Arial;"> Cheers,<br></div>
<div style="font-family:Arial;"> -nic<br></div>
<div style="font-family:Arial;"> <br></div>
<div style="font-family:Arial;"> <br></div>
<div>On 11/04/2017 10:59 PM, Bron Gondwana
wrote:<br></div>
<blockquote type="cite" cite="mid:1509832770.219005.1161749752.2A8D72E1@webmail.messagingengine.com"><div style="font-family:Arial;">Hi Nic,<br></div>
<div style="font-family:Arial;"><br></div>
<div style="font-family:Arial;">Sorry I didn't get back to
answering you on this the other day!<br></div>
<div style="font-family:Arial;"><br></div>
<div style="font-family:Arial;">So... this one is kinda tricky,
because everything is going to be on "spool", but here's how I
would do it.<br></div>
<div style="font-family:Arial;"><br></div>
<div style="font-family:Arial;">Before:<br></div>
<div style="font-family:Arial;"><br></div>
<div style="font-family:Arial;">/mnt/smalldisk/conf -> meta
files only<br></div>
<div style="font-family:Arial;">/mnt/bigdisk/spool -> all email
right now<br></div>
<div style="font-family:Arial;"><br></div>
<div style="font-family:Arial;">Stage 1: splitting:<br></div>
<div style="font-family:Arial;"><br></div>
<div style="font-family:Arial;">/mnt/smalldisk/conf<br></div>
<div style="font-family:Arial;">/mnt/bigdisk/spool<br></div>
<div style="font-family:Arial;">/mnt/bigdisk/spool-archive ->
empty<br></div>
<div style="font-family:Arial;"><br></div>
<div style="font-family:Arial;">And set archivepartition-default
to /mnt/bigdisk/spoolarchive<br></div>
<div style="font-family:Arial;"><br></div>
<div style="font-family:Arial;">Now you need to run an initial
cyr_expire. This will take a long time, but it should be able
to use hardlinks to move the files - it's using cyrus_copyfile.<br></div>
<div style="font-family:Arial;"><br></div>
<div style="font-family:Arial;">Once cyr_expire has finished an
most of your email is moved into spool-archive, shut down cyrus.<br></div>
<div style="font-family:Arial;"><br></div>
<div style="font-family:Arial;">mv /mnt/bigdisk/spool
/mnt/smalldisk/spool<br></div>
<div style="font-family:Arial;"><div style="font-family:Arial;"><br></div>
<div style="font-family:Arial;">And set partition-default to /mnt/smalldisk/spool<br></div>
</div>
<div style="font-family:Arial;"><br></div>
<div style="font-family:Arial;">That way your downtime is only
while the small remaining spool gets moved to the other disk.<br></div>
<div style="font-family:Arial;"><br></div>
<div style="font-family:Arial;">Bron.<br></div>
<div><br></div>
<div><br></div>
<div>On Sun, 5 Nov 2017, at 02:58, Nic Bernstein wrote:<br></div>
<blockquote type="cite"><div>Thanks much to you both for your comments and
suggestions. We had<br></div>
<div>already considered creating a temporary "staging" partition
and<br></div>
<div>shuffling mailboxes around, as Michael discussed, but have
the same<br></div>
<div>reservations about it. Since we're dealing with nearly 6TB
of data,<br></div>
<div>most of it old, this scheme would introduce considerable
disruption to a<br></div>
<div>very active mail system. We have a hard time getting a two
hour<br></div>
<div>maintenance window, and this would take days!<br></div>
<div><br></div>
<div>Bron, other Fastmailers, any thoughts??<br></div>
<div> -nic<br></div>
<div><br></div>
<div>On 11/03/2017 11:20 AM, Michael Menge wrote:<br></div>
<blockquote><div>Hi,<br></div>
<div><br></div>
<div>Quoting Reinaldo Gil Lima de Carvalho <<a href="mailto:reinaldoc@gmail.com">reinaldoc@gmail.com</a>>:<br></div>
<div><br></div>
<blockquote><div>I think that singleinstancestore (message hard links)
will not<br></div>
<div>survive when<br></div>
<div>moving from one partition to the other and storage
total size will<br></div>
<div>increase<br></div>
<div>significantly.<br></div>
<div><br></div>
</blockquote><div><br></div>
<div>thanks for the hint. This was not a problem while
migration to the<br></div>
<div>meta-data partition,<br></div>
<div>as the mails stayed on the same partition (as in
file-system and not<br></div>
<div>cyrus-partition)<br></div>
<div>and only hardlinks where change at all.<br></div>
<div><br></div>
<div>So one more reason for an other migration path.<br></div>
<div><br></div>
<div><br></div>
<blockquote><div><br></div>
<div>2017-11-03 12:22 GMT-03:00 Michael Menge<br></div>
<div><<a href="mailto:michael.menge@zdv.uni-tuebingen.de">michael.menge@zdv.uni-tuebingen.de</a><br></div>
<blockquote><div>:<br></div>
</blockquote><div><br></div>
<blockquote><div>Hi Nic,<br></div>
<div><br></div>
<div>Quoting Nic Bernstein <<a href="mailto:nic@onlight.com">nic@onlight.com</a>>:<br></div>
<div><br></div>
<div>Friends,<br></div>
<blockquote><div>I have a client with Cyrus 2.5.10 installed. Last
year we migrated<br></div>
<div>their<br></div>
<div>old 2.3.18 system to 2.5.10, with an eye towards an
eventual move to<br></div>
<div>3.0.x. Based on Bron's most excellent email of
last year,<br></div>
<div>([Subject: Cyrus<br></div>
<div>database and file usage data] from Cyrus Devel of 8
January 2016)<br></div>
<div>we used a<br></div>
<div>tiered layout for the storage:<br></div>
<div><br></div>
<div>The main categories are:<br></div>
<div><br></div>
<div> * Config directory (ssd) [/var/lib/imap]<br></div>
<div> o sieve<br></div>
<div> o seen<br></div>
<div> o sub<br></div>
<div> o quota<br></div>
<div> o mailboxes.db<br></div>
<div> o annotations.db<br></div>
<div> * Ephemeral [/var/run/cyrus -- in tmpfs]<br></div>
<div> o tls_sessions.db<br></div>
<div> o deliver.db<br></div>
<div> o statuscache.db<br></div>
<div> o proc (directory)<br></div>
<div> o lock (directory)<br></div>
<div> * Mailbox data [typical 2.5.X usage]<br></div>
<div> o Meta-data (ssd)<br></div>
<div> + header<br></div>
<div> + index<br></div>
<div> + cache<br></div>
<div> + expunge<br></div>
<div> + squat (search index)<br></div>
<div> + annotations<br></div>
<div> o Spool data (disk: raidX)<br></div>
<div> + messages (rfc822 blobs)<br></div>
<div><br></div>
<div>We sized the Fast SSD pool (this is three-drive
mirrors on ZFS) to be<br></div>
<div>extra large, so it could eventually handle "Hot"
data, and left<br></div>
<div>about 300GB<br></div>
<div>free there. Data, on spinning media, is currently
5.74TB with<br></div>
<div>4.8TB free<br></div>
<div>(RAID10). Metadata is 35GB and /var/lib/imap is
8GB, all of which<br></div>
<div>is in<br></div>
<div>the Fast pool.<br></div>
<div><br></div>
<div>Now the client is ready to take the dive into v3.0,
and I'm trying to<br></div>
<div>figure out how to put "archive" operation in
effect.<br></div>
<div><br></div>
<div>I have read the documentation (hell, I wrote most
of it) and<br></div>
<div>understand<br></div>
<div>the settings, but what I cannot quite wrap my brain
around is this:<br></div>
<div>There<br></div>
<div>is already all of this data sitting in all of these
data partitions<br></div>
<div>(we use<br></div>
<div>a total of 34 separate partitions each for data
& metadata) so how<br></div>
<div>do I<br></div>
<div>make the transition to separate archive partitions,
since all that<br></div>
<div>data is<br></div>
<div>on the "slow" drives? Can I just reassign all of
the current data<br></div>
<div>partitions to archivedata partitions, define the
new set of "Hot" data<br></div>
<div>partitions on the Fast pool, and let 'er rip, or
what?<br></div>
<div><br></div>
<div>I promise, if you tell me, I'll write it up as real
documentation. :-)<br></div>
<div><br></div>
<div><br></div>
</blockquote><div>We are interested in such a migration too. Our
fallback plan, if we<br></div>
<div>don't<br></div>
<div>find a<br></div>
<div>better way to do it is, do use the same method as we
introduced the ssd<br></div>
<div>meta-data<br></div>
<div>partition.<br></div>
<div><br></div>
<div><br></div>
<div>1. We created a new partition in our cyrus
configuration,<br></div>
<div>2. we moved moved the accounts from one partition to
the other one<br></div>
<div>by one.<br></div>
<div>3. (this will be new for the archive partition) run
cyrus expire to<br></div>
<div>move<br></div>
<div>the old mails back to the slow disks.<br></div>
<div><br></div>
<div>This method will have two downsides.<br></div>
<div>1. we have to copy all mails to the fast storage, and
move the old<br></div>
<div>mails<br></div>
<div> back to the slow storage. So we have to move most
of the mails<br></div>
<div>twice.<br></div>
<div>2. the path of the old mail will change so they will
be stored again in<br></div>
<div> our file based backup<br></div>
<div><br></div>
<div>so a method without these downsides will be
appreciated<br></div>
<div><br></div>
<div>Regards<br></div>
<div><br></div>
<div> Michael<br></div>
<div><br></div>
<div>------------------------------------------------------------<br></div>
<div>--------------------<br></div>
<div>M.Menge Tel.: (49)
7071/29-70316<br></div>
<div>Universität Tübingen Fax.: (49)
7071/29-5912<br></div>
<div>Zentrum für Datenverarbeitung mail:<br></div>
<div><a href="mailto:michael.menge@zdv.uni-tuebingen.de">michael.menge@zdv.uni-tuebingen.de</a><br></div>
<div>Wächterstraße 76<br></div>
<div>72074 Tübingen<br></div>
<div><br></div>
<div>----<br></div>
<div>Cyrus Home Page: <a href="http://www.cyrusimap.org/">http://www.cyrusimap.org/</a><br></div>
<div>List Archives/Info: <a href="http://lists.andrew.cmu.edu/pipermail/info-cyrus/">http://lists.andrew.cmu.edu/pipermail/info-cyrus/</a><br></div>
<div>To Unsubscribe:<br></div>
<div><a href="https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus">https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus</a><br></div>
</blockquote></blockquote><div><br></div>
<div><br></div>
<div><br></div>
<div>--------------------------------------------------------------------------------<br></div>
<div><br></div>
<div>M.Menge Tel.: (49)
7071/29-70316<br></div>
<div>Universität Tübingen Fax.: (49)
7071/29-5912<br></div>
<div>Zentrum für Datenverarbeitung mail:<br></div>
<div><a href="mailto:michael.menge@zdv.uni-tuebingen.de">michael.menge@zdv.uni-tuebingen.de</a><br></div>
<div>Wächterstraße 76<br></div>
<div>72074 Tübingen<br></div>
<div><br></div>
<div>----<br></div>
<div>Cyrus Home Page: <a href="http://www.cyrusimap.org/">http://www.cyrusimap.org/</a><br></div>
<div>List Archives/Info: <a href="http://lists.andrew.cmu.edu/pipermail/info-cyrus/">http://lists.andrew.cmu.edu/pipermail/info-cyrus/</a><br></div>
<div>To Unsubscribe:<br></div>
<div><a href="https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus">https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus</a><br></div>
</blockquote><div><br></div>
<div>--<br></div>
<div>Nic Bernstein <a href="mailto:nic@onlight.com">nic@onlight.com</a><br></div>
<div>Onlight Inc. <a href="http://www.onlight.com">www.onlight.com</a><br></div>
<div>6525 W Bluemound Rd., Ste 24 v. 414.272.4477<br></div>
<div>Milwaukee, Wisconsin 53213-4073 f. 414.290.0335<br></div>
<div><br></div>
<div>----<br></div>
<div>Cyrus Home Page: <a href="http://www.cyrusimap.org/">http://www.cyrusimap.org/</a><br></div>
<div>List Archives/Info: <a href="http://lists.andrew.cmu.edu/pipermail/info-cyrus/">http://lists.andrew.cmu.edu/pipermail/info-cyrus/</a><br></div>
<div>To Unsubscribe:<br></div>
<div><a href="https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus">https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus</a><br></div>
</blockquote><div style="font-family:Arial;"><br></div>
<div><div>--<br></div>
<div> Bron Gondwana, CEO, FastMail Pty Ltd<br></div>
<div> <a href="mailto:brong@fastmailteam.com">brong@fastmailteam.com</a><br></div>
<div><br></div>
</div>
<div style="font-family:Arial;"><br></div>
<div style="font-family:Arial;"><br></div>
<div style="font-family:Arial;"><br></div>
<pre>----
Cyrus Home Page: <a href="http://www.cyrusimap.org/">http://www.cyrusimap.org/</a>
List Archives/Info: <a href="http://lists.andrew.cmu.edu/pipermail/info-cyrus/">http://lists.andrew.cmu.edu/pipermail/info-cyrus/</a>
To Unsubscribe:
<a href="https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus">https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus</a><br></pre></blockquote><div style="font-family:Arial;"><br></div>
<pre>--
Nic Bernstein <a href="mailto:nic@onlight.com">nic@onlight.com</a>
Onlight Inc. <a href="http://www.onlight.com">www.onlight.com</a>
6525 W Bluemound Rd., Ste 24 v. 414.272.4477
Milwaukee, Wisconsin 53213-4073 f. 414.290.0335
<br></pre><div>----<br></div>
<div>Cyrus Home Page: <a href="http://www.cyrusimap.org/">http://www.cyrusimap.org/</a><br></div>
<div>List Archives/Info: <a href="http://lists.andrew.cmu.edu/pipermail/info-cyrus/">http://lists.andrew.cmu.edu/pipermail/info-cyrus/</a><br></div>
<div>To Unsubscribe:<br></div>
<div><a href="https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus">https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus</a><br></div>
</blockquote><div style="font-family:Arial;"><br></div>
<div id="sig56629417"><div class="signature">--<br></div>
<div class="signature"> Bron Gondwana, CEO, FastMail Pty Ltd<br></div>
<div class="signature"> brong@fastmailteam.com<br></div>
<div class="signature"><br></div>
</div>
<div style="font-family:Arial;"><br></div>
</body>
</html>