<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
On 12/19/2014 06:17 AM, Patrick Goetz wrote:<br>
<blockquote cite="mid:5494175F.3020409@mail.utexas.edu" type="cite">
<pre wrap="">Nic,
Thanks for that detailed explanation. I still feel myself somewhat
stymied by either the documentation (or lack thereof) or perhaps an
unfortunate case of being somewhat feeble-minded. Here are some follow
up comments/questions:
On 12/18/2014 9:59 AM, Nic Bernstein wrote:
</pre>
<blockquote type="cite">
<pre wrap="">I will say that the ability to quiesce the application without halting
it would be most desirable. Most databases have supported this sort of
thing for ages, and it would be great if one could send a signal to
Cyrus to achieve the same result.
</pre>
</blockquote>
<pre wrap="">
I wonder what would happen if you just stopped lmtp while making a
snapshot? Would postfix choke on this and start kicking messages back
to the sender, or would they get queued for later delivery?
Alternatively, maybe lmtp could temporarily divert new messages to a
dummy spool so that postfix/sendmail wouldn't have to know anything
about this. This might be the least painful way to implement quiescence
in cyrus.</pre>
</blockquote>
<br>
But LMTP is only one method affecting the mail store, IMAP and sieve
can as well. Granted one can brute-force this by shutting down
network ports and the like, but at that point why not just stop
cyrus?<br>
<br>
<blockquote cite="mid:5494175F.3020409@mail.utexas.edu" type="cite">
<pre wrap=""> > His initial suggestion -- stop cyrus, snapshot, restart cyrus -- is
> reasonable, but we feel that the later suggestion -- stop cyrus, tar
> up data, start cyrus -- is not. It takes data offline for too long.
> That's why the snapshot capability is necessary in any truly suitable
> server.
I agree. Here is a substitute proposal (and I'll come back to why I'm
pushing this point). Serially
1. rsync user mail files
2. rsync configdirectory db files
3. rsync user mail files again
That should get you reasonably close to what you get with snapshots.</pre>
</blockquote>
<br>
No, not in the least is this close to a snapshot. Snapshots are
instantaneous, or near to it. The time an rsync takes, even a
catch-up, grows with the size of the mail store and the deltas
between attempts. Also, rsync is not well suited to the
file-per-message, directory-per-mailbox storage scheme of cyrus, as
lots of fstats() result, and this just adds to the time.<br>
<br>
I don't understand why one wouldn't use snapshots? Every modern OS
and distro include filesystems or volume managers which support
snapshotting, and several, such as Ubuntu, even recommend
snapshot-capable partitioning schemes out of the box. It's just not
that hard, and it's exactly the right way to handle this sort of
staged backup.<br>
<ul>
<li>Halt cyrus</li>
<li>snapshot critical filesystems</li>
<ul>
<li>spool date (/var/spool/imap)</li>
<li>config data (/var/lib/imap or /var/imap)</li>
<li>metadata (i.e. /var/run/cyrus)</li>
</ul>
<li>start cyrus</li>
<li>mount snapshot<br>
</li>
<li>rsync or otherwise backup from snapshot</li>
<li>unmount snapshot</li>
<li>(optionally) destroy snapshot<br>
</li>
</ul>
This is so easy to handle via a cron or at job. Why would one do
this? If the answer is "legacy system," then fine, but legacies can
be upgraded or replaced.<br>
<br>
<blockquote cite="mid:5494175F.3020409@mail.utexas.edu" type="cite">
<pre wrap="">If you follow the prescribed cyrus directory structure, then this can be
simplifed (Arch linux example):
1. rsync -a --delete /var/imap/user [removable disk/other server]
2. rsync -a --delete /var/imap [removable disk/other server]
Once you've rsynced the mail files once, rsyncing them again a short
time later should be pretty fast. There does need to be a backup
solution for people who only have one server, hence can't use
replication or imapsync to do backups.</pre>
</blockquote>
<br>
There is, snapshots, or hosted mail services (like Fastmail :).<br>
<br>
<blockquote cite="mid:5494175F.3020409@mail.utexas.edu" type="cite">
<blockquote type="cite">
<pre wrap="">Lastly, as to the use of imapsync to achieve user, mailbox or server
replication,...
So your command line is much like Patrick's example, but with '--user1
<user> --authuser1 <proxyuser> --user2 <user>...'
Of course you must create a proxy user, and Cyrus supports this with the
'proxyserver' directive in imapd.conf (man imapd.conf for details),
i.e.: 'proxyservers: proxyuser'.
</pre>
</blockquote>
<pre wrap="">
Here is the imapd.conf man page entry for proxyservers:
proxyservers: <none>
A list of users and groups that are allowed to proxy for other
users, separated by spaces. Any user listed in this will be
allowed to login for any other user: use with caution. In a
standard murder this option should ONLY be set on backends.
DO NOT SET on frontends or things won't work properly.
That capitalized "DO NOT SET on frontends" would seem to be cause for
concern, especially since I don't understand how this works.</pre>
</blockquote>
<br>
Well then, get thee to a website or man page. :-) <br>
<a class="moz-txt-link-freetext" href="http://cyrusimap.web.cmu.edu/docs/cyrus-imapd/2.4.17/ag.php">http://cyrusimap.web.cmu.edu/docs/cyrus-imapd/2.4.17/ag.php</a><br>
<br>
No, seriously, this isn't an issue if you're not using a murder. A
"frontend" is the part of a murder aggregation cluster which proxies
for the backend servers which actually hold the mail store. A
murder consists of one or more frontends, one or more backends and a
single "mupdate" master, which controls the canonical copy of the
mailboxes database. In a murder, if one wants to set the
proxyservers option, one sets it only on the backend machines.<br>
<br>
The proxyservers option is exactly the right way to do this.<br>
<br>
<blockquote cite="mid:5494175F.3020409@mail.utexas.edu" type="cite">
<pre wrap="">For people who are
1. imapsync'ing between machines both behind a firewall
2. using saslauthd with pam
I thought of this solution: Temporarily block port 143 traffic on the
outward facing port of your firewall, and then add the line
auth sufficient pam_permit.so
to the top of /etc/pam.d/imap files on both the sending and receiving
imap servers. This should allow you to imapsync the mail stores for
every user without having to provide passwords. Once you're done,
simply remove these lines from the PAM configuration files and unblock
the port on the firewall. Yes, this will mean that users won't be able
to access their mail from outside the firewall while the imapsync is in
operation, and this is probably only workable for smaller organizations
where people are not concerned about their coworkers temporarily being
able to access their mail. There could probably be a desktop policy to
handle this as well.</pre>
</blockquote>
<br>
Ouch, that seems a lot harder to me than setting proxyservers.<br>
<br>
<blockquote cite="mid:5494175F.3020409@mail.utexas.edu" type="cite">
<pre wrap="">However, you are 100% correct that replication would appear to be a far
less complex solution. After reading through the available
documentation, it wasn't clear to me that it was possible to do
replication without setting up a murder, a complexity I was hoping to avoid.
So, here's the feeble-mindedness component: I didn't completely follow
your explanation for setting up a replication server. It would be
awesome to have a howto for doing this -- is anyone aware of anything
like this; i.e. howto set up a replication server outside the murder
context.</pre>
</blockquote>
<br>
Then please take a look at the replication page on the Project Cyrus
website:<br>
<a class="moz-txt-link-freetext" href="http://cyrusimap.org/docs/cyrus-imapd/2.4.17/install-replication.php">http://cyrusimap.org/docs/cyrus-imapd/2.4.17/install-replication.php</a><br>
<br>
Here's my earlier example with the murder components stripped out,
and some commenting added:<br>
<br>
Both servers (note last entry):<br>
<pre>/etc/services
</pre>
<blockquote>
<pre>lmtp 24/tcp
imap2 143/tcp
imap2 143/udp
imaps 993/tcp
imaps 993/udp
sieve 4190/tcp
<b>csync 2005/tcp</b>
</pre>
</blockquote>
Master server:<br>
<pre>/etc/imapd.conf
</pre>
<blockquote>
<pre>...
##
# These configuration parameters are for the master server
# in a replication set
# The list of userids with administrative rights
admins: cyrus
##
# Replication support
# This is how the BACKEND for this host is defined
sync_host: replica.example.com
sync_authname: mailproxy
sync_password: <password>
sync_realm: <if required for your auth scheme>
# Whether to compress the replication stream, important if using WAN links
sync_compress: true
# To enable "rolling" replication, set this to TRUE
# This causes all data altering daemons, such as imapd, lmtpd, etc. to log their
# actions for replication.
sync_log: true
# Minimum interval (in seconds) between replication runs in rolling replication mode.
sync_repeat_interval: 5
# A file whose existence will cause the sync_client to stop at its next opportunity
sync_shutdown_file: /var/run/cyrus/sync_stop
...
</pre>
</blockquote>
<pre>/etc/cyrus.conf
</pre>
<blockquote>
<pre>...
SERVICES {
...
syncclient cmd="/usr/lib/cyrus/bin/sync_client -r"
...
</pre>
</blockquote>
Replica server:<br>
<pre>## /etc/imapd.conf
</pre>
<blockquote>
<pre>...
##
# These configuration parameters are for the replica server in a
# replication cluster
# The list of userids with administrative rights
# For a replica, this must include the user with which the master
# will authenticate
admins: cyrus mailproxy
##
# Unless you're using TLS between master and replica, add this
force_sasl_client_mech: PLAIN
master_mechs: PLAIN
</pre>
</blockquote>
<pre>## /etc/cyrus.conf
</pre>
<blockquote>
<pre>...
SERVICES {
...
syncserver cmd="/usr/lib/cyrus/bin/sync_server" listen="csync"
...
</pre>
</blockquote>
Here's some extra notes:<br>
<ul>
<li>The webpage listed above on replication explains rolling
replication (think "log shipping" from the DB world) as well as
manual replication. Check that out.</li>
<li>We find that it doesn't hurt to use both rolling and periodic
replication, and have cron handle the latter</li>
<li>If the master stops listening for csync traffic, when halted
for a snapshot, for example, then the sync_server process on the
replica will die. So, we use a nanny cronjob to make sure that
one gets started if none are running.</li>
</ul>
Here's our crontabs for master and replica:<br>
<br>
Master:<br>
<blockquote>
<pre>### Ensure replication is up to date
30 5 * * * /usr/local/sbin/cyrus_user_sync.pl >/dev/null 2>&1
##
### Run quota check script
30 6 * * * /usr/local/sbin/quota-report >/dev/null 2>&1
##
### Update mailbox annotations
45 6 * * * /usr/local/sbin/set_cyrus_annotations.sh >/dev/null 2>&1
##
### Update quotas
*/5 * * * * /usr/local/sbin/cyrus_ldap_quota.pl >/dev/null 2>&1
</pre>
</blockquote>
Replica:<br>
<blockquote>
<pre>##
# ensure that the sync_client keeps running. Comment this out
# following promotion from replica to master.
@hourly /usr/local/sbin/sync_nanny.sh >/dev/null
</pre>
</blockquote>
We'll be happy to share these scripts with anyone who'd care to have
a copy, but they might be specific to our use of LDAP to manage
account details. The idea of each, however, is to leverage the
account DB, which in our case is almost always LDAP, to maintain,
update or alter the cyrus account information.<br>
<br>
<blockquote cite="mid:5494175F.3020409@mail.utexas.edu" type="cite">
<blockquote type="cite">
<pre wrap="">However, I must be honest and point out that if you're going to go to
the trouble of figuring out how to use imapsync (and possibly pay for
it, to boot) you may as well just set up a replica. As I've shown,
above, it's just not that hard.
</pre>
</blockquote>
<pre wrap="">
Imapsync is still useful for migrating individual users from one imap
server to another. In my case, I'm migrating from a cyrus 2.3.x server
using Berkeley db metadata files to a cyrus 2.4.x server which will be
entirely skiplist based. Understood that you can convert db files to
skiplists, but I feel most comfortable using imapsync for this. In this
use case there are only a handful of users, but they all have extremely
complex and massive mail folders.
My current plan is to use imapsync for the migration and then
replication to another dummy server for backup, assuming I can figure
out how to set up replication.</pre>
</blockquote>
<br>
I strongly recommend against this course of action. If you're
migrating between two boxes, which it sounds like you are, then
you're much better off rsyncing the spool data between them (once
you've halted cyrus) and then allowing cyrus to perform the
necessary DB updates. <br>
<br>
Check the Install-Upgrades page for anything else which changes
between your versions of cyrus. Since you didn't specify which
2.3.x or 2.4.x you're using, I can't tell you what you'll need, but
you'll find that info in doc/install-upgrade.html of your version.
If you're installing from packages this may not be included, so do
yourself a favor and download a copy for reference.<br>
<br>
As the upgrade guide states (emphasis added):<br>
<blockquote>The default type for all databases is now <tt>skiplist</tt>
which is very reliable now, all the bugs are ironed out! <b>Because
ctl_cyrusdb -r automatically converts databases between known
types, you shouldn't need to do anything</b>, but if you want to
keep the old defaults, you'll need to make them explicit in your
imapd.conf as follows:
<pre>duplicate_db: berkeley-nosync
ptscache_db: berkeley
statuscache_db: berkeley-nosync
tlscache_db: berkeley-nosync
</pre>
</blockquote>
You have said you want skiplist, so you needn't add those settings,
just make sure you remove any that exist if you copy your old
imapd.conf file over.<br>
<br>
If you prefer to manually convert the DB files, you can do this with
the supplied cvt_cyrusdb tool:<br>
<blockquote>
<pre>$ /usr/lib/cyrus/bin/cvt_cyrusdb /tmp/annotations.db berkeley /var/lib/imap/annotations.db skiplist
or for Ubuntu
$ cyrus cvt_cyrusdb /tmp/annotations.db berkeley /var/lib/imap/annotations.db skiplist
</pre>
</blockquote>
Note that in this case, you should NOT rsync the DB files into the
new server's /var/lib/imap (or whatever your config directory is)
but rather into a holding area, like /tmp, from which you can read
them for the DB conversion.<br>
<br>
Also, make sure you do all of this as the cyrus user, or you'll end
up with permissions problems.<br>
<br>
Good luck!<br>
-nic<br>
<br>
<blockquote cite="mid:5494175F.3020409@mail.utexas.edu" type="cite">
<pre wrap="">
Thanks again for your helpful comments!
----
Cyrus Home Page: <a class="moz-txt-link-freetext" href="http://www.cyrusimap.org/">http://www.cyrusimap.org/</a>
List Archives/Info: <a class="moz-txt-link-freetext" href="http://lists.andrew.cmu.edu/pipermail/info-cyrus/">http://lists.andrew.cmu.edu/pipermail/info-cyrus/</a>
To Unsubscribe:
<a class="moz-txt-link-freetext" href="https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus">https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus</a>
</pre>
</blockquote>
<br>
<pre class="moz-signature" cols="72">--
Nic Bernstein <a class="moz-txt-link-abbreviated" href="mailto:nic@onlight.com">nic@onlight.com</a>
Onlight, Inc. <a class="moz-txt-link-abbreviated" href="http://www.onlight.com">www.onlight.com</a>
219 N. Milwaukee St., Suite 2a v. 414.272.4477
Milwaukee, Wisconsin 53202
</pre>
</body>
</html>