Backup methods

Thu May 10 23:57:13 EDT 2018

 > There may be an argument that could be made for 2 backup stratagies

That's the point. In the context of SME environments (Small and 
Medium-sized Enterprises, i.e. from 5 to 50 employees normally, up to 
250 in some countries) that we were talking about, a replication is an 
overkill, IMO. But for large enterprises like MNCs, large universities, 
public mail providers (Fastmail) of course multiple masters and backups 
via replication is the way to go. For large deployments there are good 
backup solutions in Cyrus, but for the small businesses admins I don't 
know any to recommend.

 > If the mailboxes are on something like EXT4 you can do an LVM 
snapshot bacause of the built in auto checkpointing and for something 
like xfs there is freeze.

Yepp, this is the idea. But the data on disk should be in a consistent 
state. Something like "FLUSH TABLES WITH READ LOCK" is what is needed 
actually, i.e. to consistently write to disk the data from the current 
instance and lock.

 > The trouble is that read operations can alter a files state so it may 
not be just a simple matter of a write lock.

If you mean things like SEEN state that are changed when the user reads 
something, the implementation v1 could block it, v2 could allow it by 
queuing in memory such state-change pending operations. Anyway these are 
just implementation details that don't change the general logic and, 
taking into account the supposed duration of the lock, IMO don't even 
matter much. The significant work that blocks this feature is the global 
lock across the entire Cyrus. Once it's implemented, it would be much 
easier to introduced specific improvements here and there.

 > Cyrus has multiple databases that would also need to be frozen and 
flushed before the snapshot is taken.
 > If you spread your mailstorage and metadata storage over multiple 
file systems trying to co-ordinate snapshots becomes more complex.

Sure, Cyrus would have to lock all the databases and where to store them 
would be up to the admin. Actually, fs snapshots is just the most 
obvious use-case. There could be others... up to the admin to decide.

*From:* Alvin Starr
*Sent:* Thursday, May 10, 2018 23:55
*To:* Info-cyrus
*Subject:* Re: Backup methods

On 05/10/2018 06:29 PM, Anatoli wrote:
> Actually, mysqldump performs a lock on the records it's dumping. If 
> its for a MyISAM db, the entire table is locked. If it's for InnoDB 
> and similar, an internal snapshot is created and only the records the 
> dump is reading are unavailable for writing.
Mysql provides table consistency by locking but that is only table 
consistency.
Multiple updates across multiple tables could easily result in an 
inconsistent database even though the tables are individually consistent.
With some tables that take hours to dump locking the tables is problematic.
This is why some people use LVM snapshots combined with "FLUSH TABLES 
WITH READ LOCK".
>
> Cyrus could also implement a per-user lock, but in reality it doesn't 
> need that complex syncro mechanisms, a simple global write lock would 
> be enough (reading would not be affected, son only I, not I/O, and not 
> to stop it but just to suspend). After all, the *write* lock would 
> last only a second or so, the fs snapshots are almost instantaneous. 
> If you can't tolerate a 1 second delay for writing in Cyrus, you are 
> probably not a SME.
The trouble is that read operations can alter a files state so it may 
not be just a simple matter of a write lock.
If the mailboxes are on something like EXT4 you can do an LVM snapshot 
bacause of the built in auto checkpointing and for something like xfs 
there is freeze.

Cyrus has multiple databases that would also need to be frozen and 
flushed before the snapshot is taken.
If you spread your mailstorage and metadata storage over multiple file 
systems trying to co-ordinate snapshots becomes more complex.

>
> And you don't need to hold the data to transfer it. You can dump it 
> directly to a nfs share or pass it as stdout to ssh: mysqldump | xz -9 
> | ssh remote_server "cat > /bkp/`date +%y%m%d_%H%M`.sql.xz". With a 
> couple of pipes more you can encrypt the data on the fly so it's 
> secure to store it in a cheap VPS overseas... or you could upload it 
> to dropbox.

There may be an argument that could be made for 2 backup stratagies.
1) where the mailstoreage and metadata can exist on a single volume and 
flushing the various databases is a short duration event. Then an LVM 
snapshot could be used
2) for distribted large scale mail systems where only an online live 
backup system can be used.

Backup for 100 users has different requirements than backup for 100000 
users so why not support a few different backup strategies.

>
> *From:* Jason L Tibbitts Iii
> *Sent:* Thursday, May 10, 2018 18:41
> *To:* Anatoli
> *Cc:* Info-cyrus
> *Subject:* Re: Backup methods
>
>>>>>> "A" == Anatoli<me at anatoli.ws>  writes:
> A> What about mysqldump > dump.sql, then mysql < dump.sql? Also a wrong
> A> way and didn't have to be implemented?
>
> No, that's exactly my point.  Thanks for making it for me!  The analog
> to the way you indicated that you would like it to work would be having
> the mysql server stop IO so that you can take a filesystem snapshot
> while the database is in a consistent state.  But instead, the database
> (like cyrus) implements a backup method which you can use to extract the
> data.  And it also requires disk space to hold the backup until you can
> transfer it to your backup medium.
>
>   - J<
>
>
>
>
>
> ----
> Cyrus Home Page:http://www.cyrusimap.org/
> List Archives/Info:http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

-- 
Alvin Starr                   ||   land:  (905)513-7688
Netvel Inc.                   ||   Cell:  (416)806-0133
alvin at netvel.net               ||

----
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20180511/3dbbb01f/attachment.html>