sync_client: Fallback on master

[Ismaël Tanguy]

Hello,

During the upgrade of a master backend server (~15k mailboxes) from 
Centos7 to Centos8, we fail over on a replica.
Before the failover, we run a last time sync_client -r, to flush 
entirely sync log.

Everything went fine, except that the replica was not allowed to 
mupdatepush (ctl_mboxlist -m) and all the mailboxes dissapear in the 
murder mailboxes.db
That has been fixed as soon as we saw this error. Mailboxes.db has been 
repopulated.

Then, when the backend cames up, we fall back on it and run sync_client 
from the replica to the master.
I think now that was not the good way, because new mails start to arrive 
on the master while replication occured from the replica.

Replica's sync log size is as big as yesterday (the failover day) = 23M 
and around 1 million lines (mainly MAILBOX and APPEND commands).

replica /var/log/maillog reports various errors :


Jul  3 09:37:25 store-1-replicat-etudiant cyrus/sync_client[4000]: 
UNMAILBOX received NO response: Unknown error
Jul  3 09:37:25 store-1-replicat-etudiant cyrus/sync_client[4000]: 
sync_folder_delete(): failed: user.e21901967.Bibli-Culture 'The remote 
Server(s) denied the operation'
Jul  3 09:37:25 store-1-replicat-etudiant cyrus/sync_client[4000]: 
IOERROR: do_user_main: The remote Server(s) denied the operation for 
e21901967 to [no channel] (store-xxxxx)
Jul  3 09:37:25 store-1-replicat-etudiant cyrus/sync_client[4000]: Error 
in do_sync(): bailing out! The remote Server(s) denied the operation

Jul  3 09:42:45 store-1-replicat-etudiant cyrus/sync_client[4045]: 
SYNCNOTICE: highestmodseq higher on replica user.e21603781, updating 
8692 => 8694
Jul  3 09:42:45 store-1-replicat-etudiant cyrus/sync_client[4045]: 
SYNCNOTICE: record mismatch with replica: user.e21603781 more recent on 
replica


Jul  3 09:42:45 store-1-replicat-etudiant cyrus/sync_client[4045]: 
SYNCNOTICE: master uid:4340 modseq:8692 last_updated:1593736158 
internaldate:1593736091 flags:(\Seen) cid:0000000000000000
Jul  3 09:42:45 store-1-replicat-etudiant cyrus/sync_client[4045]: 
SYNCNOTICE: replica uid:4340 modseq:8693 last_updated:1593762055 
internaldate:1593736091 flags:(\Seen) cid:0000000000000000

Jul  3 09:43:03 store-1-replicat-etudiant cyrus/sync_client[4045]: 
MAILBOX received NO response: IMAP_SYNC_CHECKSUM Checksum Failure
Jul  3 09:43:03 store-1-replicat-etudiant cyrus/sync_client[4045]: CRC 
failure on sync for user.e21913149, trying full update


Therefore, there are 24Go of additionals mails on the master IMAP spool, 
compared to the replica.

What's is the best way to empty the replica sync_log?
I would like to be sure that every mail arrived on the replica during 
the failover has been copied on the master.
I could use external tool like imapsync, maybe Cyrus has one that 
permits that.

Thank you,

--
Ismaël Tanguy