Basic two host replication scenario, SSL failure

[Ivan Lezhnjov Jr.]

Hi,

if people from #cyrus at irc.freenode.net follow this list they might
remember me for I've been around asking many basic questions last week
there.

I'd love first to say that I appreciate all the hospitality and help
they gave, thanks :)

However, as it stands, I have this problem I'm going to describe that
boggles my mind and I can't seem to find a clear-cut answer whether what
I'm up to is even supposed to work.

Using cyrus-imapd 2.4.10, built from jmeeuwen's SRPM aka kolabsys SRPM.

Here's the basic replication scenario I've implemented.

I have got one Master (M) and one Replica (R) host. One user mailbox
that is being synced from M to R via csync/lmtp at port 2005.

First issue, and jmeeuwen in particular believes it to be a bug, but let
project developers be the judge of this, arises when R host goes down
while M is still available trying to push changes to R which has become
unavailable. What happens, essentially, is that when R becomes available
again M fails to push changes unless it [the M] is restarted.

This is not entirely critical for me but it's quite annoying and
somewhat unexpected.

Another thing I do is emulate a situation where M becomes permanently
unavailable while R remains available. I switch R's role to M's by
restarting the service with new set of configuration files then. It
continues to serve and accept incoming mails just fine. Then, when M has
been fixed after an imaginary failure and becomes available again, I
switch M's role to role of R, and try to push changes from R turned M to
the M turned R. 

This doesn't work. Is this even supposed to work? No one really answered
this question directly on IRC. So, please, tell me if that what I'm
doing here has any sense at all.

So, I tried then to revert hosts to their original roles. The problem
then was that R had new messages that M never received while it had been
down. M would accept and store new incoming mail, but it would fail to
sync with R henceforth.

This is what M output to log files (debug output turned on):
Jul  8 19:51:29 imapsite-master syncserver[12392]: EOF in SSL_accept()
-> fail
Jul  8 19:51:29 imapsite-master syncserver[12392]: STARTTLS failed:
imapsite-replica [10.10.0.188]
Jul  8 19:51:29 imapsite-master syncserver[12392]: telling master 1
Jul  8 19:51:29 imapsite-master syncserver[12392]: SSL_accept()
incomplete -> wait
Jul  8 19:52:27 imapsite-master syncserver[12392]: EOF in SSL_accept()
-> fail
Jul  8 19:52:27 imapsite-master syncserver[12392]: STARTTLS failed:
imapsite-replica [10.10.0.188]
Jul  8 19:52:27 imapsite-master syncserver[12392]: telling master 1
Jul  8 19:52:27 imapsite-master syncserver[12392]: telling master 2
Jul  8 19:52:27 imapsite-master syncserver[12392]: accepted connection
Jul  8 19:52:27 imapsite-master syncserver[12392]: telling master 3
Jul  8 19:52:27 imapsite-master syncserver[12392]: cmdloop(): startup
Jul  8 19:52:28 imapsite-master syncserver[12392]: SSL_accept()
incomplete -> wait

Jul  8 19:53:56 imapsite-master sync_client[12616]: MAILBOX received NO
response: IMAP_MAILBOX_CRC Checksum Failure
Jul  8 19:53:56 imapsite-master sync_client[12616]: CRC failure on sync
for user.zxy, trying full update
Jul  8 19:53:56 imapsite-master sync_client[12616]: SYNCERROR: guid
mismatch user.zxy 2 (9e19b5a1e93d3b3e7936044543b444ea00164649 b
b03edc18eaf8d2115510052dff24c65d894b107)
Jul  8 19:53:56 imapsite-master sync_client[12616]: SYNCERROR: guid
mismatch user.zxy 2 (9e19b5a1e93d3b3e7936044543b444ea00164649 b
b03edc18eaf8d2115510052dff24c65d894b107)
Jul  8 19:53:56 imapsite-master sync_client[12616]: user.zxy: same
message appears twice 2 3
Jul  8 19:53:56 imapsite-master sync_client[12616]: Unlinking files in
mailbox user.zxy
Jul  8 19:53:56 imapsite-master sync_client[12616]: do_folders(): update
failed: user.zxy 'Bad protocol'
Jul  8 19:53:56 imapsite-master sync_client[12616]: Error in do_sync():
bailing out! Bad protocol
Jul  8 19:53:56 imapsite-master sync_client[12616]: Processing sync log
file /var/lib/imap/sync/log-12616 failed: Bad protocol

Effectively, the whole system becomes broken and only R turned M can
continue to accept new incoming mail and serve previously stored mail.
The replication becomes broken, though.

My end goal is to have two hosts that will be able to replace one
another in case of a failure of one of them. That means I expect R to
become M, and when real M becomes available again push changes from real
R to M and continue running the service as if nothing happened. If then
real R becomes unavailable due to, say, disk controller failure, I would
revert M turned R to its original role and continue running the service.
So, both hosts would be mutually complementing in their ability to
switch between the roles of M and R, and sync up the other host.

I also tried to duplicate /var/lib/imap and /var/spool/imap with rsync
in the scenario described up above. For the sake of clarity a brief
reminder follows. 

- M pushes changes to R.
- M goes down.
- I turn R to M. 
- It receives new incoming messages. 
- M becomes available again. 
- I run rsync to copy /var/lib/imap and /var/spool/imap off R to replace
the corresponding directories on M.
- I stop R turned M.
- Change its configuration files so that it becomes R again.
- Start the real R
- Start the real M with rsync'ed /var/{lib/imap,spool/imap}

Replication doesn't work now. The question is can it work after doing
this?

So, that's all I have to say perhaps. I would really appreciate any help
with this. This seems like a basic, trivial scenario to me but I just
can't seem to get cyrus-imap working right.

-- 
With respect,
Ivan Lezhnjov Jr.

E-mail/Jabber: ivan.lezhnjov.jr at gmail.com
Phone: +38 050 7780899

Key ID: 0x5811D90C
Key Fingerprint: 2A52 5C8C 38BE C04F D8DE  A169 19E2 E49A 5811 D90C
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: This is a digitally signed message part
Url : http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20110710/cead0e80/attachment.bin