Elusive replication bug in 2.3.13

[Raphael Jaffey]

Hi Janne,

I've reproduced this with both the vanilla Cyrus tarball and the Invoca RPM.

Check out bug report 3130 at:

https://bugzilla.andrew.cmu.edu/show_bug.cgi?id=3130

and my post at:

http://lists.andrew.cmu.edu/pipermail/info-cyrus/2009-January/030279.html

for a discussion of what looks like the same problem you're
experiencing and a workaround that seems to solve the problem.

In my case it's related to how many of those inboxes have quotas  
associated with them and the use of the default 'quota_db' of  
'quotalegacy'.  I was able to work around the replication problem by  
switching 'quota_db' to 'skiplist' on the replica.  Unfortunately I  
couldn't get berkeley or berkeley-hash to work reliably and as I'm  
unsure as to which database other than 'quotalegacy' would be most  
appropriate.  So far, no one has suggested anything.

What happens when you try to delete user.jmm on the master?  Does the  
IMAP process handling the delete segfault?

Regards,
Rafe

Quoting Janne Peltonen <janne.peltonen at helsinki.fi>:

> Hi!
>
> Since I upgraded to 2.3.13 (Invoca RPM rev 4) I've been running into a
> mysterious replication bug. In some circumstances, creating a user    
> with a three
> letter long username causes the sync master process to choke, on    
> either signal
> 11 or 6. Like this:
>
> --clip--
> i08.mappi.helsinki.fi> cm user.jmm
> --clip--
> Jan 12 18:31:01 scn3 i08/syncserver[25821]: Failed to access inbox for jmm
> Jan 12 18:31:01 scn3 i08/master[31569]: process 25821 exited,    
> signaled to death by 6
> Jan 12 18:31:07 scn3 i08/syncserver[30193]: login:    
> pcn3.mappi.helsinki.fi [128.214.20.131] cyr_sync DIGEST-MD5 User    
> logged in
> Jan 12 18:31:07 scn3 i08/syncserver[30193]: Failed to access inbox for jmm
> Jan 12 18:31:07 scn3 i08/master[31569]: process 30193 exited,    
> signaled to death by 11
> [etc, the sync_client keeps retrying]
> --clip--
> [jmmpelto at pcn3 ~]$ ps -ef|grep sync.*i08
> cyrus     8255     1  0 18:16 ?        00:00:00    
> /usr/lib/cyrus-imapd/sync_client -r -C /etc/imapd.conf.i08.master
> cyrus    22092  8255  0 18:33 ?        00:00:00    
> /usr/lib/cyrus-imapd/sync_client -r -C /etc/imapd.conf.i08.master
> jmmpelto 22095 20355  0 18:33 pts/4    00:00:00 grep sync.*i08
> [jmmpelto at pcn3 ~]$ sudo kill 8255
> Password:
> [jmmpelto at pcn3 ~]$ ps -ef|grep sync.*i08
> jmmpelto 25898 20355  0 18:34 pts/4    00:00:00 grep sync.*i08
> [jmmpelto at pcn3 ~]$
> [the child died, again, because the server died]
> --clip--
> [jmmpelto at pcn3 ~]$ sudo su - cyrus -c    
> "/usr/lib/cyrus-imapd/sync_client -r -C /etc/imapd.conf.i08.master"
> [jmmpelto at pcn3 ~]$ ps -ef|grep sync.*i08
> cyrus    11483     1  0 18:35 ?        00:00:00    
> /usr/lib/cyrus-imapd/sync_client -r -C /etc/imapd.conf.i08.master
> cyrus    11484 11483  1 18:35 ?        00:00:00    
> /usr/lib/cyrus-imapd/sync_client -r -C /etc/imapd.conf.i08.master
> jmmpelto 13382 20355  0 18:35 pts/4    00:00:00 grep sync.*i08
> [after restart, rolling replication runs OK]
> --clip--
> [jmmpelto at scn3 ~]$ sudo /usr/lib/cyrus-imapd/mbpath -C    
> /etc/imapd.conf.i08.replica user.jmm
> Invalid mailbox name: user.jmm
> [but the mailbox never got replicated]
> --clip--
> -bash-3.2$ cat log-8255
> USER jmm
> -bash-3.2$ /usr/lib/cyrus-imapd/sync_client -v -o -r -C    
> /etc/imapd.conf.i08.master -f log-8255
> USER jmm
> USER jmm
> USER jmm
> --clip--
> Jan 12 18:39:19 scn3 i08/syncserver[30016]: login:    
> pcn3.mappi.helsinki.fi [128.214.20.131] cyr_sync DIGEST-MD5 User    
> logged in
> Jan 12 18:39:19 scn3 i08/syncserver[30016]: Failed to access inbox for jmm
> Jan 12 18:39:19 scn3 i08/master[31569]: process 30016 exited,    
> signaled to death by 11
> [jmmpelto at scn3 ~]$ sudo /usr/lib/cyrus-imapd/mbpath -C    
> /etc/imapd.conf.i08.replica user.jmm
> Invalid mailbox name: user.jmm
> [trying to say "USER jmm" results in sig11 death, no replication]
> --clip--
> -bash-3.2$ /usr/lib/cyrus-imapd/sync_client -v -o -r -C    
> /etc/imapd.conf.i08.master -f -
> ACL user.jmm
> SETACL user.jmm
>   Promoting: ACL user.jmm -> MAILBOX user.jmm
> MAILBOXES user.jmm
> META jmm
> [..however, doing something that gets escalated to creating the user...]
> --clip--
> [jmmpelto at scn3 ~]$ sudo /usr/lib/cyrus-imapd/mbpath -C    
> /etc/imapd.conf.i08.replica user.jmm
> /var/spool/imap/jjjr/j/user/jmm
> [...doesn't result in process death, instead, the user & mailbox do   
>  get replicated]
> --clip--
>
> The irritating thing about this is, I can't reliably recreate the    
> bug. It appears that
>
> - the user name has to be a prefix of already existing usernames
>
> - there has to be at least some (ten? twenty?) usernames the new    
> username is a
> prefix of - however, when trying to create users jmbtest1..9 and    
> jmbtes10..20,
> I wasn't able to recreate the bug with user.jmb (there were 5 'real'  
>   usernames
> with the prefix jmb), even though I was able to hit it with user.jme  
>   that only
> has 18 real usernames...
>
> I didn't have this problem with 2.3.11, so it has to be relatively new. Might
> anyone else have noticed anything similar?
>
>
> --Janne
> --
> Janne Peltonen <janne.peltonen at helsinki.fi> PGP Key ID: 0x9CFAC88B
> Please consider membership of the Hospitality Club    
> (http://www.hospitalityclub.org)
> ----
> Cyrus Home Page: http://cyrusimap.web.cmu.edu/
> Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
> List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
>