Elusive replication bug in 2.3.13

[Janne Peltonen]

Hi!

Since I upgraded to 2.3.13 (Invoca RPM rev 4) I've been running into a
mysterious replication bug. In some circumstances, creating a user with a three
letter long username causes the sync master process to choke, on either signal
11 or 6. Like this:

--clip--
i08.mappi.helsinki.fi> cm user.jmm
--clip--
Jan 12 18:31:01 scn3 i08/syncserver[25821]: Failed to access inbox for jmm
Jan 12 18:31:01 scn3 i08/master[31569]: process 25821 exited, signaled to death by 6
Jan 12 18:31:07 scn3 i08/syncserver[30193]: login: pcn3.mappi.helsinki.fi [128.214.20.131] cyr_sync DIGEST-MD5 User logged in
Jan 12 18:31:07 scn3 i08/syncserver[30193]: Failed to access inbox for jmm
Jan 12 18:31:07 scn3 i08/master[31569]: process 30193 exited, signaled to death by 11
[etc, the sync_client keeps retrying]
--clip--
[jmmpelto at pcn3 ~]$ ps -ef|grep sync.*i08
cyrus     8255     1  0 18:16 ?        00:00:00 /usr/lib/cyrus-imapd/sync_client -r -C /etc/imapd.conf.i08.master
cyrus    22092  8255  0 18:33 ?        00:00:00 /usr/lib/cyrus-imapd/sync_client -r -C /etc/imapd.conf.i08.master
jmmpelto 22095 20355  0 18:33 pts/4    00:00:00 grep sync.*i08
[jmmpelto at pcn3 ~]$ sudo kill 8255
Password:
[jmmpelto at pcn3 ~]$ ps -ef|grep sync.*i08
jmmpelto 25898 20355  0 18:34 pts/4    00:00:00 grep sync.*i08
[jmmpelto at pcn3 ~]$ 
[the child died, again, because the server died]
--clip--
[jmmpelto at pcn3 ~]$ sudo su - cyrus -c "/usr/lib/cyrus-imapd/sync_client -r -C /etc/imapd.conf.i08.master"
[jmmpelto at pcn3 ~]$ ps -ef|grep sync.*i08
cyrus    11483     1  0 18:35 ?        00:00:00 /usr/lib/cyrus-imapd/sync_client -r -C /etc/imapd.conf.i08.master
cyrus    11484 11483  1 18:35 ?        00:00:00 /usr/lib/cyrus-imapd/sync_client -r -C /etc/imapd.conf.i08.master
jmmpelto 13382 20355  0 18:35 pts/4    00:00:00 grep sync.*i08
[after restart, rolling replication runs OK]
--clip--
[jmmpelto at scn3 ~]$ sudo /usr/lib/cyrus-imapd/mbpath -C /etc/imapd.conf.i08.replica user.jmm
Invalid mailbox name: user.jmm
[but the mailbox never got replicated]
--clip--
-bash-3.2$ cat log-8255 
USER jmm
-bash-3.2$ /usr/lib/cyrus-imapd/sync_client -v -o -r -C /etc/imapd.conf.i08.master -f log-8255 
USER jmm
USER jmm
USER jmm
--clip--
Jan 12 18:39:19 scn3 i08/syncserver[30016]: login: pcn3.mappi.helsinki.fi [128.214.20.131] cyr_sync DIGEST-MD5 User logged in
Jan 12 18:39:19 scn3 i08/syncserver[30016]: Failed to access inbox for jmm
Jan 12 18:39:19 scn3 i08/master[31569]: process 30016 exited, signaled to death by 11
[jmmpelto at scn3 ~]$ sudo /usr/lib/cyrus-imapd/mbpath -C /etc/imapd.conf.i08.replica user.jmm
Invalid mailbox name: user.jmm
[trying to say "USER jmm" results in sig11 death, no replication]
--clip--
-bash-3.2$ /usr/lib/cyrus-imapd/sync_client -v -o -r -C /etc/imapd.conf.i08.master -f -
ACL user.jmm
SETACL user.jmm
  Promoting: ACL user.jmm -> MAILBOX user.jmm
MAILBOXES user.jmm
META jmm
[..however, doing something that gets escalated to creating the user...]
--clip--
[jmmpelto at scn3 ~]$ sudo /usr/lib/cyrus-imapd/mbpath -C /etc/imapd.conf.i08.replica user.jmm
/var/spool/imap/jjjr/j/user/jmm
[...doesn't result in process death, instead, the user & mailbox do get replicated]
--clip--

The irritating thing about this is, I can't reliably recreate the bug. It appears that

- the user name has to be a prefix of already existing usernames

- there has to be at least some (ten? twenty?) usernames the new username is a
prefix of - however, when trying to create users jmbtest1..9 and jmbtes10..20,
I wasn't able to recreate the bug with user.jmb (there were 5 'real' usernames
with the prefix jmb), even though I was able to hit it with user.jme that only
has 18 real usernames...

I didn't have this problem with 2.3.11, so it has to be relatively new. Might
anyone else have noticed anything similar?


--Janne
-- 
Janne Peltonen <janne.peltonen at helsinki.fi> PGP Key ID: 0x9CFAC88B
Please consider membership of the Hospitality Club (http://www.hospitalityclub.org)