IMAP/Murder mass deleting users

Jim Herbert jherbert at spsu.edu
Tue Aug 30 17:53:52 EDT 2005



Any help anyone can provide with the following issue will be greatly
appreciated.  I believe I have stumbled into a seriously dangerous
scenario with Cyrus IMAP+Murder, in which IMAP mass-deletes users.

------ CONFIGURATION -----

I am running RHEL4 kernel 2.6.9-11.ELsmp on dual XEON IBM blades.  I am
using the RPM version of Cyrus-IMAP 2.2.12-3.RHEL4.1.  I am authenticating
users via LDAP using the RPM version of saslauthd 2.1.19-5.EL4.

I have three IMAP backends, with two Murder proxys.  I run an MUpdate
master on the first proxy and slave on the second proxy.

--------------------------

Here is what happened...  The log files on both of my proxy servers slowly
filled the filesystem.  Eventually, mail stopped being delivered to the
LMTP proxys.  The following error appeared in my Exim logs:

Remote host 192.168.9.40 closed connection in response to end of data

192.168.9.40 belongs to one of the proxys.  As I would discover later,
this error was due to the filesystem being full on the master MUpdate and
proxy server.  At the time, I assumed that the proxy or MUpdate had lost
its mind, so I restarted it.  I also ran a "ctl_mboxlist -m" on each
backend to repopulate the MUpdate database.  That obviously, didn't fix
it.

After discovering that the issue was a full filesystem on the MUpdate
master/proxy, I cleaned up the log files to make more space.  I then
restarted Cyrus on the proxy.  At that point, mail delivery resumed and I
assumed all was well.

After some time, I started to notice the following errors in the Exim logs:

LMTP error after DATA: 451 4.3.0 transient system error

And the following in my /var/log/maillog on one of the three IMAP backends:

lmtp[17902]: verify_user(user.xyz) failed: Mailbox does not exist

After inspecting /var/spool/imap, I found that Cyrus had mass-deleted all
users, starting with the letter 'v' (I hash my spools), all the way to
'z'.
It also removed the top level directories once they were empty.  So, I was
left with letters a - u and a partial 'v'.  Everything else, about 300
users, were gone.  The logs in /var/log/maillog were full of the
following:

ctl_mboxlist[17856]: LOSTQUOTA: unable to remove quota root user.xyz for
user.xyz: Mailbox does not exist

It listed all users that were missing in reverse alphabetical order.  I do
not know enough about the IMAP/Murder internals to comment on why this
could happen, but it seems that each IMAP backend transferred its database
to the MUpdate database, right up to the last backend, where somewhere
around the letter 'v', the MUpdate server's filesystem filled.  The
backend may not have expected this issue mid-transfer and the result was
not pretty.

For now, I've restored all of the mailboxes from tape and manually
performed a "create mailbox" + "reconstruct" on the backend.  All is well
for the moment and as long as we keep an eye on disk usage.  But, this
sounds like something that needs to be addressed before it bites someone
else.  For me, I'd like to feel comfortable knowing that this can't happen
again.  So, any help that anyone can lend in locating and eliminating this
issue would be greatly appreciated.

      Thanks in advance!

      -Jim




-- 
Jim Herbert
Director of IT Systems, Networks & Security
Southern Polytechnic State University


-- 
Jim Herbert
Director of IT Systems, Networks & Security
Southern Polytechnic State University




More information about the Cyrus-devel mailing list