Loss of mail with Murder+LMTP (imapd 2.3.7): incoming mail rejected just after server restart

[Farzad FARID]

Hi,

In my unified murder configuration (cyrus imapd 2.3.7) with 2 imap
servers and 1 mupdate server, I have a unwanted behaviour when LMTP
tries to deliver mail for a mailbox that is not available.

Lets says I have a user, "user2", whoses mailboxes are on "imap2".

Scenario 1 :
======
 - mupdate UP
 - imap1 UP
 - imap2 UP

    * When mupdate, imap1 and imap2 are running, everything goes well.
      Even when I deliver mail via LMTP to "imap1", who doesn't own
      "users2"'s mailboxes. "imap1" then connects to "imap2" and
      correctly delivers the mail.

==> OK

Scenario 2 :
=======
 - mupdate UP
 - imap1 DOWN, then UP for less than 10 minutes
 - imap2 DOWN

Now I shut down both "imap1""imap2", and restart ONLY imap1. The mupdate
server is always running and has never been shut down.

    * Again, I send an email for "user2" via LMTP to "imap1", the
      mailbox being hosted on "imap2".

    * Instead of returning a soft error (4XX) as I would have expected,
      imap1 returns a hard error (5XX) to the SMTP/LMTP server, and the
      mail is lost!

==> Incoming Mail lost

Here are log transcripts:

- On the LMTP server (Postfix 2.0.x):

Sep 14 11:50:29 galia postfix/lmtp[24424]: 80E8473F35:
to=<user2 at xxxxx.com>, relay=imap1[10.12.17.44], delay=1154,
status=bounced (host imap1[10.12.17.44] said: 550-Mailbox unknown. 
Either there is no mailbox associated with this 550-name or you do not
have authorization to see it. 550 5.1.1 User unknown (in reply to RCPT
TO command))

- On "imap1":

Sep 14 11:50:28 imap1 cyrus/lmtp[26411]: connection from
galia.srv.in.karavel.com [10.12.17.31]
Sep 14 11:50:28 imap1 cyrus/master[26548]: about to exec
/usr/lib/cyrus/bin/lmtpd
Sep 14 11:50:28 imap1 cyrus/lmtp[26411]: login: galia.srv.in.karavel.com
[10.12.17.31] postman DIGEST-MD5 User logged in
Sep 14 11:50:28 imap1 cyrus/lmtp[26548]: executed
Sep 14 11:50:28 imap1 cyrus/lmtp[26411]: kick_mupdate: can't connect to
target: Connection refused
Sep 14 11:50:28 imap1 cyrus/lmtp[26411]: verify_user(user.user2) failed:
Mailbox does not exist
Sep 14 11:53:09 imap1 cyrus/master[26353]: process 26411 exited, status 0


Scenario 3 :
======
 - mupdate UP
 - imap1 UP for more than 10 minutes
 - imap2 DOWN

This time, if I wait *a few minutes* before resending a mail to "user2"
via LMTP on "imap1". I do get a soft error, as expected.
 
==> Incoming Mail deferred.

Here are the new log transcripts:

 - On the LMTP server:

Sep 14 11:59:54 galia postfix/lmtp[25497]: CDB3573D48:
to=<user2 at xxxx.com>, relay=imap1[10.12.17.44], delay=0, status=deferred
(host imap1[10.12.17.44] said: 451 4.4.3 Remote server unavailable (in
reply to end of DATA command))

- On the "imap1" server:

Sep 14 11:59:53 imap1 cyrus/master[26560]: about to exec
/usr/lib/cyrus/bin/lmtpd
Sep 14 11:59:53 imap1 cyrus/lmtp[26530]: connection from
galia.srv.in.karavel.com [10.12.17.31]
Sep 14 11:59:53 imap1 cyrus/lmtp[26560]: executed
Sep 14 11:59:53 imap1 cyrus/lmtp[26530]: login: galia.srv.in.karavel.com
[10.12.17.31] postman DIGEST-MD5 User logged in
Sep 14 11:59:53 imap1 cyrus/lmtp[26530]: connect(imap2) failed:
Connection refused

Question:
======

Can anybody tell me what went wrong and why "imap1" wasn't aware of the
existence of mailboxes on "imap2" just after startup?

I have another question, related to imapd startup :
 - I have 13000 mailboxes on "imap1" and 14000 mailboxes on "imap2".
 - When I start "imap1", it kicks "ctl_mboxlist -m" before spawning
other processes. This mailbox synchronisation process is very slow and
takes at least 4 minutes.

Why is this "ctl_mbox_list -m" so slow? Is this related to the fact that
even when "imap1" is up and running, it still isn't fully operational
before at least 10 minutes?

Here is some info on my configuration. I can provide more info if needed.
- All 3 servers running Debian stable+testing + home compiled Cyrus
Imapd 2.3.7
- They all have bi-XEON with at least 3 Gb of RAM
- Server load is very low, as this is a test environnment
- 13000 mailboxes on imap1, 14000 on imap2
- 1200 total users, stored in LDAP
- Database formats:
ANNOTATION skiplist
DUPLICATE berkeley-nosync
MBOX skiplist
PTS berkeley
QUOTA quotalegacy
SEEN skiplist
SUBS flat
TLS berkeley-nosync

 Regards

-- 
Farzad FARID <ffarid at pragmatic-source.com>
Architecte Open Source / Pragmatic Source
http://www.pragmatic-source.com/

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.andrew.cmu.edu/pipermail/info-cyrus/attachments/20060914/e534c575/attachment-0001.html