Repost - fetchmail/lmtpd problem

[Neal Lippman]

This is a repost of a message posted yesterday. Sorry for the repost;
however, with my mail system being messed up (see below for details,
that's the very problem I'm trying to fix) I haven't been able to be
sure that I didn't lose any incoming email over the past day or so, so I
am taking a shot by reposting. Thanks.



I have encountered a strange new problem/bug. I think I have tracked it
down in terms of WHERE in the chain of events it happens, but not why.

By way of background, I run a cyrus imap server, and use fetchmail to
retrieve email from a number of pop3 accounts and distribute the mail to
the appropriate cyrus accounts. I've had this system in operation for
about a year on a mandrake system. Recently, I switched my server to a
new debian woody (stable) based system, and with upgraded cyrus install.

Presently, I have fetchmail version 5.9.11, and the cyrus version is
2.1.12, and then cyrus-sasl version is also 2.1.12. I have fetchmail
forwarding the emails it retrieves to cyrus via lmtp.

For a while, but I am not absolutely sure dating back to when I first
switched servers, I have noted that my volume of email has gone down. I
also began realizing about a week ago or so that I wasn't getting some
email messages. At first I thought the problem was with my outside pop
server, because of the complicated way that I have some forwarding
accounts set up so that email going to one of my addresses gets
autoforwarded to TWO pop accounts, one which I use at work and one which
I use at home. Over this weekend, however, I realized that an email I
had sent directly to my pop account from work never arrived, and then as
I looked for some emails my wife had expected to receive, they weren't
here as well.

I've been testing things out, and what I can figure out is that if I
have several emails which have arrived in a particular pop account,
fetchmail seems to be receiving all the emails, but the FIRST one
retrieves never makes it into the cyrus mail store. This remains true
even after rebuilding the mailboxes. If only one email is sent to the
pop account, it never arrives at all. If there are multiple emails in
the account when fetchmail makes it's retreival connection, emails 2..n
arrive just fine. This, I think, accounts for my reduced email volume
and lost emails, since I am losing one of every few emails and if only
one email has arrived at the time of a fetchmail run, it gets lost.

Here's some info from /var/log/syslog after fetchmail has just been
restarted, with 2 messages that I have just send (with fetchmail stopped
so I will know exactly what is accumulating in the mailbox) ready to be
retrieved:

**** tail /var/log/syslog
Apr  6 17:50:40 rivendell fetchmail[27972]: starting fetchmail 5.9.11
daemon  
Apr  6 17:50:54 rivendell fetchmail[27972]: 2 messages for fred at
mail.lippman.org (1506 octets). 
Apr  6 17:50:55 rivendell fetchmail[27972]: reading message
fred at mail.lippman.org:1 of 2 (753 octets) 
Apr  6 17:50:55 rivendell lmtpd[27955]: accepted connection
Apr  6 17:50:55 rivendell lmtpd[27955]: lmtp connection preauth'd as
postman
Apr  6 17:50:55 rivendell fetchmail[27972]:  flushed 
Apr  6 17:50:55 rivendell master[27990]: about to exec
/usr/cyrus/bin/lmtpd
Apr  6 17:50:55 rivendell lmtpunix[27990]: executed
Apr  6 17:50:55 rivendell fetchmail[27972]: reading message
fred at mail.lippman.org:2 of 2 (753 octets) 
Apr  6 17:50:55 rivendell lmtpd[27955]: duplicate_check:
<1049666094.15328.14.camel at gandalf>      user.fred           0
Apr  6 17:50:55 rivendell lmtpd[27955]: mystore: starting txn 2147483666
Apr  6 17:50:55 rivendell lmtpd[27955]: mystore: committing txn
2147483666
Apr  6 17:50:55 rivendell lmtpd[27955]: duplicate_mark:
<1049666094.15328.14.camel at gandalf>      fred           1049665855
Apr  6 17:50:55 rivendell fetchmail[27972]:  flushed 
****

Note that a duplicate check occurs for message 2 of 2, but not for
message 1 of 2. If there were three messages to be retrieved, a
duplicate check would happen for 
messages 2 and 3, but not 1, and so on.

Also, note that this log occured using the following /etc/cyrus.conf,
and I've set prefork=1 for lmtpunix (figuring maybe
fetchmail was getting confused while waiting for master to start lmtpd),
but this had no effect:

**** /etc/cyrus.conf
# standard standalone server implementation

START {
  # do not delete this entry!
  recover       cmd="ctl_cyrusdb -r"

  # this is only necessary if using idled for IMAP IDLE
#  idled                cmd="idled"
}

# UNIX sockets start with a slash and are put into /var/imap/socket
SERVICES {
  # add or remove based on preferences
  imap          cmd="imapd" listen="imap" prefork=0
  imaps         cmd="imapd -s" listen="imaps" prefork=0
  pop3          cmd="pop3d" listen="pop3" prefork=0
  pop3s         cmd="pop3d -s" listen="pop3s" prefork=0
  sieve         cmd="timsieved" listen="sieve" prefork=0

  # at least one LMTP is required for delivery
#  lmtp         cmd="lmtpd" listen="lmtp" prefork=0
  lmtpunix      cmd="lmtpd" listen="/var/imap/socket/lmtp" prefork=1

  # this is only necessary if using notifications
#  notify       cmd="notifyd" listen="/var/imap/socket/notify"
proto="udp" prefork=1
}

EVENTS {
  # this is required
  checkpoint    cmd="ctl_cyrusdb -c" period=30

  # this is only necessary if using duplicate delivery suppression
  delprune      cmd="ctl_deliver -E 3" at=0400

  # this is only necessary if caching TLS sessions
  tlsprune      cmd="tls_prune" at=0400
}

****

Finally, for reference, here's my imapd.conf:
****
configdirectory: /var/imap
partition-default: /var/spool/imap
admins: cyrus root
svrtab: /var/imap/svrtab
allowanonymouslogins: no
sasl_pwcheck_method: auxprop
***

For now, I've just disabled the whole fetchmail/imap thing, but this is
really bothersome after a full year of successful use of cyrus.

I'd appreciate any help from some anyone knowledgeable enought to fix
this thing for me...I'm guessing the problem is at the lmtp/imad level
and not with fetchmail,
which seems to be trying it's best, but I guess i'm not even sure of
that yet.

nl