Strange new fetchmail / cyrus problem...can anyone figure this one out?

[Neal Lippman]

I have encountered a strange new problem/bug. I think I have tracked it
down in terms of WHERE in the chain of events it happens, but not why.

By way of background, I run a cyrus imap server, and use fetchmail to
retrieve email from a number of pop3 accounts and distribute the mail to
the appropriate cyrus accounts. I've had this system in operation for
about a year on a mandrake system. Recently, I switched my server to a
new debian woody (stable) based system, and with upgraded cyrus install.

Presently, I have fetchmail version 5.9.11, and the cyrus version is
2.1.12, and then cyrus-sasl version is also 2.1.12. I have fetchmail
forwarding the emails it retrieves to cyrus via lmtp.

For a while, but I am not absolutely sure dating back to when I first
switched servers, I have noted that my volume of email has gone down. I
also began realizing about a week ago or so that I wasn't getting some
email messages. At first I thought the problem was with my outside pop
server, because of the complicated way that I have some forwarding
accounts set up so that email going to one of my addresses gets
autoforwarded to TWO pop accounts, one which I use at work and one which
I use at home. Over this weekend, however, I realized that an email I
had sent directly to my pop account from work never arrived, and then as
I looked for some emails my wife had expected to receive, they weren't
here as well.

I've been testing things out, and what I can figure out is that if I
have several emails which have arrived in a particular pop account,
fetchmail seems to be receiving all the emails, but the FIRST one
retrieves never makes it into the cyrus mail store. This remains true
even after rebuilding the mailboxes. If only one email is sent to the
pop account, it never arrives at all. If there are multiple emails in
the account when fetchmail makes it's retreival connection, emails 2..n
arrive just fine. This, I think, accounts for my reduced email volume
and lost emails, since I am losing one of every few emails and if only
one email has arrived at the time of a fetchmail run, it gets lost.

Here's some info from /var/log/syslog after fetchmail has just been
restarted, with 2 messages that I have just send (with fetchmail stopped
so I will know exactly what is accumulating in the mailbox) ready to be
retrieved:

**** tail /var/log/syslog
Apr  6 17:50:40 rivendell fetchmail[27972]: starting fetchmail 5.9.11 daemon  
Apr  6 17:50:54 rivendell fetchmail[27972]: 2 messages for fred at mail.lippman.org (1506 octets). 
Apr  6 17:50:55 rivendell fetchmail[27972]: reading message fred at mail.lippman.org:1 of 2 (753 octets) 
Apr  6 17:50:55 rivendell lmtpd[27955]: accepted connection
Apr  6 17:50:55 rivendell lmtpd[27955]: lmtp connection preauth'd as postman
Apr  6 17:50:55 rivendell fetchmail[27972]:  flushed 
Apr  6 17:50:55 rivendell master[27990]: about to exec /usr/cyrus/bin/lmtpd
Apr  6 17:50:55 rivendell lmtpunix[27990]: executed
Apr  6 17:50:55 rivendell fetchmail[27972]: reading message fred at mail.lippman.org:2 of 2 (753 octets) 
Apr  6 17:50:55 rivendell lmtpd[27955]: duplicate_check: <1049666094.15328.14.camel at gandalf>      user.fred           0
Apr  6 17:50:55 rivendell lmtpd[27955]: mystore: starting txn 2147483666
Apr  6 17:50:55 rivendell lmtpd[27955]: mystore: committing txn 2147483666
Apr  6 17:50:55 rivendell lmtpd[27955]: duplicate_mark: <1049666094.15328.14.camel at gandalf>      fred           1049665855
Apr  6 17:50:55 rivendell fetchmail[27972]:  flushed 
****

Note that a duplicate check occurs for message 2 of 2, but not for message 1 of 2. If there were three messages to be retrieved, a duplicate check would happen for 
messages 2 and 3, but not 1, and so on.

Also, note that this log occured using the following /etc/cyrus.conf, and I've set prefork=1 for lmtpunix (figuring maybe
fetchmail was getting confused while waiting for master to start lmtpd), but this had no effect:

**** /etc/cyrus.conf
# standard standalone server implementation

START {
  # do not delete this entry!
  recover	cmd="ctl_cyrusdb -r"

  # this is only necessary if using idled for IMAP IDLE
#  idled		cmd="idled"
}

# UNIX sockets start with a slash and are put into /var/imap/socket
SERVICES {
  # add or remove based on preferences
  imap		cmd="imapd" listen="imap" prefork=0
  imaps		cmd="imapd -s" listen="imaps" prefork=0
  pop3		cmd="pop3d" listen="pop3" prefork=0
  pop3s		cmd="pop3d -s" listen="pop3s" prefork=0
  sieve		cmd="timsieved" listen="sieve" prefork=0

  # at least one LMTP is required for delivery
#  lmtp		cmd="lmtpd" listen="lmtp" prefork=0
  lmtpunix	cmd="lmtpd" listen="/var/imap/socket/lmtp" prefork=1

  # this is only necessary if using notifications
#  notify	cmd="notifyd" listen="/var/imap/socket/notify" proto="udp" prefork=1
}

EVENTS {
  # this is required
  checkpoint	cmd="ctl_cyrusdb -c" period=30

  # this is only necessary if using duplicate delivery suppression
  delprune	cmd="ctl_deliver -E 3" at=0400

  # this is only necessary if caching TLS sessions
  tlsprune	cmd="tls_prune" at=0400
}

****

Finally, for reference, here's my imapd.conf:
****
configdirectory: /var/imap
partition-default: /var/spool/imap
admins: cyrus root
svrtab: /var/imap/svrtab
allowanonymouslogins: no
sasl_pwcheck_method: auxprop
***

For now, I've just disabled the whole fetchmail/imap thing, but this is really bothersome after a full year of successful use of cyrus.

I'd appreciate any help from some anyone knowledgeable enought to fix this thing for me...I'm guessing the problem is at the lmtp/imad level and not with fetchmail,
which seems to be trying it's best, but I guess i'm not even sure of that yet.

nl