IMAP not seeing old mail present on filesystem

[Shuvam Misra]

Dear Patrick,

> How would you do the IMAP append?  Using a Perl::IMAP function?
> 
> This isn't necessarily a concern for this list, but a few days ago I 
> upgraded a site from cyrus 2.1.16 to cyrus 2.3.16 by using imapsync to 
> transfer mail from the old server to the new one.  This worked great 
> (i.e. all metadata seems to have been preserved), however the old server 
> was still collecting mail for the few hours it took to imapsync all the 
> users (note that this is very slows and probably not appropriate for 
> larger sites).  My plan was to swap the servers and then do a final 
> imapsync from the old IMAP server to the new one.  For some reason, for 
> some users the additional set of new messages is copied to the new 
> server, and others they aren't.  I couldn't figure out what the 
> difference is, and don't want to spend too much more time on this 
> because of the small number of messages in play, so I need some way to 
> transfer a couple of dozen messages "by hand".  In general, though 
> imapsync seems to be a great way to "clean up" cyrus folders when 
> switching servers.  Had I known I would have sync problem later, I would 
> have just taken the old server off line before syncing the messages.
> 
> Before someone suggests that I should have just copied /var/lib/cyrus 
> and the messages over to the new server, I didn't trust this because I 
> couldn't get anyone to confirm the database files I had on the old 
> 2.1.16 server -- the filenames were mostly not the same as this set:
> 
>    annotation_db
>    duplicate_db
>    mboxlist_db
>    ptscache_db
>    quota_db
>    seenstate_db
>    tlscache_db
> 
> and the old server experienced a couple of devastating crashes which 
> required me to cyrreconstruct all the user.user mailboxes a couple of 
> times.  It seemed pretty clear that the db files on the old server were 
> a mess, and upgrading from anywhere from libdb2 to libdb4.3 to libdb4.8 
> seemed sketchy as well.  The old server is such an old debian system 
> that apt-cache depends doesn't seem to work any more because the package 
> servers have changed.  (Yeah, I could probably find the source packages 
> somewhere, but then there are a lot of things I could do given infinite 
> time.)

We use IMAPsync all the time, manually and (more often) through automated
programs.  One of our customers uses Gmail for their corporate email
system (18,000+ user accounts, about 1000-2000 accounts of ex-employees
at any point in time).  We recently wrote a layer on top of our mailing
product for him, to download and archive all the messages of all his
ex-employees' mailboxes.  We used IMAPsync in an automated mode to
download all his mails. He gives us the usernames and passwords, and our
server-based system does the rest, creating local mirror mail accounts
and downloading the Gmail mailboxes to local Cyrus IMAP mailboxes.

We've learned a few things from all this. Firstly, an older version of
IMAPsync (IIRC it was v1.25x) used to hang at random and consume 100% CPU
indefinitely. Upgrading to a recent release (v1.34x or later) has fixed
the problem. Secondly, we have found that IMAP sessions _often_ terminate
halfway for no fault of IMAPsync. Reattempting the IMAP sync succeeds.

Therefore, for our automated IMAP download-and-archive solution, we
wrote a daemon which fires IMAPsync for each account in an infinite loop.
Whenever IMAPsync terminates with errorlevel = 0, we found that we can
safely believe the session completed successfully, and complete sync
has been done. So our daemon makes note of this, and doesn't try to sync
this user's mailboxes any more. Therefore, our IMAPsync monitor daemon
is prepared to retry sync for a recalcitrant user account an infinite
number of times. Once we understood that these errors are not repeatable,
and hits accounts at random, we gave up trying to debug why -- we just
run IMAPsync over and over till we get an all-clear. It works well. Some
users had 10+GB of messages -- all of it is pulled down eventually.

Of course, these random errors are probably more a Google IMAP issue than
IMAP in general.

If you often need to sync mailboxes among IMAP folders, I'd suggest you
write a monitor daemon and fire IMAPsync from within it in an infinite
loop the way I described.

Shuvam