Upgrading from 2.2 to 2.4 (slow cyr_expire)

[Andrew Morgan]

On Thu, 16 Oct 2014, Jay Sekora wrote:

> Hi.  I recently tried to upgrade/migrate our Cyrus deployment from
> 2.2.13 (on Debian) to 2.4.17 (on Ubuntu 14.04).  In our environment,
> user mailboxes (about 3TB of them) are on iSCSI volumes; everything else
> is on local disk (which I rsync'ed).
>
> I ran into some delays to do with the storage backend configuration, so
> I didn't actually get to the point of starting imapd on the new server
> until disturbingly close to the end of our announced downtime window.
>
> When I did, I saw that imapd wasn't responding and cyr_expire was
> running.  I was expecting that, but eyeballing what it was doing via
> strace suggested that it would have taken *over twenty hours* to walk
> all the mailboxes.  (I'm guessing that cyr_expire was doing, perhaps as
> a side effect, the "full re-parse of all messages, which may take a
> while" mentioned under "Upgrading from 2.4.3" at
> http://cyrusimap.org/docs/cyrus-imapd/2.4.17/install-upgrade.php.)  So
> we announced that we were backing out of the upgrade.
>
> However, as I was getting ready to back out, I killed the cyr_expire by
> hand, and at that point imapd started responding and I was able to log
> in and see my own mail (which I know cyr_expire hadn't gotten to).  It
> was a little slow to initially show my my mail, which suggests that
> maybe Cyrus was running cyr_expire or its equivalent after I
> authenticated and before showing my my inbox, but that led me to wonder
> whether it might be safe (when we repeat the migration) to kill the
> cyr_expire on initial startup so that Cyrus will start talking IMAP
> right away, and run it in the background.
>
> In case it matters, we have a bunch of emeritus users who occasionally
> check their mail at our site but don't use it on a day-to-day basis, and
> a bunch of users who forward their mail elsewhere and leave a copy on
> our IMAP server as a backup, and a lot of our heavy users are
> sophisticated enough not to leave all their mail in their inboxes so
> when we open the floodgates a very large fraction of that ~3TB is not
> going to be looked at immediately.

You could comment cyr_expire out of cyrus.conf before you upgrade.  After 
a few days, you could uncomment cyr_expire and send a HUP signal to the 
Cyrus master process to have it re-read cyrus.conf.

Remember, the mailbox will be upgraded anytime it is opened.  That will 
occur when a user checks their mailbox AND when new mail is delivered. 
Still, it seems reasonably safe.  Your best bet is to schedule the upgrade 
at a time of low usage and try to "touch" as many mailboxes as possible 
before things get really busy.

 	Andy