Migrate from 2.2.13 to 2.4.17 disasters

[Bron Gondwana]

a) I'm SOOO glad that we got rid of Berkeley.
b) Berkeley, bah

On Sun, Aug 23, 2015, at 11:38, Stefan Suurmeijer wrote:
> Hi all,
>
> I'm trying to migrate my old server to a new one. The old one is
> still running 2.2.13 and the new one 2.4.17. I'm running into
> trouble, however
>
> I used rsync to copy mailboxes, databases and sieve stuff to the
> new server
>
> I checked to see what type of database I was using. Most databases
> were already skiplist, except for deliver.db and tls_sessions.db. For
> some reason those two were still Berkeley.

So you pretty much have to have dumped all your DB files before you
upgraded, because it doesn't do upgrades nicely.  You might be able
to use the db_upgrade program on them.  My Ubuntu box has
db5.1_upgrade on it.

The good news about these one is that you can delete them when the
server is shut down without losing anything of value :)

> After I rsynced I ran reconstruct -n to see if it had anything to say.
> It did in fact:
>
> cyrus/reconstruct[5765]: Index upgrade: user.stefan.Spam (6 -> 12) (it
> did that for every mailbox)
>
> cyrus/reconstruct[5765]: DBERROR db5: BDB1538 Program version 5.3
> doesn't match environment version 4.7 cyrus/reconstruct[5765]:
> DBERROR: dbenv->open '/var/lib/cyrus/db' failed: BDB0091
> DB_VERSION_MISMATCH: Database environment version mismatch

With the server shut down:

rm -rf /var/lib/cyrus/db
rm -f /var/lib/cyrus/deliver.db
rm -f /var/lib/cyrus/tls_sessions.db

And of course change them to skiplist as well :)

> I got that just once (or at least I didn't see it more often)
>
> And I saw this: cyrus/ctl_cyrusdb[20917]: converting
> /var/lib/cyrus/deliver.db from berkeley to skiplist

Yeah, if you've changed the config it will do that.  It should be a good
thing, but the Berkeley environment can hold references to the old
berkeley DB still.

> Which seemed to be a good thing
>
> After that I tried to log in. Then the fun started
>
> First of: I use virtual domains. I have a default domain that I
> started out with, that was located under user (so not under a
> directory domain, just /path/to/cyrus/s/user/stefan
>
> I could't log into that account anymore: Cyrus couldn't find the user
> account.

Were you logging in as user at domain, or just 'user'?

> However, users in the virtual domains (located under
> /path/to/cyrus/domain/s/somedomain.nl are able to login.
>
> I copied all settings from the old imapd.conf to the new one, so I
> figure someting has changed in the way virtual domains are handled. Do
> I need to move everything under domain now?

That or just log in with the raw username rather than domain.  That's
for virtdomains: userid, I don't have a clue for the non-userid version.
This _might_ be a regression from 2.3 to 2.4.

> But then something even worse happened: after a reboot Cyrus
> completely freaked out:
>
> Aug 23 00:47:29 spice cyrus/master[24908]: about to exec
> /usr/sbin/cyrus Aug 23 00:47:29 spice cyrus/ctl_cyrusdb[24908]:
> DBERROR db5: BDB1521 Recovery function for LSN 91 3293614 failed Aug
> 23 00:47:29 spice cyrus/ctl_cyrusdb[24908]: DBERROR db5: BDB0061
> PANIC: Invalid argument Aug 23 00:47:29 spice
> cyrus/ctl_cyrusdb[24908]: DBERROR: critical database situation Aug 23
> 00:47:29 spice cyrus/master[24906]: process 24908 exited, status 75
> Aug 23 00:47:29 spice cyrus/master[24911]: about to exec
> /usr/sbin/cyrus Aug 23 00:47:29 spice cyrus/cyr_expire[24911]: DBERROR
> db5: BDB0060 PANIC: fatal region error detected; run recovery Aug 23
> 00:47:29 spice cyrus/cyr_expire[24911]: DBERROR: critical database
> situation Aug 23 00:47:29 spice cyrus/master[24906]: process 24911
> exited, status 75 Aug 23 00:47:29 spice cyrus/master[24914]: about to
> exec /usr/sbin/cyrus Aug 23 00:47:29 spice cyrus/tls_prune[24914]:
> DBERROR db5: BDB0060 PANIC: fatal region error detected; run recovery
> Aug 23 00:47:29 spice cyrus/tls_prune[24914]: DBERROR: critical
> database situation Aug 23 00:47:29 spice cyrus/master[24906]: process
> 24914 exited, status 75 Aug 23 00:47:29 spice cyrus/master[24906]:
> unable to setsocketopt(IP_TOS): Operation not supported Aug 23
> 00:47:29 spice cyrus/master[24906]: unable to setsocketopt(IP_TOS):
> Operation not supported Aug 23 00:47:29 spice cyrus/master[24906]:
> ready for work Aug 23 00:47:29 spice cyrus/master[24917]: about to
> exec /usr/sbin/cyrus Aug 23 00:47:29 spice cyrus/master[24918]: about
> to exec /usr/lib/cyrus/bin/notifyd Aug 23 00:47:29 spice
> cyrus/notify[24918]: DBERROR db5: BDB0060 PANIC: fatal region error
> detected; run recovery Aug 23 00:47:29 spice cyrus/notify[24918]:
> DBERROR: critical database situation Aug 23 00:47:29 spice
> cyrus/master[24906]: process 24918 exited, status 75 Aug 23 00:47:29
> spice cyrus/master[24906]: service notify pid 24918 in READY state:
> terminated abnormally Aug 23 00:47:29 spice cyrus/master[24920]: about
> to exec /usr/lib/cyrus/bin/notifyd Aug 23 00:47:29 spice
> cyrus/ctl_cyrusdb[24917]: DBERROR db5: BDB0060 PANIC: fatal region
> error detected; run recovery Aug 23 00:47:29 spice
> cyrus/ctl_cyrusdb[24917]: DBERROR: critical database situation Aug 23
> 00:47:29 spice cyrus/master[24906]: process 24917 exited, status 75
> Aug 23 00:47:29 spice cyrus/notify[24920]: DBERROR db5: BDB0060 PANIC:
> fatal region error detected; run recovery Aug 23 00:47:29 spice
> cyrus/notify[24920]: DBERROR: critical database situation Aug 23
> 00:47:29 spice cyrus/master[24906]: process 24920 exited, status 75
> Aug 23 00:47:29 spice cyrus/master[24906]: service notify pid 24920 in
> READY state: terminated abnormally Aug 23 00:47:29 spice
> cyrus/master[24922]: about to exec /usr/lib/cyrus/bin/notifyd Aug 23
> 00:47:29 spice cyrus/notify[24922]: DBERROR db5: BDB0060 PANIC: fatal
> region error detected; run recovery Aug 23 00:47:29 spice
> cyrus/notify[24922]: DBERROR: critical database situation Aug 23
> 00:47:29 spice cyrus/master[24906]: process 24922 exited, status 75
> Aug 23 00:47:29 spice cyrus/master[24906]: service notify pid 24922 in
> READY state: terminated abnormally Aug 23 00:47:29 spice
> cyrus/master[24923]: about to exec /usr/lib/cyrus/bin/notifyd Aug 23
> 00:47:29 spice cyrus/notify[24923]: DBERROR db5: BDB0060 PANIC: fatal
> region error detected; run recovery Aug 23 00:47:29 spice
> cyrus/notify[24923]: DBERROR: critical database situation Aug 23
> 00:47:29 spice cyrus/master[24906]: process 24923 exited, status 75
> Aug 23 00:47:29 spice cyrus/master[24906]: service notify pid 24923 in
> READY state: terminated abnormally
>
> This went on until I killed Cyrus. Dozens of lines of log per second.
> Can anyone tell me what went wrong? Cyrus wouldn't exit gracefully, so
> I stopped it with a kill -9 before rebooting, did that cause the
> problem?
>
> I actually repeated the entire process and got exactly the same result
>
> Any help would be GREATLY appreciated

If you follow the steps above to kill the BDB environment (again, with Cyrus
entirely stopped) you should be OK.

Bron.

-- 
  Bron Gondwana
  brong at fastmail.fm