Migrate from 2.2.13 to 2.4.17 disasters

[Stefan Suurmeijer]

Hi all,

I'm trying to migrate my old server to a new one. The old one is still 
running 2.2.13 and the new one 2.4.17.
I'm running into trouble, however

I used rsync to copy mailboxes, databases and sieve stuff to the new server

I checked to see what type of database I was using. Most databases were 
already skiplist, except for deliver.db and tls_sessions.db. For some 
reason those two were still Berkeley. 

After I rsynced I ran reconstruct -n to see if it had anything to say. It 
did in fact:

cyrus/reconstruct[5765]: Index upgrade: user.stefan.Spam (6 -> 12)
(it did that for every mailbox)

cyrus/reconstruct[5765]: DBERROR db5: BDB1538 Program version 5.3 doesn't 
match environment version 4.7
cyrus/reconstruct[5765]: DBERROR: dbenv->open '/var/lib/cyrus/db' failed: 
BDB0091 DB_VERSION_MISMATCH: Database environment version mismatch

I got that just once (or at least I didn't see it more often)

And I saw this:
cyrus/ctl_cyrusdb[20917]: converting /var/lib/cyrus/deliver.db from 
berkeley to skiplist

Which seemed to be a good thing

After that I tried to log in. Then the fun started

First of: I use virtual domains. I have a default domain that I started out 
with, that was located under user (so not under a directory domain, just 
/path/to/cyrus/s/user/stefan

I could't log into that account anymore: Cyrus couldn't find the user 
account. 

However, users in the virtual domains (located under 
/path/to/cyrus/domain/s/somedomain.nl are able to login. 

I copied all settings from the old imapd.conf to the new one, so I figure 
someting has changed in the way virtual domains are handled. Do I need to 
move everything under domain now?


But then something even worse happened: after a reboot Cyrus completely 
freaked out:

Aug 23 00:47:29 spice cyrus/master[24908]: about to exec /usr/sbin/cyrus
Aug 23 00:47:29 spice cyrus/ctl_cyrusdb[24908]: DBERROR db5: BDB1521 
Recovery function for LSN 91 3293614 failed
Aug 23 00:47:29 spice cyrus/ctl_cyrusdb[24908]: DBERROR db5: BDB0061 PANIC: 
Invalid argument
Aug 23 00:47:29 spice cyrus/ctl_cyrusdb[24908]: DBERROR: critical database 
situation
Aug 23 00:47:29 spice cyrus/master[24906]: process 24908 exited, status 75
Aug 23 00:47:29 spice cyrus/master[24911]: about to exec /usr/sbin/cyrus
Aug 23 00:47:29 spice cyrus/cyr_expire[24911]: DBERROR db5: BDB0060 PANIC: 
fatal region error detected; run recovery
Aug 23 00:47:29 spice cyrus/cyr_expire[24911]: DBERROR: critical database 
situation
Aug 23 00:47:29 spice cyrus/master[24906]: process 24911 exited, status 75
Aug 23 00:47:29 spice cyrus/master[24914]: about to exec /usr/sbin/cyrus
Aug 23 00:47:29 spice cyrus/tls_prune[24914]: DBERROR db5: BDB0060 PANIC: 
fatal region error detected; run recovery
Aug 23 00:47:29 spice cyrus/tls_prune[24914]: DBERROR: critical database 
situation
Aug 23 00:47:29 spice cyrus/master[24906]: process 24914 exited, status 75
Aug 23 00:47:29 spice cyrus/master[24906]: unable to setsocketopt(IP_TOS): 
Operation not supported
Aug 23 00:47:29 spice cyrus/master[24906]: unable to setsocketopt(IP_TOS): 
Operation not supported
Aug 23 00:47:29 spice cyrus/master[24906]: ready for work
Aug 23 00:47:29 spice cyrus/master[24917]: about to exec /usr/sbin/cyrus
Aug 23 00:47:29 spice cyrus/master[24918]: about to exec 
/usr/lib/cyrus/bin/notifyd
Aug 23 00:47:29 spice cyrus/notify[24918]: DBERROR db5: BDB0060 PANIC: 
fatal region error detected; run recovery
Aug 23 00:47:29 spice cyrus/notify[24918]: DBERROR: critical database 
situation
Aug 23 00:47:29 spice cyrus/master[24906]: process 24918 exited, status 75
Aug 23 00:47:29 spice cyrus/master[24906]: service notify pid 24918 in 
READY state: terminated abnormally
Aug 23 00:47:29 spice cyrus/master[24920]: about to exec 
/usr/lib/cyrus/bin/notifyd
Aug 23 00:47:29 spice cyrus/ctl_cyrusdb[24917]: DBERROR db5: BDB0060 PANIC: 
fatal region error detected; run recovery
Aug 23 00:47:29 spice cyrus/ctl_cyrusdb[24917]: DBERROR: critical database 
situation
Aug 23 00:47:29 spice cyrus/master[24906]: process 24917 exited, status 75
Aug 23 00:47:29 spice cyrus/notify[24920]: DBERROR db5: BDB0060 PANIC: 
fatal region error detected; run recovery
Aug 23 00:47:29 spice cyrus/notify[24920]: DBERROR: critical database 
situation
Aug 23 00:47:29 spice cyrus/master[24906]: process 24920 exited, status 75
Aug 23 00:47:29 spice cyrus/master[24906]: service notify pid 24920 in 
READY state: terminated abnormally
Aug 23 00:47:29 spice cyrus/master[24922]: about to exec 
/usr/lib/cyrus/bin/notifyd
Aug 23 00:47:29 spice cyrus/notify[24922]: DBERROR db5: BDB0060 PANIC: 
fatal region error detected; run recovery
Aug 23 00:47:29 spice cyrus/notify[24922]: DBERROR: critical database 
situation
Aug 23 00:47:29 spice cyrus/master[24906]: process 24922 exited, status 75
Aug 23 00:47:29 spice cyrus/master[24906]: service notify pid 24922 in 
READY state: terminated abnormally
Aug 23 00:47:29 spice cyrus/master[24923]: about to exec 
/usr/lib/cyrus/bin/notifyd
Aug 23 00:47:29 spice cyrus/notify[24923]: DBERROR db5: BDB0060 PANIC: 
fatal region error detected; run recovery
Aug 23 00:47:29 spice cyrus/notify[24923]: DBERROR: critical database 
situation
Aug 23 00:47:29 spice cyrus/master[24906]: process 24923 exited, status 75
Aug 23 00:47:29 spice cyrus/master[24906]: service notify pid 24923 in 
READY state: terminated abnormally

This went on until I killed Cyrus. Dozens of lines of log per second. 
Can anyone tell me what went wrong? Cyrus wouldn't exit gracefully, so I 
stopped it with a kill -9 before rebooting, did that cause the problem?

I actually repeated the entire process and got exactly the same result

Any help would be GREATLY appreciated

Thanks
Stefan