Migrate from 2.2.13 to 2.4.17 disasters

Sun Aug 23 12:17:13 EDT 2015

Hi Bron,

thanks for responding

On 23-08-15 12:10, Bron Gondwana wrote:
> a) I'm SOOO glad that we got rid of Berkeley.
> b) Berkeley, bah

LOL. Agree, always trouble when building sendmail as well

>
> On Sun, Aug 23, 2015, at 11:38, Stefan Suurmeijer wrote:
>> Hi all,
>>
>> I'm trying to migrate my old server to a new one. The old one is
>> still running 2.2.13 and the new one 2.4.17. I'm running into
>> trouble, however
>>
>> I used rsync to copy mailboxes, databases and sieve stuff to the
>> new server
>>
>> I checked to see what type of database I was using. Most databases
>> were already skiplist, except for deliver.db and tls_sessions.db. For
>> some reason those two were still Berkeley.
> So you pretty much have to have dumped all your DB files before you
> upgraded, because it doesn't do upgrades nicely.  You might be able
> to use the db_upgrade program on them.  My Ubuntu box has
> db5.1_upgrade on it.
>
> The good news about these one is that you can delete them when the
> server is shut down without losing anything of value :)
>
>> After I rsynced I ran reconstruct -n to see if it had anything to say.
>> It did in fact:
>>
>> cyrus/reconstruct[5765]: Index upgrade: user.stefan.Spam (6 -> 12) (it
>> did that for every mailbox)
>>
>> cyrus/reconstruct[5765]: DBERROR db5: BDB1538 Program version 5.3
>> doesn't match environment version 4.7 cyrus/reconstruct[5765]:
>> DBERROR: dbenv->open '/var/lib/cyrus/db' failed: BDB0091
>> DB_VERSION_MISMATCH: Database environment version mismatch
> With the server shut down:
>
> rm -rf /var/lib/cyrus/db
> rm -f /var/lib/cyrus/deliver.db
> rm -f /var/lib/cyrus/tls_sessions.db
>
> And of course change them to skiplist as well :)

Ok, did that and deleted the files in db.backup1 and db.backup2 as well,
also old berkeley stuff, and I figure deleting a backup is allright (if
you still have the files on the original server ;-) )
Everything else seems to be skiplist already.
What do you mean change them to skiplist? Do I need to re-create the
files in skiplist format? (If so, how?) Or does Cyrus create them
automatically if they don't exist?

>
>> I got that just once (or at least I didn't see it more often)
>>
>> And I saw this: cyrus/ctl_cyrusdb[20917]: converting
>> /var/lib/cyrus/deliver.db from berkeley to skiplist
> Yeah, if you've changed the config it will do that.  It should be a good
> thing, but the Berkeley environment can hold references to the old
> berkeley DB still.

Ok

>
>> Which seemed to be a good thing
>>
>> After that I tried to log in. Then the fun started
>>
>> First of: I use virtual domains. I have a default domain that I
>> started out with, that was located under user (so not under a
>> directory domain, just /path/to/cyrus/s/user/stefan
>>
>> I could't log into that account anymore: Cyrus couldn't find the user
>> account.
> Were you logging in as user at domain, or just 'user'?
>
>> However, users in the virtual domains (located under
>> /path/to/cyrus/domain/s/somedomain.nl are able to login.
>>
>> I copied all settings from the old imapd.conf to the new one, so I
>> figure someting has changed in the way virtual domains are handled. Do
>> I need to move everything under domain now?
> That or just log in with the raw username rather than domain.  That's
> for virtdomains: userid, I don't have a clue for the non-userid version.
> This _might_ be a regression from 2.3 to 2.4.

I figured this one out in the meantime: on the old server, for the
default domain I just added users without a domain to saslsb. That
doesn't seem to work anymore.
Once I added the fully qualified users to sasldb I could login with the
default domain as well. Problem solved

>
>> But then something even worse happened: after a reboot Cyrus
>> completely freaked out:
>>
>> Aug 23 00:47:29 spice cyrus/master[24908]: about to exec
>> /usr/sbin/cyrus Aug 23 00:47:29 spice cyrus/ctl_cyrusdb[24908]:
>> DBERROR db5: BDB1521 Recovery function for LSN 91 3293614 failed Aug
>> 23 00:47:29 spice cyrus/ctl_cyrusdb[24908]: DBERROR db5: BDB0061
>> PANIC: Invalid argument Aug 23 00:47:29 spice
>> cyrus/ctl_cyrusdb[24908]: DBERROR: critical database situation Aug 23
>> 00:47:29 spice cyrus/master[24906]: process 24908 exited, status 75
>> Aug 23 00:47:29 spice cyrus/master[24911]: about to exec
>> /usr/sbin/cyrus Aug 23 00:47:29 spice cyrus/cyr_expire[24911]: DBERROR
>> db5: BDB0060 PANIC: fatal region error detected; run recovery Aug 23
>> 00:47:29 spice cyrus/cyr_expire[24911]: DBERROR: critical database
>> situation Aug 23 00:47:29 spice cyrus/master[24906]: process 24911
>> exited, status 75 Aug 23 00:47:29 spice cyrus/master[24914]: about to
>> exec /usr/sbin/cyrus Aug 23 00:47:29 spice cyrus/tls_prune[24914]:
>> DBERROR db5: BDB0060 PANIC: fatal region error detected; run recovery
>> Aug 23 00:47:29 spice cyrus/tls_prune[24914]: DBERROR: critical
>> database situation Aug 23 00:47:29 spice cyrus/master[24906]: process
>> 24914 exited, status 75 Aug 23 00:47:29 spice cyrus/master[24906]:
>> unable to setsocketopt(IP_TOS): Operation not supported Aug 23
>> 00:47:29 spice cyrus/master[24906]: unable to setsocketopt(IP_TOS):
>> Operation not supported Aug 23 00:47:29 spice cyrus/master[24906]:
>> ready for work Aug 23 00:47:29 spice cyrus/master[24917]: about to
>> exec /usr/sbin/cyrus Aug 23 00:47:29 spice cyrus/master[24918]: about
>> to exec /usr/lib/cyrus/bin/notifyd Aug 23 00:47:29 spice
>> cyrus/notify[24918]: DBERROR db5: BDB0060 PANIC: fatal region error
>> detected; run recovery Aug 23 00:47:29 spice cyrus/notify[24918]:
>> DBERROR: critical database situation Aug 23 00:47:29 spice
>> cyrus/master[24906]: process 24918 exited, status 75 Aug 23 00:47:29
>> spice cyrus/master[24906]: service notify pid 24918 in READY state:
>> terminated abnormally Aug 23 00:47:29 spice cyrus/master[24920]: about
>> to exec /usr/lib/cyrus/bin/notifyd Aug 23 00:47:29 spice
>> cyrus/ctl_cyrusdb[24917]: DBERROR db5: BDB0060 PANIC: fatal region
>> error detected; run recovery Aug 23 00:47:29 spice
>> cyrus/ctl_cyrusdb[24917]: DBERROR: critical database situation Aug 23
>> 00:47:29 spice cyrus/master[24906]: process 24917 exited, status 75
>> Aug 23 00:47:29 spice cyrus/notify[24920]: DBERROR db5: BDB0060 PANIC:
>> fatal region error detected; run recovery Aug 23 00:47:29 spice
>> cyrus/notify[24920]: DBERROR: critical database situation Aug 23
>> 00:47:29 spice cyrus/master[24906]: process 24920 exited, status 75
>> Aug 23 00:47:29 spice cyrus/master[24906]: service notify pid 24920 in
>> READY state: terminated abnormally Aug 23 00:47:29 spice
>> cyrus/master[24922]: about to exec /usr/lib/cyrus/bin/notifyd Aug 23
>> 00:47:29 spice cyrus/notify[24922]: DBERROR db5: BDB0060 PANIC: fatal
>> region error detected; run recovery Aug 23 00:47:29 spice
>> cyrus/notify[24922]: DBERROR: critical database situation Aug 23
>> 00:47:29 spice cyrus/master[24906]: process 24922 exited, status 75
>> Aug 23 00:47:29 spice cyrus/master[24906]: service notify pid 24922 in
>> READY state: terminated abnormally Aug 23 00:47:29 spice
>> cyrus/master[24923]: about to exec /usr/lib/cyrus/bin/notifyd Aug 23
>> 00:47:29 spice cyrus/notify[24923]: DBERROR db5: BDB0060 PANIC: fatal
>> region error detected; run recovery Aug 23 00:47:29 spice
>> cyrus/notify[24923]: DBERROR: critical database situation Aug 23
>> 00:47:29 spice cyrus/master[24906]: process 24923 exited, status 75
>> Aug 23 00:47:29 spice cyrus/master[24906]: service notify pid 24923 in
>> READY state: terminated abnormally
>>
>> This went on until I killed Cyrus. Dozens of lines of log per second.
>> Can anyone tell me what went wrong? Cyrus wouldn't exit gracefully, so
>> I stopped it with a kill -9 before rebooting, did that cause the
>> problem?
>>
>> I actually repeated the entire process and got exactly the same result
>>
>> Any help would be GREATLY appreciated
> If you follow the steps above to kill the BDB environment (again, with Cyrus
> entirely stopped) you should be OK.

Ok. Do I need to run a reconstruct before starting? Since it did al
those index upgrades? I'm leaning toward yes, but would love to hear
from you ;-)

Have resolved to look closer into Cyrus anyway. The old server has run
without any trouble for over 3 years, makes one lazy ;-). Good software
though I guess

Thanks
Stefan