Emergancy..... db4: PANIC: fatal region error

[Simon Matter]

> On 7/27/07, Simon Matter <simon.matter at invoca.ch> wrote:
>>
>> > I have been holding back updating to cyrus 2.2.13 but this afternoon I
>> > went
>> > ahead and made the major update and really tried to follow ALL the
>> points
>> > and rules in updating but I really think I messed up the whole system
>> now.
>> > I really hope I have not hosed all the email today and have to go back
>> to
>> > an
>> > older backup :(.
>> >
>> > I really do need some help for I am so lost and I think everything I
>> have
>> > been trying has made it to worse and back again.
>>
>> Hi,
>>
>> I'm really no expert with BDB and I'm  avoiding it wherever I can. But
>> what comes to mind is this:
>> You didn't tell us what exactly you have updated. It's cyrus-imap from
>> ???
>> to 2.2.13. But what else? My guess is that you also updated BDB from db3
>> to db4 and didn't manage to upgrade you BDB files, can that be? I read
>> something like "istoric log version 3" which makes me believe you are
>> trying to access db3 files with cyrus/db4.
>> From what I know you have to dump the db3 db's with db3 and then import
>> to
>> db4 files with the db4 tools. IIRC this has been discussed on this list
>> several times and you should be able to find the correct command
>> sequence
>> for it in the archives.
>>
>> To avoid that kind of problems in future you could convert all BDB
>> databases to skiplist and enjoy beeing free of BDB version conflicts in
>> the future. However to fix it now you will for sure need the db3
>> libs/tools to get anything usefull out of your db3 files.
>>
>> Hope that helps at least a little bit.
>> Simon
>>
>> >
>> > Here is the output from syslog.
>> >
>> > Jul 26 22:37:27 jurassic cyrus/master[10684]: process started
>> > Jul 26 22:37:31 jurassic cyrus/master[10708]: about to exec
>> > /usr/sbin/ctl_cyrusdb
>> > Jul 26 22:37:32 jurassic cyrus/ctl_cyrusdb[10708]: DBERROR db4:
>> Skipping
>> > log
>> > file /var/lib/cyrus/db/log.00000000
>> > 55: historic log version 3
>> > Jul 26 22:37:33 jurassic cyrus/ctl_cyrusdb[10708]: recovering cyrus
>> > databases
>> > Jul 26 22:37:33 jurassic cyrus/ctl_cyrusdb[10708]: skiplist: recovered
>> > /var/lib/cyrus/mailboxes.db (563 records,
>> >  84908 bytes) in 0 seconds
>> > Jul 26 22:37:33 jurassic cyrus/ctl_cyrusdb[10708]: skiplist: recovered
>> > /var/lib/cyrus/annotations.db (0 records,
>> >  144 bytes) in 0 seconds
>> > Jul 26 22:37:33 jurassic cyrus/ctl_cyrusdb[10708]: DBERROR db4:
>> > /var/lib/cyrus/db/log.0000000056: log file open
>> > failed: No such file or directory
>> > Jul 26 22:37:33 jurassic cyrus/ctl_cyrusdb[10708]: DBERROR db4: PANIC:
>> No
>> > such file or directory
>> > Jul 26 22:37:33 jurassic cyrus/ctl_cyrusdb[10708]: DBERROR: critical
>> > database situation
>> > Jul 26 22:37:34 jurassic cyrus/master[10684]: process 10708 exited,
>> status
>> > 75
>> > Jul 26 22:37:34 jurassic cyrus/master[10754]: about to exec
>> > /usr/sbin/cyr_expire
>> > Jul 26 22:37:34 jurassic cyrus/cyr_expire[10754]: DBERROR db4: PANIC:
>> > fatal
>> > region error detected; run recovery
>> > Jul 26 22:37:34 jurassic cyrus/cyr_expire[10754]: DBERROR: critical
>> > database
>> > situation
>> > Jul 26 22:37:34 jurassic cyrus/master[10684]: process 10754 exited,
>> status
>> > 75
>> > Jul 26 22:37:34 jurassic cyrus/master[10755]: about to exec
>> > /usr/sbin/tls_prune
>> > Jul 26 22:37:34 jurassic cyrus/tls_prune[10755]: DBERROR db4: PANIC:
>> fatal
>> > region error detected; run recovery
>> > Jul 26 22:37:34 jurassic cyrus/tls_prune[10755]: DBERROR: critical
>> > database
>> > situation
>> > Jul 26 22:37:34 jurassic cyrus/master[10684]: process 10755 exited,
>> status
>> > 75
>> > Jul 26 22:37:34 jurassic cyrus/master[10684]: ready for work
>> > Jul 26 22:37:34 jurassic cyrus/master[10766]: about to exec
>> > /usr/sbin/ctl_cyrusdb
>> > Jul 26 22:37:34 jurassic cyrus/master[10767]: about to exec
>> > /usr/lib/cyrus/bin/notifyd
>> > Jul 26 22:37:34 jurassic cyrus/ctl_cyrusdb[10766]: DBERROR db4: PANIC:
>> > fatal
>> > region error detected; run recovery
>> > Jul 26 22:37:34 jurassic cyrus/ctl_cyrusdb[10766]: DBERROR: critical
>> > database situation
>> > Jul 26 22:37:34 jurassic cyrus/master[10684]: process 10766 exited,
>> status
>> > 75
>> >
>> >
>> >
>> > No mater what I do I just can not get passed this error.   I have
>> > tried...reconstructing, db4.3_recover, bringing in a backup db and
>> each
>> > one
>> > just does not clear the air.  Every time I try the error still
>> persists.
>> > Is
>> > this recoverable?  Everything I have read on how to fix it does not
>> work.
>> > I
>> > am wondering if i really missed something here.  I really do hope to
>> get
>> > some feed back ASAP because I want to get this fixed within the next 7
>> > hrs.
>> >
>> > -Adam
>
>
>
>
> Simon,
>
> Thank you.  Yes.. after being very tired and warry from working on this
> for
> a long while I  did forget to mention a few things.
>
> I started out with cyrus21 and moved to cyrus22 which should have been an
> easy upgrade but wow.. I really got into a mess.  This all started because
> vacation scripts were not working and trying to get them to work properly.
> Aslo I needed the virtual domain function in the new 2.2 which led me to
> this fun trip.
>
> Yes, I did update DB to 4.3 and had thought I converted everything over to
> skiplist but guess I had not.  When I used db4.3_update I did not get a
> good
> response and the moment I do not remember what it was.  I really tried to
> do
> as much as possible and maybe as much damage before I consulted the mail
> list.  It looks like now the files are skiplist however the DB files in
> the
> db directory, do I need to do anything with those? Or, are all db files I
> need to convert are in the /var/lib/cyrus/ proper directory?
>
> These are the files I have wich are skiplist.
> mailboxes.db
> annotations.db

Usually there are at least four db files:
annotations.db
deliver.db
mailboxes.db
tls_sessions.db

I think only mailboxes.db is really important here, deliver.db and
tls_sessions.db can be wiped and will be created  again, annotations.db is
empty if you don't use annotations.
You should also have a look at the db/ directory. IIRC there can be a
situation when you moved to skiplist but still have BDB files in db/ it
will still disturb and should be removed while cyrus is stopped.

The think with hashed directories you mentioned in another mail can be
controlled with hashimapspool and fulldirhash in /etc/imapd.conf. I have
no idea how the Debian packages are configured but you should find it in
the package description somehow.

Simon