DBERROR: skiplist recovery mailboxes.db 0090 - suddenly all is failing!

[Andrew Morgan]

On Mon, 19 Feb 2007, Gregor Wenkelewsky wrote:

> On Thu, 15 Feb 2007 18:15:53 +0100, Andrew Morgan <morgan at orst.edu> wrote:
>
>>> Cyrus has been installed here just a few weeks ago, and after some hard
>>> days it was working smoothly and very well. Until suddenly, sadly today
>>> it started to fail completely with this error message in mail.warn,
>>> mail.error and syslog:
>>>
>>> cyrus/imap[..]: DBERROR: skiplist recovery /var/lib/cyrus/mailboxes.db: 0090 should be ADD or DELETE
>>> cyrus/imap[..]: DBERROR: opening /var/lib/cyrus/mailboxes.db: cyrusdb error
>>
>> You'll need to fix the corruption of the mailboxes.db file.  It is a
>> skiplist format file in your case, so do a google search for
>> "skiplist.py".  You'll find a python utility that can do some better
>> recovery than the cyrus tools.  The example is for cyrus seen-state files,
>> but the same should work on the mailboxes.db as well.
>
> Fine, I succeeded with that! Did check the auto backup files before too,
> but they were all identical. It was too late probably. Then I used the
> skiplist.py from here: http://oss.netfarm.it/python-cyrus.php
>
> python ~/skiplist.py mailboxes.db >mailboxes.txt
> rn mailboxes.db mailboxes.err
> cvt_cyrusdb /var/lib/cyrus/mailboxes.txt flat /var/lib/cyrus/mailboxes.db skiplist
> chown cyrus mailboxes.db
>
> Fixed!
>
> Okay then, now it works, but how often will an error like this occur, can
> I do something to prevent it? First I thought of "0090" as some sort of
> error code, and I found only two error incidents with /line/ 0090 in
> Google... ;) ...but it is much more numerous.

The 0090 is a skiplist offset/index within the file, so the error message 
could contain any number depending where the corruption happened.

> Can this be related to shutting down and rebooting the system? Could be a
> coincidence of course, but just after rebooting the error was there.

Possibly, if Cyrus was stopped (kill -9?) in the middle of a skiplist 
operation.

> Can it be related to Squatter? By default, Squatter was not set, but some
> days before the error I set Squatter to an hourly "nice" run. Now I turned
> it off again.

I don't think squatter would have any relation, but I'm not running 
squatter here myself.

>> You should also setup a cronjob to dump the mailboxes.db file to plaintext
>> periodically (so it can be backed up).  Something like this works here:
>>
>> 58 * * * * cyrus /usr/local/cyrus/bin/ctl_mboxlist -d > /var/spool/cyrus/config/mailboxes.db.dump
>
> Yes, I'll do that, though it's more like holding ready the plaster instead
> of preventing the injury.
>
> This has to be written to /etc/crontab like Squatter, correct? How often
> should it be running? Maybe it's only neccessary when new IMAP users and/or
> folders have been created??

Yes, that command above is exactly what I have in my crontab file.  I 
can't remember why I have it run at 58 minutes after the hour.  :)

> I feel queasy with an error that has no apparent reason. I wanted to
> build a system that can run without administration for months and, maybe,
> would sustain even a rare power failure. But there was no power failure
> and no sign of a disc error either... :-(

We've been running Cyrus for a couple years now with skiplist for 
mailboxes.db.  So far we've never had a single corruption of mailboxes.db. 
Very rarely we'll get a corrupted username.seen file, which can be fixed 
using skiplist.py.

You should be using some sort of journaling filesystem (we use ext3 here) 
if you are not already, although that cannot save you from some sources of 
skiplist corruption.

 	Andy