competition

Simon Matter simon.matter at invoca.ch
Thu Sep 23 01:56:30 EDT 2010


> On Wed, Sep 22, 2010 at 04:10:15PM +1000, Bron Gondwana wrote:
>
>> Now - BDB database SHOULD be upgradable.  I want to find a BDB expert
>> to help me with that - (a) detecting that an upgrade is necessary, and
>> (b) doing the upgrade.
>
> It was quite some time ago I last upgraded a Cyrus instance, but my
> upgrade script did the following:
>
> - Run the _old_ version of db_recover. Just shutting down Cyrus is not
>   enough, without this step db_upgrade can fail.
>
> - Run the _new_ version of db_upgrade.
>
> - Run "db_checkpoint -1" to force the log to be rotated.
>
> I think it won't be hard to implement steps 2 & 3 inside Cyrus, and I
> don't remember those steps taking a significant amount of time, but I
> don't know how it would be possible to implement step 1. However if the
> BDB version change does not bring a change to the on-disk DB log format,
> then it may be possible to skip steps 1 & 3.

Unfortunately that's not enough in many situations. I did uncountable
tests with different BDB versions and came up with the following method in
our RPMs:

...
# make sure our Berkeley databases are in a sane state
# wait for db_checkpoint to end successfully or kill it after a timeout
db_checkpoint -v -1 -h ${imap_prefix}/db &
DB_CHECK_PID=$!
CNT=0
while [ $CNT -lt 60 ]; do
  if ! kill -0 $DB_CHECK_PID > /dev/null 2>&1; then
    break
  fi
  sleep 1
  let CNT+=1
done
if kill -0 $DB_CHECK_PID > /dev/null 2>&1; then
  kill -USR1 $DB_CHECK_PID > /dev/null 2>&1
  sleep 1
  kill -KILL $DB_CHECK_PID > /dev/null 2>&1
  wait $DB_CHECK_PID > /dev/null 2>&1
fi

# do a normal recovery
db_recover -v -h ${imap_prefix}/db
RETVAL=$?
if [ $RETVAL -ne 0 ]; then
  # try a catastrophic recovery instead of normal recovery
  db_recover -v -c -h ${imap_prefix}/db
  RETVAL=$?
  ERRVAL=$(( $ERRVAL + $RETVAL ))
  if [ $RETVAL -ne 0 ]; then
    echo "ERROR: catastrophic recovery of Berkeley databases failed"
  fi
fi

if [ "$EXPORT" = "export" ]; then
  # convert all db files to portable format for migration
  # TODO: quota_db, we don't touch it for now
  cvt_file ${imap_prefix}/deliver.db           "skiplist"
  cvt_file ${imap_prefix}/mailboxes.db         "skiplist"
  cvt_file ${imap_prefix}/tls_sessions.db      "skiplist"
  cvt_file ${imap_prefix}/annotations.db       "skiplist"
  cvt_file ${imap_prefix}/ptclient/ptscache.db "skiplist"
  cvt_file ${imap_prefix}/statuscache.db       "skiplist"
  cvt_file ${imap_prefix}/user_deny.db         "flat"
  rm -vf ${imap_prefix}/db/log.*
  rm -vf ${imap_prefix}/db/__db.*
else
...


This all is done not for upgrades of BDB but for simple cases where BDB is
in a bad state for some reasons.

Now, the db_upgrade may be good for those situations where it works. But
I'm quite sure it doesn't work between some major versions of BDB. And
that's exactly what we had to care in the RPMs, so you can run RHEL2.1 on
i386 and "upgrade" it to RHEL5 on x86_64 and it will just work.

It works, but I'd be very happy to get rid of this stuff, really!

Simon



More information about the Info-cyrus mailing list