Wiki, information on database back ends.

[Dan White]

On 14/09/10 14:24 -0500, Patrick Goetz wrote:
>There's been some discussion on the Debian cyrus list about how to 
>automate upgrades from cyrus 2.n.k to cyrus 2.m.j.  Jeroen van 
>Meeuwen (on both lists) suggested that the cyrus RPM package features 
>a utility called cyrus-imapd.cvt_cyrusdb_all which might be useful 
>for this.
>
>I've been looking at this script, and it mostly appears to be using 
>cvt_cyrusdb to convert particular db files to Cyrus skiplists and 
>then back again to the original db backend format.  I can't follow 
>the script completely as it seems to rely on DB configuration details 
>found in the imapd.conf file I don't have in my Debian 2.1.16 imap 
>server, and it's also not clear how the script is run.
>
>This raises a number of questions, though:
>
>1.
>Cyrus skiplists?  I thought all the DB files were in Berkeley DB 
>format.  I tried to find some documentation on skiplists, but only 
>found an old message to the developer list from Bron Gondwana 
>discussing skiplist bugs 
>(http://markmail.org/message/zbaq765brbg2acfj).

The current imapd.conf hints at the available data types for each cyrus
database. Depending on the option, the following database types are
available (in 2.3.16):

annotation_db: skiplist
   Allowed values: berkeley, berkeley-hash, skiplist

duplicate_db: berkeley-nosync
   Allowed values: berkeley, berkeley-nosync, berkeley-hash,
berkeley-hash-nosync, skiplist, sql

mboxkey_db: skiplist
   Allowed values: berkeley, skiplist

mboxlist_db: skiplist
   Allowed values: flat, berkeley, berkeley-hash, skiplist

ptscache_db: berkeley
   Allowed values: berkeley, berkeley-hash, skiplist

quota_db: quotalegacy
   Allowed values: flat, berkeley, berkeley-hash, skiplist, sql, quotalegacy

seenstate_db: skiplist
   Allowed values: flat, berkeley, berkeley-hash, skiplist

subscription_db: flat
   Allowed values: flat, berkeley, berkeley-hash, skiplist

statuscache_db: berkeley-nosync
   Allowed values: berkeley, berkeley-nosync, berkeley-hash,
berkeley-hash-nosync, skiplist

tlscache_db: berkeley-nosync
   Allowed values: berkeley, berkeley-nosync, berkeley-hash,
berkeley-hash-nosync, skiplist, sql

userdeny_db: flat
   Allowed values: flat, berkeley, berkeley-hash, skiplist, sql

http://www.cyrusimap.org/docs/cyrus-imapd/2.3.16/internal/database-formats.php
has some discussion of the databases.

I don't know why the (rpm) upgrade script chose skiplist as an
intermediate format, rather than flat. Perhaps skiplist hasn't changed much
between major versions, and offers a speed benefit.

The script seems to use the file command to determine the current database
types, since they can be changed by an admin.

>2.
>The Redhat cvt_cyrusdb_all script seems to assume a specific set of 
>database files.  Is the set of cyrus imap DB files fixed, and if so 
>what are they?  Is there any documentation on what each database file 
>contains? This would be very useful to people trying to convert older 
>cyrus IMAP installations to new ones.

As far as skiplist, that seems to be a proprietary format.

As far as I can tell, all databases use the same underlying structure
(except for quotalegacy), which is defined in the database-formats above.

flat: ASCII text file

berkeley: berkeley db (with safer values).

berkeley-nosync: berkeley db with the DB_TXN_NOSYNC flag set.

berkeley-hash: berkeley db with the DB_HASH flag set.

berkeley-hash-nosync: berkeley db with both DB_TXN_NOSYNC and DB_HASH
flags?

sql: A mysql, pgsql, or sqlite store.

quotalegacy: See database-formats

skiplist: ?

With cyrus version 2.3.13 and above, the cyr_dbtool can be used to insert
and remove records manually for each database.

-- 
Dan White