<!DOCTYPE html>
<html>
<head>
<title></title>
<style type="text/css">p.MsoNormal,p.MsoNoSpacing{margin:0}</style>
</head>
<body><div style="font-family:Arial;">Hi All,<br></div>
<div style="font-family:Arial;"><br></div>
<div style="font-family:Arial;">By popular request, a primer on the cyrusdb interface.<br></div>
<div style="font-family:Arial;"><br></div>
<div style="font-family:Arial;">The cyrusdb interface is a common API for accessing key-value datastores from within cyrus code. It's part of the libcyrus shared object (but not libcyrus_min), used by all cyrus binaries. The use of the cyrusdb API abstracts the details of the underlying data store, allowing sites to use different database types depending on their needs, and allowing Cyrus developers to use a common API for data storage needs.<br></div>
<div style="font-family:Arial;"><br></div>
<div style="font-family:Arial;">The entire cyrusdb source lives in the lib/ directory of the cyrus-imapd repository, in the following files:<br></div>
<div style="font-family:Arial;"><br></div>
<div style="font-family:Arial;">lib/cyrusdb.h<br></div>
<div style="font-family:Arial;"><div style="font-family:Arial;"><div style="font-family:Arial;">lib/cyrusdb.c</div>
</div>
</div>
<div style="font-family:Arial;">lib/cyrusdb_flat.c<br></div>
<div style="font-family:Arial;">lib/cyrusdb_quotalegacy.c<br></div>
<div style="font-family:Arial;">lib/cyrusdb_skiplist.c<br></div>
<div style="font-family:Arial;">lib/cyrusdb_sql.c<br></div>
<div style="font-family:Arial;">lib/cyrusdb_twoskip.c<br></div>
<div style="font-family:Arial;"><br></div>
<div style="font-family:Arial;">cyrusdb.h:<br></div>
<div style="font-family:Arial;"><br></div>
<div style="font-family:Arial;">* interface definitions (all access to cyrusdb databases happens through the functions defined here)<br></div>
<div style="font-family:Arial;">* the "struct cyrusdb_backend" data structure which defines the interface implemented by each backend.<br></div>
<div style="font-family:Arial;">* constants for flags to the cyrusdb_open call, and return codes. Cyrusdb functions all return their own CYRUSDB_* error codes, which are not compatible with the r = IMAP_* return codes used throughout much of the rest of the codebase.<br></div>
<div style="font-family:Arial;"><br></div>
<div style="font-family:Arial;">cyrusdb.c:<br></div>
<div style="font-family:Arial;"><br></div>
<div style="font-family:Arial;">* implementations of the wrapper functions around the backends, including default implementations of some functions which are common to many backends but overridden by some.<br></div>
<div style="font-family:Arial;">* a wrapper to initialise and cleanup the state of each backend (if needed) during Cyrus set up / tear down.<br></div>
<div style="font-family:Arial;"><br></div>
<div style="font-family:Arial;">cyrusdb_*.c:<br></div>
<div style="font-family:Arial;"><br></div>
<div style="font-family:Arial;">* the actual implementations of each backend! We'll look at some in a sec.<br></div>
<div style="font-family:Arial;"><br></div>
<div style="font-family:Arial;">There are also some tools to work with and support cyrus databases:<br></div>
<div style="font-family:Arial;"><br></div>
<div style="font-family:Arial;">imap/ctl_cyrusdb:<br></div>
<div style="font-family:Arial;"><br></div>
<div style="font-family:Arial;">* performs maintenance on the cyrusdb subsystem. This is called in two places:<br></div>
<div style="font-family:Arial;">* START: "ctl_cyrusdb -r" (recovery). This is the ONLY PLACE that code is guaranteed to be run at startup on every cyrus installation, so you'll find quite a lot of detritus has built up in this codepath over the years.<br></div>
<div style="font-family:Arial;">* EVENTS: "ctl_cyrusdb -c" (checkpoint). This is run regularly (period=180 at FastMail, examples in the codebase have period=5 or period=30). Both this codepath and cyr_expire tend to run periodically on cyrus systems, and cleanup code is spread between those two locations.<br></div>
<div style="font-family:Arial;"><br></div>
<div style="font-family:Arial;">imap/cvt_cyrusdb:<br></div>
<div style="font-family:Arial;"><br></div>
<div style="font-family:Arial;">* used for converting a database between versions. This is often used to prepare for upgrade, particularly in the past when Cyrus supported berkeley DB which didn't upgrade cleanly across OS versions, it was common to use cvt_cyrusdb to turn databases into a very portable format (flat or skiplist) before upgrading, upgrade the OS, convert back to the fast format (berkeley) and then restart.<br></div>
<div style="font-family:Arial;"><br></div>
<div style="font-family:Arial;">imap/cyr_dbtool:<br></div>
<div style="font-family:Arial;"><br></div>
<div style="font-family:Arial;">* once known as brontool, this is the first piece of Cyrus code I ever wrote! It's a fairly dumb wrapper around the CyrusDB interface, and able to be used to read, write, or iterate any parts of a database. Its interactive mode is not special-character clean, but it can also be used in batch mode, which uses IMAP atom-string literal8 for input/output, and hence can roundtrip data reliably.<br></div>
<div style="font-family:Arial;"><br></div>
<div style="font-family:Arial;">There are also tools like: ctl_conversationsdb, dav_reconstruct and ctl_mboxlist which can be used to manage individual databases through a more specific interface which understands the context as well as just the raw key/value.<br></div>
<div style="font-family:Arial;"><br></div>
<div style="font-family:Arial;">How to use CyrusDB:<br></div>
<div style="font-family:Arial;"><div style="font-family:Arial;">===<br></div>
<div style="font-family:Arial;"><div style="font-family:Arial;"><br></div>
</div>
</div>
<div style="font-family:Arial;">Assuming that cyrus_init() has been called, which calls cyrusdb_init(), you can assume that databases will work in any Cyrus code.<br></div>
<div style="font-family:Arial;"><br></div>
<div style="font-family:Arial;">The first step is to open a database. Databases have a filename - this might be a literal filename on the backend, a directly containing data, or an opaque token used by the backend to locate a datasource.<br></div>
<div style="font-family:Arial;"><br></div>
<div style="font-family:Arial;"> int flags = 0;<br></div>
<div style="font-family:Arial;"> struct db *mydb = NULL;<br></div>
<div style="font-family:Arial;"> int r = cyrusdb_open("skiplist", "/tmp/database.db", flags, &mydb);<br></div>
<div style="font-family:Arial;"> if (!r) return mydb; // if (r == CYRUSDB_OK) { ... }<br></div>
<div style="font-family:Arial;"> /* XXX: error handling */<br></div>
<div style="font-family:Arial;"><br></div>
<div style="font-family:Arial;">Accepted flags:<br></div>
<div style="font-family:Arial;"><br></div>
<div style="font-family:Arial;">CYRUSDB_CREATE - if the named database doesn't exist, create a blank database.<br></div>
<div style="font-family:Arial;"><br></div>
<div style="font-family:Arial;">CYRUSDB_MBOXSORT - use the abomination called improved_mboxlist_sort which re-orders a couple of characters to allow "foo.bar" to sort before "foo bar", for perfectly good reasons, but we're going to fix it a better way. Not every engine supports arbitrary collation, and if many engines corrupt horribly if the same database is opened with different choices for this flag. Ouch.<br></div>
<div style="font-family:Arial;"><br></div>
<div style="font-family:Arial;">CYRUSDB_CONVERT - if set and the database fails to open, attempt a magic detection on the file content and try to convert the database to the requested backend type before opening it. In-place upgrades! If this is NOT set, then instead the magic detection will still be performed, but the open database will be returned using the correct engine for that database rather than converted. Magic detection only currently works for single-file database formats.<br></div>
<div style="font-family:Arial;"><br></div>
<div style="font-family:Arial;">CYRUSDB_NOCOMPACT - if the database format supports automatic compacting, don't use it. Handy for when you want to read without causing any possible issues (e.g. read-only filesystem during recovery) or when performance is critical and you don't want to risk waiting while a recompact happens.<br></div>
<div style="font-family:Arial;"><br></div>
<div style="font-family:Arial;">All the remaining functions take that "struct db" pointer.<br></div>
<div style="font-family:Arial;"><br></div>
<div style="font-family:Arial;">There's also a cyrusdb_lockopen() interface which takes a transaction pointer and returns with the transaction already active. This isn't actually being used yet, but is intended to allow slightly more efficient single-operation database use. Right now, open returns an unlocked database, but may need to lock as part of the setup, so keeping that lock would avoid one extra unlock/lock cycle.<br></div>
<div style="font-family:Arial;"><br></div>
<div style="font-family:Arial;">Reading, writing, transactions:<br></div>
<div style="font-family:Arial;"><div style="font-family:Arial;">===<br></div>
<div style="font-family:Arial;"><div style="font-family:Arial;"><br></div>
<div style="font-family:Arial;">CyrusDB supports both transactional and non-transactional access. Transactions are always exclusive. This is arguably a deficiency in the interface, particularly since many engines implement a non-exclusive (read) lock internally anyway.<br></div>
<div style="font-family:Arial;"><br></div>
<div style="font-family:Arial;">There are now 4 interfaces to read data. Two of which are original cyrusdb and two of which are more recently added.<br></div>
<div style="font-family:Arial;"><br></div>
<div style="font-family:Arial;">original:<br></div>
<div style="font-family:Arial;">* cyrusdb_fetch() - fetch a single value by exact key.<br></div>
<div style="font-family:Arial;">* cyrusdb_foreach() - given a prefix, iterate over all the keys with that prefix (including exactly that key) in order.<br></div>
<div style="font-family:Arial;"><br></div>
<div style="font-family:Arial;">newer:<br></div>
<div style="font-family:Arial;">* cyrusdb_fetchnext() - given an exact key, fetch the following key and value (regardless of whether the key exists), e.g given keys "f" and "g", fetchnext "foo" would return "g", as would fetchnext "f". This can be used to implement foreach (indeed, the skips do exactly that).<br></div>
<div style="font-family:Arial;">* cyrusdb_forone() - given an exact key, act like "cyrusdb_foreach" but only for that one key. This is a convenience wrapper around fetch to allow doing things like:<br></div>
<div style="font-family:Arial;"><br></div>
<div style="font-family:Arial;"> r = cyrusdb_forone(mydb, "folder", 6, p, cb, rock, &tid);<br></div>
<div style="font-family:Arial;"> if (!r) r = cyrusdb_foreach(mydb, "folder.", 7, p, cb, rock, &tid);<br></div>
<div style="font-family:Arial;"><br></div>
<div style="font-family:Arial;">Which does precisely "folder" and its children without visiting any other keys that have "folder" as a prefix.<br></div>
<div style="font-family:Arial;"><br></div>
<div style="font-family:Arial;">Since the cyrusdb interface always takes both a pointer and a length, it's also possible to use:<br></div>
<div style="font-family:Arial;"><br></div>
<div style="font-family:Arial;"> char *key = "folder.";<br></div>
<div style="font-family:Arial;"> r = cyrusdb_forone(mydb, key, 6, p, cb, rock, &tid);<br></div>
<div style="font-family:Arial;"> if (!r) r = cyrusdb_foreach(mydb, key, 7, p, cb, rock, &tid);<br></div>
<div style="font-family:Arial;"><div style="font-family:Arial;"><br></div>
<div style="font-family:Arial;">You may have noticed that 'tid' at the end. Every function for acting on the database takes as its last argument a "struct txn **". You can pass one of three things to this:<br></div>
<div style="font-family:Arial;"><br></div>
<div style="font-family:Arial;">NULL - non-transactional request. Do whatever you need for internal locking, but starts with an unlocked database and ends with an unlocked database. NOTE: at least skiplist and twoskip implement a hack where if the database IS locked for a non-transactional read request, they will act as if you'd passed the current transaction in for the NULL case. This is a hack around layering violations and kind of sucks.<br></div>
<div style="font-family:Arial;"><br></div>
<div style="font-family:Arial;">&NULL - e.g:<br></div>
<div style="font-family:Arial;"><br></div>
<div style="font-family:Arial;"> struct txn *tid = NULL;<br></div>
<div style="font-family:Arial;"> const char *data = NULL;<br></div>
<div style="font-family:Arial;"> size_t datalen = 0;<br></div>
<div style="font-family:Arial;"> int r = cyrusdb_fetch(mydb, key, keylen, &data, &datalen, &tid);<br></div>
<div style="font-family:Arial;"><br></div>
<div style="font-family:Arial;">After calling this, tid will have an opaque value allocated by the database backend, which must be passed to all further cyrusdb operations on that database until either cyrusdb_commit() or cyrusdb_abort() are called.<br></div>
<div style="font-family:Arial;"><br></div>
<div style="font-family:Arial;">&tid - e.g.<br></div>
<div style="font-family:Arial;"><br></div>
<div style="font-family:Arial;"> if (r == CYRUSDB_NOTFOUND) {<br></div>
<div style="font-family:Arial;"> r = cyrusdb_store(mydb, key, keylen, "DEFAULT", 7, &tid); // set a default value<br></div>
<div style="font-family:Arial;"> }<br></div>
<div style="font-family:Arial;"><br></div>
<div style="font-family:Arial;">Given an existing transaction, perform this call in the context of the transaction.<br></div>
<div style="font-family:Arial;"><br></div>
<div style="font-family:Arial;">OK, foreach. Foreach is very tricky, because it takes TWO callbacks. The callbacks have an identical signature, but different return codes!<br></div>
<div style="font-family:Arial;"><br></div>
</div>
<div style="font-family:Arial;">typedef int foreach_p(void *rock,<br></div>
<div style="font-family:Arial;"> const char *key, size_t keylen,<br></div>
</div>
</div>
<div style="font-family:Arial;"> const char *data, size_t datalen);<br></div>
<div style="font-family:Arial;"><br></div>
<div style="font-family:Arial;">typedef int foreach_cb(void *rock,<br></div>
<div style="font-family:Arial;"> const char *key, size_t keylen,<br></div>
<div style="font-family:Arial;"> const char *data, size_t datalen);<br></div>
<div style="font-family:Arial;"><br></div>
<div style="font-family:Arial;"><div style="font-family:Arial;"><div style="font-family:Arial;"><div style="font-family:Arial;"><br></div>
<div style="font-family:Arial;">The difference is this: foreach_p is called with the database locked, always - even if called without a transaction. foreach_p returns 1 or 0. 0 means "skip this record", 1 means "process this record". This is useful to pre-filter records when called without a transaction, because otherwise you lock and unlock all the time.<br></div>
<div style="font-family:Arial;"><br></div>
<div style="font-family:Arial;">NULL for foreach_p is treated like a test which always returns '1', so you can pass NULL if you don't need filtering.<br></div>
<div style="font-family:Arial;"><br></div>
<div style="font-family:Arial;">If foreach_p returns 1, then with an unlocked transaction, the database is now unlocked BEFORE calling foreach_cb, the callback. foreach_cb returns a CYRUSDB_ response. If zero, the foreach will continue. If non-zero, the foreach will abort and return the non-zero response. This is both useful for error cases, and useful for short-circuiting, if you only care that a key exists, you can do something like:<br></div>
<div style="font-family:Arial;"><br></div>
</div>
</div>
</div>
<div style="font-family:Arial;">static int exists_cb(void *rock __attribute__((unused)), [...])<br></div>
<div style="font-family:Arial;">{<br></div>
<div style="font-family:Arial;"> return CYRUSDB_DONE; /* one is enough */<br></div>
<div style="font-family:Arial;">}<br></div>
<div style="font-family:Arial;"><div style="font-family:Arial;"><div style="font-family:Arial;"><div style="font-family:Arial;"><br></div>
<div style="font-family:Arial;">and then use exists_cb as your foreach_cb and check if the return code is CYRUSDB_DONE to know if the foreach found a key.<br></div>
<div style="font-family:Arial;"><br></div>
<div style="font-family:Arial;">If foreach is called with a transaction pointer, then it is your responsibility as the caller to also pass that pointer (and a pointer to the database) in that rock, so that callees can make further operations within the same transaction. A foreach with a transaction does NOT unlock before calling its callback.<br></div>
<div style="font-family:Arial;"><br></div>
<div style="font-family:Arial;">Writing:<br></div>
<div style="font-family:Arial;">===<br></div>
<div style="font-family:Arial;"><br></div>
<div style="font-family:Arial;">There are three write operations:<br></div>
<div style="font-family:Arial;"><br></div>
<div style="font-family:Arial;">cyrusdb_create<br></div>
<div style="font-family:Arial;">cyrusdb_store<br></div>
<div style="font-family:Arial;">cyrusdb_delete<br></div>
<div style="font-family:Arial;"><br></div>
<div style="font-family:Arial;">cyrusdb_store will either create or overwrite an existing key. cyrusdb_create will abort if the key already exists. cyrusdb_delete takes a flag 'force' which just makes it return CYRUSDB_OK (0) rather than CYRUSDB_NOTFOUND if they key doesn't exist. Strangely, 'force' is after &tid, making it the only cyrusdb API that does that, but hey - keeps you on your toes.<br></div>
<div style="font-family:Arial;"><br></div>
<div style="font-family:Arial;">&tid behaves exactly the same for the write APIs. If not passed, then the database engine will behave as it if creates a writable transaction, does the operation, then commits all within the cyrusdb_* call.<br></div>
<div style="font-family:Arial;"><br></div>
<div style="font-family:Arial;">Gotchas!<br></div>
<div style="font-family:Arial;">===<br></div>
<div style="font-family:Arial;"><br></div>
<div style="font-family:Arial;">* NULL is permitted in both keys and values, though 'flat' and 'quotalegacy' have 8-bit cleanliness issues.<br></div>
<div style="font-family:Arial;"><br></div>
<div style="font-family:Arial;">* zero-length keys are not supported<br></div>
<div style="font-family:Arial;"><br></div>
<div style="font-family:Arial;">* zero-length values are theoretically supported, but a little interesting. Certainly, pass "" rather than NULL as the value when writing or things will get weird. I'm pretty sure at least the *skip databases assert on these kinds of weirdness.<br></div>
<div style="font-family:Arial;"><br></div>
<div style="font-family:Arial;">* unlocked foreach: this is the land of the gotcha! They key and data pointers (const char *) passed to your foreach_cb are only valid <b>UNTIL YOU TOUCH THE DATABASE</b>. A common cause of rare and hard to diagnose bugs is writing something to the same database in the same process (<b>OR EVEN READING FROM IT AGAIN</b>). I cannot emphasise this enough. If you want to zero-copy access that data, you need to access it first, before touching that DB again. Otherwise the map in which the data was a pointer may have been replaced as the next read found a new file and mapped it in!<br></div>
<div style="font-family:Arial;"><br></div>
<div style="font-family:Arial;">also: if you're implementing a backend. Unlocked foreach must find future records created by the current callback. Consider a database containing 4 keys:<br></div>
<div style="font-family:Arial;"><br></div>
<div style="font-family:Arial;">A B C D<br></div>
<div style="font-family:Arial;"><br></div>
<div style="font-family:Arial;">if you are at key B and insert a key BB, then it must be iterated over. If you insert AA while at B, it must NOT be iterated over.<br></div>
<div style="font-family:Arial;"><br></div>
<div style="font-family:Arial;">* Opening the same database multiple times. In the bad old days, opening the same database multiple times in the same process led to locking bugs (fcntl is braindead). Each database engine is responsible for making sure this doesn't happen. Most engines keep a linked lists of open databases. If you try to open the same database again, they will just return the existing opened copy and bump a refcount. Beware. If a database is locked and you try to lock again - thinking you were opening it brand new, it will assertion fail and/or error.<br></div>
<div style="font-family:Arial;"><br></div>
<div style="font-family:Arial;"><br></div>
<div style="font-family:Arial;">I think that covers about everything! Cyrusdb is used just about everywhere that sorted key-value databases give what's needed, including mailboxes.db, annotations.db (global and per mailbox databases), seen state (non-owner), subscriptions, cyrus.indexed.db for Xapian, and the rather massive (and increasingly inaccurately named) user.conversations.<br></div>
<div style="font-family:Arial;"><br></div>
<div style="font-family:Arial;">Future plans are to increase the usage of cyrusdb databases, possibly by building an indexing layer on top and using that instead of the sqldb interface used for sqlite databases by DAV code, and possibly also moving other custom file formats into a cyrusdb to allow easier stateless server builds on a distributed backend.<br></div>
<div style="font-family:Arial;"><br></div>
<div style="font-family:Arial;"><br></div>
<div style="font-family:Arial;">Bron.<br></div>
<div style="font-family:Arial;"><br></div>
</div>
</div>
</div>
<div id="sig56629417"><div class="signature">--<br></div>
<div class="signature"> Bron Gondwana, CEO, FastMail Pty Ltd<br></div>
<div class="signature"> brong@fastmailteam.com<br></div>
<div class="signature"><br></div>
</div>
<div style="font-family:Arial;"><br></div>
</body>
</html>