Laying groundwork

[Bron Gondwana]

Way back in the distant past (2010) I rewrote everything which read records from cyrus.index to go through one API call:

mailbox_read_index_record(mailbox, recno, &record);

So the way to work through all the records in a mailbox was:

int r;
uint32_t recno;
struct index_record record;

for (recno = 1; recno <= mailbox->i.num_records; recno++) {
    r = mailbox_read_index_record(mailbox, recno, &record);
    if (r) continue;

    /* if not looking at expunged records */
    if (record.system_flags & FLAG_EXPUGNED) continue;

    ...
}

The problem with this pattern is that it won't be efficient for a sql backed (or any other cursor backed) method of accessing the records.  There were two choices for how to change it, one is a foreach style callback, and the other is an iterator.  I chose an iterator.  So presenting the new API.  Note that "record" is now a const pointer to data stored inside the iterator, and there's a bit of magic under the hood to allow you do do a lazy cache read while still pretending it's not changing.

const struct index_record *record;
struct mailbox_iter *iter = mailbox_iter_init(mailbox, modseq, iterflags);
while ((record = mailbox_iter_step(iter))) {
   ...
}
mailbox_iter_done(&iter);

This allows both a fetch of only records changed since a particular modseq, or a fetch of only records without certain flags - check some of the uses for examples.  A common pattern is init(mailbox, 0, ITER_SKIP_EXPUNGED).

In parallel with this, mailbox_read_index_record has been entirely removed.  You can't address records by recno from outside mailbox.c any more.  Instead, you use mailbox_reload_index_record and it pulls the recno or uid from the record itself, or if you're coming from a UID you can still use mailbox_find_index_record (which lost the oldrecord pointer that isn't being used).

The full series of commits just went to master (like this, I missed a couple of rebases, oh well)

353751c mailbox_find_index_record: reimplement on top of mailbox_finduid
33d57a6 mailbox: remove unused oldrecord param from find_index_record
9d61ec7 cunit: test mailbox
e069ea6 sync_client record sending logic
0d73da6 sync_support: remove prev list
45bc1a3 sync_support - add back missing else
d596f30 mailbox: make read_index_record a static method!  Yay
b84cf55 sync_support: use iter
4e21ae0 dav: use reload_index_record
088a743 iter: keep num_records from the start, so we don't iterate over anything we append ourselves
c272425 nntpd: use iter (bugfix record fetch, been broken since forever)
093cb37 index: remove the last iterator
0bafa60 search_engines: use iter
b59ec23 mailbox: startuid for iterators
8b3c3b5 pop3d: use reload_record
91734e2 dav: use reload from index map
f9e72cb index: export index_reload_record
639e35d message_test: use reload_record
66096f5 rss: use reload_record
b2c6656 index: use reload_record and some name tidying
7514083 cyrdump: use reload_record
93ca9cb message: use reload_record
15dc0c0 mailbox: handle uid too
724e4bc mailbox: reload_index_record API
1bc977f fixer
34ade4a to keep
5499849 mailbox: use iter
e7ef08a mailbox: more const APIs
8995b4a fud: use iter
07f9a75 index: use iter
6ed70e4 mailbox: skip no-uid records
be0ee3f mbtool: use iter
8690d5e ctl_conversationsdb: use iter
53eb96c unexpunge: use iter
d7e6d6a dav: use iter (note: still bogus per-recno reads)
8362f6a statuscache: use iter
966ce84 doc: update mailbox-api example
ec4a985 sync: more const support
fa1292c dlist: const support for guids
6cbb112 message_guid: const interfaces (even if dodgy inside)
5331a3f xapian: use iter
ee6ff58 mbexamine: use iter
985d91e mailbox: backdoor const stripper to allow mailbox_cacherecord on const records
4b484a2 mailbox: const struct_index_record for all cache readers
4cd0cf0 mbdump: use iter
710b2fa pop3: use iter
f2f6049 caldav: use iter
4cee903 imapd: change to new iter
4d431c4 mailbox: add mailbox_iter API
e70b7b8 ical_support: add const to index_record usage
396e919 mailbox: add const to index_record usage
66625a9 remove unused sync_mailbox_finduid API
421bfb7 ptclient: fixed typo

===

In addition to the changed way of accessing mailboxes, I also pushed a refactor of sqldb out from dav_db.c to make it more generic.  This has been running in production on FastMail for a few days now.  The generic implementation of sqldb will allow us to use sqlite databases with less boilerplate code anywhere else we need them.  It includes the upgrade framework.  It requires sqlite 3.6.0+ (though I haven't updated the detection code yet) - it's in commits:

a8a81e7 Makefile: only include sqldb if required
4528a39 dav_db: use sqldb
e3373cc caldav_alarm: use sqldb
a2feb10 sqldb: add sqldb_lastid API
fbffb11 sqldb API

===

This is all groundwork for larger changes later :)

Bron.




-- 
  Bron Gondwana
  brong at fastmail.fm