From brong at fastmailteam.com Mon Jun 3 11:53:23 2019 From: brong at fastmailteam.com (Bron Gondwana) Date: Tue, 04 Jun 2019 01:53:23 +1000 Subject: squatter -F increases the index size In-Reply-To: <20190531183343.Horde.NqDuNlWbxJXSqZCJ2rWmPGG@webmail.aegee.org> References: <20190531183343.Horde.NqDuNlWbxJXSqZCJ2rWmPGG@webmail.aegee.org> Message-ID: <0a8d05c3-0a1b-491f-8d2e-3982272807b0@www.fastmail.com> On Sat, Jun 1, 2019, at 04:34, Dilyan Palauzov wrote: > Hello, > > I gave squatter -F a try. > > Before I run it for a user tier T1 was not compacted and allocated 3,4 > MB (mega), T2 was compacted and contained 3.7GB (giga). After > removing the records of the deteled messages, say running squatter -F > T2 was 5.7GB and squatter printed ?filtering? instead of ?compacting?. > Then I run again ?squatter -t T1,T2 -z T2? without -F, without -X > and squatter reindexed all messages, to create a 3.0 GB index. > > I expected, that using -F the resulting database will be compacted and > on the second call there will be no reindexing. I discovered some bad bugs in -F recently, so I suspect that's why. They should be fixed on master now. > When does squatter decide on its own to reindex? When the DB version is too old (which is one of the -F bugs - it wasn't setting the DB version, so it assumed the data was all version zero!) > What do G records in conversations.db contain? G records contain a mapping from GUID to folder number (offset into the $FOLDER_NAMES key) and UID and optionally IMAP part number as the key - mapping to values which contain some keywords and modseq from the original record as well. > My reading is that the way to create a Xapian index of an indexed > mailbox, is that first squatter has to be run in INDEX mode and then > in COMPACT mode. In particular it is not possible to create in one > step a compacted database. No, it's not - due to the way to compact API works. At least, I haven't figured out how. > Does squatter -R -S sleep after each mailbox or after each message indexed? It sleeps after each mailbox. > When compacting, squatter deals just with messages and on search or > reindex the conversations.db is used to map the messages to mailboxes. > How does squatter -S sleep after each mailbox during compacting, if > it knows nothing about mailboxes? -S is not used when compacting. > What does mean a tier name in a xapianactive file without a number? that shouldn't happen. It will be parsed as the same as tier:0 I believe. > What are XAPIAN_DBW_CONVINDEXED and _XAPINDEXED? Two different ways to know if a document is indexed. CONVINDEXED uses the conversations DB to look up mailbox and uid and then the cyrus.indexed.db databases to see if the message has already been seen. XAPINDEXED uses the metadata inside the Xapian databases to know if a particular message has been indexed based on the cyrusid.*G* metadata values which are identical to the GUIDs themselves. > What does the file sync/squatter? It's a sync/$channel directory which squatter watches on. This is a method for providing a queue of mailboxes to look at based on the APPEND sync_log statements. > squatter can print ?Xapian: truncating text from message mailbox > user.... uid 7309?. When are messages truncated for the purposes of > indexing? When they are too long! The comment in the source code says this: /* Maximum size of a query, determined empirically, is a little bit * under 8MB. That seems like more than enough, so let's limit the * total amount of parts text to 4 MB. */ #define MAX_PARTS_SIZE (4*1024*1024) This is a holdover from when Greg was working on it. We could switch this to be a configurable option. > Do I understand correctly, that for a Xapianactive file with "A B C D > E", to remove C one has to call "squatter -t C,D -z D". But A cannot > be removed, if it the defaultsearchtier. Is the defaultsearchtier > always included in the xapianactive file, if the tier is missing, > whenever the file is modified (and the only way to modify it is to > call squatter in COMPACT mode)? When you do any compact, if it includes the first item (the writable database) then a new writable database will be created on the default tier. So if you try to compact the default tier away, a new default tier item will be created. Bron. -- Bron Gondwana, CEO, FastMail Pty Ltd brong at fastmailteam.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From Dilyan.Palauzov at aegee.org Mon Jun 3 14:15:43 2019 From: Dilyan.Palauzov at aegee.org (Dilyan Palauzov) Date: Mon, 03 Jun 2019 18:15:43 +0000 Subject: squatter -F increases the index size In-Reply-To: <0a8d05c3-0a1b-491f-8d2e-3982272807b0@www.fastmail.com> References: <20190531183343.Horde.NqDuNlWbxJXSqZCJ2rWmPGG@webmail.aegee.org> <0a8d05c3-0a1b-491f-8d2e-3982272807b0@www.fastmail.com> Message-ID: <20190603181543.Horde.q5ZVGuOUcTm8eh5OFkE17H8@webmail.aegee.org> Hello Bron, imap/squatter.c:do_compact() does call `if (sleepmicroseconds) usleep(sleepmicroseconds);` so -S number is honoured with `squatter -t? -z?`. Will `squatter -F -z? -t?` be fixed on the stable branch, or shall calling `squatter -F -t? -z` be discouraged with 3.0? Providing that currently after `squatter -F -t? -z?` calling `squatter -t? -z` does reindex all messages and therefore creates a new xapian index, it must be possible to to create a compacted database directly, without creating an bloated index first. My understaning to the rolling mode is that once a new message appears/arrives/is APPENDed or deliver(ed), it is added to the sync log and then indexed in rolling mode. Then arrives a message at a different place, it is added to the log and then indexed. Whether the first and second messages are in the same mailbox is completely random. Why does squatter not sleep, if the two messages are in the same mailbox and works non-stop otherwise, say why does it sleep depending on random circumstances? https://wiki.dovecot.org/Plugins/FTS/Squat says for DoveCot that IMAP requires that SEARCH is done also on substings, no IMAP server implements this requirement, and dovecot does implement it only when Squat indices are used. Is the same valid for Cyrus Imap (Squat index offers substring search, Xapian index does not offer substring search)? Runnig squatter once printed ?compressing X:0,X,Y:0 to Y:3 for ? (active Y:0,X:0,X,Y:0,Y:1,Y:2)? (https://github.com/cyrusimap/cyrus-imapd/issues/2764) so I suspect a tiername without a number was in the .xapianactive file. If I do any compact (-o, -F, -X, just -t -z), where the first tier is not referenced, does squatter ensure that the default tier according to imapd.conf is inserted in the xapianactive file. Or asking in other ways, it I change imapd.conf and create a new tier T6 and declare T5 to be the default tier, which of the following will insert a reference to T5:0 in .xapianactive and which will not: squatter -t T2 -o -z T2 squatter -t T5,T2 -z T2 squatter -t T5 -o T4 squatter -t T2 -F T3 sqautter -t T2 -X T3 or what else? (The name T5 is declared, and the root directory exist, but neither there is data in the directory, nor is T5 yet in any .xapianactive file). Regards ????? ----- Message from Bron Gondwana --------- Date: Tue, 04 Jun 2019 01:53:23 +1000 From: Bron Gondwana Subject: Re: squatter -F increases the index size To: Cyrus Devel > On Sat, Jun 1, 2019, at 04:34, Dilyan Palauzov wrote: >> Hello, >> >> I gave squatter -F a try. >> >> Before I run it for a user tier T1 was not compacted and allocated 3,4 >> MB (mega), T2 was compacted and contained 3.7GB (giga). After >> removing the records of the deteled messages, say running squatter -F >> T2 was 5.7GB and squatter printed ?filtering? instead of ?compacting?. >> Then I run again ?squatter -t T1,T2 -z T2? without -F, without -X >> and squatter reindexed all messages, to create a 3.0 GB index. >> >> I expected, that using -F the resulting database will be compacted and >> on the second call there will be no reindexing. > > I discovered some bad bugs in -F recently, so I suspect that's why. > They should be fixed on master now. > >> When does squatter decide on its own to reindex? > > When the DB version is too old (which is one of the -F bugs - it > wasn't setting the DB version, so it assumed the data was all > version zero!) > >> What do G records in conversations.db contain? > > G records contain a mapping from GUID to folder number (offset into > the $FOLDER_NAMES key) and UID and optionally IMAP part number as > the key - mapping to values which contain some keywords and modseq > from the original record as well. > >> My reading is that the way to create a Xapian index of an indexed >> mailbox, is that first squatter has to be run in INDEX mode and then >> in COMPACT mode. In particular it is not possible to create in one >> step a compacted database. > > No, it's not - due to the way to compact API works. At least, I > haven't figured out how. > >> Does squatter -R -S sleep after each mailbox or after each message indexed? > > It sleeps after each mailbox. > >> When compacting, squatter deals just with messages and on search or >> reindex the conversations.db is used to map the messages to mailboxes. >> How does squatter -S sleep after each mailbox during compacting, if >> it knows nothing about mailboxes? > > -S is not used when compacting. > >> What does mean a tier name in a xapianactive file without a number? > > that shouldn't happen. It will be parsed as the same as tier:0 I believe. > >> What are XAPIAN_DBW_CONVINDEXED and _XAPINDEXED? > > Two different ways to know if a document is indexed. CONVINDEXED > uses the conversations DB to look up mailbox and uid and then the > cyrus.indexed.db databases to see if the message has already been > seen. > > XAPINDEXED uses the metadata inside the Xapian databases to know if > a particular message has been indexed based on the cyrusid.*G* > metadata values which are identical to the GUIDs themselves. > >> What does the file sync/squatter? > > It's a sync/$channel directory which squatter watches on. This is a > method for providing a queue of mailboxes to look at based on the > APPEND sync_log statements. > >> squatter can print ?Xapian: truncating text from message mailbox >> user.... uid 7309?. When are messages truncated for the purposes of >> indexing? > > When they are too long! The comment in the source code says this: > > /* Maximum size of a query, determined empirically, is a little bit > * under 8MB. That seems like more than enough, so let's limit the > * total amount of parts text to 4 MB. */ > #define MAX_PARTS_SIZE (4*1024*1024) > > This is a holdover from when Greg was working on it. We could switch > this to be a configurable option. > >> Do I understand correctly, that for a Xapianactive file with "A B C D >> E", to remove C one has to call "squatter -t C,D -z D". But A cannot >> be removed, if it the defaultsearchtier. Is the defaultsearchtier >> always included in the xapianactive file, if the tier is missing, >> whenever the file is modified (and the only way to modify it is to >> call squatter in COMPACT mode)? > > When you do any compact, if it includes the first item (the writable > database) then a new writable database will be created on the > default tier. So if you try to compact the default tier away, a new > default tier item will be created. > > Bron. > > -- > Bron Gondwana, CEO, FastMail Pty Ltd > brong at fastmailteam.com ----- End message from Bron Gondwana ----- From brong at fastmailteam.com Fri Jun 7 15:11:39 2019 From: brong at fastmailteam.com (Bron Gondwana) Date: Sat, 08 Jun 2019 05:11:39 +1000 Subject: squatter -F increases the index size In-Reply-To: <20190603181543.Horde.q5ZVGuOUcTm8eh5OFkE17H8@webmail.aegee.org> References: <20190531183343.Horde.NqDuNlWbxJXSqZCJ2rWmPGG@webmail.aegee.org> <0a8d05c3-0a1b-491f-8d2e-3982272807b0@www.fastmail.com> <20190603181543.Horde.q5ZVGuOUcTm8eh5OFkE17H8@webmail.aegee.org> Message-ID: I saw your ticket about that - I'll check it out soon. Sorry, been busy at the calconnect conference in the UK this week. I also did this: https://github.com/cyrusimap/cyrus-imapd/commit/27513a9bc3f217f388bac163820f9879178071fb Which I believe will mean that if you change the defaultsearchtier, it will immediately start indexing to the new location. You'll definitely want to restart a server over changing that config option though, and not have it be different between different invocations of squatter, or you'll wind up creating a lot of xapainactive entries! Bron. On Tue, Jun 4, 2019, at 04:19, Dilyan Palauzov wrote: > Hello Bron, > > imap/squatter.c:do_compact() does call `if (sleepmicroseconds) > usleep(sleepmicroseconds);` so -S number is honoured with `squatter > -t? -z?`. > > Will `squatter -F -z? -t?` be fixed on the stable branch, or shall > calling `squatter -F -t? -z` be discouraged with 3.0? > > Providing that currently after `squatter -F -t? -z?` calling `squatter > -t? -z` does reindex all messages and therefore creates a new xapian > index, it must be possible to to create a compacted database directly, > without creating an bloated index first. > > My understaning to the rolling mode is that once a new message > appears/arrives/is APPENDed or deliver(ed), it is added to the sync > log and then indexed in rolling mode. Then arrives a message at a > different place, it is added to the log and then indexed. Whether the > first and second messages are in the same mailbox is completely > random. Why does squatter not sleep, if the two messages are in the > same mailbox and works non-stop otherwise, say why does it sleep > depending on random circumstances? > > https://wiki.dovecot.org/Plugins/FTS/Squat says for DoveCot that IMAP > requires that SEARCH is done also on substings, no IMAP server > implements this requirement, and dovecot does implement it only when > Squat indices are used. Is the same valid for Cyrus Imap (Squat index > offers substring search, Xapian index does not offer substring search)? > > Runnig squatter once printed ?compressing X:0,X,Y:0 to Y:3 for ? > (active Y:0,X:0,X,Y:0,Y:1,Y:2)? > (https://github.com/cyrusimap/cyrus-imapd/issues/2764) so I suspect a > tiername without a number was in the .xapianactive file. > > If I do any compact (-o, -F, -X, just -t -z), where the first tier is > not referenced, does squatter ensure that the default tier according > to imapd.conf is inserted in the xapianactive file. Or asking in > other ways, it I change imapd.conf and create a new tier T6 and > declare T5 to be the default tier, which of the following will insert > a reference to T5:0 in .xapianactive and which will not: > > squatter -t T2 -o -z T2 > squatter -t T5,T2 -z T2 > squatter -t T5 -o T4 > squatter -t T2 -F T3 > sqautter -t T2 -X T3 > or what else? (The name T5 is declared, and the root directory exist, > but neither there is data in the directory, nor is T5 yet in any > .xapianactive file). > > Regards > ????? > ----- Message from Bron Gondwana --------- > Date: Tue, 04 Jun 2019 01:53:23 +1000 > From: Bron Gondwana > Subject: Re: squatter -F increases the index size > To: Cyrus Devel > > > > On Sat, Jun 1, 2019, at 04:34, Dilyan Palauzov wrote: > >> Hello, > >> > >> I gave squatter -F a try. > >> > >> Before I run it for a user tier T1 was not compacted and allocated 3,4 > >> MB (mega), T2 was compacted and contained 3.7GB (giga). After > >> removing the records of the deteled messages, say running squatter -F > >> T2 was 5.7GB and squatter printed ?filtering? instead of ?compacting?. > >> Then I run again ?squatter -t T1,T2 -z T2? without -F, without -X > >> and squatter reindexed all messages, to create a 3.0 GB index. > >> > >> I expected, that using -F the resulting database will be compacted and > >> on the second call there will be no reindexing. > > > > I discovered some bad bugs in -F recently, so I suspect that's why. > > They should be fixed on master now. > > > >> When does squatter decide on its own to reindex? > > > > When the DB version is too old (which is one of the -F bugs - it > > wasn't setting the DB version, so it assumed the data was all > > version zero!) > > > >> What do G records in conversations.db contain? > > > > G records contain a mapping from GUID to folder number (offset into > > the $FOLDER_NAMES key) and UID and optionally IMAP part number as > > the key - mapping to values which contain some keywords and modseq > > from the original record as well. > > > >> My reading is that the way to create a Xapian index of an indexed > >> mailbox, is that first squatter has to be run in INDEX mode and then > >> in COMPACT mode. In particular it is not possible to create in one > >> step a compacted database. > > > > No, it's not - due to the way to compact API works. At least, I > > haven't figured out how. > > > >> Does squatter -R -S sleep after each mailbox or after each message indexed? > > > > It sleeps after each mailbox. > > > >> When compacting, squatter deals just with messages and on search or > >> reindex the conversations.db is used to map the messages to mailboxes. > >> How does squatter -S sleep after each mailbox during compacting, if > >> it knows nothing about mailboxes? > > > > -S is not used when compacting. > > > >> What does mean a tier name in a xapianactive file without a number? > > > > that shouldn't happen. It will be parsed as the same as tier:0 I believe. > > > >> What are XAPIAN_DBW_CONVINDEXED and _XAPINDEXED? > > > > Two different ways to know if a document is indexed. CONVINDEXED > > uses the conversations DB to look up mailbox and uid and then the > > cyrus.indexed.db databases to see if the message has already been > > seen. > > > > XAPINDEXED uses the metadata inside the Xapian databases to know if > > a particular message has been indexed based on the cyrusid.*G* > > metadata values which are identical to the GUIDs themselves. > > > >> What does the file sync/squatter? > > > > It's a sync/$channel directory which squatter watches on. This is a > > method for providing a queue of mailboxes to look at based on the > > APPEND sync_log statements. > > > >> squatter can print ?Xapian: truncating text from message mailbox > >> user.... uid 7309?. When are messages truncated for the purposes of > >> indexing? > > > > When they are too long! The comment in the source code says this: > > > > /* Maximum size of a query, determined empirically, is a little bit > > * under 8MB. That seems like more than enough, so let's limit the > > * total amount of parts text to 4 MB. */ > > #define MAX_PARTS_SIZE (4*1024*1024) > > > > This is a holdover from when Greg was working on it. We could switch > > this to be a configurable option. > > > >> Do I understand correctly, that for a Xapianactive file with "A B C D > >> E", to remove C one has to call "squatter -t C,D -z D". But A cannot > >> be removed, if it the defaultsearchtier. Is the defaultsearchtier > >> always included in the xapianactive file, if the tier is missing, > >> whenever the file is modified (and the only way to modify it is to > >> call squatter in COMPACT mode)? > > > > When you do any compact, if it includes the first item (the writable > > database) then a new writable database will be created on the > > default tier. So if you try to compact the default tier away, a new > > default tier item will be created. > > > > Bron. > > > > -- > > Bron Gondwana, CEO, FastMail Pty Ltd > > brong at fastmailteam.com > > > ----- End message from Bron Gondwana ----- > > > -- Bron Gondwana, CEO, FastMail Pty Ltd brong at fastmailteam.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From dilyan.palauzov at aegee.org Mon Jun 10 13:11:51 2019 From: dilyan.palauzov at aegee.org (=?UTF-8?B?0JTQuNC70Y/QvSDQn9Cw0LvQsNGD0LfQvtCy?=) Date: Mon, 10 Jun 2019 20:11:51 +0300 Subject: Cyrus Roadmap as VTASK set Message-ID: <3BC04BF4-8C11-4DC3-A149-50AFBE7221EE@aegee.org> Hello, since Cyrus Imap supports publishing Tasks = Roadmaps (VTODOs over CalDAV or over HTTP/iCalendar), will it be feasible to integrate VTODOs into the workflow? How does a Github issue differ from a VTASK, apart from tracking the references in git and the automatic closing? Regarda ????? -------------- next part -------------- An HTML attachment was scrubbed... URL: From brong at fastmailteam.com Sun Jun 16 08:35:11 2019 From: brong at fastmailteam.com (Bron Gondwana) Date: Sun, 16 Jun 2019 22:35:11 +1000 Subject: syncronous replication and replication cache Message-ID: <5c1dd251-804c-4014-b169-89517579ba25@www.fastmail.com> (this was originally written for a FastMail internal mailing list, but it has more general interest, so I'm posting it here too) I figured I should write some of this up so we have a design we can all talk about, and so we know what's roughly involved in getting there. *The goal*** We'd love to be able to run no-raid IMAP servers, but that means that if we lose a drive, we lose data unless we have a real time replica. There are basically two ways to do this: 1) drbd (aka: cross-machine RAID!) 2) synchronous replication at the Cyrus level We consider that drbd would be too poor performance (though this isn't tested) and would mean tricky things for failover because our tooling isn't designed for it, so we're looking at synchronous replication in Cyrus. *Making replication more efficient*** Here's the current network traffic for 3 common scenarios: C is the client (aka master) and S is the server (replica). 1) delivery of a new message to mailbox A: C: GET MAILBOXES (A) S: * MAILBOX (A) { header details } S: OK C: APPLY RESERVE (A) (GUID) S: OK RESERVED () C: APPLY MESSAGE (GUID) {n+} data S: OK C: APPLY MAILBOX A { header details } RECORD (%(UID GUID ...)) S: OK 2) move of a message from mailbox A to mailbox B C: GET MAILBOXES (A B) S: * MAILBOX (A) { header details } S: * MAILBOX (B) { header details } S: OK C: APPLY RESERVE (A B) (GUID) S: OK RESERVED (GUID) C: APPLY MAILBOX B { header details } RECORD (%(UID GUID ...)) S: OK 3) touch a flag on a record (e.g. mark it seen as the owner - seen as non-owner is more complex and involves an APPLY META) C: GET MAILBOXES (A) S: * MAILBOX (A) { header details } S: OK C: APPLY MAILBOX A { header details } RECORD (%(UID GUID FLAGS...)) S: OK in order to speed all these up, we have proposed a replication cache. This would know, for each mailbox, the last known state at the replica end (aka: the GET MAILBOXES response) so: 1) becomes C: APPLY MESSAGE (GUID) {n+} data S: OK C: APPLY MAILBOX A { header details } RECORD (%(UID GUID ...)) S: OK 2) becomes C: APPLY MAILBOX B { header details } RECORD (%(UID GUID ...)) S: OK and; 3) becomes C: APPLY MAILBOX A { header details } RECORD (%(UID GUID FLAGS...)) S: OK What happened to the "APPLY RESERVE" in (2) you ask? We do on-the-fly reserve on the server by scanning through any new appends and looking for the record using the conversations DB. Yep. And the client knows it's likely there by checking createdmodseqs on existing records in ITS conversations DB, because we can do that. *Sanity checking*** If we're going to just be doing a direct APPLY command with a cached state, then the existing race condition between the GET MAILBOXES and the APPLY MAILBOX becomes much wider. This is fine if nothing else ever modifies your replica, but sometimes things change. I already have a branch to address this with the ifInState equivalent for the replication protocol, which is three new keys in the APPLY MAILBOX: SINCE_MODSEQ, SINCE_CRC and SINCE_CRC_ANNOT, plus a new error code. This branch is already done and passes tests. *The cache*** This would be a per-channel twoskip file on tmpfs probably, into which a dlist per mailbox would be stored. The place to hook writing this would be after the OK response just before 'done:' in update_mailbox_once(). And also hook wiping it into any failure of update_mailbox_once so that the retries could fix things up. The place to hook reading would probably just be sync_do_mailboxes, where the loop copying the sync_folder_list data into the MAILBOXES dlist could check the cache for each name and just transcribe that into the replica_folders list, and obviously not send a backend request if the entire list was satisfied from cache. We could also write cache with this result of the MAILBOXES call just in case it's already up to date and hence update_mailbox_once doesn't need to be called! *The new reserve*** Skipping the current reserve call requires making the sync_server able to use its local conversations db to resolve GUIDs as needed from the conversations.db and copy the files over. This should be viable using logic from the JMAP bulk-update, but it's going to need to be copied, because the JMAP code requires lots of logic which is deep inside the jmap modules, so we can't just call it from the replication subsystem. This saves roundtrips for the reserve call. It does depend on delayed expunge to some level, but that's OK so long as it can recover from a "missing GUID on the replica" failure because its estimate of what was in other folders was wrong! *A local lock*** While any sync_client is replicating a mailbox, it will need to take a local lock on $SYNC-$channel-$mboxname! This is so that we don't have two processes trying to sync the same channel at the same time! Rolling sync could try non-blocking for that name and just punt it (aka: sync_log it to itself again) if the lock is busy. *The realtime sync*** And now we get to the meat of this! We want to allow multiple parallel real-time sync calls, but probably not hundreds. I suggest that we use a sync_client-style tool which runs as a daemon within cyrus, listening probably on a UNIX socket and keeping a long running backend connection open, so there may be a handful of syncd / sync_server parings running at any one time, servicing random requests. The config would be something like sync_realtime_socket: [% confdir %]/socket/syncd and in cyrus.conf syncd cmd="syncd [% CONF %] -n [ %buddyslotchannel %]" listen="[% confdir %]/socket/syncd" (aka: use the named channel to choose the backend for realtime sync) The client code would keep EXACTLY the current sync_log support to all channels, but have the following very small modification added: at the end of mailbox_unlock_index or mailbox_close or wherever seems the most sane, we add a synchronous callout. It might even go up a layer, though there's many more places to think about. Anyway, this callout connects to the local syncd on the specified port and asks it to replicate just the one mailbox name to the backend, and waits for the reply. If the reply is a failure, then that gets syslogged, but otherwise the return is still success to the rest of the code. The end result of hooking at this point is that EVERY protocol which replies success after committing the changes will not reply back to the client until the data is replicated to the buddy machine (buddy naming is hearkening back to the short-lived career of Max at FastMail and his buddyslot idea!) The rolling sync_client will pick up the logged name still, but the good news is, it will either still be locked (local lock) and hence try later, or it will already be up to date, and the local state will match the cached state, so there's no network IO or heavy calculations at all! This lack of network IO at all for the followup case is why I think the cache and remote conversations-based GUID lookup (and local validation that it's worth trying) is worth doing first, because that means that synchronous replication stands a chance of being efficient enough to be sane! I _think_ this design is good. Please let me know if anything about how it's supposed to work is unclear, or if you have a better idea. In terms of the split if work: I envisage Ken or ellie writing the syncd stuff. I've already written the ifInState stuff, so I'll just ship that. The caching could be anybody, it's pretty easy, and the GUID lookup for master/replica could be me, or could be someone else if they're keen to have a look at it - I have a very clear idea of how that would look. Cheers, Bron. -- Bron Gondwana, CEO, FastMail Pty Ltd brong at fastmailteam.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From brong at fastmailteam.com Mon Jun 24 07:45:21 2019 From: brong at fastmailteam.com (Bron Gondwana) Date: Mon, 24 Jun 2019 21:45:21 +1000 Subject: Notes 24 June Message-ID: <368db19a-33a2-4709-abc1-eedbb69cb63a@www.fastmail.com> Ken: * Spend last week pottering around with rebase of uuidmailbox work. * Also working on libical * Last week issue with broken libicu, now crashing on invalid data when before it ignored it. - might patch around it in Cyruslibs version of libicu. Robert: * Cyrus work: - some of the Xapian stuff would be good to test with older index db formats. - It would be good to run tests with two different binaries, one to create old-style data and the other one to see that things still work! - this would be a good general thing, both for replication and for our own testing! - The Xapian issues in the last little while would have been found if we had legacy version testing. * Query normalisation for contact groups - may need to cache. * JSCalendar and JSContact drafts! - JSCalendar unlikely to finish by the end of the week, so Bron to take on pushing it from here. - JSContact - will only be an interim draft, so will continue when back. * Will be bringing Cyrus up to date with JSCalendar - should we bring it up to date on master first, or wait until approved? Bron: * Wrote an email about SINCE_* keys in replication and replication cache. * Have shipped SINCE_* to FM production, working fine. * Next steps are exciting, but can't work on that just yet! ellie: * not sure if Cassandane is the right vehicle for general multiple version support * spent some time with imaptest this week and has opened an issue with their github - working on integrating the non-scripted benchmark/race condition searching tests. Times with Robert away - we'll stick with this time for now, it's still working for everyone! -- Bron Gondwana, CEO, FastMail Pty Ltd brong at fastmailteam.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From ellie at fastmail.com Thu Jun 27 23:57:43 2019 From: ellie at fastmail.com (ellie timoney) Date: Fri, 28 Jun 2019 13:57:43 +1000 Subject: Notes 24 June In-Reply-To: <368db19a-33a2-4709-abc1-eedbb69cb63a@www.fastmail.com> References: <368db19a-33a2-4709-abc1-eedbb69cb63a@www.fastmail.com> Message-ID: <32a4a124-171a-448b-867e-00752885e69e@www.fastmail.com> So, > Robert: > * Cyrus work: > - some of the Xapian stuff would be good to test with older index db formats. > - It would be good to run tests with two different binaries, one to create old-style data and the other one to see that things still work! > - this would be a good general thing, both for replication and for our own testing! > - The Xapian issues in the last little while would have been found if we had legacy version testing. > I did a bunch of work on this ages ago, but I forget about it a lot, and I'm not sure anyone but me really knows it's there because we all treat our Cassandane config as set-and-forget. Cassandane _does_, _sort of_, have support for multiple-version testing. If you have a look in your cassandane.ini (or cassandane.ini.example), you will see that there are separate sections for [cyrus default], [cyrus replica] and [cyrus murder]. When Cassandane spins up multiple Cyrus instances for the replication/murder/etc tests, it'll usually use the [cyrus default] cyrus installation, but if there are configured [cyrus replica] and/or [cyrus murder] installations, it will use those to start up any replica or murder instances needed by the tests. So, to do multiple-version testing, you install your desired versions of Cyrus, tell Cassandane which install to use for the default instance and which to use for replica/murder instances, then run testrunner.pl normally. The tests themselves have no awareness that there are multiple versions being run, they just know they have a main instance and a replica (or murder) instance. (The gory details are in Cassandane/Instance.pm iirc.) What it doesn't have support for, is for an individual test to manage the multiple versions (you are either running multiple versions for all tests, or none), or for a test to test any more than one pair of versions, or for a single instance to run two different versions of some Cyrus component. It needs you to manage the Cyrus installs, and tell it which to use for what purpose, and then it delivers those instances when a test asks for a replica or murder instance. Any tests you write specifically to verify cross-version behaviour will, in a normal situation (i.e. with [cyrus default] only), not be testing cross-version behaviour because both instances will be from the same install. A test _can_ interrogate which Cyrus version is being used for each instance by calling Cassandane::Instance->get_version($instance->{installation}) (installation will be one of "default", "replica", "murder"... i.e. the foo in the [cyrus foo] section name from your config) if you need to assert specific behaviours between specific versions. The :min_version and :max_version magics only look at the default instance, fyi. I can't remember what the details of the Xapian issue were, but I think to make a test for something like this, you'd make a test that sets up the world on the main instance, then replicates it and runs squatter on the replica, then asserts that the replica's state is what we expect. If you ran this with "the older version" as your default cyrus and "the newer version" as your replica cyrus, then you'd be able to see whether old version -> new version is sane (with the caveat that the test inherently needs to replicate it, so if replicating it would mask the problem, then you can't test for it like this...) > ellie: > * not sure if Cassandane is the right vehicle for general multiple version support > The Test::Unit CPAN suite it fundamentally depends on is incomplete and abandonware, so the whole thing is already fairly precarious. I'm worried that a substantial refactor will make it unreliable, and if it's unreliable, it's useless. Cheers, ellie -------------- next part -------------- An HTML attachment was scrubbed... URL: