Xapian locking rewrite

[Bron Gondwana]

It's become clear trying to fix the Xapian locking mess that we
actually want two locks, like we have for mailboxes.  A namelock that
is long lasting and doesn't block regular indexing or searching, but
guarantees that things won't be deleted out underneath, and the
existing short term locks on xapianactive that are used to guarantee
clean reads and safe indexing.
Xapian Locking:
---

* Xapian per-user NAMELOCK (shared or exclusive)
* xapianactive file lock (shared or exclusive)

Shared namelock holds the following invariants:
* xapianactive file contents are not changed
* directories mentioned in xapianactive are not cleaned up

Xapianactive exclusive lock holds the following invariants:
* owner may write to first database mentioned in xapianactive

Xapianactive shared lock holds:
* all databases in xapianactive are readable and a consistent
  read can be made across all of them, even with multiple requests
  while the lock is held.


Locking orders:

SHARED case:
* user conversations db <=== possibly reversible with SHARED
  xapian namelock* SHARED xapian namelock
* xapianactive lock (shared to search, exclusive to write)
* cyrus.index may be locked either side of the xapianactive lock,
  because the  conversationsdb lock protects it from races.

EXCLUSIVE case:
* EXCLUSIVE xapian namelock
  -> that's it.  While you've got this, you can add or delete items from     the xapianactive file, and delete paths on disk for
     directories that     you have removed (either during or after locking).  No other locks
     are     permitted.

If you hold a SHARED xapian namelock, you may write to a .NEW
folder for axapianactive entry that you created without taking any additional locks,because nothing can clean it under you, and nothing else can read it.
Thisis how the repack case works.

---------------------------------

Places that will need to be changed:

begin_search: needs to take a SHARED namelock before taking a SHARED xapianactive lock.
end_search: needs to release the namelock after unlocking the activefile.
begin_mailbox_update: needs to take a SHARED namelock before taking an EXCLUSIVE xapianactive lock.
end_mailbox_update: needs to release the namelock.

list_files: needs to take a SHARED namelock and release it again.

delete_user: needs to take an EXCLUSIVE namelock as well as the existing EXCLUSIVE xapianactive lock.
compact_dbs: this is the complex one!

1) take an EXCLUSIVE namelock while doing the following:
   -> add a new first target tier:number to the xapianactive file
   (always temp)   -> add a new tier:number target to the tail of the xapianactive file
   (squatter -z option)2) release the EXCLUSIVE namelock
3) take a SHARED namelock and SHARED xapianactive lock to read
   the contents of the xapianactive file and make sure we didn't
   lose a race.4) release the SHARED xapianactive lock.
5) compact or reindex into target:number xapian.<number>.NEW directory
   (see above, this is safe without locks)6) release the SHARED namelock
7) take an EXCLUSIVE namelock and check again that we have the same
   xapianactive contents, if we lost the race here, we still need to
   delete all the xapian.<number>.NEW and abort.8)  rename xapian.<number>.NEW to xapian.<number>
9) remove the source tier:number entries from xapianactive
10) release the EXCLUSIVE namelock.
11) take a SHARED namelock
12) It's now safe to delete all the directories which were source
    tier:number, as well as any other directory not mentioned in
    xapianactive any more.13) release everything, you're done :)


--
  Bron Gondwana, CEO, FastMail Pty Ltd
  brong at fastmailteam.com


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/cyrus-devel/attachments/20171120/9e0f9386/attachment.html>