From dilyan.palauzov at aegee.org  Mon Oct  5 05:44:11 2020
From: dilyan.palauzov at aegee.org (=?UTF-8?Q?=D0=94=D0=B8=D0=BB=D1=8F=D0=BD_?= =?UTF-8?Q?=D0=9F=D0=B0=D0=BB=D0=B0=D1=83=D0=B7=D0=BE=D0=B2?=)
Date: Mon, 05 Oct 2020 12:44:11 +0300
Subject: Deleting stale git-branches
Message-ID: <c0a216c1436e99f124a390a2e70783f67dd56402.camel@aegee.org>

Dear all,

when I work on the command line with git and want from git to
autocomplete the words ?sieve? or ?master?, git does not know if I mean
the branches ?sieve-1.2/master? or the directories ?sieve/master?.
 This ambiguity makes thing unnecessary complicated.

? Please delete all stale branches, like sieve-1.2 (last updated in
1999).

? Either rename the master branch, or rename the master directory, so
that ?git log -p ma<TAB>" is unambiguous.

Greetings
  ?????


From dilyan.palauzov at aegee.org  Mon Oct  5 11:36:34 2020
From: dilyan.palauzov at aegee.org (=?UTF-8?Q?=D0=94=D0=B8=D0=BB=D1=8F=D0=BD_?= =?UTF-8?Q?=D0=9F=D0=B0=D0=BB=D0=B0=D1=83=D0=B7=D0=BE=D0=B2?=)
Date: Mon, 05 Oct 2020 18:36:34 +0300
Subject: =?UTF-8?Q?=E2=80=9Cctl=5Fmboxlist?= -d | ctl_mboxlist
 =?UTF-8?Q?-u=E2=80=9D?= makes some fields disappear
Message-ID: <8e802ecf0d3255dcd760ac96dfc6d09f0d2ba25e.camel@aegee.org>

Hello,

for turning on improved_mboxlist_sort I turned off cyrus imap,  called
`ctl_mboxlist -d > FILE`, renamed mailboxes.db to mailboxes.db2,
switched in imapd.conf 'improved_mboxlist_sort:1', and called
`ctl_mboxlist -u < FILE`, cf 
https://www.cyrusimap.org/imap/developer/thoughts/improved_mboxlist_sort.html
 .

This created a new mailboxes.db file, which is 2.33 times smaller, that
the original (renamed).

The database format is twoskip.  

Converting both the old and new mailboxes.db to flat with
  $cvt_cyrusdb MAILBOXES.dbX twoskip MM flat
and comparing the results, the old file contains


user.ttt.#addressbooks.MMM	%(A %(ttt lrswipkxtecdan) I a0ab91ff-
6de8-4961-b9cf-646b8037e87a P default T a V 1562330471 F 674 M
1599389319)
 
whereas the new file contains

user.ttt.#addressbooks.MMM	%(A %(ttt lrswipkxtecdan) P default T a
M 1601752550)

The fields:
I a0ab91ff-6de8-4961-b9cf-646b8037e87a 
V 1562330471
F 674

are missing from the new file.

Shall I be worried?

Greetings
  ?????


From rjbs at fastmailteam.com  Mon Oct 12 09:44:06 2020
From: rjbs at fastmailteam.com (Ricardo Signes)
Date: Mon, 12 Oct 2020 09:44:06 -0400
Subject: minutes, cyrus call, 2020-10-12
Message-ID: <ebb5467d-5d9c-4b43-bcc6-e451737693d2@dogfood.fastmail.com>

Largely unedited, here are the minutes from this morning's dev call:

 * murch 
   * discussion of expanded header search semantics (prefix/suffix/substr match)
     * it's unclear whether we need or want this, given the yet unknown cost v. complexity v. benefit ratios
     * This isn't Xapian related at all, we'll be processing fields from the cyrus.cache.
   * Discuss replication of JMAPSieve script ids 
     * Given our current replication protocol, SieveScript object ids will not replicate. *Does this matter?* (This is: replicas will have the same script with a different id.)
     * First: *does* it get replicated? We need to double-check that it doesn't, but consensus seems to be that it isn't replicated.
     * We should fix it, if it isn't replicated, but we don't really *care* as far as Fastmail operation is concerned. It won't affect user-visible behavior.
   * Discuss expectations of cyr_ls 
     * always use unixhierarchysep?
     * let's use admin namespace
     * (we have other tools that use internal ns, it's been a pain, and we're fixing it as we go)
   * fixed cross-domain searching (new term added to Xapian)
   * still waiting on CR for 8bit characters in C-D
 * ellie 
   * PR #3166 contains code from OpenBSD, NetBSD, DragonflyBSD and FreeBSD's top(1) utilities (for turning process state flags into human-readable strings). Can we accept this? ? We think so.
   * we should be expecting lots of small MRs for xsyslog conversions
 * rjbs 
   * we're working on cyrus ML conversion; waiting on Dave
 * brong 
   * locking! conversationsdb has locking problems, annotations are "a whole locking nightmare"
   * there's a lock inversion between JMAP and other calls in the locking of convdb vs. mailboxes
   * we can change how convdb locking works internally so you can take/release a lock without closing the entire db, add a user lock, then we're done!(?)
   * ?but it's a bunch of work, and that will be Bron's next project for Cyrus
   * but this week is CalConnect week

-- 
Ricardo Signes (rjbs)
CTO, Fastmail
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/cyrus-devel/attachments/20201012/4ba475fe/attachment.html>

From brong at fastmailteam.com  Mon Oct 12 11:20:47 2020
From: brong at fastmailteam.com (Bron Gondwana)
Date: Tue, 13 Oct 2020 02:20:47 +1100
Subject: minutes, cyrus call, 2020-10-12
In-Reply-To: <ebb5467d-5d9c-4b43-bcc6-e451737693d2@dogfood.fastmail.com>
References: <ebb5467d-5d9c-4b43-bcc6-e451737693d2@dogfood.fastmail.com>
Message-ID: <de919400-7eda-4b91-9109-3178339671b7@dogfood.fastmail.com>

On Tue, Oct 13, 2020, at 00:44, Ricardo Signes wrote:
>  * brong 
>    * locking! conversationsdb has locking problems, annotations are "a whole locking nightmare"
>    * there's a lock inversion between JMAP and other calls in the locking of convdb vs. mailboxes
>    * we can change how convdb locking works internally so you can take/release a lock without closing the entire db, add a user lock, then we're done!(?)
>    * ?but it's a bunch of work, and that will be Bron's next project for Cyrus

Some more about this...

So... I just reverted my attempt from https://github.com/cyrusimap/cyrus-imapd/pull/3232 to fix the deadlock problems.

Here's the root problem:

mailbox.c:mailbox_open_advanced() takes a *shared NAMELOCK on the mailbox name*, then opens the index and first locks the user.conversations file before locking the indexes.

So far so good.

BUT: JMAP (and other things which need a long lock) takes a *(shared/exclusive) lock on the user.conversations file BEFORE it opens mailboxes*.

This leads to a lock inversion.

My attempted fix was to release the namelock while locking the conversations - which reverted everything to the order that the JMAP code does.  This caused errors because the mailbox could be deleted during that time that the lock is missing, and that broke invarients.

There are a couple of options:

*1) always lock conversations first*

Refactor mailbox handling to always lock the conversations.db BEFORE it tries to lock the mailbox name.

This means that we probably need to find a bunch of places in the code where we're doing copy and rename actions and lock one or both of the conversations databases early as well, so the lock is held across the entire action.

*2) create a new user-level lock for JMAP*

This would be useful for other things and would stop our current abuse of the conversations.db as a user-level lock.  Instead we could open and close it along with mailboxes and commit more frequently, which would give better transaction safety.  It does mean yet another lock, which would need to be managed in both mailbox_open_* and anywhere which is currently using the conversations database as a proxy lock.

...

There are pros and cons of both.  I'm leaning a bit towards the second one because it allows us to commit the conversations DB along with the mailbox in each case.

...

Here's the log information that led to knowing about this, using https://github.com/cyrusimap/cyrus-imapd/pull/3179 in a build:

2020-10-07T07:30:37.415533-04:00 fastmail1 slotf1t01/sync_client[4156099]: LOCKORDER: 0=<E:6:/slot1/conf/lock/q/*S*10^37^129^209*brong at fastmaildev^com.lock>* 1=<S:8:/slot1/conf/lock/domain/f/**fastmaildev.com/b/user/brong.lock**> 2=<S:10:/slot1/conf/domain/f/**fastmaildev.com/user/b/brong.conversations**>*
?
2020-10-07T07:30:37.078907-04:00 fastmail1 slotf1t01/httpjmap[4156617]: LOCKORDER: *0=<S:11:/slot1/conf/domain/f/**fastmaildev.com/user/b/brong.conversations**> 1=<S:13:/slot1/conf/lock/domain/f/**fastmaildev.com/b/user/brong.lock**>*

...

So the plan appears to be:

*1) do a new user-level lock* that is held when any mailbox is opened as well as held for the entire command by JMAP.  This will probably be a reuse of the existing user_namespacelock which is used for rename safety.

*2) make conversations lock inside mailboxes* - for efficiency this will be done by making it not locked for the entire time it's open, so making it more like every other cyrusdb which can be held open for efficiency and only locked when used.  Conversations will be lazyloaded inside mailbox if not already open, and it will just use refcounting like every other DB.

*3) BONUS: move per-mailbox annotations to only be a mailbox internal thing too* - and just like cache, lazyload it when needed.

I'm going to get started on that when I get a chance.

Bron.

--
  Bron Gondwana, CEO, Fastmail Pty Ltd
  brong at fastmailteam.com

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/cyrus-devel/attachments/20201013/c516aff1/attachment.html>

From dave64 at andrew.cmu.edu  Thu Oct 15 12:42:26 2020
From: dave64 at andrew.cmu.edu (Dave McMurtrie)
Date: Thu, 15 Oct 2020 12:42:26 -0400 (EDT)
Subject: Moving Cyrus mailing lists to Topicbox
Message-ID: <6ec7e4f-78be-dbc8-5463-b4b589be9dc3@andrew.cmu.edu>

Hi,

I'm working with Fastmail to transition the Cyrus mailing lists from 
lists.andrew.cmu.edu over to cyrus.topicbox.com.

Our plan is to transition the active mailing lists as follows:

cyrus-announce at lists.andrew.cmu.edu -> announce at cyrus.topicbox.com
cyrus-devel at lists.andrew.cmu.edu -> devel at cyrus.topicbox.com
info-cyrus at lists.andrew.cmu.edu -> info at cyrus.topicbox.com
cyrus-sasl at lists.andrew.cmu.edu -> sasl at cyrus.topicbox.com

We have already copied over subscriber information and list archives to 
Topicbox for these lists.  You should begin using Topicbox immediately. 
If you have questions about using Topicbox, please see:

http://www.topicbox.com/helpspot

I will unsubscribe everyone from the lists.andrew.cmu.edu mailing lists 
later today.

I will be deleting the following mailing lists which are no longer in use:

cyrus-bugzilla at lists.andrew.cmu.edu
cyrus-cvs at lists.andrew.cmu.edu
cyrus-project at lists.andrew.cmu.edu

Thanks!

Dave