From dilyan.palauzov at aegee.org Wed Jul 17 04:49:15 2019 From: dilyan.palauzov at aegee.org (=?UTF-8?Q?=D0=94=D0=B8=D0=BB=D1=8F=D0=BD_?= =?UTF-8?Q?=D0=9F=D0=B0=D0=BB=D0=B0=D1=83=D0=B7=D0=BE=D0=B2?=) Date: Wed, 17 Jul 2019 08:49:15 +0000 Subject: squatter -F increases the index size In-Reply-To: <0a8d05c3-0a1b-491f-8d2e-3982272807b0@www.fastmail.com> References: <20190531183343.Horde.NqDuNlWbxJXSqZCJ2rWmPGG@webmail.aegee.org> <0a8d05c3-0a1b-491f-8d2e-3982272807b0@www.fastmail.com> Message-ID: <72d8ad6c1b3263f3df8821210efe67b7f33eae46.camel@aegee.org> Hello, for more than a month it is known, that on the stable branch, ?squatter -F? does increase the index size, but not on the instable branch. Will the fix be backported, or shall ?sqatter -F? be deleted from the documentation on the stable branch? Regards ????? On Tue, 2019-06-04 at 01:53 +1000, Bron Gondwana wrote: > On Sat, Jun 1, 2019, at 04:34, Dilyan Palauzov wrote: > > Hello, > > > > I gave squatter -F a try. > > > > Before I run it for a user tier T1 was not compacted and allocated 3,4 > > MB (mega), T2 was compacted and contained 3.7GB (giga). After > > removing the records of the deteled messages, say running squatter -F > > T2 was 5.7GB and squatter printed ?filtering? instead of ?compacting?. > > Then I run again ?squatter -t T1,T2 -z T2? without -F, without -X > > and squatter reindexed all messages, to create a 3.0 GB index. > > > > I expected, that using -F the resulting database will be compacted and > > on the second call there will be no reindexing. > > I discovered some bad bugs in -F recently, so I suspect that's why. They should be fixed on master now. > > > When does squatter decide on its own to reindex? > > When the DB version is too old (which is one of the -F bugs - it wasn't setting the DB version, so it assumed the data was all version zero!) > > > What do G records in conversations.db contain? > > G records contain a mapping from GUID to folder number (offset into the $FOLDER_NAMES key) and UID and optionally IMAP part number as the key - mapping to values which contain some keywords and modseq from the original record as well. > > > My reading is that the way to create a Xapian index of an indexed > > mailbox, is that first squatter has to be run in INDEX mode and then > > in COMPACT mode. In particular it is not possible to create in one > > step a compacted database. > > No, it's not - due to the way to compact API works. At least, I haven't figured out how. > > > Does squatter -R -S sleep after each mailbox or after each message indexed? > > It sleeps after each mailbox. > > > When compacting, squatter deals just with messages and on search or > > reindex the conversations.db is used to map the messages to mailboxes. > > How does squatter -S sleep after each mailbox during compacting, if > > it knows nothing about mailboxes? > > -S is not used when compacting. > > > What does mean a tier name in a xapianactive file without a number? > > that shouldn't happen. It will be parsed as the same as tier:0 I believe. > > > What are XAPIAN_DBW_CONVINDEXED and _XAPINDEXED? > > Two different ways to know if a document is indexed. CONVINDEXED uses the conversations DB to look up mailbox and uid and then the cyrus.indexed.db databases to see if the message has already been seen. > > XAPINDEXED uses the metadata inside the Xapian databases to know if a particular message has been indexed based on the cyrusid.*G* metadata values which are identical to the GUIDs themselves. > > > What does the file sync/squatter? > > It's a sync/$channel directory which squatter watches on. This is a method for providing a queue of mailboxes to look at based on the APPEND sync_log statements. > > > squatter can print ?Xapian: truncating text from message mailbox > > user.... uid 7309?. When are messages truncated for the purposes of > > indexing? > > When they are too long! The comment in the source code says this: > > /* Maximum size of a query, determined empirically, is a little bit > * under 8MB. That seems like more than enough, so let's limit the > * total amount of parts text to 4 MB. */ > #define MAX_PARTS_SIZE (4*1024*1024) > > This is a holdover from when Greg was working on it. We could switch this to be a configurable option. > > > Do I understand correctly, that for a Xapianactive file with "A B C D > > E", to remove C one has to call "squatter -t C,D -z D". But A cannot > > be removed, if it the defaultsearchtier. Is the defaultsearchtier > > always included in the xapianactive file, if the tier is missing, > > whenever the file is modified (and the only way to modify it is to > > call squatter in COMPACT mode)? > > When you do any compact, if it includes the first item (the writable database) then a new writable database will be created on the default tier. So if you try to compact the default tier away, a new default tier item will be created. > > Bron. > > -- > Bron Gondwana, CEO, FastMail Pty Ltd > brong at fastmailteam.com > > From quanah at symas.com Thu Jul 25 14:16:10 2019 From: quanah at symas.com (Quanah Gibson-Mount) Date: Thu, 25 Jul 2019 11:16:10 -0700 Subject: Cyrus-SASL development Message-ID: It appears that once again development on cyrus-sasl has come to a standstill. I understand that Fastmail's priority is cyrus-imapd, but there are many software projects that use and depend on cyrus-sasl. At this point, can it be clarified if: a) People should just assume that cyrus-sasl is dead and look at transitioning to other libraries (such as libgsasl) or b) Fastmail would be willing to hand over control of cyrus-sasl to those parties interested in actually maintaining cyrus-sasl or c) Fastmail is actually going to start actively responding to and dealing with the growing number of open issues (many with provided patch fixes) in a timely fashion Thanks, Quanah -- Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: From murch at fastmail.com Thu Jul 25 14:27:38 2019 From: murch at fastmail.com (Ken Murchison) Date: Thu, 25 Jul 2019 14:27:38 -0400 Subject: Cyrus-SASL development In-Reply-To: References: Message-ID: <6bac139f-b0f7-7ea3-017c-91cc2d59a80d@fastmail.com> I'm away at a conference, but I will look at whatever urgent issues/PRs are pending next week. On 7/25/19 2:16 PM, Quanah Gibson-Mount wrote: > It appears that once again development on cyrus-sasl has come to a > standstill.? I understand that Fastmail's priority is cyrus-imapd, but > there are many software projects that use and depend on cyrus-sasl.? > At this point, can it be clarified if: > > a) People should just assume that cyrus-sasl is dead and look at > transitioning to other libraries (such as libgsasl) > > or > > b) Fastmail would be willing to hand over control of cyrus-sasl to > those parties interested in actually maintaining cyrus-sasl > > or > > c) Fastmail is actually going to start actively responding to and > dealing with the growing number of open issues (many with provided > patch fixes) in a timely fashion > > Thanks, > Quanah > > -- > > Quanah Gibson-Mount > Product Architect > Symas Corporation > Packaged, certified, and supported LDAP solutions powered by OpenLDAP: > > -- Ken Murchison Cyrus Development Team Fastmail US LLC -------------- next part -------------- A non-text attachment was scrubbed... Name: murch.vcf Type: text/x-vcard Size: 4 bytes Desc: not available URL: From brong at fastmailteam.com Mon Jul 29 07:36:15 2019 From: brong at fastmailteam.com (Bron Gondwana) Date: Mon, 29 Jul 2019 21:36:15 +1000 Subject: Notes 29 July Message-ID: <9527a9a4-e056-4c3f-86fd-dde8c3db5bd2@www.fastmail.com> Present: ellie, Ken, Bron ellie: * 3.0.11 release - hopefully tomorrow * 3.1 checkpoint release? - Let?s do one now-ish! - Then we can do another one after the delayed send changes land. Ken: * Mailbox by UUID status - Needs to be rebased. - Then - need to do some testing! Stand up a new one, replicate over. * Going to keep working on Delayed Send, including GUID from mailbox record. Bron: * Cyruslibs-v26+: - Need to fix tzdata issue and fixed libicu. * Calendar Sharing: - ?Shared via JMAP" is clearly bad UI - Neil and Ken have discussed notifications Inbox and how to deal with these. - Have created a task to work on design for the various workflows. * Handling Calendar scheduling and X-Schedule-User-Address on updates - A user had issues with a complex mailflow causing participantId to be null because the schedule address didn't match. - For now we?ll fix this in the Fastmail middleware for our weird usecase. - Need to build a more general way to determine "IsYou" inside Cyrus. -- Bron Gondwana, CEO, Fastmail Pty Ltd brong at fastmailteam.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From me at anatoli.ws Mon Jul 29 09:09:08 2019 From: me at anatoli.ws (Anatoli) Date: Mon, 29 Jul 2019 10:09:08 -0300 Subject: Notes 29 July In-Reply-To: <9527a9a4-e056-4c3f-86fd-dde8c3db5bd2@www.fastmail.com> References: <9527a9a4-e056-4c3f-86fd-dde8c3db5bd2@www.fastmail.com> Message-ID: Hi All, Ken, what's the /Delayed Send/ feature you mentioned? Is it something like Send Later extension in TB but on the server? How would the clients use it? Bron, do you still pursue the goal to close most of the GitHub issues? Do you have any roadmap for which issues to be closed in the near future? There are a lot of issues with both 3.0, 3.1 and 3.2 tags (some even have 2.5 tag), it's not clear which are scheduled to be closed for which release. Regards, Anatoli *From:* Bron Gondwana *Sent:* Monday, July 29, 2019 08:36 *To:* Cyrus Devel *Subject:* Notes 29 July Present: ellie, Ken, Bron ellie: * 3.0.11 release - hopefully tomorrow * 3.1 checkpoint release? ? - Let?s do one now-ish! ? - Then we can do another one after the delayed send changes land. Ken: * Mailbox by UUID status ? - Needs to be rebased. ? - Then - need to do some testing! ?Stand up a new one, replicate over. * Going to keep working on Delayed Send, including GUID from mailbox record. Bron: * Cyruslibs-v26+: ? - Need to fix tzdata issue and fixed libicu. * Calendar Sharing: ? - ?Shared via JMAP" is clearly bad UI ? - Neil and Ken have discussed notifications Inbox and how to deal with these. ? - Have created a task to work on design for the various workflows. * Handling Calendar scheduling and X-Schedule-User-Address on updates ? - A user had issues with a complex mailflow causing participantId to be null because the schedule address didn't match. ? - For now we?ll fix this in the Fastmail middleware for our weird usecase. ? - Need to build a more general way to determine "IsYou" inside Cyrus. -- ? Bron Gondwana, CEO, Fastmail Pty Ltd ? brong at fastmailteam.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From murch at fastmail.com Mon Jul 29 09:47:19 2019 From: murch at fastmail.com (Ken Murchison) Date: Mon, 29 Jul 2019 09:47:19 -0400 Subject: Notes 29 July In-Reply-To: References: <9527a9a4-e056-4c3f-86fd-dde8c3db5bd2@www.fastmail.com> Message-ID: <8e36fbd4-3cd1-16fe-f85c-61d5985dc674@fastmail.com> On 7/29/19 9:09 AM, Anatoli via Cyrus-devel wrote: > Hi All, > > Ken, what's the /Delayed Send/ feature you mentioned? Is it something > like Send Later extension in TB but on the server? How would the > clients use it? Its a new JMAP feature where when a client creates an email submission, it can also specify a time in the future when the message will actually be sent.? Until that point, it remains in a special mailbox in Cyrus and can be canceled prior to sending. > > Bron, do you still pursue the goal to close most of the GitHub issues? > Do you have any roadmap for which issues to be closed in the near > future? There are a lot of issues with both 3.0, 3.1 and 3.2 tags > (some even have 2.5 tag), it's not clear which are scheduled to be > closed for which release. > > Regards, > Anatoli > > *From:* Bron Gondwana > *Sent:* Monday, July 29, 2019 08:36 > *To:* Cyrus Devel > *Subject:* Notes 29 July > > Present: ellie, Ken, Bron > > ellie: > * 3.0.11 release - hopefully tomorrow > * 3.1 checkpoint release? > ? - Let?s do one now-ish! > ? - Then we can do another one after the delayed send changes land. > > Ken: > * Mailbox by UUID status > ? - Needs to be rebased. > ? - Then - need to do some testing! ?Stand up a new one, replicate over. > * Going to keep working on Delayed Send, including GUID from mailbox > record. > > Bron: > * Cyruslibs-v26+: > ? - Need to fix tzdata issue and fixed libicu. > * Calendar Sharing: > ? - ?Shared via JMAP" is clearly bad UI > ? - Neil and Ken have discussed notifications Inbox and how to deal > with these. > ? - Have created a task to work on design for the various workflows. > * Handling Calendar scheduling and X-Schedule-User-Address on updates > ? - A user had issues with a complex mailflow causing participantId to > be null because the schedule address didn't match. > ? - For now we?ll fix this in the Fastmail middleware for our weird > usecase. > ? - Need to build a more general way to determine "IsYou" inside Cyrus. > > -- > ? Bron Gondwana, CEO, Fastmail Pty Ltd > brong at fastmailteam.com > > > -- Ken Murchison Cyrus Development Team Fastmail US LLC -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: murch.vcf Type: text/x-vcard Size: 4 bytes Desc: not available URL: From quanah at symas.com Mon Jul 29 10:42:45 2019 From: quanah at symas.com (Quanah Gibson-Mount) Date: Mon, 29 Jul 2019 07:42:45 -0700 Subject: Cyrus-SASL development In-Reply-To: <6bac139f-b0f7-7ea3-017c-91cc2d59a80d@fastmail.com> References: <6bac139f-b0f7-7ea3-017c-91cc2d59a80d@fastmail.com> Message-ID: --On Thursday, July 25, 2019 3:27 PM -0400 Ken Murchison wrote: > I'm away at a conference, but I will look at whatever urgent issues/PRs > are pending next week. Thanks Ken, that's appreciated. However, I'd still like an overall answer to my questions for the long term, so cyrus-sasl is not just an afterthought. ;) Regards, Quanah -- Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: From ellie at fastmail.com Mon Jul 29 19:28:07 2019 From: ellie at fastmail.com (ellie timoney) Date: Tue, 30 Jul 2019 09:28:07 +1000 Subject: Notes 29 July In-Reply-To: References: <9527a9a4-e056-4c3f-86fd-dde8c3db5bd2@www.fastmail.com> Message-ID: <646f9752-5d30-4995-8093-2f8aca98266e@www.fastmail.com> On Mon, Jul 29, 2019, at 11:10 PM, Anatoli via Cyrus-devel wrote: > There are a lot of issues with both 3.0, 3.1 and 3.2 tags (some even have 2.5 tag), it's not clear which are scheduled to be closed for which release. > The tags for existing versions (i.e. 2.4, 2.5, 3.0, 3.1) indicate which versions are known (or suspected) to be affected by those issues, but they don't indicate any particular release schedule for fixes. The 3.2 tag currently indicates stuff we definitely need to fix/implement on master before forking the 3.2 stable branch, but after this occurs the 3.2 tag will continue like the rest. The main person working through the github issues tracker is myself, but I've been working reduced hours for medical reasons for quite a while so there's some weeds growing. Sorry about that! -------------- next part -------------- An HTML attachment was scrubbed... URL: From me at anatoli.ws Mon Jul 29 22:33:25 2019 From: me at anatoli.ws (Anatoli) Date: Mon, 29 Jul 2019 23:33:25 -0300 Subject: Notes 29 July In-Reply-To: <646f9752-5d30-4995-8093-2f8aca98266e@www.fastmail.com> References: <9527a9a4-e056-4c3f-86fd-dde8c3db5bd2@www.fastmail.com> <646f9752-5d30-4995-8093-2f8aca98266e@www.fastmail.com> Message-ID: <1759b542-c9f9-5205-6724-fc29abe2e8cf@anatoli.ws> Ellie, Ken, thanks for the clarifications! Ellie, I hope you recover soon! Best regards, Anatoli *From:* Ellie Timoney *Sent:* Monday, July 29, 2019 20:28 *To:* Cyrus Devel *Subject:* Re: Notes 29 July On Mon, Jul 29, 2019, at 11:10 PM, Anatoli via Cyrus-devel wrote: > There are a lot of issues with both 3.0, 3.1 and 3.2 tags (some even > have 2.5 tag), it's not clear which are scheduled to be closed for > which release. > The tags for existing versions (i.e. 2.4, 2.5, 3.0, 3.1) indicate which versions are known (or suspected) to be affected by those issues, but they don't indicate any particular release schedule for fixes. The 3.2 tag currently indicates stuff we definitely need to fix/implement on master before forking the 3.2 stable branch, but after this occurs the 3.2 tag will continue like the rest. The main person working through the github issues tracker is myself, but I've been working reduced hours for medical reasons for quite a while so there's some weeds growing. ?Sorry about that! -------------- next part -------------- An HTML attachment was scrubbed... URL: From dilyan.palauzov at aegee.org Tue Jul 30 09:01:10 2019 From: dilyan.palauzov at aegee.org (=?UTF-8?Q?=D0=94=D0=B8=D0=BB=D1=8F=D0=BD_?= =?UTF-8?Q?=D0=9F=D0=B0=D0=BB=D0=B0=D1=83=D0=B7=D0=BE=D0=B2?=) Date: Tue, 30 Jul 2019 13:01:10 +0000 Subject: Notes 29 July In-Reply-To: <646f9752-5d30-4995-8093-2f8aca98266e@www.fastmail.com> References: <9527a9a4-e056-4c3f-86fd-dde8c3db5bd2@www.fastmail.com> <646f9752-5d30-4995-8093-2f8aca98266e@www.fastmail.com> Message-ID: <9bcd9d673072749c6b6d1ea7bf0a7d523a94dc86.camel@aegee.org> Hello Ellie, your personal problems are mentioned for quite some time now. I do not know the details, but I personally wish you to recover soon permanently. In any case, this thread is not about you. The accent is about finding a way where contributions are considered in a timely manner. Currently it is absolutely unclear and unpredictable, whether submitted fixes, being clarifying the documentation, adding features or fixing bugs will be consideted for intergration, so that other users can benefit from them. Whatever conditions are fulfilled, it is still unpredictable. This uncertainty questions whether it makes sense to contribute upstream to make cyrus imap better. As an example, uploading a meeting with ATTENDEE;CN="A, B":mailto:b.a at examle.orggenerates an email/iMIP invitation with: To: A, B which is converted by the MTA/MSA in To: A at thisdomain, B which is wrong. Correct is: To: "A, B" https://github.com/cyrusimap/cyrus-imapd/pull/2693 contains a fix from April 2019. The question is not what happened with this particular example, but how to avoid such stalls in the future. Besides if a long time passes between a notice on github and a reaction, the one who wrote the notice might not remember anymore the details and this complicates the situation more. A further challenge, that needs to be approached, on which contributors have no impact, is that most users do not run the ?master? version, but a stable one and in turn the provided patches are towards the stable version. In some parts, the master and stable versions have diverged significantly and continue to diverge further. Regards ????? On Tue, 2019-07-30 at 09:28 +1000, ellie timoney wrote: > On Mon, Jul 29, 2019, at 11:10 PM, Anatoli via Cyrus-devel wrote: > > There are a lot of issues with both 3.0, 3.1 and 3.2 tags (some even have 2.5 tag), it's not clear which are scheduled to be closed for which release. > > > > The tags for existing versions (i.e. 2.4, 2.5, 3.0, 3.1) indicate which versions are known (or suspected) to be affected by those issues, but they don't indicate any particular release schedule for fixes. > > The 3.2 tag currently indicates stuff we definitely need to fix/implement on master before forking the 3.2 stable branch, but after this occurs the 3.2 tag will continue like the rest. > > The main person working through the github issues tracker is myself, but I've been working reduced hours for medical reasons for quite a while so there's some weeds growing. Sorry about that!