From me at anatoli.ws Thu Jul 2 23:50:30 2020 From: me at anatoli.ws (Anatoli) Date: Fri, 3 Jul 2020 00:50:30 -0300 Subject: new features with no documentation In-Reply-To: References: <04ee2b89-7c24-2947-b4f9-8b094f2a9e3f@anatoli.ws> <88cce492-632e-439a-ac7b-f68ef9c3381c@www.fastmail.com> <5427d59b-503b-5ba6-76e9-ba66f853262c@anatoli.ws> <20132e86-e439-8821-347e-b1adfb56ee24@anatoli.ws> <3e1f0f55-4fc9-b7ae-069f-7b3f453dfda1@anatoli.ws> <8ace84e8-c7dd-4618-a527-11579a3bdd08@www.fastmail.com> Message-ID: Ellie, Ken, All, I have prepared a PR [1] with some changes to the compiling page. Your feedback is highly appreciated. The changes are as follows: 1. Completed the descriptions of a number of libraries with the explanations of when they are needed and what happens if they are not included (based on the information from this thread). For this I also added to the tables a new column "Required". 2. Completed the libs package names on Debian & RH for all libs (except for the perl packages). 3. Reordered some libs, in particular brotli, nghttp2, shapelib, wslay, etc. (they were in Other, moved them to DAV/JMAP/httpd and renamed a bit that section). 4. Added links to the libs official websites where they were missing. 5. Other minor fixes & cleanup. 6. Removed 2 libs that IMO are not used anymore: 1. net-snmp: in my understanding, it was used for the SNMP code in master which is pending removal (BTW, is it possible to close the issues/1765 [2]? It looks like everything is ready); so no need to include obsolete features in the description; 2. tcp_wrappers: not 100% sure, but it looks like it's not used anymore. Ellie, with respect to the point 6, please take this into account if you would like to backport the changes to the previous versions. I guess 3.x is ok, but not sure about 2.x. For those who have RedHat instances, please confirm the libs package names are correct. I have a Debian one so those should all be ok, but for RH I had to look for the package names on the web. On the other hand, Ellie, I see that the cld2 detection code is still in configure.ac. Shouldn't it be removed based on what we've discussed here (i.e. cld2 is not used anywhere and was included by mistake)? And related, recently there was a thread ("cyrus-imap buildin: sse extention") in info-cyrus@ about SSE4.2 compilation for CRC32c algo. I've commented (but got no reply) that maybe the detection and informational code should also be removed from configure.ac as it is not used in any way at this moment and there are no plans to make it work, so to remove the confusion and simplify the code. Regards, Anatoli [1]: https://github.com/cyrusimap/cyrus-imapd/pull/3095 [2]: https://github.com/cyrusimap/cyrus-imapd/issues/1765 On 19/6/20 02:09, Anatoli wrote: > Ellie, > > Thanks for the explanation! > > I'll try to prepare a PR for the compiling page these days. > > Regards, > Anatoli > > On 19/6/20 00:29, ellie timoney wrote: >> I think transfig is the package that provides fig2dev? I think that's an artifact from the old days, and is only used for generation of a couple of png files in the legacy documentation (doc/legacy/murder.png and doc/legacy/netnews.png). I think it's only needed when configure is run with --enable-maintainer-mode (i.e. if you're building a release or package, rather than just building the software). Those files are pre-built in the release tarballs, so additionally you would only need this when building from git. It would be nice to one day finish merging this legacy documentation into the current documentation, cause then we can get rid of it: https://github.com/cyrusimap/cyrus-imapd/issues/1769 >> >> libsrs2 looks like it's used for implementing Sender Rewriting Scheme functionality for messages forwarded by sieve script. Without it, messages forwarded by sieve script will not have this functionality (and, I suppose, might have difficulty delivering to servers that insist on it). >> >> The section of configure.ac that looks for libsrs2 is missing an AC_MSG_RESULT call to match the AC_MSG_CHECKING one. It probably also wants an AC_NOTICE when it's not found, explaining the consequences of it being left out, like we have for the various http dependencies. I'll fix this up in a moment. >> >> shapelib is already explained by configure. For example, I don't have it installed myself, so I see: >> >>> checking for SHAPELIB... no >>> configure: tzdist will not have geolocation support. Consider installing shapelib >> >> If you don't run a tzdist service, you don't need it. If you do, you probably also know whether geolocation support is useful; I don't. >> >> I've already updated configure.ac to provide a similar notice when chardet is not found (https://github.com/cyrusimap/cyrus-imapd/commit/30fef51485); it'll be in 3.2.2. :) >> >> If you want to have a go at making the compiling page a bit clearer, I'd be happy to review/merge a pull request. Please submit it against the master, and once we're happy with it, I'll get it cherry-picked appropriately to the other branches. >> >> Cheers, >> >> ellie >> >> On Fri, Jun 19, 2020, at 12:50 PM, Anatoli wrote: >>> Ellie (or anyone else, Ken?), >>> >>> Could you please let us know what's the purpose of transfig, libsrs2 and >>> shapelib libs in cyrus-imapd, and if they are required for any >>> functionality (and if not (i.e. they are optional), what's their >>> benefit)? >>> >>> Also, Ellie, would it be useful to add to configure checks with warnings >>> for these libs (as well as for chardet) to inform those users that build >>> cyrus-imapd from sources that these libs are actually needed if certain >>> features are enabled (like Cal/CardDAV & JMAP)? I could prepare a patch. >>> >>> And I'd like to add these clarifications to the compiling page and >>> reformat a bit the tables. If you consider it appropriate, I could >>> prepare a patch too. >>> >>> Regards, >>> Anatoli >>> >>> On 5/6/20 05:45, Anatoli wrote: >>>>> Thanks for the explanation. What happens is that at least Cal/CardDAV >>>>> compile and work well without libchardet (at least under normal >>>>> circumstances) and, as it was not marked as required, I thought it's an >>>>> optional lib and here is the question of why to use it. >>>> >>>> Actually, the same happens with transfig, libsrs2 and shapelib. >>>> >>>> The page mentiones these libs, but "Required for `make check`" (not sure >>>> what that means in relation to the lib being reqiured for cyrus-imapd as >>>> such) for all of them is "no". >>>> >>>> What would help is a column "required" yes/no and another column "usage >>>> in cyrus-imapd" explaining for what exactly it's used. Say if it's for >>>> tzdata processing in the TZDist module, then I probably won't build with >>>> it as I don't see much use for it (or maybe package it as a flavor). >>>> >>>> But if it's something essential for a broader functionality like CalDAV >>>> itself, then definitely it would be included. But at this moment the >>>> Compiling page doesn't provide these details and here we are. >>>> >>>> >>>> On 5/6/20 05:30, Anatoli wrote: >>>>> Ellie, >>>>> >>>>>> libchardet is already listed on the compiling page as being used by >>>>>> the CalDAV, CardDAV, and/or JMAP features. You would decide to >>>>>> install libchardet based on whether you need these features: if you >>>>>> don't enable these features, you don't need it; if you do, you >>>>>> probably do. >>>>>> >>>>>> If you don't have libchardet, Cyrus will not do character-set >>>>>> detection. If some piece of data has no character set coming in, it >>>>>> will have no character set. I don't know why you would decide to not >>>>>> use this, and it may simply become a mandatory requirement for these >>>>>> features in 3.4. It might have been mandatory in 3.2, if this had >>>>>> been brought to our attention during the nearly 3 month beta window; >>>>>> but now that it's in a stable release, it will remain as it is. >>>>> >>>>> Thanks for the explanation. What happens is that at least Cal/CardDAV >>>>> compile and work well without libchardet (at least under normal >>>>> circumstances) and, as it was not marked as required, I thought it's an >>>>> optional lib and here is the question of why to use it. >>>>> >>>>> I guess what you could do for 3.2 is to include a warning in configure >>>>> explaining the issue, so the maintainers see it and take appropriate >>>>> action. I suppose there are not that many installations with 3.2 yet so >>>>> if the warning is included with 3.2.2 it would be an appropriate time. >>>>> >>>>> Regards, >>>>> Anatoli >>>>> >>>>> On 4/6/20 21:58, ellie timoney wrote: >>>>>> On Thu, Jun 4, 2020, at 12:52 PM, Anatoli wrote: >>>>>>>> chardet is used by the JMAP module of httpd to detect the character >>>>>>> set of untagged 8-bit headers. >>>>>>> >>>>>>> Does that mean that these libs are required? If not, what would happen >>>>>>> if not included? How Cyrus would detect the charset? Or put in other >>>>>>> words, if they are not required, what's the benefit to use them? >>>>>>> >>>>>>> On the other hand, may I suggest to please add these details to the >>>>>>> Compiling [1] page? There's already a lot of useful information, but >>>>>>> some of the libs lack any details on what's their purpose in Cyrus and >>>>>>> why they are beneficial/required (and if they are not used yet, maybe >>>>>>> they shouldn't be mentioned there at all?). This could help porters and >>>>>>> maintainers to make more educated decisions on how to package Cyrus. >>>>>> >>>>>> libchardet is already listed on the compiling page as being used by the CalDAV, CardDAV, and/or JMAP features. You would decide to install libchardet based on whether you need these features: if you don't enable these features, you don't need it; if you do, you probably do. >>>>>> >>>>>> If you don't have libchardet, Cyrus will not do character-set detection. If some piece of data has no character set coming in, it will have no character set. I don't know why you would decide to not use this, and it may simply become a mandatory requirement for these features in 3.4. It might have been mandatory in 3.2, if this had been brought to our attention during the nearly 3 month beta window; but now that it's in a stable release, it will remain as it is. >>>>>> >>>>>> cld2 was used by one (or maybe both) of the experimental search features that were accidentally included in 3.2.0, and removed in 3.2.1 (see the release notes). Looks like we forgot to remove the configure checks for the library when removing the feature(s) that used it. You don't need it, and if you have it, it won't be used anyway. >>>>>> >>>>>> Cheers, >>>>>> >>>>>> ellie >>>>>> >>> From rjbs at fastmailteam.com Fri Jul 10 17:50:07 2020 From: rjbs at fastmailteam.com (Ricardo Signes) Date: Fri, 10 Jul 2020 17:50:07 -0400 Subject: minutes, cyrus dev call, 2020-06 Message-ID: <44e3cda6-48a9-4dfa-bf10-e87ccdeeacd6@dogfood.fastmail.com> We have been keeping minutes, but not getting them out in a timely fashion. I've adjusted my calendar to make this happen more reliably. (The basic deal is that these meetings happen at 7:30 a.m. my time, and as soon as they're over I go to get some caffeine, and then they've fallen down my priority list.) I realize these are less useful when they're sent late, but I'll learn my lesson better by sending them anyway. *2020-06-01* * *Present*: Bron, Robert, ellie, Ken, Rik, Neil * [ discussion about reindexing Fastmail's large install ] * Bron * been doing some small changes * wants to get things shepherded to production * soon: sync replication code!! * need to give sync client a state struct to pass around between funcs * ellie * v3.2.1 on Friday! * experimental Debian build passing all the cunit tests on all the Debian platforms * it was the tests more than the code under testing being a problem * v3.3.0 dev tag should happen soon! * some backports from master are pending (probably) * Cass failing on v3.2 failing because of mismatch between expectations/fixes on v3.2 * rjbs: can we put Cassandane into the cyrus repo? * yes but it sounds like for complex reasons we might want to; rjbs will follow up * also what will we do with the uuid mailboxes test code? * neilj * rules question: fromContactGroupId, convert errors to false ? has just hit production * status of backup/restore bugs? * Ken: as far as I know, all tasks are done * What about the ?contacts don?t get restored back to groups? * for now, we?ll say this is expect, and count people who notice/complain * will ask Matthew to do a review of code and/or tests? * murch * uuid mailboxes all rebased by Friday before Memorial Day * but since, been working on other things * would like to move Blob/get capability out of the core capabilities (because it isn?t core) * did a fix for mime parameter wrapping (*0* stuff) for boundary string (wtf??) * did some work on AuthIndicator (BIMI) blob retrieval; had led to ?we should try to unify some branches of blob handling code? * hopefully uuid mailboxes is back to top slot (assuming for example that restore work doesn?t turn out to be a problem) * rsto * deleted mailboxes need to retain an entry in convdb * we can know mailbox has been deleted, which affects how to understand the flags of emails in mailbox [rjbs: not sure I got this *exactly* right] *2020-06-08* * moving Blob/get to JMAP_BLOB_EXTENSION capability in cyrus master, while adding a revert to FM builds until our middleware is ready for it: coordinate * Ken * UUID mailboxes rebased, subs db. code has been updated; will upgrade on first open on UUID mailbox code * ready to start testing??? * fixed a crasher in vacation responses (free of const str) * what?s next? maybe sieve in mailboxes; rjbs says: maybe let?s get back to Sieve JMAP, for which we wanted a fewer-http-round-trip API (for Fm?s usage pattern) * Bron * seems like something in the mailbox delete code path doesn?t always clean up code on disk * clearly still live; we?re seeing recent should-be-deleted data still on disk * possibly related to duff locking code, possibly in sync client * rjbs: how did this get found? * bron: audit_slot, an internal Fastmail tool * slow slog of license update continues; no clear progress atm *2020-06-15* * rjbs * to discuss: time to create v3.3.x branch so we can merge usermeta-by-id-bis to master * discussion result: we?ll just keep the branches like they are unless we think of a way in which a second temporary main branch is better than this * testing of usermeta-by-id-bis on a vm is now happening * will post a public ?give ken a heads up? if you have an intended thing to merge * when we know this can run safely on a replica, we can merge to master * Ken * managed attachments; seem to work okay in Apple! * CalConnect later today: will talk DAV caching * ellie * mostly working through 3.2 bugs, but maybe fixed the last one today * working on release notes for 3.3 dev so she can make proper v3.3.0 * Bron * working on rename bugs: something with intermediates and renames is causing bugs * rjbs hopes we can rip out the intermediate-supporting code from cyrus entirely [someday] * rjbs again * what?s the effort required to move to single copy of metadata per email * bron: the convdb is not yet appropriate, as it?s a secondary source * general consensus seems that we will likely just replace how messages are stored (hash of emailid) *2020-06-22* 2020-06-22 * Discuss design of JMAP Sieve * Ken: still working on UUID mailboxes * ellie * v3.3.0 ? about halfway done with release notes, will send an email about places we?re stuck * v3.2.2 ? released today; last known new bug in v3.2 may be fixed! * Bron * main thing: working on fixing behavior of user delete/rename; it creates intermediates! * [ ed: *INTERMEDIATES AUUUURURGH* ] * there?s something wrong with users in deleted namespace * next up: back to the big ol? pile of calendar reports * or? synchronous replication. * Rik * how do we get to a ?every email has one set of metadata?? * one option: updates are more complex * another option: put all metadata in one db * conv. db becomes primary store for some data * cyrus.index and cyrus.header go away * rjbs will send an email to cyrus-devel about this * circled back to caldav discussion * libical does support multiple rrule/exdate/rdate * ?so we?ll probably have this in jscalendar *2020-06-29* * Present: Bron, ellie, Ken, Rik * JMAP Sieve: * If we want to use Blob/set, we?ll need to standardise that first! Bron will look at. * Bron will look at the spec too and see if has opinions. * Ken: * Rebased UUID stuff again last week and fixed rebase issue. * Fixed issue where upgrade of mailboxes.db had a double translation which messed up separator and domain names. Hard to track down, simple fix. * Fixed issue where new user with no subscriptions didn?t get upgraded. * On laptop, list subscriptions is not working. Maybe issue with existing Cyrus data. * Reviewed some pull requests and implemented SAFE for HTTP methods. * ellie: * still plugging away on release notes for 3.3 * added some quality of life features to Cassandane * Lots of mailing list activity. * People upgrading from 2.3! * Bron: * Added readonly mode. * Fixed issue with intermediates and renames WITHIN a user breaking replication. * Threeskip design! * Xapian -- rjbs -------------- next part -------------- An HTML attachment was scrubbed... URL: From marc at marclaporte.com Fri Jul 10 18:56:43 2020 From: marc at marclaporte.com (marc at marclaporte.com) Date: Fri, 10 Jul 2020 18:56:43 -0400 Subject: minutes, cyrus dev call, 2020-06 In-Reply-To: <44e3cda6-48a9-4dfa-bf10-e87ccdeeacd6@dogfood.fastmail.com> Message-ID: <5e755cdb06a2ac8fa60a6b814e592ac5@avan.tech> Thank you Ricardo! This is very impressive! Marc On Fri, 10 Jul 2020 17:50:07 -0400 Ricardo Signes rjbs at fastmailteam.com said > We have been keeping minutes, but not getting them out in a timely fashion. > I've adjusted my calendar to make this happen more reliably. (The basic deal > is that these meetings happen at 7:30 a.m. my time, and as soon as they're > over I go to get some caffeine, and then they've fallen down my priority > list.) > > I realize these are less useful when they're sent late, but I'll learn my > lesson better by sending them anyway. > > *2020-06-01* > * *Present*: Bron, Robert, ellie, Ken, Rik, Neil > * [ discussion about reindexing Fastmail's large install ] > * Bron > * been doing some small changes > * wants to get things shepherded to production > * soon: sync replication code!! > * need to give sync client a state struct to pass around between funcs > * ellie > * v3.2.1 on Friday! > * experimental Debian build passing all the cunit tests on all the Debian > platforms > * it was the tests more than the code under testing being a problem > * v3.3.0 dev tag should happen soon! > * some backports from master are pending (probably) > * Cass failing on v3.2 failing because of mismatch between > expectations/fixes on v3.2 > * rjbs: can we put Cassandane into the cyrus repo? > * yes but it sounds like for complex reasons we might want to; rjbs will > follow up > * also what will we do with the uuid mailboxes test code? > * neilj > * rules question: fromContactGroupId, convert errors to false ? has just > hit production > * status of backup/restore bugs? > * Ken: as far as I know, all tasks are done > * What about the ?contacts don?t get restored back to groups? > * for now, we?ll say this is expect, and count people who > notice/complain > * will ask Matthew to do a review of code and/or tests? > * murch > * uuid mailboxes all rebased by Friday before Memorial Day > * but since, been working on other things > * would like to move Blob/get capability out of the core capabilities > (because it isn?t core) > * did a fix for mime parameter wrapping (*0* stuff) for boundary string > (wtf??) > * did some work on AuthIndicator (BIMI) blob retrieval; had led to ?we > should try to unify some branches of blob handling code? > * hopefully uuid mailboxes is back to top slot (assuming for example that > restore work doesn?t turn out to be a problem) > * rsto > * deleted mailboxes need to retain an entry in convdb > * we can know mailbox has been deleted, which affects how to understand > the flags of emails in mailbox [rjbs: not sure I got this *exactly* right] > > *2020-06-08* > * moving Blob/get to JMAP_BLOB_EXTENSION capability in cyrus master, while > adding a revert to FM builds until our middleware is ready for it: > coordinate > * Ken > * UUID mailboxes rebased, subs db. code has been updated; will upgrade on > first open on UUID mailbox code > * ready to start testing??? > * fixed a crasher in vacation responses (free of const str) > * what?s next? maybe sieve in mailboxes; rjbs says: maybe let?s get > back to Sieve JMAP, for which we wanted a fewer-http-round-trip API (for > Fm?s usage pattern) > * Bron > * seems like something in the mailbox delete code path doesn?t always > clean up code on disk > * clearly still live; we?re seeing recent should-be-deleted data still > on disk > * possibly related to duff locking code, possibly in sync client > * rjbs: how did this get found? > * bron: audit_slot, an internal Fastmail tool > * slow slog of license update continues; no clear progress atm > > *2020-06-15* > * rjbs > * to discuss: time to create v3.3.x branch so we can merge > usermeta-by-id-bis to master > * discussion result: we?ll just keep the branches like they are unless > we think of a way in which a second temporary main branch is better than > this > * testing of usermeta-by-id-bis on a vm is now happening > * will post a public ?give ken a heads up? if you have an intended > thing to merge > * when we know this can run safely on a replica, we can merge to master > * Ken > * managed attachments; seem to work okay in Apple! > * CalConnect later today: will talk DAV caching > * ellie > * mostly working through 3.2 bugs, but maybe fixed the last one today > * working on release notes for 3.3 dev so she can make proper v3.3.0 > * Bron > * working on rename bugs: something with intermediates and renames is > causing bugs > * rjbs hopes we can rip out the intermediate-supporting code from cyrus > entirely [someday] > * rjbs again > * what?s the effort required to move to single copy of metadata per > email > * bron: the convdb is not yet appropriate, as it?s a secondary source > * general consensus seems that we will likely just replace how messages > are stored (hash of emailid) > > *2020-06-22* > 2020-06-22 > * Discuss design of JMAP Sieve > * Ken: still working on UUID mailboxes > * ellie > * v3.3.0 ? about halfway done with release notes, will send an email > about places we?re stuck > * v3.2.2 ? released today; last known new bug in v3.2 may be fixed! > * Bron > * main thing: working on fixing behavior of user delete/rename; it creates > intermediates! > * [ ed: *INTERMEDIATES AUUUURURGH* ] > * there?s something wrong with users in deleted namespace > * next up: back to the big ol? pile of calendar reports > * or? synchronous replication. > * Rik > * how do we get to a ?every email has one set of metadata?? > * one option: updates are more complex > * another option: put all metadata in one db > * conv. db becomes primary store for some data > * cyrus.index and cyrus.header go away > * rjbs will send an email to cyrus-devel about this > * circled back to caldav discussion > * libical does support multiple rrule/exdate/rdate > * ?so we?ll probably have this in jscalendar > > *2020-06-29* > * Present: Bron, ellie, Ken, Rik > * JMAP Sieve: > * If we want to use Blob/set, we?ll need to standardise that first! Bron > will look at. > * Bron will look at the spec too and see if has opinions. > * Ken: > * Rebased UUID stuff again last week and fixed rebase issue. > * Fixed issue where upgrade of mailboxes.db had a double translation > which messed up separator and domain names. Hard to track down, simple fix. > * Fixed issue where new user with no subscriptions didn?t get > upgraded. > * On laptop, list subscriptions is not working. Maybe issue with existing > Cyrus data. > * Reviewed some pull requests and implemented SAFE for HTTP methods. > * ellie: > * still plugging away on release notes for 3.3 > * added some quality of life features to Cassandane > * Lots of mailing list activity. > * People upgrading from 2.3! > * Bron: > * Added readonly mode. > * Fixed issue with intermediates and renames WITHIN a user breaking > replication. > * Threeskip design! > * Xapian > > -- > rjbs From ellie at fastmail.com Mon Jul 13 01:36:31 2020 From: ellie at fastmail.com (ellie timoney) Date: Mon, 13 Jul 2020 15:36:31 +1000 Subject: Cyrus IMAPd version 3.3.0 Message-ID: The Cyrus team is pleased to announce the immediate availability of a new version of Cyrus IMAP: 3.3.0 This is a snapshot of the master branch, and should be considered for testing purposes and bleeding-edge features only. It is available as a git tag, which can be found here: https://github.com/cyrusimap/cyrus-imapd/releases/tag/cyrus-imapd-3.3.0 Join us on Github at https://github.com/cyrusimap/cyrus-imapd to report issues, join in the deliberations of new features for the next Cyrus IMAP release, and to contribute to the documentation. On behalf of the Cyrus team, ellie -------------- next part -------------- An HTML attachment was scrubbed... URL: From me at anatoli.ws Tue Jul 28 03:42:58 2020 From: me at anatoli.ws (Anatoli) Date: Tue, 28 Jul 2020 04:42:58 -0300 Subject: "format specifies type" compilation warnings with clang on OpenBSD Message-ID: <4260bad3-c9ea-c48c-ea15-8758f6ce3d1d@anatoli.ws> Hi All, I just created an issue [1] about subj. I can create a PR if the proposed fix is ok (I'd go without the type cast). Regards, Anatoli [1] https://github.com/cyrusimap/cyrus-imapd/issues/3128