From me at anatoli.ws Tue Jun 2 03:48:36 2020 From: me at anatoli.ws (Anatoli) Date: Tue, 2 Jun 2020 04:48:36 -0300 Subject: deflatePending not available in zlib on OpenBSD (undefined symbol) Message-ID: <8b142e4e-ce7b-c509-2dbc-7d2753ea60c9@anatoli.ws> Cyrus developers, Is it possible to somehow rework the code in imap/httpd.c lines 158-169 in order to NOT use deflatePending as this func is not available on OpenBSD? The zlib version there is 1.2.3 and deflatePending appeared in 1.2.5 so the code doesn't compile with --enable-http (undefined symbol: deflatePending). The packaged version disables http for now. Is it safe to reduce these lines: 158 if (!zstrm->avail_out) { 159 unsigned pending; 160 161 zr = deflatePending(zstrm, &pending, Z_NULL); 162 if (zr != Z_OK) { 163 /* something went wrong */ 164 syslog(LOG_ERR, "zlib deflate error: %d %s", zr, zstrm->msg); 165 return -1; 166 } 167 168 buf_ensure(&txn->zbuf, pending); 169 } to something like: 158 if (!zstrm->avail_out) { 159 buf_ensure(&txn->zbuf, 256 * 1024); 160 } If I understand it correctly, deflatePending in this case is only used to get the needed buffer size and we could replace it with a hardcoded buffer size (like in my example of 256K). Thanks, Anatoli From murch at fastmail.com Tue Jun 2 16:36:17 2020 From: murch at fastmail.com (Ken Murchison) Date: Tue, 2 Jun 2020 16:36:17 -0400 Subject: deflatePending not available in zlib on OpenBSD (undefined symbol) In-Reply-To: <8b142e4e-ce7b-c509-2dbc-7d2753ea60c9@anatoli.ws> References: <8b142e4e-ce7b-c509-2dbc-7d2753ea60c9@anatoli.ws> Message-ID: Hi Anatoli, Thanks for the report.? I'm not sure that we even need the deflatePending() call, since we use deflateBound() to create an appropriately-sized buffer to hold the entire compressed response body.? Let me do some testing. On 6/2/20 3:48 AM, Anatoli wrote: > Cyrus developers, > > Is it possible to somehow rework the code in imap/httpd.c lines 158-169 > in order to NOT use deflatePending as this func is not available on > OpenBSD? The zlib version there is 1.2.3 and deflatePending appeared in > 1.2.5 so the code doesn't compile with --enable-http (undefined symbol: > deflatePending). The packaged version disables http for now. > > Is it safe to reduce these lines: > > 158 if (!zstrm->avail_out) { > 159 unsigned pending; > 160 > 161 zr = deflatePending(zstrm, &pending, Z_NULL); > 162 if (zr != Z_OK) { > 163 /* something went wrong */ > 164 syslog(LOG_ERR, "zlib deflate error: %d %s", zr, > zstrm->msg); > 165 return -1; > 166 } > 167 > 168 buf_ensure(&txn->zbuf, pending); > 169 } > > > to something like: > > 158 if (!zstrm->avail_out) { > 159 buf_ensure(&txn->zbuf, 256 * 1024); > 160 } > > If I understand it correctly, deflatePending in this case is only used > to get the needed buffer size and we could replace it with a hardcoded > buffer size (like in my example of 256K). > > > Thanks, > Anatoli -- Kenneth Murchison Senior Software Developer Fastmail US LLC From me at anatoli.ws Tue Jun 2 18:19:00 2020 From: me at anatoli.ws (Anatoli) Date: Tue, 2 Jun 2020 19:19:00 -0300 Subject: deflatePending not available in zlib on OpenBSD (undefined symbol) In-Reply-To: References: <8b142e4e-ce7b-c509-2dbc-7d2753ea60c9@anatoli.ws> Message-ID: <59b40e91-1f0e-e1ef-046d-ad0626510dca@anatoli.ws> Hi Ken, According to the docs [1]: If deflate returns Z_OK and with zero avail_out, it must be called again after making room in the buffer because there might be more output pending. On the other hand it says: Z_FINISH can be used in the first deflate call after deflateInit if all the compression is to be done in a single step. In order to complete in one call, avail_out must be at least the value returned by deflateBound. Then deflate is guaranteed to return Z_STREAM_END. But: Note that it is possible for the compressed size to be larger than the value returned by deflateBound() if flush options other than Z_FINISH or Z_NO_FLUSH are used. At the beginning of zlib_compress() there's code that decides with which flush option to call deflate(). /* Only flush for static content or on last (zero-length) chunk */ If it's possible to make flush to be always Z_FINISH, then I guess the entire do { ... } while (!zstrm->avail_out); loop could be simplified to just: zstrm->next_out = (Bytef *) txn->zbuf.s + txn->zbuf.len; zstrm->avail_out = txn->zbuf.alloc - txn->zbuf.len; zr = deflate(zstrm, flush); if (zr != Z_STREAM_END) { /* something went wrong */ syslog(LOG_ERR, "zlib deflate error: %d %s", zr, zstrm->msg); return -1; } I changed the "if (zr)" condition as the only good value would be Z_STREAM_END. But if the flush option can't be always Z_FINISH, then I believe the loop should be kept with the checks for !zstrm->avail_out as it's possible that there would be not enough buffer for deflate() to complete in one call. Regards, Anatoli [1] https://www.zlib.net/manual.html On 2/6/20 17:36, Ken Murchison wrote: > Hi Anatoli, > > Thanks for the report.? I'm not sure that we even need the > deflatePending() call, since we use deflateBound() to create an > appropriately-sized buffer to hold the entire compressed response body.? > Let me do some testing. > > > On 6/2/20 3:48 AM, Anatoli wrote: >> Cyrus developers, >> >> Is it possible to somehow rework the code in imap/httpd.c lines 158-169 >> in order to NOT use deflatePending as this func is not available on >> OpenBSD? The zlib version there is 1.2.3 and deflatePending appeared in >> 1.2.5 so the code doesn't compile with --enable-http (undefined symbol: >> deflatePending). The packaged version disables http for now. >> >> Is it safe to reduce these lines: >> >> ? 158???????? if (!zstrm->avail_out) { >> ? 159???????????? unsigned pending; >> ? 160 >> ? 161???????????? zr = deflatePending(zstrm, &pending, Z_NULL); >> ? 162???????????? if (zr != Z_OK) { >> ? 163???????????????? /* something went wrong */ >> ? 164???????????????? syslog(LOG_ERR, "zlib deflate error: %d %s", zr, >> zstrm->msg); >> ? 165???????????????? return -1; >> ? 166???????????? } >> ? 167 >> ? 168???????????? buf_ensure(&txn->zbuf, pending); >> ? 169???????? } >> >> >> to something like: >> >> ? 158???????? if (!zstrm->avail_out) { >> ? 159???????????? buf_ensure(&txn->zbuf, 256 * 1024); >> ? 160???????? } >> >> If I understand it correctly, deflatePending in this case is only used >> to get the needed buffer size and we could replace it with a hardcoded >> buffer size (like in my example of 256K). >> >> >> Thanks, >> Anatoli > From murch at fastmail.com Tue Jun 2 18:31:19 2020 From: murch at fastmail.com (Ken Murchison) Date: Tue, 2 Jun 2020 18:31:19 -0400 Subject: deflatePending not available in zlib on OpenBSD (undefined symbol) In-Reply-To: <59b40e91-1f0e-e1ef-046d-ad0626510dca@anatoli.ws> References: <8b142e4e-ce7b-c509-2dbc-7d2753ea60c9@anatoli.ws> <59b40e91-1f0e-e1ef-046d-ad0626510dca@anatoli.ws> Message-ID: <13a34d04-b43f-666c-f92d-f1ba42e891fc@fastmail.com> Yes, you're correct.? We have a comment in prot.c that quotes the same passage ion the docs. I think this might do the trick: diff --git a/imap/httpd.c b/imap/httpd.c index fc430d935..b5014b97e 100644 --- a/imap/httpd.c +++ b/imap/httpd.c @@ -149,24 +149,12 @@ HIDDEN int zlib_compress(struct transaction_t *txn, unsigned flags, ???? zstrm->next_in = (Bytef *) buf; ???? zstrm->avail_in = len; -??? buf_ensure(&txn->zbuf, deflateBound(zstrm, zstrm->avail_in)); ???? buf_reset(&txn->zbuf); ???? do { ???????? int zr; -??????? if (!zstrm->avail_out) { -??????????? unsigned pending; - -??????????? zr = deflatePending(zstrm, &pending, Z_NULL); -??????????? if (zr != Z_OK) { -??????????????? /* something went wrong */ -??????????????? syslog(LOG_ERR, "zlib deflate error: %d %s", zr, zstrm->msg); -??????????????? return -1; -??????????? } - -??????????? buf_ensure(&txn->zbuf, pending); -??????? } +??????? buf_ensure(&txn->zbuf, deflateBound(zstrm, len)); ???????? zstrm->next_out = (Bytef *) txn->zbuf.s + txn->zbuf.len; ???????? zstrm->avail_out = txn->zbuf.alloc - txn->zbuf.len; On 6/2/20 6:19 PM, Anatoli wrote: > Hi Ken, > > According to the docs [1]: If deflate returns Z_OK and with zero > avail_out, it must be called again after making room in the buffer > because there might be more output pending. > > On the other hand it says: Z_FINISH can be used in the first deflate > call after deflateInit if all the compression is to be done in a single > step. In order to complete in one call, avail_out must be at least the > value returned by deflateBound. Then deflate is guaranteed to return > Z_STREAM_END. > > But: Note that it is possible for the compressed size to be larger than > the value returned by deflateBound() if flush options other than > Z_FINISH or Z_NO_FLUSH are used. > > > At the beginning of zlib_compress() there's code that decides with which > flush option to call deflate(). > > /* Only flush for static content or on last (zero-length) chunk */ > > If it's possible to make flush to be always Z_FINISH, then I guess the > entire > > do { ... } while (!zstrm->avail_out); > > loop could be simplified to just: > > zstrm->next_out = (Bytef *) txn->zbuf.s + txn->zbuf.len; > zstrm->avail_out = txn->zbuf.alloc - txn->zbuf.len; > > zr = deflate(zstrm, flush); > if (zr != Z_STREAM_END) { > /* something went wrong */ > syslog(LOG_ERR, "zlib deflate error: %d %s", zr, zstrm->msg); > return -1; > } > > > I changed the "if (zr)" condition as the only good value would be > Z_STREAM_END. > > But if the flush option can't be always Z_FINISH, then I believe the > loop should be kept with the checks for !zstrm->avail_out as it's > possible that there would be not enough buffer for deflate() to complete > in one call. > > Regards, > Anatoli > > [1] https://www.zlib.net/manual.html > > > > On 2/6/20 17:36, Ken Murchison wrote: >> Hi Anatoli, >> >> Thanks for the report.? I'm not sure that we even need the >> deflatePending() call, since we use deflateBound() to create an >> appropriately-sized buffer to hold the entire compressed response body. >> Let me do some testing. >> >> >> On 6/2/20 3:48 AM, Anatoli wrote: >>> Cyrus developers, >>> >>> Is it possible to somehow rework the code in imap/httpd.c lines 158-169 >>> in order to NOT use deflatePending as this func is not available on >>> OpenBSD? The zlib version there is 1.2.3 and deflatePending appeared in >>> 1.2.5 so the code doesn't compile with --enable-http (undefined symbol: >>> deflatePending). The packaged version disables http for now. >>> >>> Is it safe to reduce these lines: >>> >>> ? 158???????? if (!zstrm->avail_out) { >>> ? 159???????????? unsigned pending; >>> ? 160 >>> ? 161???????????? zr = deflatePending(zstrm, &pending, Z_NULL); >>> ? 162???????????? if (zr != Z_OK) { >>> ? 163???????????????? /* something went wrong */ >>> ? 164???????????????? syslog(LOG_ERR, "zlib deflate error: %d %s", zr, >>> zstrm->msg); >>> ? 165???????????????? return -1; >>> ? 166???????????? } >>> ? 167 >>> ? 168???????????? buf_ensure(&txn->zbuf, pending); >>> ? 169???????? } >>> >>> >>> to something like: >>> >>> ? 158???????? if (!zstrm->avail_out) { >>> ? 159???????????? buf_ensure(&txn->zbuf, 256 * 1024); >>> ? 160???????? } >>> >>> If I understand it correctly, deflatePending in this case is only used >>> to get the needed buffer size and we could replace it with a hardcoded >>> buffer size (like in my example of 256K). >>> >>> >>> Thanks, >>> Anatoli -- Kenneth Murchison Senior Software Developer Fastmail US LLC From me at anatoli.ws Tue Jun 2 19:34:16 2020 From: me at anatoli.ws (Anatoli) Date: Tue, 2 Jun 2020 20:34:16 -0300 Subject: deflatePending not available in zlib on OpenBSD (undefined symbol) In-Reply-To: <13a34d04-b43f-666c-f92d-f1ba42e891fc@fastmail.com> References: <8b142e4e-ce7b-c509-2dbc-7d2753ea60c9@anatoli.ws> <59b40e91-1f0e-e1ef-046d-ad0626510dca@anatoli.ws> <13a34d04-b43f-666c-f92d-f1ba42e891fc@fastmail.com> Message-ID: Looks good to me and compiles correctly on OpenBSD. Could it be included in the next 3.2 release (3.2.2)? On 2/6/20 19:31, Ken Murchison wrote: > Yes, you're correct.? We have a comment in prot.c that quotes the same > passage ion the docs. > > I think this might do the trick: > > > diff --git a/imap/httpd.c b/imap/httpd.c > index fc430d935..b5014b97e 100644 > --- a/imap/httpd.c > +++ b/imap/httpd.c > @@ -149,24 +149,12 @@ HIDDEN int zlib_compress(struct transaction_t > *txn, unsigned flags, > ???? zstrm->next_in = (Bytef *) buf; > ???? zstrm->avail_in = len; > > -??? buf_ensure(&txn->zbuf, deflateBound(zstrm, zstrm->avail_in)); > ???? buf_reset(&txn->zbuf); > > ???? do { > ???????? int zr; > > -??????? if (!zstrm->avail_out) { > -??????????? unsigned pending; > - > -??????????? zr = deflatePending(zstrm, &pending, Z_NULL); > -??????????? if (zr != Z_OK) { > -??????????????? /* something went wrong */ > -??????????????? syslog(LOG_ERR, "zlib deflate error: %d %s", zr, > zstrm->msg); > -??????????????? return -1; > -??????????? } > - > -??????????? buf_ensure(&txn->zbuf, pending); > -??????? } > +??????? buf_ensure(&txn->zbuf, deflateBound(zstrm, len)); > > ???????? zstrm->next_out = (Bytef *) txn->zbuf.s + txn->zbuf.len; > ???????? zstrm->avail_out = txn->zbuf.alloc - txn->zbuf.len; > > > On 6/2/20 6:19 PM, Anatoli wrote: >> Hi Ken, >> >> According to the docs [1]: If deflate returns Z_OK and with zero >> avail_out, it must be called again after making room in the buffer >> because there might be more output pending. >> >> On the other hand it says: Z_FINISH can be used in the first deflate >> call after deflateInit if all the compression is to be done in a single >> step. In order to complete in one call, avail_out must be at least the >> value returned by deflateBound. Then deflate is guaranteed to return >> Z_STREAM_END. >> >> But: Note that it is possible for the compressed size to be larger than >> the value returned by deflateBound() if flush options other than >> Z_FINISH or Z_NO_FLUSH are used. >> >> >> At the beginning of zlib_compress() there's code that decides with which >> flush option to call deflate(). >> >> /* Only flush for static content or on last (zero-length) chunk */ >> >> If it's possible to make flush to be always Z_FINISH, then I guess the >> entire >> >> do { ... } while (!zstrm->avail_out); >> >> loop could be simplified to just: >> >> zstrm->next_out = (Bytef *) txn->zbuf.s + txn->zbuf.len; >> zstrm->avail_out = txn->zbuf.alloc - txn->zbuf.len; >> >> zr = deflate(zstrm, flush); >> if (zr != Z_STREAM_END) { >> ????/* something went wrong */ >> ????syslog(LOG_ERR, "zlib deflate error: %d %s", zr, zstrm->msg); >> ????return -1; >> } >> >> >> I changed the "if (zr)" condition as the only good value would be >> Z_STREAM_END. >> >> But if the flush option can't be always Z_FINISH, then I believe the >> loop should be kept with the checks for !zstrm->avail_out as it's >> possible that there would be not enough buffer for deflate() to complete >> in one call. >> >> Regards, >> Anatoli >> >> [1] https://www.zlib.net/manual.html >> >> >> >> On 2/6/20 17:36, Ken Murchison wrote: >>> Hi Anatoli, >>> >>> Thanks for the report.? I'm not sure that we even need the >>> deflatePending() call, since we use deflateBound() to create an >>> appropriately-sized buffer to hold the entire compressed response body. >>> Let me do some testing. >>> >>> >>> On 6/2/20 3:48 AM, Anatoli wrote: >>>> Cyrus developers, >>>> >>>> Is it possible to somehow rework the code in imap/httpd.c lines 158-169 >>>> in order to NOT use deflatePending as this func is not available on >>>> OpenBSD? The zlib version there is 1.2.3 and deflatePending appeared in >>>> 1.2.5 so the code doesn't compile with --enable-http (undefined symbol: >>>> deflatePending). The packaged version disables http for now. >>>> >>>> Is it safe to reduce these lines: >>>> >>>> ?? 158???????? if (!zstrm->avail_out) { >>>> ?? 159???????????? unsigned pending; >>>> ?? 160 >>>> ?? 161???????????? zr = deflatePending(zstrm, &pending, Z_NULL); >>>> ?? 162???????????? if (zr != Z_OK) { >>>> ?? 163???????????????? /* something went wrong */ >>>> ?? 164???????????????? syslog(LOG_ERR, "zlib deflate error: %d %s", zr, >>>> zstrm->msg); >>>> ?? 165???????????????? return -1; >>>> ?? 166???????????? } >>>> ?? 167 >>>> ?? 168???????????? buf_ensure(&txn->zbuf, pending); >>>> ?? 169???????? } >>>> >>>> >>>> to something like: >>>> >>>> ?? 158???????? if (!zstrm->avail_out) { >>>> ?? 159???????????? buf_ensure(&txn->zbuf, 256 * 1024); >>>> ?? 160???????? } >>>> >>>> If I understand it correctly, deflatePending in this case is only used >>>> to get the needed buffer size and we could replace it with a hardcoded >>>> buffer size (like in my example of 256K). >>>> >>>> >>>> Thanks, >>>> Anatoli > From me at anatoli.ws Wed Jun 3 00:13:25 2020 From: me at anatoli.ws (Anatoli) Date: Wed, 3 Jun 2020 01:13:25 -0300 Subject: configure: wslay v1.1.1 required but the latest one is 1.1.0 Message-ID: Cyrus developers, The configure script checks for wslay lib version 1.1.1, but the latest version released is 1.1.0. So when it is installed, it reports: checking for WSLAY... no configure: httpd will not have support for WebSockets. Consider installing libwslay The wslay's github repo has a mention of a 1.1.1-DEV version. Not sure if cyrus-imapd httpd requires something from it or if it was just a typo and 1.1.0 is ok. For the later case below is a patch. Regards, Anatoli diff --git a/configure.ac b/configure.ac index dc0e0fff2..30e925c60 100644 --- a/configure.ac +++ b/configure.ac @@ -1631,7 +1631,7 @@ dnl AC_MSG_WARN([Your version of OpenDKIM can not support iSchedu AC_ARG_WITH(wslay, [AS_HELP_STRING([--without-wslay], [disable WebSockets support (check)])],,[with_wslay="check"]) if test "x$with_wslay" = "xyes" -o "x$with_wslay" = "xcheck"; then - PKG_CHECK_MODULES([WSLAY], [libwslay >= 1.1.1], [ + PKG_CHECK_MODULES([WSLAY], [libwslay >= 1.1.0], [ AC_DEFINE(HAVE_WSLAY,[], [Build WebSockets support into httpd?]) with_wslay=yes From ellie at fastmail.com Wed Jun 3 00:35:21 2020 From: ellie at fastmail.com (ellie timoney) Date: Wed, 03 Jun 2020 14:35:21 +1000 Subject: configure: wslay v1.1.1 required but the latest one is 1.1.0 In-Reply-To: References: Message-ID: <18e0a63d-31bf-45b5-a001-efc92237ba16@www.fastmail.com> In our "cyruslibs" package, the wslay submodule is at this commit: commit 4a937cd (HEAD, origin/master, origin/HEAD, master) Author: Tatsuhiro Tsujikawa AuthorDate: Fri Jun 8 23:19:03 2018 +0900 Commit: Tatsuhiro Tsujikawa CommitDate: Fri Jun 8 23:19:03 2018 +0900 Bump up version number to 1.1.1-DEV Which is the commit immediately following the release-1.1.0 tag. So, presumably, we're not dependent on any feature/fix that's only in the unreleased version, because otherwise we would've bumped the cyruslibs submodule to include those commits? So, "1.1.1" might be a typo, or an anticipatory thing that didn't go anywhere yet, I'm not sure. When you configure it with your patch applied and 1.1.0 installed, does Cyrus build okay? Cheers, ellie On Wed, Jun 3, 2020, at 2:13 PM, Anatoli wrote: > Cyrus developers, > > The configure script checks for wslay lib version 1.1.1, but the latest > version released is 1.1.0. So when it is installed, it reports: > > checking for WSLAY... no > configure: httpd will not have support for WebSockets. Consider > installing libwslay > > The wslay's github repo has a mention of a 1.1.1-DEV version. Not sure > if cyrus-imapd httpd requires something from it or if it was just a > typo and 1.1.0 is ok. > > For the later case below is a patch. > > Regards, > Anatoli > > > diff --git a/configure.ac b/configure.ac > index dc0e0fff2..30e925c60 100644 > --- a/configure.ac > +++ b/configure.ac > @@ -1631,7 +1631,7 @@ dnl AC_MSG_WARN([Your version of > OpenDKIM can not support iSchedu > > AC_ARG_WITH(wslay, [AS_HELP_STRING([--without-wslay], [disable > WebSockets support (check)])],,[with_wslay="check"]) > if test "x$with_wslay" = "xyes" -o "x$with_wslay" = "xcheck"; > then > - PKG_CHECK_MODULES([WSLAY], [libwslay >= 1.1.1], [ > + PKG_CHECK_MODULES([WSLAY], [libwslay >= 1.1.0], [ > AC_DEFINE(HAVE_WSLAY,[], > [Build WebSockets support into httpd?]) > with_wslay=yes > From me at anatoli.ws Wed Jun 3 00:58:41 2020 From: me at anatoli.ws (Anatoli) Date: Wed, 3 Jun 2020 01:58:41 -0300 Subject: configure: wslay v1.1.1 required but the latest one is 1.1.0 In-Reply-To: <18e0a63d-31bf-45b5-a001-efc92237ba16@www.fastmail.com> References: <18e0a63d-31bf-45b5-a001-efc92237ba16@www.fastmail.com> Message-ID: <810d8452-edd0-7128-4071-444cbf324d8d@anatoli.ws> Hi Ellie, > When you configure it with your patch applied and 1.1.0 installed, does Cyrus build okay? Yes, it builds without errors. Configure prints: checking for WSLAY... yes wslay: yes And -lwslay is passed as arg numerous times during build process. And effectively httpd binary includes references to wslay_event_xxx in its symbols table. Regards, Anatoli On 3/6/20 01:35, ellie timoney wrote: > In our "cyruslibs" package, the wslay submodule is at this commit: > > commit 4a937cd (HEAD, origin/master, origin/HEAD, master) > Author: Tatsuhiro Tsujikawa > AuthorDate: Fri Jun 8 23:19:03 2018 +0900 > Commit: Tatsuhiro Tsujikawa > CommitDate: Fri Jun 8 23:19:03 2018 +0900 > > Bump up version number to 1.1.1-DEV > > Which is the commit immediately following the release-1.1.0 tag. So, presumably, we're not dependent on any feature/fix that's only in the unreleased version, because otherwise we would've bumped the cyruslibs submodule to include those commits? > > So, "1.1.1" might be a typo, or an anticipatory thing that didn't go anywhere yet, I'm not sure. > > When you configure it with your patch applied and 1.1.0 installed, does Cyrus build okay? > > Cheers, > > ellie > > On Wed, Jun 3, 2020, at 2:13 PM, Anatoli wrote: >> Cyrus developers, >> >> The configure script checks for wslay lib version 1.1.1, but the latest >> version released is 1.1.0. So when it is installed, it reports: >> >> checking for WSLAY... no >> configure: httpd will not have support for WebSockets. Consider >> installing libwslay >> >> The wslay's github repo has a mention of a 1.1.1-DEV version. Not sure >> if cyrus-imapd httpd requires something from it or if it was just a >> typo and 1.1.0 is ok. >> >> For the later case below is a patch. >> >> Regards, >> Anatoli >> >> >> diff --git a/configure.ac b/configure.ac >> index dc0e0fff2..30e925c60 100644 >> --- a/configure.ac >> +++ b/configure.ac >> @@ -1631,7 +1631,7 @@ dnl AC_MSG_WARN([Your version of >> OpenDKIM can not support iSchedu >> >> AC_ARG_WITH(wslay, [AS_HELP_STRING([--without-wslay], [disable >> WebSockets support (check)])],,[with_wslay="check"]) >> if test "x$with_wslay" = "xyes" -o "x$with_wslay" = "xcheck"; >> then >> - PKG_CHECK_MODULES([WSLAY], [libwslay >= 1.1.1], [ >> + PKG_CHECK_MODULES([WSLAY], [libwslay >= 1.1.0], [ >> AC_DEFINE(HAVE_WSLAY,[], >> [Build WebSockets support into httpd?]) >> with_wslay=yes >> From ellie at fastmail.com Wed Jun 3 01:19:52 2020 From: ellie at fastmail.com (ellie timoney) Date: Wed, 03 Jun 2020 15:19:52 +1000 Subject: configure: wslay v1.1.1 required but the latest one is 1.1.0 In-Reply-To: <810d8452-edd0-7128-4071-444cbf324d8d@anatoli.ws> References: <18e0a63d-31bf-45b5-a001-efc92237ba16@www.fastmail.com> <810d8452-edd0-7128-4071-444cbf324d8d@anatoli.ws> Message-ID: Cool, thanks for confirming that. So far it's sounding like 1.1.0 is probably adequate, but I'll wait a little bit to see if Ken has any input once he's been online On Wed, Jun 3, 2020, at 2:58 PM, Anatoli wrote: > Hi Ellie, > > > When you configure it with your patch applied and 1.1.0 installed, > does Cyrus build okay? > > Yes, it builds without errors. > > Configure prints: > > checking for WSLAY... yes > wslay: yes > > And -lwslay is passed as arg numerous times during build process. > > And effectively httpd binary includes references to wslay_event_xxx in > its symbols table. > > Regards, > Anatoli > > On 3/6/20 01:35, ellie timoney wrote: > > In our "cyruslibs" package, the wslay submodule is at this commit: > > > > commit 4a937cd (HEAD, origin/master, origin/HEAD, master) > > Author: Tatsuhiro Tsujikawa > > AuthorDate: Fri Jun 8 23:19:03 2018 +0900 > > Commit: Tatsuhiro Tsujikawa > > CommitDate: Fri Jun 8 23:19:03 2018 +0900 > > > > Bump up version number to 1.1.1-DEV > > > > Which is the commit immediately following the release-1.1.0 tag. So, presumably, we're not dependent on any feature/fix that's only in the unreleased version, because otherwise we would've bumped the cyruslibs submodule to include those commits? > > > > So, "1.1.1" might be a typo, or an anticipatory thing that didn't go anywhere yet, I'm not sure. > > > > When you configure it with your patch applied and 1.1.0 installed, does Cyrus build okay? > > > > Cheers, > > > > ellie > > > > On Wed, Jun 3, 2020, at 2:13 PM, Anatoli wrote: > >> Cyrus developers, > >> > >> The configure script checks for wslay lib version 1.1.1, but the latest > >> version released is 1.1.0. So when it is installed, it reports: > >> > >> checking for WSLAY... no > >> configure: httpd will not have support for WebSockets. Consider > >> installing libwslay > >> > >> The wslay's github repo has a mention of a 1.1.1-DEV version. Not sure > >> if cyrus-imapd httpd requires something from it or if it was just a > >> typo and 1.1.0 is ok. > >> > >> For the later case below is a patch. > >> > >> Regards, > >> Anatoli > >> > >> > >> diff --git a/configure.ac b/configure.ac > >> index dc0e0fff2..30e925c60 100644 > >> --- a/configure.ac > >> +++ b/configure.ac > >> @@ -1631,7 +1631,7 @@ dnl AC_MSG_WARN([Your version of > >> OpenDKIM can not support iSchedu > >> > >> AC_ARG_WITH(wslay, [AS_HELP_STRING([--without-wslay], [disable > >> WebSockets support (check)])],,[with_wslay="check"]) > >> if test "x$with_wslay" = "xyes" -o "x$with_wslay" = "xcheck"; > >> then > >> - PKG_CHECK_MODULES([WSLAY], [libwslay >= 1.1.1], [ > >> + PKG_CHECK_MODULES([WSLAY], [libwslay >= 1.1.0], [ > >> AC_DEFINE(HAVE_WSLAY,[], > >> [Build WebSockets support into httpd?]) > >> with_wslay=yes > >> > From me at anatoli.ws Wed Jun 3 03:48:19 2020 From: me at anatoli.ws (Anatoli) Date: Wed, 3 Jun 2020 04:48:19 -0300 Subject: run-time dependencies Message-ID: Cyrus developers, I couldn't find in the documentation the *run-time* dependencies for cyrus-imapd. In particular, are any of these not required to be present for the correct execution of cyrus-imapd (with --enable-http)? pcre e2fsprogs/uuid jansson sqlite3 icu4c libical nghttpd2 brotli wslay Thanks, Anatoli From me at anatoli.ws Wed Jun 3 04:24:31 2020 From: me at anatoli.ws (Anatoli) Date: Wed, 3 Jun 2020 05:24:31 -0300 Subject: new features with no documentation Message-ID: <04ee2b89-7c24-2947-b4f9-8b094f2a9e3f@anatoli.ws> Cyrus developers, What is the purpose/benefit of zeroskip? What is the purpose of chardet and cld2 in cyrus-imapd? Should any be considered in production environments running 3.2 branch? Thanks, Anatoli From murch at fastmail.com Wed Jun 3 06:39:32 2020 From: murch at fastmail.com (Ken Murchison) Date: Wed, 3 Jun 2020 06:39:32 -0400 Subject: run-time dependencies In-Reply-To: References: Message-ID: On 6/3/20 3:48 AM, Anatoli wrote: > Cyrus developers, > > I couldn't find in the documentation the *run-time* dependencies for > cyrus-imapd. > > In particular, are any of these not required to be present for the > correct execution of cyrus-imapd (with --enable-http)? > > pcre e2fsprogs/uuid jansson sqlite3 icu4c libical nghttpd2 brotli wslay pcre is NOT required for httpd at all as far as I recall nghttpd2, brotli, wslay are entirely OPTIONAL sqlite3 is required if you want to use any DAV functionality libical is required if you want to use CalDAV icu4c is required if libical was compiled with support for non-Gregorian calendars jansson is definitely required for JMAP functionality, and may be required for other modules uuid may be prevasive enough to be required for multiple modules -- Kenneth Murchison Senior Software Developer Fastmail US LLC From murch at fastmail.com Wed Jun 3 06:44:55 2020 From: murch at fastmail.com (Ken Murchison) Date: Wed, 3 Jun 2020 06:44:55 -0400 Subject: configure: wslay v1.1.1 required but the latest one is 1.1.0 In-Reply-To: References: <18e0a63d-31bf-45b5-a001-efc92237ba16@www.fastmail.com> <810d8452-edd0-7128-4071-444cbf324d8d@anatoli.ws> Message-ID: Yes, 1.1.0 is probably sufficient, unless this bug is an issue with Cyrus: https://github.com/tatsuhiro-t/wslay/pull/47 On 6/3/20 1:19 AM, ellie timoney wrote: > Cool, thanks for confirming that. So far it's sounding like 1.1.0 is probably adequate, but I'll wait a little bit to see if Ken has any input once he's been online > > On Wed, Jun 3, 2020, at 2:58 PM, Anatoli wrote: >> Hi Ellie, >> >>> When you configure it with your patch applied and 1.1.0 installed, >> does Cyrus build okay? >> >> Yes, it builds without errors. >> >> Configure prints: >> >> checking for WSLAY... yes >> wslay: yes >> >> And -lwslay is passed as arg numerous times during build process. >> >> And effectively httpd binary includes references to wslay_event_xxx in >> its symbols table. >> >> Regards, >> Anatoli >> >> On 3/6/20 01:35, ellie timoney wrote: >>> In our "cyruslibs" package, the wslay submodule is at this commit: >>> >>> commit 4a937cd (HEAD, origin/master, origin/HEAD, master) >>> Author: Tatsuhiro Tsujikawa >>> AuthorDate: Fri Jun 8 23:19:03 2018 +0900 >>> Commit: Tatsuhiro Tsujikawa >>> CommitDate: Fri Jun 8 23:19:03 2018 +0900 >>> >>> Bump up version number to 1.1.1-DEV >>> >>> Which is the commit immediately following the release-1.1.0 tag. So, presumably, we're not dependent on any feature/fix that's only in the unreleased version, because otherwise we would've bumped the cyruslibs submodule to include those commits? >>> >>> So, "1.1.1" might be a typo, or an anticipatory thing that didn't go anywhere yet, I'm not sure. >>> >>> When you configure it with your patch applied and 1.1.0 installed, does Cyrus build okay? >>> >>> Cheers, >>> >>> ellie >>> >>> On Wed, Jun 3, 2020, at 2:13 PM, Anatoli wrote: >>>> Cyrus developers, >>>> >>>> The configure script checks for wslay lib version 1.1.1, but the latest >>>> version released is 1.1.0. So when it is installed, it reports: >>>> >>>> checking for WSLAY... no >>>> configure: httpd will not have support for WebSockets. Consider >>>> installing libwslay >>>> >>>> The wslay's github repo has a mention of a 1.1.1-DEV version. Not sure >>>> if cyrus-imapd httpd requires something from it or if it was just a >>>> typo and 1.1.0 is ok. >>>> >>>> For the later case below is a patch. >>>> >>>> Regards, >>>> Anatoli >>>> >>>> >>>> diff --git a/configure.ac b/configure.ac >>>> index dc0e0fff2..30e925c60 100644 >>>> --- a/configure.ac >>>> +++ b/configure.ac >>>> @@ -1631,7 +1631,7 @@ dnl AC_MSG_WARN([Your version of >>>> OpenDKIM can not support iSchedu >>>> >>>> AC_ARG_WITH(wslay, [AS_HELP_STRING([--without-wslay], [disable >>>> WebSockets support (check)])],,[with_wslay="check"]) >>>> if test "x$with_wslay" = "xyes" -o "x$with_wslay" = "xcheck"; >>>> then >>>> - PKG_CHECK_MODULES([WSLAY], [libwslay >= 1.1.1], [ >>>> + PKG_CHECK_MODULES([WSLAY], [libwslay >= 1.1.0], [ >>>> AC_DEFINE(HAVE_WSLAY,[], >>>> [Build WebSockets support into httpd?]) >>>> with_wslay=yes >>>> -- Kenneth Murchison Senior Software Developer Fastmail US LLC From murch at fastmail.com Wed Jun 3 16:47:05 2020 From: murch at fastmail.com (Ken Murchison) Date: Wed, 3 Jun 2020 16:47:05 -0400 Subject: deflatePending not available in zlib on OpenBSD (undefined symbol) In-Reply-To: References: <8b142e4e-ce7b-c509-2dbc-7d2753ea60c9@anatoli.ws> <59b40e91-1f0e-e1ef-046d-ad0626510dca@anatoli.ws> <13a34d04-b43f-666c-f92d-f1ba42e891fc@fastmail.com> Message-ID: This is my latest proposed fix: https://github.com/cyrusimap/cyrus-imapd/pull/3061 On 6/2/20 7:34 PM, Anatoli wrote: > Looks good to me and compiles correctly on OpenBSD. > > Could it be included in the next 3.2 release (3.2.2)? > > > On 2/6/20 19:31, Ken Murchison wrote: >> Yes, you're correct.? We have a comment in prot.c that quotes the same >> passage ion the docs. >> >> I think this might do the trick: >> >> >> diff --git a/imap/httpd.c b/imap/httpd.c >> index fc430d935..b5014b97e 100644 >> --- a/imap/httpd.c >> +++ b/imap/httpd.c >> @@ -149,24 +149,12 @@ HIDDEN int zlib_compress(struct transaction_t >> *txn, unsigned flags, >> ???? zstrm->next_in = (Bytef *) buf; >> ???? zstrm->avail_in = len; >> >> -??? buf_ensure(&txn->zbuf, deflateBound(zstrm, zstrm->avail_in)); >> ???? buf_reset(&txn->zbuf); >> >> ???? do { >> ???????? int zr; >> >> -??????? if (!zstrm->avail_out) { >> -??????????? unsigned pending; >> - >> -??????????? zr = deflatePending(zstrm, &pending, Z_NULL); >> -??????????? if (zr != Z_OK) { >> -??????????????? /* something went wrong */ >> -??????????????? syslog(LOG_ERR, "zlib deflate error: %d %s", zr, >> zstrm->msg); >> -??????????????? return -1; >> -??????????? } >> - >> -??????????? buf_ensure(&txn->zbuf, pending); >> -??????? } >> +??????? buf_ensure(&txn->zbuf, deflateBound(zstrm, len)); >> >> ???????? zstrm->next_out = (Bytef *) txn->zbuf.s + txn->zbuf.len; >> ???????? zstrm->avail_out = txn->zbuf.alloc - txn->zbuf.len; >> >> >> On 6/2/20 6:19 PM, Anatoli wrote: >>> Hi Ken, >>> >>> According to the docs [1]: If deflate returns Z_OK and with zero >>> avail_out, it must be called again after making room in the buffer >>> because there might be more output pending. >>> >>> On the other hand it says: Z_FINISH can be used in the first deflate >>> call after deflateInit if all the compression is to be done in a single >>> step. In order to complete in one call, avail_out must be at least the >>> value returned by deflateBound. Then deflate is guaranteed to return >>> Z_STREAM_END. >>> >>> But: Note that it is possible for the compressed size to be larger than >>> the value returned by deflateBound() if flush options other than >>> Z_FINISH or Z_NO_FLUSH are used. >>> >>> >>> At the beginning of zlib_compress() there's code that decides with which >>> flush option to call deflate(). >>> >>> /* Only flush for static content or on last (zero-length) chunk */ >>> >>> If it's possible to make flush to be always Z_FINISH, then I guess the >>> entire >>> >>> do { ... } while (!zstrm->avail_out); >>> >>> loop could be simplified to just: >>> >>> zstrm->next_out = (Bytef *) txn->zbuf.s + txn->zbuf.len; >>> zstrm->avail_out = txn->zbuf.alloc - txn->zbuf.len; >>> >>> zr = deflate(zstrm, flush); >>> if (zr != Z_STREAM_END) { >>> ????/* something went wrong */ >>> ????syslog(LOG_ERR, "zlib deflate error: %d %s", zr, zstrm->msg); >>> ????return -1; >>> } >>> >>> >>> I changed the "if (zr)" condition as the only good value would be >>> Z_STREAM_END. >>> >>> But if the flush option can't be always Z_FINISH, then I believe the >>> loop should be kept with the checks for !zstrm->avail_out as it's >>> possible that there would be not enough buffer for deflate() to complete >>> in one call. >>> >>> Regards, >>> Anatoli >>> >>> [1] https://www.zlib.net/manual.html >>> >>> >>> >>> On 2/6/20 17:36, Ken Murchison wrote: >>>> Hi Anatoli, >>>> >>>> Thanks for the report.? I'm not sure that we even need the >>>> deflatePending() call, since we use deflateBound() to create an >>>> appropriately-sized buffer to hold the entire compressed response body. >>>> Let me do some testing. >>>> >>>> >>>> On 6/2/20 3:48 AM, Anatoli wrote: >>>>> Cyrus developers, >>>>> >>>>> Is it possible to somehow rework the code in imap/httpd.c lines 158-169 >>>>> in order to NOT use deflatePending as this func is not available on >>>>> OpenBSD? The zlib version there is 1.2.3 and deflatePending appeared in >>>>> 1.2.5 so the code doesn't compile with --enable-http (undefined symbol: >>>>> deflatePending). The packaged version disables http for now. >>>>> >>>>> Is it safe to reduce these lines: >>>>> >>>>> ?? 158???????? if (!zstrm->avail_out) { >>>>> ?? 159???????????? unsigned pending; >>>>> ?? 160 >>>>> ?? 161???????????? zr = deflatePending(zstrm, &pending, Z_NULL); >>>>> ?? 162???????????? if (zr != Z_OK) { >>>>> ?? 163???????????????? /* something went wrong */ >>>>> ?? 164???????????????? syslog(LOG_ERR, "zlib deflate error: %d %s", zr, >>>>> zstrm->msg); >>>>> ?? 165???????????????? return -1; >>>>> ?? 166???????????? } >>>>> ?? 167 >>>>> ?? 168???????????? buf_ensure(&txn->zbuf, pending); >>>>> ?? 169???????? } >>>>> >>>>> >>>>> to something like: >>>>> >>>>> ?? 158???????? if (!zstrm->avail_out) { >>>>> ?? 159???????????? buf_ensure(&txn->zbuf, 256 * 1024); >>>>> ?? 160???????? } >>>>> >>>>> If I understand it correctly, deflatePending in this case is only used >>>>> to get the needed buffer size and we could replace it with a hardcoded >>>>> buffer size (like in my example of 256K). >>>>> >>>>> >>>>> Thanks, >>>>> Anatoli -- Kenneth Murchison Senior Software Developer Fastmail US LLC From me at anatoli.ws Wed Jun 3 17:41:35 2020 From: me at anatoli.ws (Anatoli) Date: Wed, 3 Jun 2020 18:41:35 -0300 Subject: run-time dependencies In-Reply-To: References: Message-ID: <3df9bcab-aecc-fbcc-f8d0-db6860883ae4@anatoli.ws> Hi Ken, Thanks for the explanation, though my question was a bit of another scope. Please, let me reformulate it. If we use all these elements during build time, i.e. cyrus-imapd becomes compiled with all of them (when using --enable-http), then... will all of them be also required (as OS packages) during runtime or we can avoid installing some of them on the systems where the cyrus-imapd package would be running? In other words which of them do not constitute runtime dependencies once they were used as build-time dependencies? If all of them are used as shared libraries or some executables invoked by some component of cyrus-imapd, then I guess all of them would be required during runtime. If some of them are only used for building the executables or linked as static libs, then they should not be needed during runtime, I guess, but I'm not sure if there are any. Thanks, Anatoli On 3/6/20 07:39, Ken Murchison wrote: > > On 6/3/20 3:48 AM, Anatoli wrote: >> Cyrus developers, >> >> I couldn't find in the documentation the *run-time* dependencies for >> cyrus-imapd. >> >> In particular, are any of these not required to be present for the >> correct execution of cyrus-imapd (with --enable-http)? >> >> pcre e2fsprogs/uuid jansson sqlite3 icu4c libical nghttpd2 brotli wslay > > > pcre is NOT required for httpd at all as far as I recall > > nghttpd2, brotli, wslay are entirely OPTIONAL > > sqlite3 is required if you want to use any DAV functionality > > libical is required if you want to use CalDAV > > icu4c is required if libical was compiled with support for non-Gregorian calendars > > jansson is definitely required for JMAP functionality, and may be required for other modules > > uuid may be prevasive enough to be required for multiple modules > From partha at fastmailteam.com Wed Jun 3 17:44:57 2020 From: partha at fastmailteam.com (Partha Susarla) Date: Thu, 04 Jun 2020 07:44:57 +1000 Subject: new features with no documentation In-Reply-To: <04ee2b89-7c24-2947-b4f9-8b094f2a9e3f@anatoli.ws> References: <04ee2b89-7c24-2947-b4f9-8b094f2a9e3f@anatoli.ws> Message-ID: Hello Anatoli, On Wed, 3 Jun 2020, at 6:24 PM, Anatoli wrote: > Cyrus developers, > > What is the purpose/benefit of zeroskip? > Zeroskip is an append-only key-value DB, currently in experimental stage. > What is the purpose of chardet and cld2 in cyrus-imapd? > Both the character detection and language detection libraries although are listed in the dependencies, i don't think are being used. But I will let ellie/Ken respond to this. > Should any be considered in production environments running 3.2 branch? > You can skip installing Zeroskip, as it is not being used in Cyrus-imapd currently. ..Partha From murch at fastmail.com Wed Jun 3 17:52:29 2020 From: murch at fastmail.com (Ken Murchison) Date: Wed, 3 Jun 2020 17:52:29 -0400 Subject: run-time dependencies In-Reply-To: <3df9bcab-aecc-fbcc-f8d0-db6860883ae4@anatoli.ws> References: <3df9bcab-aecc-fbcc-f8d0-db6860883ae4@anatoli.ws> Message-ID: <9c5ed80c-eb8f-a15d-ad88-0384de014c04@fastmail.com> Ahh, I didn't read your first email well enough.? If you configure the build so that those libraries are linked, then they will be needed at runtime.? I'm not aware of a way to disable functionality in binary based on what runtime libs are available, but would be willing to make that work with Cyrus is possible (pull requests welcome). On 6/3/20 5:41 PM, Anatoli wrote: > Hi Ken, > > Thanks for the explanation, though my question was a bit of another > scope. Please, let me reformulate it. > > If we use all these elements during build time, i.e. cyrus-imapd becomes > compiled with all of them (when using --enable-http), then... will all > of them be also required (as OS packages) during runtime or we can avoid > installing some of them on the systems where the cyrus-imapd package > would be running? > > In other words which of them do not constitute runtime dependencies once > they were used as build-time dependencies? > > If all of them are used as shared libraries or some executables invoked > by some component of cyrus-imapd, then I guess all of them would be > required during runtime. > > If some of them are only used for building the executables or linked as > static libs, then they should not be needed during runtime, I guess, but > I'm not sure if there are any. > > Thanks, > Anatoli > > On 3/6/20 07:39, Ken Murchison wrote: >> On 6/3/20 3:48 AM, Anatoli wrote: >>> Cyrus developers, >>> >>> I couldn't find in the documentation the *run-time* dependencies for >>> cyrus-imapd. >>> >>> In particular, are any of these not required to be present for the >>> correct execution of cyrus-imapd (with --enable-http)? >>> >>> pcre e2fsprogs/uuid jansson sqlite3 icu4c libical nghttpd2 brotli wslay >> >> pcre is NOT required for httpd at all as far as I recall >> >> nghttpd2, brotli, wslay are entirely OPTIONAL >> >> sqlite3 is required if you want to use any DAV functionality >> >> libical is required if you want to use CalDAV >> >> icu4c is required if libical was compiled with support for non-Gregorian calendars >> >> jansson is definitely required for JMAP functionality, and may be required for other modules >> >> uuid may be prevasive enough to be required for multiple modules >> -- Kenneth Murchison Senior Software Developer Fastmail US LLC From murch at fastmail.com Wed Jun 3 18:01:01 2020 From: murch at fastmail.com (Ken Murchison) Date: Wed, 3 Jun 2020 18:01:01 -0400 Subject: new features with no documentation In-Reply-To: References: <04ee2b89-7c24-2947-b4f9-8b094f2a9e3f@anatoli.ws> Message-ID: On 6/3/20 5:44 PM, Partha Susarla wrote: > Hello Anatoli, > On Wed, 3 Jun 2020, at 6:24 PM, Anatoli wrote: >> Cyrus developers, >> >> What is the purpose/benefit of zeroskip? >> > Zeroskip is an append-only key-value DB, currently in experimental stage. > >> What is the purpose of chardet and cld2 in cyrus-imapd? >> > Both the character detection and language detection libraries although are listed in the dependencies, i don't think are being used. But I will let ellie/Ken respond to this. chardet is used by the JMAP module of httpd to detect the character set of untagged 8-bit headers. cld2 is also used by JMAP for language detection but it doesn't look like its being used by any code on the master branch yet.? I believe that Robert S. has a dev branch that uses it.? We can't ask him right now because he's on holiday the rest of June. -- Kenneth Murchison Senior Software Developer Fastmail US LLC From me at anatoli.ws Wed Jun 3 20:18:12 2020 From: me at anatoli.ws (Anatoli) Date: Wed, 3 Jun 2020 21:18:12 -0300 Subject: configure: wslay v1.1.1 required but the latest one is 1.1.0 In-Reply-To: References: <18e0a63d-31bf-45b5-a001-efc92237ba16@www.fastmail.com> <810d8452-edd0-7128-4071-444cbf324d8d@anatoli.ws> Message-ID: <711a3731-39fa-0fcb-2b57-6da08b372dc0@anatoli.ws> Ken, Why do you believe it could be an issue with Cyrus? It appears the fix was commited 4 years ago and that part was revorked later, so it should not be a problem anymore in 1.1.0. On 3/6/20 07:44, Ken Murchison wrote: > Yes, 1.1.0 is probably sufficient, unless this bug is an issue with Cyrus: https://github.com/tatsuhiro-t/wslay/pull/47 > > > On 6/3/20 1:19 AM, ellie timoney wrote: >> Cool, thanks for confirming that.? So far it's sounding like 1.1.0 is probably adequate, but I'll wait a little bit to see if Ken has any input once he's been online >> >> On Wed, Jun 3, 2020, at 2:58 PM, Anatoli wrote: >>> Hi Ellie, >>> >>>> When you configure it with your patch applied and 1.1.0 installed, >>> does Cyrus build okay? >>> >>> Yes, it builds without errors. >>> >>> Configure prints: >>> >>> checking for WSLAY... yes >>> ??? wslay:????????????? yes >>> >>> And -lwslay is passed as arg numerous times during build process. >>> >>> And effectively httpd binary includes references to wslay_event_xxx in >>> its symbols table. >>> >>> Regards, >>> Anatoli >>> >>> On 3/6/20 01:35, ellie timoney wrote: >>>> In our "cyruslibs" package, the wslay submodule is at this commit: >>>> >>>> commit 4a937cd (HEAD, origin/master, origin/HEAD, master) >>>> Author:???? Tatsuhiro Tsujikawa >>>> AuthorDate: Fri Jun 8 23:19:03 2018 +0900 >>>> Commit:???? Tatsuhiro Tsujikawa >>>> CommitDate: Fri Jun 8 23:19:03 2018 +0900 >>>> >>>> ???? Bump up version number to 1.1.1-DEV >>>> >>>> Which is the commit immediately following the release-1.1.0 tag.? So, presumably, we're not dependent on any feature/fix that's only in the unreleased version, because otherwise we would've bumped the cyruslibs submodule to include those commits? >>>> >>>> So, "1.1.1" might be a typo, or an anticipatory thing that didn't go anywhere yet, I'm not sure. >>>> >>>> When you configure it with your patch applied and 1.1.0 installed, does Cyrus build okay? >>>> >>>> Cheers, >>>> >>>> ellie >>>> >>>> On Wed, Jun 3, 2020, at 2:13 PM, Anatoli wrote: >>>>> Cyrus developers, >>>>> >>>>> The configure script checks for wslay lib version 1.1.1, but the latest >>>>> version released is 1.1.0. So when it is installed, it reports: >>>>> >>>>> checking for WSLAY... no >>>>> configure: httpd will not have support for WebSockets.? Consider >>>>> installing libwslay >>>>> >>>>> The wslay's github repo has a mention of a 1.1.1-DEV version. Not sure >>>>> if cyrus-imapd httpd requires something from it or if it was just a >>>>> typo and 1.1.0 is ok. >>>>> >>>>> For the later case below is a patch. >>>>> >>>>> Regards, >>>>> Anatoli >>>>> >>>>> >>>>> diff --git a/configure.ac b/configure.ac >>>>> index dc0e0fff2..30e925c60 100644 >>>>> --- a/configure.ac >>>>> +++ b/configure.ac >>>>> @@ -1631,7 +1631,7 @@ dnl??????????????? AC_MSG_WARN([Your version of >>>>> OpenDKIM can not support iSchedu >>>>> >>>>> ????????? AC_ARG_WITH(wslay, [AS_HELP_STRING([--without-wslay], [disable >>>>> WebSockets support (check)])],,[with_wslay="check"]) >>>>> ????????? if test "x$with_wslay" = "xyes" -o "x$with_wslay" = "xcheck"; >>>>> then >>>>> -??????????????? PKG_CHECK_MODULES([WSLAY], [libwslay >= 1.1.1], [ >>>>> +??????????????? PKG_CHECK_MODULES([WSLAY], [libwslay >= 1.1.0], [ >>>>> ????????????????????????? AC_DEFINE(HAVE_WSLAY,[], >>>>> ????????????????????????????????? [Build WebSockets support into httpd?]) >>>>> ????????????????????????? with_wslay=yes >>>>> From me at anatoli.ws Wed Jun 3 22:36:50 2020 From: me at anatoli.ws (Anatoli) Date: Wed, 3 Jun 2020 23:36:50 -0300 Subject: run-time dependencies In-Reply-To: <9c5ed80c-eb8f-a15d-ad88-0384de014c04@fastmail.com> References: <3df9bcab-aecc-fbcc-f8d0-db6860883ae4@anatoli.ws> <9c5ed80c-eb8f-a15d-ad88-0384de014c04@fastmail.com> Message-ID: Thanks, Ken, it's clear now. I wasn't intending to make it adaptable at run-time, just wasn't sure all of them were used as dynamic libraries. I'm accommodating cyrus-imapd package to 3.2 + http, just wanted to confirm this to specify the RUN_DEPENDS list correctly. On 3/6/20 18:52, Ken Murchison wrote: > Ahh, I didn't read your first email well enough.? If you configure the build so that those libraries are linked, then they will be needed at runtime.? I'm not aware of a way to disable functionality in binary based on what runtime libs are available, but would be willing to make that work with Cyrus is possible (pull requests welcome). > > > On 6/3/20 5:41 PM, Anatoli wrote: >> Hi Ken, >> >> Thanks for the explanation, though my question was a bit of another >> scope. Please, let me reformulate it. >> >> If we use all these elements during build time, i.e. cyrus-imapd becomes >> compiled with all of them (when using --enable-http), then... will all >> of them be also required (as OS packages) during runtime or we can avoid >> installing some of them on the systems where the cyrus-imapd package >> would be running? >> >> In other words which of them do not constitute runtime dependencies once >> they were used as build-time dependencies? >> >> If all of them are used as shared libraries or some executables invoked >> by some component of cyrus-imapd, then I guess all of them would be >> required during runtime. >> >> If some of them are only used for building the executables or linked as >> static libs, then they should not be needed during runtime, I guess, but >> I'm not sure if there are any. >> >> Thanks, >> Anatoli >> >> On 3/6/20 07:39, Ken Murchison wrote: >>> On 6/3/20 3:48 AM, Anatoli wrote: >>>> Cyrus developers, >>>> >>>> I couldn't find in the documentation the *run-time* dependencies for >>>> cyrus-imapd. >>>> >>>> In particular, are any of these not required to be present for the >>>> correct execution of cyrus-imapd (with --enable-http)? >>>> >>>> pcre e2fsprogs/uuid jansson sqlite3 icu4c libical nghttpd2 brotli wslay >>> >>> pcre is NOT required for httpd at all as far as I recall >>> >>> nghttpd2, brotli, wslay are entirely OPTIONAL >>> >>> sqlite3 is required if you want to use any DAV functionality >>> >>> libical is required if you want to use CalDAV >>> >>> icu4c is required if libical was compiled with support for non-Gregorian calendars >>> >>> jansson is definitely required for JMAP functionality, and may be required for other modules >>> >>> uuid may be prevasive enough to be required for multiple modules >>> From me at anatoli.ws Wed Jun 3 22:52:46 2020 From: me at anatoli.ws (Anatoli) Date: Wed, 3 Jun 2020 23:52:46 -0300 Subject: new features with no documentation In-Reply-To: References: <04ee2b89-7c24-2947-b4f9-8b094f2a9e3f@anatoli.ws> Message-ID: Ken, Partha, Thanks for the explanation. > chardet is used by the JMAP module of httpd to detect the character set of untagged 8-bit headers. Does that mean that these libs are required? If not, what would happen if not included? How Cyrus would detect the charset? Or put in other words, if they are not required, what's the benefit to use them? On the other hand, may I suggest to please add these details to the Compiling [1] page? There's already a lot of useful information, but some of the libs lack any details on what's their purpose in Cyrus and why they are beneficial/required (and if they are not used yet, maybe they shouldn't be mentioned there at all?). This could help porters and maintainers to make more educated decisions on how to package Cyrus. Thanks, Anatoli [1] https://www.cyrusimap.org/imap/developer/compiling.html On 3/6/20 19:01, Ken Murchison wrote: > > On 6/3/20 5:44 PM, Partha Susarla wrote: >> Hello Anatoli, >> On Wed, 3 Jun 2020, at 6:24 PM, Anatoli wrote: >>> Cyrus developers, >>> >>> What is the purpose/benefit of zeroskip? >>> >> Zeroskip is an append-only key-value DB, currently in experimental stage. >> >>> What is the purpose of chardet and cld2 in cyrus-imapd? >>> >> Both the character detection and language detection libraries although are listed in the dependencies, i don't think are being used. But I will let ellie/Ken respond to this. > > > chardet is used by the JMAP module of httpd to detect the character set of untagged 8-bit headers. > > cld2 is also used by JMAP for language detection but it doesn't look like its being used by any code on the master branch yet.? I believe that Robert S. has a dev branch that uses it.? We can't ask him right now because he's on holiday the rest of June. > > From dilyan.palauzov at aegee.org Thu Jun 4 08:19:20 2020 From: dilyan.palauzov at aegee.org (=?UTF-8?Q?=D0=94=D0=B8=D0=BB=D1=8F=D0=BD_?= =?UTF-8?Q?=D0=9F=D0=B0=D0=BB=D0=B0=D1=83=D0=B7=D0=BE=D0=B2?=) Date: Thu, 04 Jun 2020 12:19:20 +0000 Subject: Two communication channels Message-ID: <492ea2be89f5315b28a5e136115767ce14048dcd.camel@aegee.org> Hello, for Cyrus Imap there are now two communication channels: cyrus-devel at lists.andrew.cmu.edu and https://github.com/cyrusimap/cyrus-imapd/issues . Based on what criteria shall one use the one or the other channel, to get the things tackled with higher probability? Yesterday there were questions on cyrus-devel concerning the role of pcre and zeroskip. The dependency on pcre is discussed at https://github.com/cyrusimap/cyrus-imapd/issues/2891 : as far as I remember pcre is supposed to be used for UTF-8. The zeroskip state is discussed at https://github.com/cyrusimap/cyrus-imapd/issues/3007#issuecomment-597431775 . I propose that somebody unifies these comminication channels, e.g. by posting everything from github to cyrus-devel or alike. This will lead to some limited extend avoiding discussing the same thing on github and on cyrus-devel. Opinions welcome. Greetings ????? From me at anatoli.ws Thu Jun 4 18:57:59 2020 From: me at anatoli.ws (Anatoli) Date: Thu, 4 Jun 2020 19:57:59 -0300 Subject: Two communication channels In-Reply-To: <492ea2be89f5315b28a5e136115767ce14048dcd.camel@aegee.org> References: <492ea2be89f5315b28a5e136115767ce14048dcd.camel@aegee.org> Message-ID: Hi All, > for Cyrus Imap there are now two communication channels: > cyrus-devel at lists.andrew.cmu.edu and > https://github.com/cyrusimap/cyrus-imapd/issues . Based on what > criteria shall one use the one or the other channel, to get the things > tackled with higher probability? I guess it's outlined in the Contribute code docs [1], maybe not that explicitly, but.. TL;DR: communicate here, send patches to small fixes here, larger contributions go to GitHub (with issue describing the problem first, then a mail here with the link to the PR). What I do believe (and what I have commented in the other mail) is that it would be beneficial to keep the dev docs updated and with enough detail, i.e. summarizing all the conversations here and on GitHub. Also, replying in a reasonable time to all the PRs would encourage more contributions I believe. Aaand.. seizing the opportunity and as 3.2 got its own branch, any chance to close issues/1765 (remove SNMP) [2] by merging pull/2100 [3]? This would open the road to work on chroot & other security improvements. And it would be nice to finally decide/proceed on issues/1731 [4], what ????? has mentioned (PCRE). Regards, Anatoli [1] https://www.cyrusimap.org/imap/developer/process.html [2] https://github.com/cyrusimap/cyrus-imapd/issues/1765 [3] https://github.com/cyrusimap/cyrus-imapd/pull/2100 [4] https://github.com/cyrusimap/cyrus-imapd/issues/1731 On 4/6/20 09:19, ????? ???????? wrote: > Hello, > > for Cyrus Imap there are now two communication channels: > cyrus-devel at lists.andrew.cmu.edu and > https://github.com/cyrusimap/cyrus-imapd/issues . Based on what > criteria shall one use the one or the other channel, to get the things > tackled with higher probability? > > Yesterday there were questions on cyrus-devel concerning the role of > pcre and zeroskip. > > The dependency on pcre is discussed at > https://github.com/cyrusimap/cyrus-imapd/issues/2891 : as far as I > remember pcre is supposed to be used for UTF-8. > > The zeroskip state is discussed at > https://github.com/cyrusimap/cyrus-imapd/issues/3007#issuecomment-597431775 > . > > I propose that somebody unifies these comminication channels, e.g. by > posting everything from github to cyrus-devel or alike. This will lead > to some limited extend avoiding discussing the same thing on github and > on cyrus-devel. > > Opinions welcome. > > Greetings > ????? > From ellie at fastmail.com Thu Jun 4 20:10:32 2020 From: ellie at fastmail.com (ellie timoney) Date: Fri, 05 Jun 2020 10:10:32 +1000 Subject: configure: wslay v1.1.1 required but the latest one is 1.1.0 In-Reply-To: <711a3731-39fa-0fcb-2b57-6da08b372dc0@anatoli.ws> References: <18e0a63d-31bf-45b5-a001-efc92237ba16@www.fastmail.com> <810d8452-edd0-7128-4071-444cbf324d8d@anatoli.ws> <711a3731-39fa-0fcb-2b57-6da08b372dc0@anatoli.ws> Message-ID: <6fd3855a-d422-4ed7-838e-09efad88105c@www.fastmail.com> The commit was authored in 2015, but the pull request was only merged to wslay master last November. The fix is not in 1.1.0, but it's also not in our cyruslibs version (unless my submodule is out of date, which it might be, but I thought I updated it correctly the other day). So, since the fix is not in 1.1.0, that would be a problem for Cyrus if Cyrus depends on the fix being there. I don't know whether it does, or how to test it. On Thu, Jun 4, 2020, at 10:18 AM, Anatoli wrote: > Ken, > > Why do you believe it could be an issue with Cyrus? It appears the fix > was commited 4 years ago and that part was revorked later, so it should > not be a problem anymore in 1.1.0. > > > On 3/6/20 07:44, Ken Murchison wrote: > > Yes, 1.1.0 is probably sufficient, unless this bug is an issue with Cyrus: https://github.com/tatsuhiro-t/wslay/pull/47 > > > > > > On 6/3/20 1:19 AM, ellie timoney wrote: > >> Cool, thanks for confirming that.? So far it's sounding like 1.1.0 is probably adequate, but I'll wait a little bit to see if Ken has any input once he's been online > >> > >> On Wed, Jun 3, 2020, at 2:58 PM, Anatoli wrote: > >>> Hi Ellie, > >>> > >>>> When you configure it with your patch applied and 1.1.0 installed, > >>> does Cyrus build okay? > >>> > >>> Yes, it builds without errors. > >>> > >>> Configure prints: > >>> > >>> checking for WSLAY... yes > >>> ??? wslay:????????????? yes > >>> > >>> And -lwslay is passed as arg numerous times during build process. > >>> > >>> And effectively httpd binary includes references to wslay_event_xxx in > >>> its symbols table. > >>> > >>> Regards, > >>> Anatoli > >>> > >>> On 3/6/20 01:35, ellie timoney wrote: > >>>> In our "cyruslibs" package, the wslay submodule is at this commit: > >>>> > >>>> commit 4a937cd (HEAD, origin/master, origin/HEAD, master) > >>>> Author:???? Tatsuhiro Tsujikawa > >>>> AuthorDate: Fri Jun 8 23:19:03 2018 +0900 > >>>> Commit:???? Tatsuhiro Tsujikawa > >>>> CommitDate: Fri Jun 8 23:19:03 2018 +0900 > >>>> > >>>> ???? Bump up version number to 1.1.1-DEV > >>>> > >>>> Which is the commit immediately following the release-1.1.0 tag.? So, presumably, we're not dependent on any feature/fix that's only in the unreleased version, because otherwise we would've bumped the cyruslibs submodule to include those commits? > >>>> > >>>> So, "1.1.1" might be a typo, or an anticipatory thing that didn't go anywhere yet, I'm not sure. > >>>> > >>>> When you configure it with your patch applied and 1.1.0 installed, does Cyrus build okay? > >>>> > >>>> Cheers, > >>>> > >>>> ellie > >>>> > >>>> On Wed, Jun 3, 2020, at 2:13 PM, Anatoli wrote: > >>>>> Cyrus developers, > >>>>> > >>>>> The configure script checks for wslay lib version 1.1.1, but the latest > >>>>> version released is 1.1.0. So when it is installed, it reports: > >>>>> > >>>>> checking for WSLAY... no > >>>>> configure: httpd will not have support for WebSockets.? Consider > >>>>> installing libwslay > >>>>> > >>>>> The wslay's github repo has a mention of a 1.1.1-DEV version. Not sure > >>>>> if cyrus-imapd httpd requires something from it or if it was just a > >>>>> typo and 1.1.0 is ok. > >>>>> > >>>>> For the later case below is a patch. > >>>>> > >>>>> Regards, > >>>>> Anatoli > >>>>> > >>>>> > >>>>> diff --git a/configure.ac b/configure.ac > >>>>> index dc0e0fff2..30e925c60 100644 > >>>>> --- a/configure.ac > >>>>> +++ b/configure.ac > >>>>> @@ -1631,7 +1631,7 @@ dnl??????????????? AC_MSG_WARN([Your version of > >>>>> OpenDKIM can not support iSchedu > >>>>> > >>>>> ????????? AC_ARG_WITH(wslay, [AS_HELP_STRING([--without-wslay], [disable > >>>>> WebSockets support (check)])],,[with_wslay="check"]) > >>>>> ????????? if test "x$with_wslay" = "xyes" -o "x$with_wslay" = "xcheck"; > >>>>> then > >>>>> -??????????????? PKG_CHECK_MODULES([WSLAY], [libwslay >= 1.1.1], [ > >>>>> +??????????????? PKG_CHECK_MODULES([WSLAY], [libwslay >= 1.1.0], [ > >>>>> ????????????????????????? AC_DEFINE(HAVE_WSLAY,[], > >>>>> ????????????????????????????????? [Build WebSockets support into httpd?]) > >>>>> ????????????????????????? with_wslay=yes > >>>>> > From ellie at fastmail.com Thu Jun 4 20:58:18 2020 From: ellie at fastmail.com (ellie timoney) Date: Fri, 05 Jun 2020 10:58:18 +1000 Subject: new features with no documentation In-Reply-To: References: <04ee2b89-7c24-2947-b4f9-8b094f2a9e3f@anatoli.ws> Message-ID: <88cce492-632e-439a-ac7b-f68ef9c3381c@www.fastmail.com> On Thu, Jun 4, 2020, at 12:52 PM, Anatoli wrote: > > chardet is used by the JMAP module of httpd to detect the character > set of untagged 8-bit headers. > > Does that mean that these libs are required? If not, what would happen > if not included? How Cyrus would detect the charset? Or put in other > words, if they are not required, what's the benefit to use them? > > On the other hand, may I suggest to please add these details to the > Compiling [1] page? There's already a lot of useful information, but > some of the libs lack any details on what's their purpose in Cyrus and > why they are beneficial/required (and if they are not used yet, maybe > they shouldn't be mentioned there at all?). This could help porters and > maintainers to make more educated decisions on how to package Cyrus. libchardet is already listed on the compiling page as being used by the CalDAV, CardDAV, and/or JMAP features. You would decide to install libchardet based on whether you need these features: if you don't enable these features, you don't need it; if you do, you probably do. If you don't have libchardet, Cyrus will not do character-set detection. If some piece of data has no character set coming in, it will have no character set. I don't know why you would decide to not use this, and it may simply become a mandatory requirement for these features in 3.4. It might have been mandatory in 3.2, if this had been brought to our attention during the nearly 3 month beta window; but now that it's in a stable release, it will remain as it is. cld2 was used by one (or maybe both) of the experimental search features that were accidentally included in 3.2.0, and removed in 3.2.1 (see the release notes). Looks like we forgot to remove the configure checks for the library when removing the feature(s) that used it. You don't need it, and if you have it, it won't be used anyway. Cheers, ellie From ellie at fastmail.com Thu Jun 4 21:09:55 2020 From: ellie at fastmail.com (ellie timoney) Date: Fri, 05 Jun 2020 11:09:55 +1000 Subject: run-time dependencies In-Reply-To: References: Message-ID: <8d01ebd1-7918-4fd9-a592-2e44ba19f630@www.fastmail.com> The build-time dependencies are listed on the Compiling page under the "Developers only" heading: https://www.cyrusimap.org/imap/developer/compiling.html#developers-only Additionally, these dependencies are only required when building from git sources. In our official release files (with filename like cyrus-imapd-[version].tar.gz), the files they are needed for are pre-built and included in the package, and therefore these dependencies are generally not needed. You can safely assume that every other dependency is both build-time AND run-time. :) Cheers, ellie On Wed, Jun 3, 2020, at 5:48 PM, Anatoli wrote: > Cyrus developers, > > I couldn't find in the documentation the *run-time* dependencies for > cyrus-imapd. > > In particular, are any of these not required to be present for the > correct execution of cyrus-imapd (with --enable-http)? > > pcre e2fsprogs/uuid jansson sqlite3 icu4c libical nghttpd2 brotli wslay > > Thanks, > Anatoli > From me at anatoli.ws Fri Jun 5 04:11:25 2020 From: me at anatoli.ws (Anatoli) Date: Fri, 5 Jun 2020 05:11:25 -0300 Subject: configure: wslay v1.1.1 required but the latest one is 1.1.0 In-Reply-To: <6fd3855a-d422-4ed7-838e-09efad88105c@www.fastmail.com> References: <18e0a63d-31bf-45b5-a001-efc92237ba16@www.fastmail.com> <810d8452-edd0-7128-4071-444cbf324d8d@anatoli.ws> <711a3731-39fa-0fcb-2b57-6da08b372dc0@anatoli.ws> <6fd3855a-d422-4ed7-838e-09efad88105c@www.fastmail.com> Message-ID: <7c8025e2-7718-b7e4-dcf3-21d9228bb0a0@anatoli.ws> Ellie, You're right, I haven't checked well the dates! I just asked the author of Wslay if he could tag a new release. Hope we get an answer soon. Regards, Anatoli On 4/6/20 21:10, ellie timoney wrote: > The commit was authored in 2015, but the pull request was only merged to wslay master last November. The fix is not in 1.1.0, but it's also not in our cyruslibs version (unless my submodule is out of date, which it might be, but I thought I updated it correctly the other day). > > So, since the fix is not in 1.1.0, that would be a problem for Cyrus if Cyrus depends on the fix being there. I don't know whether it does, or how to test it. > > On Thu, Jun 4, 2020, at 10:18 AM, Anatoli wrote: >> Ken, >> >> Why do you believe it could be an issue with Cyrus? It appears the fix >> was commited 4 years ago and that part was revorked later, so it should >> not be a problem anymore in 1.1.0. >> >> >> On 3/6/20 07:44, Ken Murchison wrote: >>> Yes, 1.1.0 is probably sufficient, unless this bug is an issue with Cyrus: https://github.com/tatsuhiro-t/wslay/pull/47 >>> >>> >>> On 6/3/20 1:19 AM, ellie timoney wrote: >>>> Cool, thanks for confirming that.? So far it's sounding like 1.1.0 is probably adequate, but I'll wait a little bit to see if Ken has any input once he's been online >>>> >>>> On Wed, Jun 3, 2020, at 2:58 PM, Anatoli wrote: >>>>> Hi Ellie, >>>>> >>>>>> When you configure it with your patch applied and 1.1.0 installed, >>>>> does Cyrus build okay? >>>>> >>>>> Yes, it builds without errors. >>>>> >>>>> Configure prints: >>>>> >>>>> checking for WSLAY... yes >>>>> ??? wslay:????????????? yes >>>>> >>>>> And -lwslay is passed as arg numerous times during build process. >>>>> >>>>> And effectively httpd binary includes references to wslay_event_xxx in >>>>> its symbols table. >>>>> >>>>> Regards, >>>>> Anatoli >>>>> >>>>> On 3/6/20 01:35, ellie timoney wrote: >>>>>> In our "cyruslibs" package, the wslay submodule is at this commit: >>>>>> >>>>>> commit 4a937cd (HEAD, origin/master, origin/HEAD, master) >>>>>> Author:???? Tatsuhiro Tsujikawa >>>>>> AuthorDate: Fri Jun 8 23:19:03 2018 +0900 >>>>>> Commit:???? Tatsuhiro Tsujikawa >>>>>> CommitDate: Fri Jun 8 23:19:03 2018 +0900 >>>>>> >>>>>> ???? Bump up version number to 1.1.1-DEV >>>>>> >>>>>> Which is the commit immediately following the release-1.1.0 tag.? So, presumably, we're not dependent on any feature/fix that's only in the unreleased version, because otherwise we would've bumped the cyruslibs submodule to include those commits? >>>>>> >>>>>> So, "1.1.1" might be a typo, or an anticipatory thing that didn't go anywhere yet, I'm not sure. >>>>>> >>>>>> When you configure it with your patch applied and 1.1.0 installed, does Cyrus build okay? >>>>>> >>>>>> Cheers, >>>>>> >>>>>> ellie >>>>>> >>>>>> On Wed, Jun 3, 2020, at 2:13 PM, Anatoli wrote: >>>>>>> Cyrus developers, >>>>>>> >>>>>>> The configure script checks for wslay lib version 1.1.1, but the latest >>>>>>> version released is 1.1.0. So when it is installed, it reports: >>>>>>> >>>>>>> checking for WSLAY... no >>>>>>> configure: httpd will not have support for WebSockets.? Consider >>>>>>> installing libwslay >>>>>>> >>>>>>> The wslay's github repo has a mention of a 1.1.1-DEV version. Not sure >>>>>>> if cyrus-imapd httpd requires something from it or if it was just a >>>>>>> typo and 1.1.0 is ok. >>>>>>> >>>>>>> For the later case below is a patch. >>>>>>> >>>>>>> Regards, >>>>>>> Anatoli >>>>>>> >>>>>>> >>>>>>> diff --git a/configure.ac b/configure.ac >>>>>>> index dc0e0fff2..30e925c60 100644 >>>>>>> --- a/configure.ac >>>>>>> +++ b/configure.ac >>>>>>> @@ -1631,7 +1631,7 @@ dnl??????????????? AC_MSG_WARN([Your version of >>>>>>> OpenDKIM can not support iSchedu >>>>>>> >>>>>>> ????????? AC_ARG_WITH(wslay, [AS_HELP_STRING([--without-wslay], [disable >>>>>>> WebSockets support (check)])],,[with_wslay="check"]) >>>>>>> ????????? if test "x$with_wslay" = "xyes" -o "x$with_wslay" = "xcheck"; >>>>>>> then >>>>>>> -??????????????? PKG_CHECK_MODULES([WSLAY], [libwslay >= 1.1.1], [ >>>>>>> +??????????????? PKG_CHECK_MODULES([WSLAY], [libwslay >= 1.1.0], [ >>>>>>> ????????????????????????? AC_DEFINE(HAVE_WSLAY,[], >>>>>>> ????????????????????????????????? [Build WebSockets support into httpd?]) >>>>>>> ????????????????????????? with_wslay=yes >>>>>>> >> From me at anatoli.ws Fri Jun 5 04:30:16 2020 From: me at anatoli.ws (Anatoli) Date: Fri, 5 Jun 2020 05:30:16 -0300 Subject: new features with no documentation In-Reply-To: <88cce492-632e-439a-ac7b-f68ef9c3381c@www.fastmail.com> References: <04ee2b89-7c24-2947-b4f9-8b094f2a9e3f@anatoli.ws> <88cce492-632e-439a-ac7b-f68ef9c3381c@www.fastmail.com> Message-ID: <5427d59b-503b-5ba6-76e9-ba66f853262c@anatoli.ws> Ellie, > libchardet is already listed on the compiling page as being used by > the CalDAV, CardDAV, and/or JMAP features. You would decide to > install libchardet based on whether you need these features: if you > don't enable these features, you don't need it; if you do, you > probably do. > > If you don't have libchardet, Cyrus will not do character-set > detection. If some piece of data has no character set coming in, it > will have no character set. I don't know why you would decide to not > use this, and it may simply become a mandatory requirement for these > features in 3.4. It might have been mandatory in 3.2, if this had > been brought to our attention during the nearly 3 month beta window; > but now that it's in a stable release, it will remain as it is. Thanks for the explanation. What happens is that at least Cal/CardDAV compile and work well without libchardet (at least under normal circumstances) and, as it was not marked as required, I thought it's an optional lib and here is the question of why to use it. I guess what you could do for 3.2 is to include a warning in configure explaining the issue, so the maintainers see it and take appropriate action. I suppose there are not that many installations with 3.2 yet so if the warning is included with 3.2.2 it would be an appropriate time. Regards, Anatoli On 4/6/20 21:58, ellie timoney wrote: > On Thu, Jun 4, 2020, at 12:52 PM, Anatoli wrote: >>> chardet is used by the JMAP module of httpd to detect the character >> set of untagged 8-bit headers. >> >> Does that mean that these libs are required? If not, what would happen >> if not included? How Cyrus would detect the charset? Or put in other >> words, if they are not required, what's the benefit to use them? >> >> On the other hand, may I suggest to please add these details to the >> Compiling [1] page? There's already a lot of useful information, but >> some of the libs lack any details on what's their purpose in Cyrus and >> why they are beneficial/required (and if they are not used yet, maybe >> they shouldn't be mentioned there at all?). This could help porters and >> maintainers to make more educated decisions on how to package Cyrus. > > libchardet is already listed on the compiling page as being used by the CalDAV, CardDAV, and/or JMAP features. You would decide to install libchardet based on whether you need these features: if you don't enable these features, you don't need it; if you do, you probably do. > > If you don't have libchardet, Cyrus will not do character-set detection. If some piece of data has no character set coming in, it will have no character set. I don't know why you would decide to not use this, and it may simply become a mandatory requirement for these features in 3.4. It might have been mandatory in 3.2, if this had been brought to our attention during the nearly 3 month beta window; but now that it's in a stable release, it will remain as it is. > > cld2 was used by one (or maybe both) of the experimental search features that were accidentally included in 3.2.0, and removed in 3.2.1 (see the release notes). Looks like we forgot to remove the configure checks for the library when removing the feature(s) that used it. You don't need it, and if you have it, it won't be used anyway. > > Cheers, > > ellie > From me at anatoli.ws Fri Jun 5 04:45:47 2020 From: me at anatoli.ws (Anatoli) Date: Fri, 5 Jun 2020 05:45:47 -0300 Subject: new features with no documentation In-Reply-To: <5427d59b-503b-5ba6-76e9-ba66f853262c@anatoli.ws> References: <04ee2b89-7c24-2947-b4f9-8b094f2a9e3f@anatoli.ws> <88cce492-632e-439a-ac7b-f68ef9c3381c@www.fastmail.com> <5427d59b-503b-5ba6-76e9-ba66f853262c@anatoli.ws> Message-ID: <20132e86-e439-8821-347e-b1adfb56ee24@anatoli.ws> > Thanks for the explanation. What happens is that at least Cal/CardDAV > compile and work well without libchardet (at least under normal > circumstances) and, as it was not marked as required, I thought it's an > optional lib and here is the question of why to use it. Actually, the same happens with transfig, libsrs2 and shapelib. The page mentiones these libs, but "Required for `make check`" (not sure what that means in relation to the lib being reqiured for cyrus-imapd as such) for all of them is "no". What would help is a column "required" yes/no and another column "usage in cyrus-imapd" explaining for what exactly it's used. Say if it's for tzdata processing in the TZDist module, then I probably won't build with it as I don't see much use for it (or maybe package it as a flavor). But if it's something essential for a broader functionality like CalDAV itself, then definitely it would be included. But at this moment the Compiling page doesn't provide these details and here we are. On 5/6/20 05:30, Anatoli wrote: > Ellie, > >> libchardet is already listed on the compiling page as being used by >> the CalDAV, CardDAV, and/or JMAP features. You would decide to >> install libchardet based on whether you need these features: if you >> don't enable these features, you don't need it; if you do, you >> probably do. >> >> If you don't have libchardet, Cyrus will not do character-set >> detection. If some piece of data has no character set coming in, it >> will have no character set. I don't know why you would decide to not >> use this, and it may simply become a mandatory requirement for these >> features in 3.4. It might have been mandatory in 3.2, if this had >> been brought to our attention during the nearly 3 month beta window; >> but now that it's in a stable release, it will remain as it is. > > Thanks for the explanation. What happens is that at least Cal/CardDAV > compile and work well without libchardet (at least under normal > circumstances) and, as it was not marked as required, I thought it's an > optional lib and here is the question of why to use it. > > I guess what you could do for 3.2 is to include a warning in configure > explaining the issue, so the maintainers see it and take appropriate > action. I suppose there are not that many installations with 3.2 yet so > if the warning is included with 3.2.2 it would be an appropriate time. > > Regards, > Anatoli > > On 4/6/20 21:58, ellie timoney wrote: >> On Thu, Jun 4, 2020, at 12:52 PM, Anatoli wrote: >>>> chardet is used by the JMAP module of httpd to detect the character >>> set of untagged 8-bit headers. >>> >>> Does that mean that these libs are required? If not, what would happen >>> if not included? How Cyrus would detect the charset? Or put in other >>> words, if they are not required, what's the benefit to use them? >>> >>> On the other hand, may I suggest to please add these details to the >>> Compiling [1] page? There's already a lot of useful information, but >>> some of the libs lack any details on what's their purpose in Cyrus and >>> why they are beneficial/required (and if they are not used yet, maybe >>> they shouldn't be mentioned there at all?). This could help porters and >>> maintainers to make more educated decisions on how to package Cyrus. >> >> libchardet is already listed on the compiling page as being used by the CalDAV, CardDAV, and/or JMAP features. You would decide to install libchardet based on whether you need these features: if you don't enable these features, you don't need it; if you do, you probably do. >> >> If you don't have libchardet, Cyrus will not do character-set detection. If some piece of data has no character set coming in, it will have no character set. I don't know why you would decide to not use this, and it may simply become a mandatory requirement for these features in 3.4. It might have been mandatory in 3.2, if this had been brought to our attention during the nearly 3 month beta window; but now that it's in a stable release, it will remain as it is. >> >> cld2 was used by one (or maybe both) of the experimental search features that were accidentally included in 3.2.0, and removed in 3.2.1 (see the release notes). Looks like we forgot to remove the configure checks for the library when removing the feature(s) that used it. You don't need it, and if you have it, it won't be used anyway. >> >> Cheers, >> >> ellie >> From me at anatoli.ws Thu Jun 18 22:50:24 2020 From: me at anatoli.ws (Anatoli) Date: Thu, 18 Jun 2020 23:50:24 -0300 Subject: new features with no documentation In-Reply-To: <20132e86-e439-8821-347e-b1adfb56ee24@anatoli.ws> References: <04ee2b89-7c24-2947-b4f9-8b094f2a9e3f@anatoli.ws> <88cce492-632e-439a-ac7b-f68ef9c3381c@www.fastmail.com> <5427d59b-503b-5ba6-76e9-ba66f853262c@anatoli.ws> <20132e86-e439-8821-347e-b1adfb56ee24@anatoli.ws> Message-ID: <3e1f0f55-4fc9-b7ae-069f-7b3f453dfda1@anatoli.ws> Ellie (or anyone else, Ken?), Could you please let us know what's the purpose of transfig, libsrs2 and shapelib libs in cyrus-imapd, and if they are required for any functionality (and if not (i.e. they are optional), what's their benefit)? Also, Ellie, would it be useful to add to configure checks with warnings for these libs (as well as for chardet) to inform those users that build cyrus-imapd from sources that these libs are actually needed if certain features are enabled (like Cal/CardDAV & JMAP)? I could prepare a patch. And I'd like to add these clarifications to the compiling page and reformat a bit the tables. If you consider it appropriate, I could prepare a patch too. Regards, Anatoli On 5/6/20 05:45, Anatoli wrote: >> Thanks for the explanation. What happens is that at least Cal/CardDAV >> compile and work well without libchardet (at least under normal >> circumstances) and, as it was not marked as required, I thought it's an >> optional lib and here is the question of why to use it. > > Actually, the same happens with transfig, libsrs2 and shapelib. > > The page mentiones these libs, but "Required for `make check`" (not sure > what that means in relation to the lib being reqiured for cyrus-imapd as > such) for all of them is "no". > > What would help is a column "required" yes/no and another column "usage > in cyrus-imapd" explaining for what exactly it's used. Say if it's for > tzdata processing in the TZDist module, then I probably won't build with > it as I don't see much use for it (or maybe package it as a flavor). > > But if it's something essential for a broader functionality like CalDAV > itself, then definitely it would be included. But at this moment the > Compiling page doesn't provide these details and here we are. > > > On 5/6/20 05:30, Anatoli wrote: >> Ellie, >> >>> libchardet is already listed on the compiling page as being used by >>> the CalDAV, CardDAV, and/or JMAP features. You would decide to >>> install libchardet based on whether you need these features: if you >>> don't enable these features, you don't need it; if you do, you >>> probably do. >>> >>> If you don't have libchardet, Cyrus will not do character-set >>> detection. If some piece of data has no character set coming in, it >>> will have no character set. I don't know why you would decide to not >>> use this, and it may simply become a mandatory requirement for these >>> features in 3.4. It might have been mandatory in 3.2, if this had >>> been brought to our attention during the nearly 3 month beta window; >>> but now that it's in a stable release, it will remain as it is. >> >> Thanks for the explanation. What happens is that at least Cal/CardDAV >> compile and work well without libchardet (at least under normal >> circumstances) and, as it was not marked as required, I thought it's an >> optional lib and here is the question of why to use it. >> >> I guess what you could do for 3.2 is to include a warning in configure >> explaining the issue, so the maintainers see it and take appropriate >> action. I suppose there are not that many installations with 3.2 yet so >> if the warning is included with 3.2.2 it would be an appropriate time. >> >> Regards, >> Anatoli >> >> On 4/6/20 21:58, ellie timoney wrote: >>> On Thu, Jun 4, 2020, at 12:52 PM, Anatoli wrote: >>>>> chardet is used by the JMAP module of httpd to detect the character >>>> set of untagged 8-bit headers. >>>> >>>> Does that mean that these libs are required? If not, what would happen >>>> if not included? How Cyrus would detect the charset? Or put in other >>>> words, if they are not required, what's the benefit to use them? >>>> >>>> On the other hand, may I suggest to please add these details to the >>>> Compiling [1] page? There's already a lot of useful information, but >>>> some of the libs lack any details on what's their purpose in Cyrus and >>>> why they are beneficial/required (and if they are not used yet, maybe >>>> they shouldn't be mentioned there at all?). This could help porters and >>>> maintainers to make more educated decisions on how to package Cyrus. >>> >>> libchardet is already listed on the compiling page as being used by the CalDAV, CardDAV, and/or JMAP features. You would decide to install libchardet based on whether you need these features: if you don't enable these features, you don't need it; if you do, you probably do. >>> >>> If you don't have libchardet, Cyrus will not do character-set detection. If some piece of data has no character set coming in, it will have no character set. I don't know why you would decide to not use this, and it may simply become a mandatory requirement for these features in 3.4. It might have been mandatory in 3.2, if this had been brought to our attention during the nearly 3 month beta window; but now that it's in a stable release, it will remain as it is. >>> >>> cld2 was used by one (or maybe both) of the experimental search features that were accidentally included in 3.2.0, and removed in 3.2.1 (see the release notes). Looks like we forgot to remove the configure checks for the library when removing the feature(s) that used it. You don't need it, and if you have it, it won't be used anyway. >>> >>> Cheers, >>> >>> ellie >>> From ellie at fastmail.com Thu Jun 18 23:29:58 2020 From: ellie at fastmail.com (ellie timoney) Date: Fri, 19 Jun 2020 13:29:58 +1000 Subject: new features with no documentation In-Reply-To: <3e1f0f55-4fc9-b7ae-069f-7b3f453dfda1@anatoli.ws> References: <04ee2b89-7c24-2947-b4f9-8b094f2a9e3f@anatoli.ws> <88cce492-632e-439a-ac7b-f68ef9c3381c@www.fastmail.com> <5427d59b-503b-5ba6-76e9-ba66f853262c@anatoli.ws> <20132e86-e439-8821-347e-b1adfb56ee24@anatoli.ws> <3e1f0f55-4fc9-b7ae-069f-7b3f453dfda1@anatoli.ws> Message-ID: <8ace84e8-c7dd-4618-a527-11579a3bdd08@www.fastmail.com> I think transfig is the package that provides fig2dev? I think that's an artifact from the old days, and is only used for generation of a couple of png files in the legacy documentation (doc/legacy/murder.png and doc/legacy/netnews.png). I think it's only needed when configure is run with --enable-maintainer-mode (i.e. if you're building a release or package, rather than just building the software). Those files are pre-built in the release tarballs, so additionally you would only need this when building from git. It would be nice to one day finish merging this legacy documentation into the current documentation, cause then we can get rid of it: https://github.com/cyrusimap/cyrus-imapd/issues/1769 libsrs2 looks like it's used for implementing Sender Rewriting Scheme functionality for messages forwarded by sieve script. Without it, messages forwarded by sieve script will not have this functionality (and, I suppose, might have difficulty delivering to servers that insist on it). The section of configure.ac that looks for libsrs2 is missing an AC_MSG_RESULT call to match the AC_MSG_CHECKING one. It probably also wants an AC_NOTICE when it's not found, explaining the consequences of it being left out, like we have for the various http dependencies. I'll fix this up in a moment. shapelib is already explained by configure. For example, I don't have it installed myself, so I see: > checking for SHAPELIB... no > configure: tzdist will not have geolocation support. Consider installing shapelib If you don't run a tzdist service, you don't need it. If you do, you probably also know whether geolocation support is useful; I don't. I've already updated configure.ac to provide a similar notice when chardet is not found (https://github.com/cyrusimap/cyrus-imapd/commit/30fef51485); it'll be in 3.2.2. :) If you want to have a go at making the compiling page a bit clearer, I'd be happy to review/merge a pull request. Please submit it against the master, and once we're happy with it, I'll get it cherry-picked appropriately to the other branches. Cheers, ellie On Fri, Jun 19, 2020, at 12:50 PM, Anatoli wrote: > Ellie (or anyone else, Ken?), > > Could you please let us know what's the purpose of transfig, libsrs2 and > shapelib libs in cyrus-imapd, and if they are required for any > functionality (and if not (i.e. they are optional), what's their > benefit)? > > Also, Ellie, would it be useful to add to configure checks with warnings > for these libs (as well as for chardet) to inform those users that build > cyrus-imapd from sources that these libs are actually needed if certain > features are enabled (like Cal/CardDAV & JMAP)? I could prepare a patch. > > And I'd like to add these clarifications to the compiling page and > reformat a bit the tables. If you consider it appropriate, I could > prepare a patch too. > > Regards, > Anatoli > > On 5/6/20 05:45, Anatoli wrote: > >> Thanks for the explanation. What happens is that at least Cal/CardDAV > >> compile and work well without libchardet (at least under normal > >> circumstances) and, as it was not marked as required, I thought it's an > >> optional lib and here is the question of why to use it. > > > > Actually, the same happens with transfig, libsrs2 and shapelib. > > > > The page mentiones these libs, but "Required for `make check`" (not sure > > what that means in relation to the lib being reqiured for cyrus-imapd as > > such) for all of them is "no". > > > > What would help is a column "required" yes/no and another column "usage > > in cyrus-imapd" explaining for what exactly it's used. Say if it's for > > tzdata processing in the TZDist module, then I probably won't build with > > it as I don't see much use for it (or maybe package it as a flavor). > > > > But if it's something essential for a broader functionality like CalDAV > > itself, then definitely it would be included. But at this moment the > > Compiling page doesn't provide these details and here we are. > > > > > > On 5/6/20 05:30, Anatoli wrote: > >> Ellie, > >> > >>> libchardet is already listed on the compiling page as being used by > >>> the CalDAV, CardDAV, and/or JMAP features. You would decide to > >>> install libchardet based on whether you need these features: if you > >>> don't enable these features, you don't need it; if you do, you > >>> probably do. > >>> > >>> If you don't have libchardet, Cyrus will not do character-set > >>> detection. If some piece of data has no character set coming in, it > >>> will have no character set. I don't know why you would decide to not > >>> use this, and it may simply become a mandatory requirement for these > >>> features in 3.4. It might have been mandatory in 3.2, if this had > >>> been brought to our attention during the nearly 3 month beta window; > >>> but now that it's in a stable release, it will remain as it is. > >> > >> Thanks for the explanation. What happens is that at least Cal/CardDAV > >> compile and work well without libchardet (at least under normal > >> circumstances) and, as it was not marked as required, I thought it's an > >> optional lib and here is the question of why to use it. > >> > >> I guess what you could do for 3.2 is to include a warning in configure > >> explaining the issue, so the maintainers see it and take appropriate > >> action. I suppose there are not that many installations with 3.2 yet so > >> if the warning is included with 3.2.2 it would be an appropriate time. > >> > >> Regards, > >> Anatoli > >> > >> On 4/6/20 21:58, ellie timoney wrote: > >>> On Thu, Jun 4, 2020, at 12:52 PM, Anatoli wrote: > >>>>> chardet is used by the JMAP module of httpd to detect the character > >>>> set of untagged 8-bit headers. > >>>> > >>>> Does that mean that these libs are required? If not, what would happen > >>>> if not included? How Cyrus would detect the charset? Or put in other > >>>> words, if they are not required, what's the benefit to use them? > >>>> > >>>> On the other hand, may I suggest to please add these details to the > >>>> Compiling [1] page? There's already a lot of useful information, but > >>>> some of the libs lack any details on what's their purpose in Cyrus and > >>>> why they are beneficial/required (and if they are not used yet, maybe > >>>> they shouldn't be mentioned there at all?). This could help porters and > >>>> maintainers to make more educated decisions on how to package Cyrus. > >>> > >>> libchardet is already listed on the compiling page as being used by the CalDAV, CardDAV, and/or JMAP features. You would decide to install libchardet based on whether you need these features: if you don't enable these features, you don't need it; if you do, you probably do. > >>> > >>> If you don't have libchardet, Cyrus will not do character-set detection. If some piece of data has no character set coming in, it will have no character set. I don't know why you would decide to not use this, and it may simply become a mandatory requirement for these features in 3.4. It might have been mandatory in 3.2, if this had been brought to our attention during the nearly 3 month beta window; but now that it's in a stable release, it will remain as it is. > >>> > >>> cld2 was used by one (or maybe both) of the experimental search features that were accidentally included in 3.2.0, and removed in 3.2.1 (see the release notes). Looks like we forgot to remove the configure checks for the library when removing the feature(s) that used it. You don't need it, and if you have it, it won't be used anyway. > >>> > >>> Cheers, > >>> > >>> ellie > >>> > From me at anatoli.ws Fri Jun 19 01:09:52 2020 From: me at anatoli.ws (Anatoli) Date: Fri, 19 Jun 2020 02:09:52 -0300 Subject: new features with no documentation In-Reply-To: <8ace84e8-c7dd-4618-a527-11579a3bdd08@www.fastmail.com> References: <04ee2b89-7c24-2947-b4f9-8b094f2a9e3f@anatoli.ws> <88cce492-632e-439a-ac7b-f68ef9c3381c@www.fastmail.com> <5427d59b-503b-5ba6-76e9-ba66f853262c@anatoli.ws> <20132e86-e439-8821-347e-b1adfb56ee24@anatoli.ws> <3e1f0f55-4fc9-b7ae-069f-7b3f453dfda1@anatoli.ws> <8ace84e8-c7dd-4618-a527-11579a3bdd08@www.fastmail.com> Message-ID: Ellie, Thanks for the explanation! I'll try to prepare a PR for the compiling page these days. Regards, Anatoli On 19/6/20 00:29, ellie timoney wrote: > I think transfig is the package that provides fig2dev? I think that's an artifact from the old days, and is only used for generation of a couple of png files in the legacy documentation (doc/legacy/murder.png and doc/legacy/netnews.png). I think it's only needed when configure is run with --enable-maintainer-mode (i.e. if you're building a release or package, rather than just building the software). Those files are pre-built in the release tarballs, so additionally you would only need this when building from git. It would be nice to one day finish merging this legacy documentation into the current documentation, cause then we can get rid of it: https://github.com/cyrusimap/cyrus-imapd/issues/1769 > > libsrs2 looks like it's used for implementing Sender Rewriting Scheme functionality for messages forwarded by sieve script. Without it, messages forwarded by sieve script will not have this functionality (and, I suppose, might have difficulty delivering to servers that insist on it). > > The section of configure.ac that looks for libsrs2 is missing an AC_MSG_RESULT call to match the AC_MSG_CHECKING one. It probably also wants an AC_NOTICE when it's not found, explaining the consequences of it being left out, like we have for the various http dependencies. I'll fix this up in a moment. > > shapelib is already explained by configure. For example, I don't have it installed myself, so I see: > >> checking for SHAPELIB... no >> configure: tzdist will not have geolocation support. Consider installing shapelib > > If you don't run a tzdist service, you don't need it. If you do, you probably also know whether geolocation support is useful; I don't. > > I've already updated configure.ac to provide a similar notice when chardet is not found (https://github.com/cyrusimap/cyrus-imapd/commit/30fef51485); it'll be in 3.2.2. :) > > If you want to have a go at making the compiling page a bit clearer, I'd be happy to review/merge a pull request. Please submit it against the master, and once we're happy with it, I'll get it cherry-picked appropriately to the other branches. > > Cheers, > > ellie > > On Fri, Jun 19, 2020, at 12:50 PM, Anatoli wrote: >> Ellie (or anyone else, Ken?), >> >> Could you please let us know what's the purpose of transfig, libsrs2 and >> shapelib libs in cyrus-imapd, and if they are required for any >> functionality (and if not (i.e. they are optional), what's their >> benefit)? >> >> Also, Ellie, would it be useful to add to configure checks with warnings >> for these libs (as well as for chardet) to inform those users that build >> cyrus-imapd from sources that these libs are actually needed if certain >> features are enabled (like Cal/CardDAV & JMAP)? I could prepare a patch. >> >> And I'd like to add these clarifications to the compiling page and >> reformat a bit the tables. If you consider it appropriate, I could >> prepare a patch too. >> >> Regards, >> Anatoli >> >> On 5/6/20 05:45, Anatoli wrote: >>>> Thanks for the explanation. What happens is that at least Cal/CardDAV >>>> compile and work well without libchardet (at least under normal >>>> circumstances) and, as it was not marked as required, I thought it's an >>>> optional lib and here is the question of why to use it. >>> >>> Actually, the same happens with transfig, libsrs2 and shapelib. >>> >>> The page mentiones these libs, but "Required for `make check`" (not sure >>> what that means in relation to the lib being reqiured for cyrus-imapd as >>> such) for all of them is "no". >>> >>> What would help is a column "required" yes/no and another column "usage >>> in cyrus-imapd" explaining for what exactly it's used. Say if it's for >>> tzdata processing in the TZDist module, then I probably won't build with >>> it as I don't see much use for it (or maybe package it as a flavor). >>> >>> But if it's something essential for a broader functionality like CalDAV >>> itself, then definitely it would be included. But at this moment the >>> Compiling page doesn't provide these details and here we are. >>> >>> >>> On 5/6/20 05:30, Anatoli wrote: >>>> Ellie, >>>> >>>>> libchardet is already listed on the compiling page as being used by >>>>> the CalDAV, CardDAV, and/or JMAP features. You would decide to >>>>> install libchardet based on whether you need these features: if you >>>>> don't enable these features, you don't need it; if you do, you >>>>> probably do. >>>>> >>>>> If you don't have libchardet, Cyrus will not do character-set >>>>> detection. If some piece of data has no character set coming in, it >>>>> will have no character set. I don't know why you would decide to not >>>>> use this, and it may simply become a mandatory requirement for these >>>>> features in 3.4. It might have been mandatory in 3.2, if this had >>>>> been brought to our attention during the nearly 3 month beta window; >>>>> but now that it's in a stable release, it will remain as it is. >>>> >>>> Thanks for the explanation. What happens is that at least Cal/CardDAV >>>> compile and work well without libchardet (at least under normal >>>> circumstances) and, as it was not marked as required, I thought it's an >>>> optional lib and here is the question of why to use it. >>>> >>>> I guess what you could do for 3.2 is to include a warning in configure >>>> explaining the issue, so the maintainers see it and take appropriate >>>> action. I suppose there are not that many installations with 3.2 yet so >>>> if the warning is included with 3.2.2 it would be an appropriate time. >>>> >>>> Regards, >>>> Anatoli >>>> >>>> On 4/6/20 21:58, ellie timoney wrote: >>>>> On Thu, Jun 4, 2020, at 12:52 PM, Anatoli wrote: >>>>>>> chardet is used by the JMAP module of httpd to detect the character >>>>>> set of untagged 8-bit headers. >>>>>> >>>>>> Does that mean that these libs are required? If not, what would happen >>>>>> if not included? How Cyrus would detect the charset? Or put in other >>>>>> words, if they are not required, what's the benefit to use them? >>>>>> >>>>>> On the other hand, may I suggest to please add these details to the >>>>>> Compiling [1] page? There's already a lot of useful information, but >>>>>> some of the libs lack any details on what's their purpose in Cyrus and >>>>>> why they are beneficial/required (and if they are not used yet, maybe >>>>>> they shouldn't be mentioned there at all?). This could help porters and >>>>>> maintainers to make more educated decisions on how to package Cyrus. >>>>> >>>>> libchardet is already listed on the compiling page as being used by the CalDAV, CardDAV, and/or JMAP features. You would decide to install libchardet based on whether you need these features: if you don't enable these features, you don't need it; if you do, you probably do. >>>>> >>>>> If you don't have libchardet, Cyrus will not do character-set detection. If some piece of data has no character set coming in, it will have no character set. I don't know why you would decide to not use this, and it may simply become a mandatory requirement for these features in 3.4. It might have been mandatory in 3.2, if this had been brought to our attention during the nearly 3 month beta window; but now that it's in a stable release, it will remain as it is. >>>>> >>>>> cld2 was used by one (or maybe both) of the experimental search features that were accidentally included in 3.2.0, and removed in 3.2.1 (see the release notes). Looks like we forgot to remove the configure checks for the library when removing the feature(s) that used it. You don't need it, and if you have it, it won't be used anyway. >>>>> >>>>> Cheers, >>>>> >>>>> ellie >>>>> >> From me at anatoli.ws Sat Jun 20 06:11:36 2020 From: me at anatoli.ws (Anatoli) Date: Sat, 20 Jun 2020 07:11:36 -0300 Subject: configure: wslay v1.1.1 required but the latest one is 1.1.0 In-Reply-To: <7c8025e2-7718-b7e4-dcf3-21d9228bb0a0@anatoli.ws> References: <18e0a63d-31bf-45b5-a001-efc92237ba16@www.fastmail.com> <810d8452-edd0-7128-4071-444cbf324d8d@anatoli.ws> <711a3731-39fa-0fcb-2b57-6da08b372dc0@anatoli.ws> <6fd3855a-d422-4ed7-838e-09efad88105c@www.fastmail.com> <7c8025e2-7718-b7e4-dcf3-21d9228bb0a0@anatoli.ws> Message-ID: <77924f98-c651-6718-70e9-da76c912c426@anatoli.ws> FYI, the wslay developer just released the 1.1.1 version with the commit in question so the dependency is ok now. More details here: https://github.com/cyrusimap/cyrus-imapd/issues/3070 On 5/6/20 05:11, Anatoli wrote: > Ellie, > > You're right, I haven't checked well the dates! > > I just asked the author of Wslay if he could tag a new release. Hope we get an answer soon. > > Regards, > Anatoli > > On 4/6/20 21:10, ellie timoney wrote: >> The commit was authored in 2015, but the pull request was only merged to wslay master last November. The fix is not in 1.1.0, but it's also not in our cyruslibs version (unless my submodule is out of date, which it might be, but I thought I updated it correctly the other day). >> >> So, since the fix is not in 1.1.0, that would be a problem for Cyrus if Cyrus depends on the fix being there. I don't know whether it does, or how to test it. >> >> On Thu, Jun 4, 2020, at 10:18 AM, Anatoli wrote: >>> Ken, >>> >>> Why do you believe it could be an issue with Cyrus? It appears the fix >>> was commited 4 years ago and that part was revorked later, so it should >>> not be a problem anymore in 1.1.0. >>> >>> >>> On 3/6/20 07:44, Ken Murchison wrote: >>>> Yes, 1.1.0 is probably sufficient, unless this bug is an issue with Cyrus: https://github.com/tatsuhiro-t/wslay/pull/47 >>>> >>>> >>>> On 6/3/20 1:19 AM, ellie timoney wrote: >>>>> Cool, thanks for confirming that.? So far it's sounding like 1.1.0 is probably adequate, but I'll wait a little bit to see if Ken has any input once he's been online >>>>> >>>>> On Wed, Jun 3, 2020, at 2:58 PM, Anatoli wrote: >>>>>> Hi Ellie, >>>>>> >>>>>>> When you configure it with your patch applied and 1.1.0 installed, >>>>>> does Cyrus build okay? >>>>>> >>>>>> Yes, it builds without errors. >>>>>> >>>>>> Configure prints: >>>>>> >>>>>> checking for WSLAY... yes >>>>>> ??? wslay:????????????? yes >>>>>> >>>>>> And -lwslay is passed as arg numerous times during build process. >>>>>> >>>>>> And effectively httpd binary includes references to wslay_event_xxx in >>>>>> its symbols table. >>>>>> >>>>>> Regards, >>>>>> Anatoli >>>>>> >>>>>> On 3/6/20 01:35, ellie timoney wrote: >>>>>>> In our "cyruslibs" package, the wslay submodule is at this commit: >>>>>>> >>>>>>> commit 4a937cd (HEAD, origin/master, origin/HEAD, master) >>>>>>> Author:???? Tatsuhiro Tsujikawa >>>>>>> AuthorDate: Fri Jun 8 23:19:03 2018 +0900 >>>>>>> Commit:???? Tatsuhiro Tsujikawa >>>>>>> CommitDate: Fri Jun 8 23:19:03 2018 +0900 >>>>>>> >>>>>>> ???? Bump up version number to 1.1.1-DEV >>>>>>> >>>>>>> Which is the commit immediately following the release-1.1.0 tag.? So, presumably, we're not dependent on any feature/fix that's only in the unreleased version, because otherwise we would've bumped the cyruslibs submodule to include those commits? >>>>>>> >>>>>>> So, "1.1.1" might be a typo, or an anticipatory thing that didn't go anywhere yet, I'm not sure. >>>>>>> >>>>>>> When you configure it with your patch applied and 1.1.0 installed, does Cyrus build okay? >>>>>>> >>>>>>> Cheers, >>>>>>> >>>>>>> ellie >>>>>>> >>>>>>> On Wed, Jun 3, 2020, at 2:13 PM, Anatoli wrote: >>>>>>>> Cyrus developers, >>>>>>>> >>>>>>>> The configure script checks for wslay lib version 1.1.1, but the latest >>>>>>>> version released is 1.1.0. So when it is installed, it reports: >>>>>>>> >>>>>>>> checking for WSLAY... no >>>>>>>> configure: httpd will not have support for WebSockets.? Consider >>>>>>>> installing libwslay >>>>>>>> >>>>>>>> The wslay's github repo has a mention of a 1.1.1-DEV version. Not sure >>>>>>>> if cyrus-imapd httpd requires something from it or if it was just a >>>>>>>> typo and 1.1.0 is ok. >>>>>>>> >>>>>>>> For the later case below is a patch. >>>>>>>> >>>>>>>> Regards, >>>>>>>> Anatoli >>>>>>>> >>>>>>>> >>>>>>>> diff --git a/configure.ac b/configure.ac >>>>>>>> index dc0e0fff2..30e925c60 100644 >>>>>>>> --- a/configure.ac >>>>>>>> +++ b/configure.ac >>>>>>>> @@ -1631,7 +1631,7 @@ dnl??????????????? AC_MSG_WARN([Your version of >>>>>>>> OpenDKIM can not support iSchedu >>>>>>>> >>>>>>>> ????????? AC_ARG_WITH(wslay, [AS_HELP_STRING([--without-wslay], [disable >>>>>>>> WebSockets support (check)])],,[with_wslay="check"]) >>>>>>>> ????????? if test "x$with_wslay" = "xyes" -o "x$with_wslay" = "xcheck"; >>>>>>>> then >>>>>>>> -??????????????? PKG_CHECK_MODULES([WSLAY], [libwslay >= 1.1.1], [ >>>>>>>> +??????????????? PKG_CHECK_MODULES([WSLAY], [libwslay >= 1.1.0], [ >>>>>>>> ????????????????????????? AC_DEFINE(HAVE_WSLAY,[], >>>>>>>> ????????????????????????????????? [Build WebSockets support into httpd?]) >>>>>>>> ????????????????????????? with_wslay=yes >>>>>>>> >>> From me at anatoli.ws Mon Jun 22 16:28:01 2020 From: me at anatoli.ws (Anatoli) Date: Mon, 22 Jun 2020 17:28:01 -0300 Subject: deflatePending not available in zlib on OpenBSD (undefined symbol) In-Reply-To: References: <8b142e4e-ce7b-c509-2dbc-7d2753ea60c9@anatoli.ws> <59b40e91-1f0e-e1ef-046d-ad0626510dca@anatoli.ws> <13a34d04-b43f-666c-f92d-f1ba42e891fc@fastmail.com> Message-ID: <8cae03a5-65c3-0512-f32c-573e6a1b1c77@anatoli.ws> Hi Ken, Is there anything preventing the merge of your change in the PR? I thought it would be included in 3.2.2. Regards, Anatoli On 3/6/20 17:47, Ken Murchison wrote: > This is my latest proposed fix: https://github.com/cyrusimap/cyrus-imapd/pull/3061 > > > On 6/2/20 7:34 PM, Anatoli wrote: >> Looks good to me and compiles correctly on OpenBSD. >> >> Could it be included in the next 3.2 release (3.2.2)? >> >> >> On 2/6/20 19:31, Ken Murchison wrote: >>> Yes, you're correct.? We have a comment in prot.c that quotes the same >>> passage ion the docs. >>> >>> I think this might do the trick: >>> >>> >>> diff --git a/imap/httpd.c b/imap/httpd.c >>> index fc430d935..b5014b97e 100644 >>> --- a/imap/httpd.c >>> +++ b/imap/httpd.c >>> @@ -149,24 +149,12 @@ HIDDEN int zlib_compress(struct transaction_t >>> *txn, unsigned flags, >>> ????? zstrm->next_in = (Bytef *) buf; >>> ????? zstrm->avail_in = len; >>> >>> -??? buf_ensure(&txn->zbuf, deflateBound(zstrm, zstrm->avail_in)); >>> ????? buf_reset(&txn->zbuf); >>> >>> ????? do { >>> ????????? int zr; >>> >>> -??????? if (!zstrm->avail_out) { >>> -??????????? unsigned pending; >>> - >>> -??????????? zr = deflatePending(zstrm, &pending, Z_NULL); >>> -??????????? if (zr != Z_OK) { >>> -??????????????? /* something went wrong */ >>> -??????????????? syslog(LOG_ERR, "zlib deflate error: %d %s", zr, >>> zstrm->msg); >>> -??????????????? return -1; >>> -??????????? } >>> - >>> -??????????? buf_ensure(&txn->zbuf, pending); >>> -??????? } >>> +??????? buf_ensure(&txn->zbuf, deflateBound(zstrm, len)); >>> >>> ????????? zstrm->next_out = (Bytef *) txn->zbuf.s + txn->zbuf.len; >>> ????????? zstrm->avail_out = txn->zbuf.alloc - txn->zbuf.len; >>> >>> >>> On 6/2/20 6:19 PM, Anatoli wrote: >>>> Hi Ken, >>>> >>>> According to the docs [1]: If deflate returns Z_OK and with zero >>>> avail_out, it must be called again after making room in the buffer >>>> because there might be more output pending. >>>> >>>> On the other hand it says: Z_FINISH can be used in the first deflate >>>> call after deflateInit if all the compression is to be done in a single >>>> step. In order to complete in one call, avail_out must be at least the >>>> value returned by deflateBound. Then deflate is guaranteed to return >>>> Z_STREAM_END. >>>> >>>> But: Note that it is possible for the compressed size to be larger than >>>> the value returned by deflateBound() if flush options other than >>>> Z_FINISH or Z_NO_FLUSH are used. >>>> >>>> >>>> At the beginning of zlib_compress() there's code that decides with which >>>> flush option to call deflate(). >>>> >>>> /* Only flush for static content or on last (zero-length) chunk */ >>>> >>>> If it's possible to make flush to be always Z_FINISH, then I guess the >>>> entire >>>> >>>> do { ... } while (!zstrm->avail_out); >>>> >>>> loop could be simplified to just: >>>> >>>> zstrm->next_out = (Bytef *) txn->zbuf.s + txn->zbuf.len; >>>> zstrm->avail_out = txn->zbuf.alloc - txn->zbuf.len; >>>> >>>> zr = deflate(zstrm, flush); >>>> if (zr != Z_STREAM_END) { >>>> ?????/* something went wrong */ >>>> ?????syslog(LOG_ERR, "zlib deflate error: %d %s", zr, zstrm->msg); >>>> ?????return -1; >>>> } >>>> >>>> >>>> I changed the "if (zr)" condition as the only good value would be >>>> Z_STREAM_END. >>>> >>>> But if the flush option can't be always Z_FINISH, then I believe the >>>> loop should be kept with the checks for !zstrm->avail_out as it's >>>> possible that there would be not enough buffer for deflate() to complete >>>> in one call. >>>> >>>> Regards, >>>> Anatoli >>>> >>>> [1] https://www.zlib.net/manual.html >>>> >>>> >>>> >>>> On 2/6/20 17:36, Ken Murchison wrote: >>>>> Hi Anatoli, >>>>> >>>>> Thanks for the report.? I'm not sure that we even need the >>>>> deflatePending() call, since we use deflateBound() to create an >>>>> appropriately-sized buffer to hold the entire compressed response body. >>>>> Let me do some testing. >>>>> >>>>> >>>>> On 6/2/20 3:48 AM, Anatoli wrote: >>>>>> Cyrus developers, >>>>>> >>>>>> Is it possible to somehow rework the code in imap/httpd.c lines 158-169 >>>>>> in order to NOT use deflatePending as this func is not available on >>>>>> OpenBSD? The zlib version there is 1.2.3 and deflatePending appeared in >>>>>> 1.2.5 so the code doesn't compile with --enable-http (undefined symbol: >>>>>> deflatePending). The packaged version disables http for now. >>>>>> >>>>>> Is it safe to reduce these lines: >>>>>> >>>>>> ??? 158???????? if (!zstrm->avail_out) { >>>>>> ??? 159???????????? unsigned pending; >>>>>> ??? 160 >>>>>> ??? 161???????????? zr = deflatePending(zstrm, &pending, Z_NULL); >>>>>> ??? 162???????????? if (zr != Z_OK) { >>>>>> ??? 163???????????????? /* something went wrong */ >>>>>> ??? 164???????????????? syslog(LOG_ERR, "zlib deflate error: %d %s", zr, >>>>>> zstrm->msg); >>>>>> ??? 165???????????????? return -1; >>>>>> ??? 166???????????? } >>>>>> ??? 167 >>>>>> ??? 168???????????? buf_ensure(&txn->zbuf, pending); >>>>>> ??? 169???????? } >>>>>> >>>>>> >>>>>> to something like: >>>>>> >>>>>> ??? 158???????? if (!zstrm->avail_out) { >>>>>> ??? 159???????????? buf_ensure(&txn->zbuf, 256 * 1024); >>>>>> ??? 160???????? } >>>>>> >>>>>> If I understand it correctly, deflatePending in this case is only used >>>>>> to get the needed buffer size and we could replace it with a hardcoded >>>>>> buffer size (like in my example of 256K). >>>>>> >>>>>> >>>>>> Thanks, >>>>>> Anatoli > From murch at fastmail.com Mon Jun 22 16:31:40 2020 From: murch at fastmail.com (Ken Murchison) Date: Mon, 22 Jun 2020 16:31:40 -0400 Subject: deflatePending not available in zlib on OpenBSD (undefined symbol) In-Reply-To: <8cae03a5-65c3-0512-f32c-573e6a1b1c77@anatoli.ws> References: <8b142e4e-ce7b-c509-2dbc-7d2753ea60c9@anatoli.ws> <59b40e91-1f0e-e1ef-046d-ad0626510dca@anatoli.ws> <13a34d04-b43f-666c-f92d-f1ba42e891fc@fastmail.com> <8cae03a5-65c3-0512-f32c-573e6a1b1c77@anatoli.ws> Message-ID: I'd like to get it tested by someone other than me and probably under load. On 6/22/20 4:28 PM, Anatoli wrote: > Hi Ken, > > Is there anything preventing the merge of your change in the PR? > > I thought it would be included in 3.2.2. > > Regards, > Anatoli > > On 3/6/20 17:47, Ken Murchison wrote: >> This is my latest proposed fix: https://github.com/cyrusimap/cyrus-imapd/pull/3061 >> >> >> On 6/2/20 7:34 PM, Anatoli wrote: >>> Looks good to me and compiles correctly on OpenBSD. >>> >>> Could it be included in the next 3.2 release (3.2.2)? >>> >>> >>> On 2/6/20 19:31, Ken Murchison wrote: >>>> Yes, you're correct.? We have a comment in prot.c that quotes the same >>>> passage ion the docs. >>>> >>>> I think this might do the trick: >>>> >>>> >>>> diff --git a/imap/httpd.c b/imap/httpd.c >>>> index fc430d935..b5014b97e 100644 >>>> --- a/imap/httpd.c >>>> +++ b/imap/httpd.c >>>> @@ -149,24 +149,12 @@ HIDDEN int zlib_compress(struct transaction_t >>>> *txn, unsigned flags, >>>> ????? zstrm->next_in = (Bytef *) buf; >>>> ????? zstrm->avail_in = len; >>>> >>>> -??? buf_ensure(&txn->zbuf, deflateBound(zstrm, zstrm->avail_in)); >>>> ????? buf_reset(&txn->zbuf); >>>> >>>> ????? do { >>>> ????????? int zr; >>>> >>>> -??????? if (!zstrm->avail_out) { >>>> -??????????? unsigned pending; >>>> - >>>> -??????????? zr = deflatePending(zstrm, &pending, Z_NULL); >>>> -??????????? if (zr != Z_OK) { >>>> -??????????????? /* something went wrong */ >>>> -??????????????? syslog(LOG_ERR, "zlib deflate error: %d %s", zr, >>>> zstrm->msg); >>>> -??????????????? return -1; >>>> -??????????? } >>>> - >>>> -??????????? buf_ensure(&txn->zbuf, pending); >>>> -??????? } >>>> +??????? buf_ensure(&txn->zbuf, deflateBound(zstrm, len)); >>>> >>>> ????????? zstrm->next_out = (Bytef *) txn->zbuf.s + txn->zbuf.len; >>>> ????????? zstrm->avail_out = txn->zbuf.alloc - txn->zbuf.len; >>>> >>>> >>>> On 6/2/20 6:19 PM, Anatoli wrote: >>>>> Hi Ken, >>>>> >>>>> According to the docs [1]: If deflate returns Z_OK and with zero >>>>> avail_out, it must be called again after making room in the buffer >>>>> because there might be more output pending. >>>>> >>>>> On the other hand it says: Z_FINISH can be used in the first deflate >>>>> call after deflateInit if all the compression is to be done in a single >>>>> step. In order to complete in one call, avail_out must be at least the >>>>> value returned by deflateBound. Then deflate is guaranteed to return >>>>> Z_STREAM_END. >>>>> >>>>> But: Note that it is possible for the compressed size to be larger than >>>>> the value returned by deflateBound() if flush options other than >>>>> Z_FINISH or Z_NO_FLUSH are used. >>>>> >>>>> >>>>> At the beginning of zlib_compress() there's code that decides with which >>>>> flush option to call deflate(). >>>>> >>>>> /* Only flush for static content or on last (zero-length) chunk */ >>>>> >>>>> If it's possible to make flush to be always Z_FINISH, then I guess the >>>>> entire >>>>> >>>>> do { ... } while (!zstrm->avail_out); >>>>> >>>>> loop could be simplified to just: >>>>> >>>>> zstrm->next_out = (Bytef *) txn->zbuf.s + txn->zbuf.len; >>>>> zstrm->avail_out = txn->zbuf.alloc - txn->zbuf.len; >>>>> >>>>> zr = deflate(zstrm, flush); >>>>> if (zr != Z_STREAM_END) { >>>>> ?????/* something went wrong */ >>>>> ?????syslog(LOG_ERR, "zlib deflate error: %d %s", zr, zstrm->msg); >>>>> ?????return -1; >>>>> } >>>>> >>>>> >>>>> I changed the "if (zr)" condition as the only good value would be >>>>> Z_STREAM_END. >>>>> >>>>> But if the flush option can't be always Z_FINISH, then I believe the >>>>> loop should be kept with the checks for !zstrm->avail_out as it's >>>>> possible that there would be not enough buffer for deflate() to complete >>>>> in one call. >>>>> >>>>> Regards, >>>>> Anatoli >>>>> >>>>> [1] https://www.zlib.net/manual.html >>>>> >>>>> >>>>> >>>>> On 2/6/20 17:36, Ken Murchison wrote: >>>>>> Hi Anatoli, >>>>>> >>>>>> Thanks for the report.? I'm not sure that we even need the >>>>>> deflatePending() call, since we use deflateBound() to create an >>>>>> appropriately-sized buffer to hold the entire compressed response body. >>>>>> Let me do some testing. >>>>>> >>>>>> >>>>>> On 6/2/20 3:48 AM, Anatoli wrote: >>>>>>> Cyrus developers, >>>>>>> >>>>>>> Is it possible to somehow rework the code in imap/httpd.c lines 158-169 >>>>>>> in order to NOT use deflatePending as this func is not available on >>>>>>> OpenBSD? The zlib version there is 1.2.3 and deflatePending appeared in >>>>>>> 1.2.5 so the code doesn't compile with --enable-http (undefined symbol: >>>>>>> deflatePending). The packaged version disables http for now. >>>>>>> >>>>>>> Is it safe to reduce these lines: >>>>>>> >>>>>>> ??? 158???????? if (!zstrm->avail_out) { >>>>>>> ??? 159???????????? unsigned pending; >>>>>>> ??? 160 >>>>>>> ??? 161???????????? zr = deflatePending(zstrm, &pending, Z_NULL); >>>>>>> ??? 162???????????? if (zr != Z_OK) { >>>>>>> ??? 163???????????????? /* something went wrong */ >>>>>>> ??? 164???????????????? syslog(LOG_ERR, "zlib deflate error: %d %s", zr, >>>>>>> zstrm->msg); >>>>>>> ??? 165???????????????? return -1; >>>>>>> ??? 166???????????? } >>>>>>> ??? 167 >>>>>>> ??? 168???????????? buf_ensure(&txn->zbuf, pending); >>>>>>> ??? 169???????? } >>>>>>> >>>>>>> >>>>>>> to something like: >>>>>>> >>>>>>> ??? 158???????? if (!zstrm->avail_out) { >>>>>>> ??? 159???????????? buf_ensure(&txn->zbuf, 256 * 1024); >>>>>>> ??? 160???????? } >>>>>>> >>>>>>> If I understand it correctly, deflatePending in this case is only used >>>>>>> to get the needed buffer size and we could replace it with a hardcoded >>>>>>> buffer size (like in my example of 256K). >>>>>>> >>>>>>> >>>>>>> Thanks, >>>>>>> Anatoli -- Kenneth Murchison Senior Software Developer Fastmail US LLC From rjbs at fastmailteam.com Wed Jun 24 10:57:40 2020 From: rjbs at fastmailteam.com (Ricardo Signes) Date: Wed, 24 Jun 2020 10:57:40 -0400 Subject: expanding allowsetacl Message-ID: <898d2778-bb35-4449-a3d6-f06aa2d474c1@dogfood.fastmail.com> Behold this commit: commit da8305164877735ba29b078151c70455f1aa6eea Author: Bron Gondwana Date: Wed Oct 30 14:13:38 2019 +1100 imapd: allow disabling the SETACL command I'm not sure what the intent here was, but I *assume* it was related to our plan to kill of the ability of users to set their own ACLs. If so, I think we need two small changes which I'd like to get out pretty soon. 1. relax the restriction so the admin user can still use SETACL 2. tighten the restriction so it also restricts DELETEACL That's it! Then we'll move on to rolling that out and cleaning up users' existing ACLs. I have tentatively made a task for this . Bron: I just want to make sure we're not stepping on work intended for something else! -- Ricardo Signes (rjbs) CTO, Fastmail -------------- next part -------------- An HTML attachment was scrubbed... URL: From dilyan.palauzov at aegee.org Wed Jun 24 16:25:22 2020 From: dilyan.palauzov at aegee.org (=?UTF-8?Q?=D0=94=D0=B8=D0=BB=D1=8F=D0=BD_?= =?UTF-8?Q?=D0=9F=D0=B0=D0=BB=D0=B0=D1=83=D0=B7=D0=BE=D0=B2?=) Date: Wed, 24 Jun 2020 20:25:22 +0000 Subject: expanding allowsetacl In-Reply-To: <898d2778-bb35-4449-a3d6-f06aa2d474c1@dogfood.fastmail.com> References: <898d2778-bb35-4449-a3d6-f06aa2d474c1@dogfood.fastmail.com> Message-ID: <292e86150cbfd91950c8f70989f938b3fc1287f9.camel@aegee.org> Hello Ricardo, Horde/IMP allow the user to edit their own IMAP ACLs. Unfortunately I do not know of any WebDAV client which users can utilize to edit their own ACLs. For WebDAV editing ACLs is one way to share (a caldav) collection. Why is there a plan to kill the ability of users to set their own ACLs? Greetings ????? On Wed, 2020-06-24 at 10:57 -0400, Ricardo Signes wrote: > Behold this commit: > > commit da8305164877735ba29b078151c70455f1aa6eea > Author: Bron Gondwana > Date: Wed Oct 30 14:13:38 2019 +1100 > > ?imapd: allow disabling the SETACL command > > I'm not sure what the intent here was, but I?assume it was related to our plan to kill of the ability of users to set > their own ACLs.? If so, I think we need two small changes which I'd like to get out pretty soon. > > ???1. relax the restriction so the admin user can still use SETACL > ???2. tighten the restriction so it also restricts DELETEACL > > That's it!? Then we'll move on to rolling that out and cleaning up users' existing ACLs.? I have tentatively made a > task for this. > > Bron:? I just want to make sure we're not stepping on work intended for something else! > > --? > Ricardo Signes (rjbs) > CTO, Fastmail > From rjbs at fastmailteam.com Wed Jun 24 17:01:52 2020 From: rjbs at fastmailteam.com (Ricardo Signes) Date: Wed, 24 Jun 2020 17:01:52 -0400 Subject: expanding allowsetacl In-Reply-To: <292e86150cbfd91950c8f70989f938b3fc1287f9.camel@aegee.org> References: <898d2778-bb35-4449-a3d6-f06aa2d474c1@dogfood.fastmail.com> <292e86150cbfd91950c8f70989f938b3fc1287f9.camel@aegee.org> Message-ID: <164f1099-e386-458f-8870-78133b100190@dogfood.fastmail.com> On Wed, Jun 24, 2020, at 16:25, ????? ???????? wrote: > Horde/IMP allow the user to edit their own IMAP ACLs. Unfortunately I do not know of any WebDAV client which users can utilize to edit their own ACLs. For WebDAV editing ACLs is one way to share (a caldav) collection. > > Why is there a plan to kill the ability of users to set their own ACLs? The intent here is only to make it *possible* for an individual server to restrict this permission, for the sake of a more structured set of managed ACLs. That is: to provide a control panel for ACL management with presets, rather than full control to every user. I would not suggest that ACL management be removed for all installs! -- rjbs -------------- next part -------------- An HTML attachment was scrubbed... URL: From brong at fastmailteam.com Thu Jun 25 21:59:19 2020 From: brong at fastmailteam.com (Bron Gondwana) Date: Fri, 26 Jun 2020 11:59:19 +1000 Subject: expanding allowsetacl In-Reply-To: <898d2778-bb35-4449-a3d6-f06aa2d474c1@dogfood.fastmail.com> References: <898d2778-bb35-4449-a3d6-f06aa2d474c1@dogfood.fastmail.com> Message-ID: <699b6dfc-4008-4643-b669-c4800aae6633@dogfood.fastmail.com> On Thu, Jun 25, 2020, at 00:57, Ricardo Signes wrote: > Behold this commit: > > commit da8305164877735ba29b078151c70455f1aa6eea Author: Bron Gondwana Date: Wed Oct 30 14:13:38 2019 +1100 imapd: allow disabling the SETACL command > > I'm not sure what the intent here was, but I *assume* it was related to our plan to kill of the ability of users to set their own ACLs. If so, I think we need two small changes which I'd like to get out pretty soon. > > 1. relax the restriction so the admin user can still use SETACL > 2. tighten the restriction so it also restricts DELETEACL > > That's it! Then we'll move on to rolling that out and cleaning up users' existing ACLs. I have tentatively made a task for this . > > Bron: I just want to make sure we're not stepping on work intended for something else! So... this does also handle DELETEACL, because they both call cmd_setacl, just with different paramenters. Maybe it should be called cmd_frobacl or something. As for allowing the admin - the way to do that is for the admin to connect to a different imapd which has no such restrictions. At least the way this is intended for use at Fastmail, it will only be on set on particular services entries rather than globally, so the `imapadmin` service won't have it set. Bron. -- Bron Gondwana, CEO, Fastmail Pty Ltd brong at fastmailteam.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From rjbs at fastmailteam.com Thu Jun 25 22:07:51 2020 From: rjbs at fastmailteam.com (Ricardo Signes) Date: Thu, 25 Jun 2020 22:07:51 -0400 Subject: expanding allowsetacl In-Reply-To: <699b6dfc-4008-4643-b669-c4800aae6633@dogfood.fastmail.com> References: <898d2778-bb35-4449-a3d6-f06aa2d474c1@dogfood.fastmail.com> <699b6dfc-4008-4643-b669-c4800aae6633@dogfood.fastmail.com> Message-ID: <2d802daf-77df-457e-9a78-2eceffe24756@dogfood.fastmail.com> On Thu, Jun 25, 2020, at 21:59, Bron Gondwana wrote: > So... this does also handle DELETEACL, because they both call cmd_setacl, just with different paramenters. Maybe it should be called cmd_frobacl or something. Foolishly I read the commit message and stopped there instead of looking at the code. Thanks for the clarification! -- rjbs -------------- next part -------------- An HTML attachment was scrubbed... URL: From brong at fastmailteam.com Sun Jun 28 09:55:28 2020 From: brong at fastmailteam.com (Bron Gondwana) Date: Sun, 28 Jun 2020 23:55:28 +1000 Subject: =?UTF-8?Q?A_"Threeskip"_database_format_(because_there_can_never_be_too_?= =?UTF-8?Q?many_skips)?= Message-ID: <5b494c23-9d41-41ba-b1c2-bc4525940a19@dogfood.fastmail.com> I've been thinking about this one for a very long time. It's still not going to be as excellent as zeroskip, but it has the advantage that it's still a single file, and it fixes two of the three defects of twoskip: Those three defects are: 1) poor write locality - updates happen throughout the file 2) long repack freeze gives unpredictable performance 3) no MVCC reads - writers block all readers. Threeskip will fix two of those! Numbers 2 and 3. For comparison, here's the twoskip ondisk format: * HEADER: 64 bytes * magic: 20 bytes: "4 bytes same as skiplist" "twoskip file\0\0\0\0" * version: 4 bytes * generation: 8 bytes * num_records: 8 bytes * repack_size: 8 bytes * current_size: 8 bytes * flags: 4 bytes * crc32: 4 bytes * * RECORDS: * type 1 byte * level: 1 byte * keylen: 2 bytes * vallen: 4 bytes * * * ptrs: 8 bytes * (level+1) * crc32_head: 4 bytes * crc32_tail: 4 bytes * key: (keylen bytes) * val: (vallen bytes) * padding: enough zeros to round up to an 8 byte multiple Threeskip is almost exactly the same, with three types (and of course with 'threeskip file\0\0\0' in the header) RECORD NEWVALUE DELETE RECORD: * type 1 byte * level: 1 byte * keylen: 2 bytes * vallen: 4 bytes * * ** latestptr: 8 bytes (new)* * ptrs: 8 bytes * (level+1) * crc32_head: 4 bytes * crc32_tail: 4 bytes * key: (keylen bytes) * val: (vallen bytes) * padding: enough zeros to round up to an 8 byte multiple NEWVALUE: * type: 1 byte * padding: 3 bytes * vallen: 4 bytes * ** prevptr: 8 bytes* * crc32_head: 4 bytes * crc32_tail: 4 bytes * val: (vallen bytes) * padding: enough zeros to round up to an 8 byte multiple +1 DELETE: * type: 1 byte * ZEROS: 7 bytes * prevptr: 8 bytes * crc32_head: 4 bytes * crc32_tail: 4 bytes (ZERO) This gives us MVCC! When you start a read transaction, the transaction includes the length of the file at the time of transaction start. The reader holds the same filehandle open, meaning it can keep reading even after a repack. If it reads past the transaction length, it just moves on to the next record. If it follows the latestptr past the transaction length, it just follows the prevptr back until it's at the latest value within the transaction. (DELETE already had a prevptr, but it was less powerful) *DUAL LOCKING * With this comes a second MVCC help. We use two different writelocks, which can be achieved with ranged flock within the header. Lock number 1 is held for the entire transaction and avoids any other writer. Lock number 2 is ONLY held for the duration of writing a single record and updating the related pointers. Readers take a shared lock on number 2 only, which allows them to guarantee clean reads. *NON-BLOCKING REPACK* The repack process can take an MVCC read and stream those records to a new file $DB.new (on which it holds an exclusive lock to avoid racing repacks), and then it can replay the log from there. Even the log replay can be done with read locks if the log length is over a certain amount (though this can lead to starvation, so should be a limited number of times) . The final fsync and rename needs to be done with a writelock on the source file of course. Reading the log, if we hit a NEWVALUE or DELETE, we follow the prevptrs back to find out the key that it references. It might be necessary to have a "repackd" which does the actual repacking, or some other way of a process saying "I don't mind waiting on the repack" because otherwise the repack freeze will still be visible to the current client, even if other readers and writers can continue. *STORAGE COST COMPARISON** * This adds an extra 8 bytes to the overhead for every CREATE record. The DELETE record is the same size. The NEWVALUE record no longer has the cost of an extract copy of the key in it. So it will use slightly more storage on newly repacked databases. (The base-level average overhead of a twoskip record is 40 bytes, this makes it 48) *COMPUTATIONAL AND IO COST COMPARISON** * Calculating "repack_size" will be a little more costly, but not too bad. Replacing values will be much cheaper because we don't need to update pointers. In theory we could even "row lock" or "gap lock" if we wanted other concurrency levels. *CRC ALGORITHM** * Since we have a choice with a new format, clearly we'd go with something fast! CRC32c with hardware acceleration all the way. *CRASH SAFETY AND THE OTHER TWOSKIP BENEFITS* All still there just as much as ever. Of course, write locality is helped slightly (overwrite and delete only touch the current record) but it's still the issue that will make zeroskip a better DB format. Bron. -- Bron Gondwana, CEO, Fastmail Pty Ltd brong at fastmailteam.com -------------- next part -------------- An HTML attachment was scrubbed... URL: