v3.0

ellie timoney ellie at fastmail.com
Tue May 17 00:15:48 EDT 2016 

>> I made a patch (configure.ac.patch, attached)
>> [...]
>> Please apply the fixes as you see fit.
>
> Thanks, those look good to me.  I'll push it up, but I'll split it
> into separate commits for each piece (sieve/RSCALE/com_err) for
> historical clarity.  I'll also fix the "distribuion" typo while I'm in
> there, that's been annoying me for ages. :P
 
The configure.ac changes are now (mostly) on master.
 
I left out one of the WARN=>NOTICE changes for com_err because it looks
like its intent is to alert the user that we're going against their
wishes, and using our com_err when they explicitly asked us  to not use
any com_err at all.  That seems like it should still be at least a
warning, if not an error.
 
Though, I did uncover a (largely inconsequential) bug, whereby this
warning doesn't fire when it ought to anyway:
https://git.cyrus.foundation/T240
 
On Tue, May 17, 2016, at 01:16 PM, ellie timoney via Cyrus-devel wrote:
> Hi Anatoli,
>
>> Ellie, the patch that you've sent for the backup linking problem
>> doesn't solve the issue
>
> Does anything different happen, or is it exactly the same as without
> it?  Can you provide output?  I'm pretty sure my patch is on the right
> track, but there's almost certainly a detail I've missed since I can't
> reproduce it myself.
>
> The other thing I wanted to ask was if you could run make with "-k".
> I have a hunch the other backup tools will currently fail to link for
> you in the exact same way, but if make stops at the first failure then
> we don't get to see that, nor whether they fail or succeed for some
> other reason (which might provide a clue).
>
>> There is a detailed explanation of the issue here:
>> https://wiki.debian.org/ToolChain/DSOLinking#Only_link_with_needed_libraries
>> and here: https://wiki.ubuntu.com/NattyNarwhal/ToolchainTransition.
>
> Thanks for tracking down those links, I'll look through them.
>
>> I've already proposed to Ken a slightly different patch than yours
>> (Makefile.am.patch, attached to this email is its version with
>> adaptations to apply smoothly to the latest master commit 9c8e8ea)
>> that works well under these rules.
>>
>> The changes are: to add SSL_LIBS to the list of the libs for
>> libcyrus_backup (currently there are none) and for the second error
>> (that was mentioned in the initial mail about these problems) to add
>> lib/libcyrus.la to the list of the libs for cyr_backup.
>>
>
> Yep, I saw that patch.  The reasons my patch is different are that:
>
> * libcyrus_backup.la is a no-install library. At present, I expect
> anything linking against it to also link against anything it needs
> (hence not having any LDADD of its own).  It sounds like this needs to
> change because of the SHA1_Update call in the library.
> * libcyrus.la is in $LD_BASIC_ADD, which is in $LD_UTILITY_ADD, and
> thus is already on the list for cyr_backup and the other backup tools
> (but clearly in the wrong order)
> * I want, as much as possible, to avoid having to manually specify
> lists of library dependencies for build targets. (That gets
> unmaintainable very quickly)
>
> So I'm looking for a way to get the libraries specified in the correct
> order, portably, and with minimal hand maintenance of dependency
> lists.  Given that the existing system works for everything but the
> backup tools (with their internal API library), and given that
> literally nobody but me uses the backup tools at the moment (afaik),
> there's an opportunity here to integrate them correctly, if a way can
> be found to do so.
>
> If we just manually add them to the lists, then next time something
> like this shows up we'll still have no idea how to fix it properly
>
>> I made a patch (configure.ac.patch, attached)
>> [...]
>> Please apply the fixes as you see fit.
>
> Thanks, those look good to me.  I'll push it up, but I'll split it
> into separate commits for each piece (sieve/RSCALE/com_err) for
> historical clarity.  I'll also fix the "distribuion" typo while I'm in
> there, that's been annoying me for ages. :P
>
> Cheers,
>
> ellie
>
> On Tue, May 17, 2016, at 10:21 AM, Anatoli via Cyrus-devel wrote:
>> Hi All,
>>
>> Ellie, the patch that you've sent for the backup linking problem
>> doesn't solve the issue, but you're right, I've just dug deeper: it's
>> a matter of order of the library inclusion options and it's because
>> in Ubuntu starting from 11.04 (and some versions of Debian too) the
>> gcc invokes ld with --as-needed flag that makes the order of
>> libraries quite important. There is a detailed explanation of the
>> issue here:
>> https://wiki.debian.org/ToolChain/DSOLinking#Only_link_with_needed_libraries
>> and here: https://wiki.ubuntu.com/NattyNarwhal/ToolchainTransition.
>>
>> Also it looks like newer versions of gcc/ld (mine is 4.8.4/2.24)
>> treat the order of includes much more strictly than previous versions
>> on most platforms.
>>
>> In a nutshell, a binary/library (A) which calls an external function
>> defined in another library (B) should appear *before* the library (B)
>> containing the function. The reason is that gcc/ld with --as-needed
>> flag optimizes symbols that aren't referenced, so if it sees library
>> B first, and no one has referenced it at that point then it won't
>> link in anything from it at all. This is what happens with -lcrypto
>> going before libcyrus_backup.a: the SHA1_Update function is only used
>> inside libcyrus_backup.a but the linker has already stripped it from
>> -lcrypto. If we include -lcrypto again *after* libcyrus_backup.a,
>> everything builds fine.
>>
>>
>> I've already proposed to Ken a slightly different patch than yours
>> (Makefile.am.patch, attached to this email is its version with
>> adaptations to apply smoothly to the latest master commit 9c8e8ea)
>> that works well under these rules.
>>
>> The changes are: to add SSL_LIBS to the list of the libs for
>> libcyrus_backup (currently there are none) and for the second error
>> (that was mentioned in the initial mail about these problems) to add
>> lib/libcyrus.la to the list of the libs for cyr_backup.
>>
>> So, instead of expecting the include libs, that are required for
>> certain parts of the binary/lib we're building, to make random
>> appearance in the gcc line (in sometimes incorrect order under the
>> strict/as-needed rules), we add in the Makefile.am the needed libs
>> for each and every lib/block of code we use to compose the final
>> output. As a result, at the linking stage they appear in the
>> correct order.
>>
>>
>>
>>
>> Warning (mostly harmless): No library found for -lcrypto
>>
>> Indeed, it's a Makefile.PL warning as my perl installation LD path is
>> not complete (doesn't include /usr/lib64 where crypto/ssl libs are).
>> It should be fixed locally in my perl deployment. In any case it's
>> indeed harmless as gcc/ld knows about this path and this is where
>> it's used.
>>
>>
>>
>>
>> The yacc/lex dependency is actually not checked. Better to say
>> configure tries to determine what yacc/lex progs are installed, but
>> if it founds none, it continues as if everything's OK. The config.log
>> I've sent you was for the build with the packages installed
>> (otherwise you can't proceed with the make stage). Without the
>> packages, in the config.log I get:
>>
>> configure:19090: checking for bison
>> configure:19120: result: no
>> configure:19090: checking for byacc
>> configure:19120: result: no
>> configure:19133: checking for flex
>> configure:19163: result: no
>> configure:19133: checking for lex
>> configure:19163: result: no
>>
>> but nothing more. So, configure reports no warnings/errors with
>> respect to the missing tools and then with make I get:
>>
>> ./ylwrap: line 175: yacc: command not found
>> make: *** [sieve/addr.c] Error 127
>> make: *** Waiting for unfinished jobs....
>> ./ylwrap: line 175: yacc: command not found
>> make: *** [sieve/sieve.c] Error 127
>>
>>
>> I made a patch (configure.ac.patch, attached) to check for yacc/lex
>> programs inside the Sieve option block as it's the only component (if
>> I'm not wrong) that requires yacc/lex.
>>
>>
>> Then I went ahead and also changed the following (I've already
>> mentioned both suggestions to Ken/list but received no response):
>>
>> Changed the behavior when no ICU4C lib is detected: from warning to
>> error as its absence causes an error during make stage, and other
>> similar required packages are reported inside configure with an error
>> message.
>>
>> Changed the type of message when com_err is absent or incomplete
>> (lacking compile_et): from warning to notice, as this is the most
>> common situation I believe (no or incomplete com_err) and it doesn't
>> affect in anything the result of the build process (its sources are
>> shipped with Cyrus sources and used when needed).
>>
>>
>> Inside this patch there is also a fix for a typo in a comment (says
>> "Build in *Sieve* support?" instead of "Build in *Squat* support?")
>> introduced with my previous patch for --disable-squat option fix.
>> This typo doesn't affect anything (it's just a comment for
>> config.h.in), but seizing the opportunity of fixing Makefile.am,
>> here it is.
>>
>>
>> With all this applied, I see no warning/errors for the entire build
>> process when all needed packages are installed, and when not,
>> configure notifies me about all of them, not allowing to continue to
>> the make stage until all requirements are satisfied. Please apply the
>> fixes as you see fit.
>>
>>
>> Regards,
>> Anatoli
>>
>>
>> -----Original Message----- From: Cyrus-devel [mailto:cyrus-devel-
>> bounces+me=anatoli.ws at lists.andrew.cmu.edu] On Behalf Of ellie
>> timoney via Cyrus-devel Sent: Sunday, May 15, 2016 23:28 To: cyrus-
>> devel at lists.andrew.cmu.edu Subject: Re: v3.0
>>
>> Hi Anatoli,
>>
>> Thanks for sending through the config.log and make output.
>>
>> For what it's worth, if you configure with --enable-silent-rules, the
>> make output will be a lot less noisy (and a lot easier to read).
>>
>> > Ken, the 2 errors that don't happen to you only happen to me when
>> > compiling
>> > with --enable-backup. I've tested the beta2 tarball as well as the
>> > current
>> > git, both errors still happen to me on 2 different machines.
>> > Without the
>> > --enable-backup option these errors don't occur.
>>
>> As I mentioned in private email, I don't get these errors either
>> (Debian).
>>
>> Poking around on Google, I see a few other projects experiencing the
>> "DSO missing from command line" error for Ubuntu builds.  It
>> looks like
>> maybe Ubuntu's linker setup is more sensitive to correct order of
>> dependencies?
>>
>> I've attached an experimental patch which I think corrects the link
>> dependency order for the backup tools.  Can you try it out?
>> The patch
>> is against 3.0.0beta2 but should also apply cleanly against
>> latest git
>> master.  This should hopefully resolve the "DSO missing from command
>> line" errors when building with --enable-backup.  Let me know how it
>> goes.
>>
>> > Warning (mostly harmless): No library found for -lcrypto
>> > (same for -lssl)
>>
>> I dug around a bit and I think these come from the perl module build
>> system.  I'm not sure what they mean exactly, might be a red herring.
>> Your config.log shows it's correctly detecting OpenSSL, anyway.
>>
>> > I found another issue with the dependencies: the build requires
>> > yacc/lex
>> > when building sieve, though in the Release Notes there is the
>> > following
>> > text: Replaced the et (error table) libary with a version that
>> > doesn't
>> > require lex or yacc. Remove the lex/yacc checking from Configure.
>>
>> That release note item is ancient (version 1.4!) and I doubt it's of
>> much relevance now.  Your config.log says:
>>
>> > configure:19090: checking for bison
>> > configure:19106: found /usr/bin/bison
>> > configure:19117: result: bison -y
>> > configure:19133: checking for flex
>> > configure:19149: found /usr/bin/flex
>> > configure:19160: result: flex
>>
>> so configure is looking for, and finding, bison (supercedes yacc) and
>> flex (supercedes lex).
>>
>> What problem are you seeing here?
>>
>> Cheers,
>>
>> ellie
>>
>> On Fri, May 13, 2016, at 05:15 PM, Anatoli via Cyrus-devel wrote:
>> > Hi all,
>> >
>> > Ken, the 2 errors that don't happen to you only happen to me when
>> > compiling
>> > with --enable-backup. I've tested the beta2 tarball as well as the
>> > current
>> > git, both errors still happen to me on 2 different machines.
>> > Without the
>> > --enable-backup option these errors don't occur.
>> >
>> > Ellie, I'll send the config.log and the resulting make output to
>> > you and
>> > Ken
>> > directly.
>> >
>> >
>> > I can confirm that the latest master from the dev git has the
>> > patch for
>> > --disable-squat correctly working and the ICU checking in
>> > configure is
>> > present now too, but maybe it should be converted from a warning
>> > to an
>> > error
>> > when with --enable-http?
>> >
>> > DKIM warning doesn't appearing anymore. It would be great to
>> > solve the
>> > "Parts of com_err distribuion were found, but not compile_et"
>> > warning
>> > too.
>> >
>> > I'm also getting:
>> > checking for BIO_accept in -lcrypto... yes
>> > checking for crypt... no
>> > checking for crypt in -lcrypt... yes
>> > Warning (mostly harmless): No library found for -lcrypto
>> > (same for -lssl)
>> >
>> > I found another issue with the dependencies: the build requires
>> > yacc/lex
>> > when building sieve, though in the Release Notes there is the
>> > following
>> > text: Replaced the et (error table) libary with a version that
>> > doesn't
>> > require lex or yacc. Remove the lex/yacc checking from Configure.
>> >
>> > I guess the lex/yacc checking should be placed in the configure
>> > again,
>> > inside the enable-sieve block.
>> >
>> > Regards,
>> > Anatoli
>> >
>> >
>> > -----Original Message-----
>> > From: Ken Murchison [mailto:murch at andrew.cmu.edu]
>> > Sent: Wednesday, May 11, 2016 15:44
>> > To: Anatoli; cyrus-devel at lists.andrew.cmu.edu
>> > Subject: Re: v3.0
>> >
>> >
>> >
>> > On 05/09/2016 02:57 AM, Anatoli via Cyrus-devel wrote:
>> > > Hi all,
>> > >
>> > > I'm testing v3.0.0 beta2. Here goes the feedback, this time for
>> > > the build
>> > > process.
>> > >
>> > > 1. --disable-squat option in configure has no effect. Please see
>> > >    attached
>> > a
>> > > patch (configure.ac.patch).
>> >
>> > Patch applied.
>> >
>> >
>> > > 2. Without icu-dev package make fails with:
>> > >
>> > > unicode/ucal.h No such file or directory
>> > >
>> > > It somehow depends on libical, i.e. it looks like icu-dev
>> > > should be
>> > > installed before building libical. In any case it should be
>> > > detected in
>> > > configure to avoid build errors.
>> >
>> > A check for ICU4C has been added to configure.ac
>> >
>> >
>> > > 3. With --enable-backup make fails with:
>> > >
>> > > /usr/bin/ld: backup/.libs/libcyrus_backup.a(lcb_append.o):
>> > > undefined
>> > > reference to symbol 'SHA1_Update@@OPENSSL_1.0.0'
>> > > //usr/lib64/libcrypto.so: error adding symbols: DSO missing from
>> > > command
>> > > line
>> > >
>> > > Problem: missing -lcrypto for libcyrus_backup.a
>> > >
>> > > /usr/bin/ld: imap/.libs/libcyrus_imap.so: undefined reference to
>> > > symbol
>> > > 'cyrusdb_fetch'
>> > > lib/.libs/libcyrus.so: error adding symbols: DSO missing from
>> > > command line
>> > >
>> > > Problem: missing lib/libcyrus.la for cyr_backup.
>> > >
>> > > Please see attached a patch for both errors (Makefile.am.patch).
>> > >
>> > >
>> > > With these fixes build completes without errors.
>> >
>> > I'm not seeing this problem compiling on my machine.  I wonder why?
>> >
>> >
>> > > Then I got this new warning:
>> > >
>> > > configure: WARNING: Your version of OpenDKIM can not support
>> > > iSchedule.
>> > > Consider upgrading to OpenDKIM >= 2.7.0
>> > >
>> > > I don't have DKIM on the box where Cyrus runs, in 2.5.7 there
>> > > was no
>> > warning
>> > > about DKIM. Haven't investigated the details yet. DKIM is part of
>> > > the
>> > > iSchedule draft, but is it required? And should it be OpenDKIM
>> > > only or any
>> > > other DKIM package works too?
>> >
>> > I have commented out the check for DKIM since its not needed for
>> > anything that sites will deploy any time soon.
>> >
>> >
>> > > And then there is an old warning (also happens in 2.5.7):
>> > >
>> > > configure: WARNING: Parts of com_err distribuion were found, but
>> > > not
>> > > compile_et.
>> > > configure: WARNING: Will build com_err from included sources.
>> > >
>> > > What should be installed/changed to avoid it? Or maybe the
>> > > warning itself
>> > > should be converted to a notice as compile_et is shipped with cyrus-
>> > > imap
>> > and
>> > > everything works as expected?
>> > >
>> > > Regards,
>> > > Anatoli
>> > >
>> > >
>> > > -----Original Message-----
>> > > From: Anatoli [mailto:me at anatoli.ws]
>> > > Sent: Tuesday, April 19, 2016 03:39
>> > > To: cyrus-devel at lists.andrew.cmu.edu
>> > > Subject: RE: v3.0
>> > >
>> > > Hi Ellie,
>> > >
>> > > Thanks for the link! This is the information I was looking for.
>> > > Great,
>> > > there's a new beta! I'll test it these days and if it behaves
>> > > reasonably
>> > > well, I'll try to deploy it in a small production environment
>> > > where the
>> > > users are OK being beta-testers. I'll post here any issues found.
>> > >
>> > > With respect to the specialuse flags issue, it's not possible to
>> > > set these
>> > > flags from cyradm in the latest release (2.5.7). I'll check if
>> > > it's fixed
>> > in
>> > > the 3.0 beta.
>> > >
>> > > I have a feature suggestion with respect to these flags. I
>> > > suppose that
>> > most
>> > > of the deployments use these flags on some folders from the
>> > > autocreate_inbox_folders list. So, instead of writing scripts
>> > > that set
>> > these
>> > > flags somehow, what if it would be possible to specify the
>> > > specialuse flag
>> > > for each autocreate folder as an optional param (with some
>> > > (invalid for
>> > > folder names) char (e.g. ':') as the delimiter)?
>> > >
>> > > Something like this:
>> > > autocreate_inbox_folders:
>> > > Sent:Sent|Trash:trash|Drafts:DRAFTS|Spam:Junk|OtherFolder1|Other-
>> > > Folder2
>> > >
>> > > The format would be defines as:
>> > > folder[:<specialuse_flag>][|folder[:<specialuse_flag>]]... and
>> > > <specialuse_flag> would be one of the options from the RFC6154
>> > > (section
>> > 2),
>> > > interpreted case-insensitively.
>> > >
>> > > So when the user logs in for the first time, he/she has all the
>> > > folders
>> > > created with the necessary flags. IMO, a significant
>> > > simplification of the
>> > > mbox creation process.
>> > >
>> > > I haven't analyzed the code for this feature yet, but it should
>> > > be quite
>> > > simple to implement. Just parse the optional params, check if the
>> > > value is
>> > > in a predefined array and after creation of the folder, set the
>> > > requested
>> > > flag. Every flag could be validated for uniqueness or it may be
>> > > left up to
>> > > the Cyrus administrator to decide and specify the correct values
>> > > (I would
>> > > prefer the later, the RFC explicitly says it's up to each server
>> > > implementation (section 3)).
>> > >
>> > > Regards,
>> > > Anatoli
>> > >
>> > > -----Original Message-----
>> > > From: Cyrus-devel
>> > > [mailto:cyrus-devel-bounces+me=anatoli.ws at lists.andrew.cmu.edu]
>> > > On Behalf
>> > Of
>> > > ellie timoney via Cyrus-devel
>> > > Sent: Sunday, April 17, 2016 22:37
>> > > To: cyrus-devel at lists.andrew.cmu.edu
>> > > Subject: Re: v3.0
>> > >
>> > > Hi Anatoli,
>> > >
>> > >> I'm quite interested in this release and I'd
>> > >> like to help with testing, simple problems investigation and
>> > >> fixing, and
>> > >> similar tasks,
>> > > That would be greatly appreciated :)
>> > >
>> > >> but at least some insight to the current state of the
>> > >> master is needed for that, i.e. how close it is to an RC, new
>> > >> functionality expected to work, functionality/configuration
>> > >> changes
>> > >> compared to v2.5.7, known limitations, etc.
>> > > Have you looked at the 3.0.0-beta2 that was released last week?
>> > > Its
>> > > release notes compare it against the 2.5 series:
>> > > http://cyrusimap.org/imap/release-notes/3.0/x/3.0.0-beta2.html
>> > >
>> > >> I see the T232 by Ellie has 4 issues that are apparently
>> > >> stopping this
>> > >> release. Are they the only remaining issues for production-ready
>> > >> state?
>> > > These are just the very narrow intersection of a) what I'm aware
>> > > of,
>> > > that has b) been logged at all, and c) is logged in phabricator
>> > > where I
>> > > can mark it as blocking (rather than in bugzilla, the mailing
>> > > list,
>> > > private email, etc).
>> > >
>> > >> The roadmap
>> > >> (https://cyrusimap.org/overview/cyrus_roadmap.html#cyrus-roadmap)
>> > >> says
>> > >> nothing about v3.0 and looks a little outdated.
>> > > This page looks like a direct import of the page from the old
>> > > website.
>> > > I'm not sure quite how old it is, but given it's referencing
>> > > "2.6" as
>> > > future, that suggests that it's over a year old...
>> > >
>> > >> I'm personally interested in resolving the "specialuse flags not
>> > >> working
>> > >> from cyradm" (T199, 198, 191, 121; looks like still pending)
>> > >> issue
>> > > Are these still issues?  Or are they stale tasks that have been
>> > > fixed
>> > > but not closed?  We've had a number of cyradm metadata patches
>> > > contributed by a few different people over the last year, so it
>> > > seems
>> > > probable that at least some are fixed but the assignees don't
>> > > know.
>> > >
>> > >> Also I made a raw chroot patch for 2.5.7,
>> > >> I'd like to polish and submit it for review and inclusion in
>> > >> v3.0.
>> > > That'd be great!
>> > >
>> > >> Should I write to someone in particular to discuss the subject?
>> > > Everyone working on Cyrus 3.0 is active on this list, so this is
>> > > probably the best place for it.
>> > >
>> > > You're also welcome to join our conference calls, they happen at
>> > > 11am
>> > > UTC most Mondays on Google Hangouts.  Probably the easiest way to
>> > > join
>> > > is to come into #cyrus on Freenode IRC at the meeting time and
>> > > ask for
>> > > the hangouts link, because it changes occasionally.
>> > >
>> > > Cheers,
>> > >
>> > > ellie
>> > >
>> > > On Sat, Apr 16, 2016, at 04:41 AM, Anatoli via Cyrus-devel wrote:
>> > >> Hi folks,
>> > >>
>> > >> Sorry for bothering you again with subj, I haven't received any
>> > >> answer to
>> > >> the previous mails about it. I'm quite interested in this
>> > >> release and I'd
>> > >> like to help with testing, simple problems investigation and
>> > >> fixing, and
>> > >> similar tasks, but at least some insight to the current state of
>> > >> the
>> > >> master is needed for that, i.e. how close it is to an RC, new
>> > >> functionality expected to work, functionality/configuration
>> > >> changes
>> > >> compared to v2.5.7, known limitations, etc.
>> > >>
>> > >> I see the T232 by Ellie has 4 issues that are apparently
>> > >> stopping this
>> > >> release. Are they the only remaining issues for production-ready
>> > >> state?
>> > >> The roadmap
>> > >> (https://cyrusimap.org/overview/cyrus_roadmap.html#cyrus-roadmap)
>> > >> says
>> > >> nothing about v3.0 and looks a little outdated.
>> > >>
>> > >> I'm personally interested in resolving the "specialuse flags not
>> > >> working
>> > >> from cyradm" (T199, 198, 191, 121; looks like still pending)
>> > >> issue and in
>> > >> XAPPLEPUSHSERVICE feature, but I know there are a lot of other
>> > >> improvements and new libraries support (like LibiCal2.0) that
>> > >> are worth
>> > >> the effort releasing it ASAP. Also I made a raw chroot patch for
>> > >> 2.5.7,
>> > >> I'd like to polish and submit it for review and inclusion in
>> > >> v3.0.
>> > >>
>> > >> Should I write to someone in particular to discuss the subject?
>> > >>
>> > >> Regards,
>> > >> Anatoli
>> > >>
>> > >>
>> >
>> > --
>> > Kenneth Murchison
>> > Principal Systems Software Engineer
>> > Carnegie Mellon University
>> >
>> Email had 2 attachments:


>>  * Makefile.am.patch  1k (text/plain)
>>  * configure.ac.patch  3k (text/plain)
>
 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/cyrus-devel/attachments/20160517/44f79acc/attachment-0001.html>

More information about the Cyrus-devel mailing list