From brong at fastmail.fm  Mon May  4 07:22:02 2009
From: brong at fastmail.fm (Bron Gondwana)
Date: Mon, 4 May 2009 21:22:02 +1000
Subject: CONDSTORE by default?
Message-ID: <20090504112202.GA22300@brong.net>

Hi Ken, everyone,

Can anyone think of a good reason why CONDSTORE isn't enabled
by default for everyone all the time?  Back when it wasn't
stable, fair enough - but I think it's pretty close now, and
am willing to put a bit more effort in to finishing the job...

Thunderbird 3 will support CONDSTORE to reduce the FLAGS traffic
quite considerably, and hopefully other clients will follow suit
if there is enough server support to be worth the effort.

Looking at the code, the "bookkeeping" cost is quite low, and
it always touches records that are being updated anyway, so the
writes will be combined in the same block most of the time.

Besides, it will simplify the code considerably to remove all
the tests for "CONDSTORE ENABLED" everywhere.

Bron ( looking at turning it on for all new mailboxes soon,
       and then enabling it for all the old ones )

From murch at andrew.cmu.edu  Mon May  4 07:27:32 2009
From: murch at andrew.cmu.edu (Ken Murchison)
Date: Mon, 04 May 2009 07:27:32 -0400
Subject: CONDSTORE by default?
In-Reply-To: <20090504112202.GA22300@brong.net>
References: <20090504112202.GA22300@brong.net>
Message-ID: <49FED124.5040008@andrew.cmu.edu>

There is a higher bookkeeping cost when only \Seen state changes because 
rather than just updating seen.db, we also have to tweak cyrus.index. 
The cost may not be enough to worry about, but since \Seen state is most 
likely the most frequently used flag (and most likely is changed 
separately from other flags), I didn't want to force CONDSTORE on people 
until we had a good handle on the performance hit.


Bron Gondwana wrote:
> Hi Ken, everyone,
> 
> Can anyone think of a good reason why CONDSTORE isn't enabled
> by default for everyone all the time?  Back when it wasn't
> stable, fair enough - but I think it's pretty close now, and
> am willing to put a bit more effort in to finishing the job...
> 
> Thunderbird 3 will support CONDSTORE to reduce the FLAGS traffic
> quite considerably, and hopefully other clients will follow suit
> if there is enough server support to be worth the effort.
> 
> Looking at the code, the "bookkeeping" cost is quite low, and
> it always touches records that are being updated anyway, so the
> writes will be combined in the same block most of the time.
> 
> Besides, it will simplify the code considerably to remove all
> the tests for "CONDSTORE ENABLED" everywhere.
> 
> Bron ( looking at turning it on for all new mailboxes soon,
>        and then enabling it for all the old ones )
> 

-- 
Kenneth Murchison
Systems Programmer
Carnegie Mellon University

From michael.menge at zdv.uni-tuebingen.de  Mon May  4 08:20:53 2009
From: michael.menge at zdv.uni-tuebingen.de (Michael Menge)
Date: Mon, 04 May 2009 14:20:53 +0200
Subject: CONDSTORE by default?
In-Reply-To: <49FED124.5040008@andrew.cmu.edu>
References: <20090504112202.GA22300@brong.net>
	<49FED124.5040008@andrew.cmu.edu>
Message-ID: <20090504142053.16782974p201en51@webmail.uni-tuebingen.de>

Hi,

doc/readme.html in 2.3.14 still says:

Upgrade Caveats

     This section reserved for WARNING WARNING WARNING comments.
     Note that the replication protocol currently does not have the facility
     to support the IMAP CONDSTORE extension (modification sequences). It is
     recommended that you do not try to use both CONDSTORE and replication
     at this time. The deficiencies in the replication protocol will be
     fixed in version 2.3.9.

Is this problem fixed or is it still valid?



Quoting Ken Murchison <murch at andrew.cmu.edu>:

> There is a higher bookkeeping cost when only \Seen state changes  
> because rather than just updating seen.db, we also have to tweak  
> cyrus.index. The cost may not be enough to worry about, but since  
> \Seen state is most likely the most frequently used flag (and most  
> likely is changed separately from other flags), I didn't want to  
> force CONDSTORE on people until we had a good handle on the  
> performance hit.
>
>
> Bron Gondwana wrote:
>> Hi Ken, everyone,
>>
>> Can anyone think of a good reason why CONDSTORE isn't enabled
>> by default for everyone all the time?  Back when it wasn't
>> stable, fair enough - but I think it's pretty close now, and
>> am willing to put a bit more effort in to finishing the job...
>>
>> Thunderbird 3 will support CONDSTORE to reduce the FLAGS traffic
>> quite considerably, and hopefully other clients will follow suit
>> if there is enough server support to be worth the effort.
>>
>> Looking at the code, the "bookkeeping" cost is quite low, and
>> it always touches records that are being updated anyway, so the
>> writes will be combined in the same block most of the time.
>>
>> Besides, it will simplify the code considerably to remove all
>> the tests for "CONDSTORE ENABLED" everywhere.
>>
>> Bron ( looking at turning it on for all new mailboxes soon,
>>       and then enabling it for all the old ones )
>>
>
> -- 
> Kenneth Murchison
> Systems Programmer
> Carnegie Mellon University
>



--------------------------------------------------------------------------------
M.Menge                                Tel.: (49) 7071/29-70316
Universit?t T?bingen                   Fax.: (49) 7071/29-5912
Zentrum f?r Datenverarbeitung          mail:  
michael.menge at zdv.uni-tuebingen.de
W?chterstra?e 76
72074 T?bingen
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 5339 bytes
Desc: S/MIME krytographische Unterschrift
Url : http://lists.andrew.cmu.edu/pipermail/cyrus-devel/attachments/20090504/1e44e55f/attachment.bin 

From brong at fastmail.fm  Mon May  4 09:36:32 2009
From: brong at fastmail.fm (Bron Gondwana)
Date: Mon, 4 May 2009 23:36:32 +1000
Subject: CONDSTORE by default?
In-Reply-To: <20090504142053.16782974p201en51@webmail.uni-tuebingen.de>
References: <20090504112202.GA22300@brong.net>
	<49FED124.5040008@andrew.cmu.edu>
	<20090504142053.16782974p201en51@webmail.uni-tuebingen.de>
Message-ID: <20090504133632.GA2565@brong.net>

On Mon, May 04, 2009 at 02:20:53PM +0200, Michael Menge wrote:
> Hi,
>
> doc/readme.html in 2.3.14 still says:
>
> Upgrade Caveats
>
>     This section reserved for WARNING WARNING WARNING comments.
>     Note that the replication protocol currently does not have the facility
>     to support the IMAP CONDSTORE extension (modification sequences). It is
>     recommended that you do not try to use both CONDSTORE and replication
>     at this time. The deficiencies in the replication protocol will be
>     fixed in version 2.3.9.
>
> Is this problem fixed or is it still valid?

I think it's pretty much fixed now.  I will know better soon, since I've
just enabled it for all new folders and am bringing it in on old folders
bit by bit (tends to spam the replication logs if you do it too fast!)

I've also added highestmodseq and modseq tests to our replication checking
suite, so they'll show up if anything isn't perfect!

Bron.

From selsky at columbia.edu  Wed May  6 21:03:01 2009
From: selsky at columbia.edu (Matt Selsky)
Date: Wed, 6 May 2009 21:03:01 -0400
Subject: [Cyrus-CVS] src/cyrus/lib by brong
In-Reply-To: <200903310443.n2V4hLuN028673@cyrus-devel-01.andrew.cmu.edu>
References: <200903310443.n2V4hLuN028673@cyrus-devel-01.andrew.cmu.edu>
Message-ID: <C8BEFC8A-91C2-476F-9CA8-50D219E12CF6@columbia.edu>

In libcyr_cfg.h, the comment should say "ON" to match libcyr_cfg.c,  
correct?

/* Checkpoint after every recovery (ON) */
CYRUSOPT_SKIPLIST_ALWAYS_CHECKPOINT,

On Mar 31, 2009, at 12:43 AM, brong at andrew.cmu.edu wrote:

> Update of /afs/andrew.cmu.edu/system/cvs/src/cyrus/lib
> In directory cyrus-devel-01.andrew.cmu.edu:/afs/andrew.cmu.edu/usr3/ 
> brong/src/cyrus/lib
>
> Modified Files:
> 	cyrusdb_skiplist.c imapoptions libcyr_cfg.c libcyr_cfg.h
> Log Message:
> skiplist always checkpoint
>
> Adds an option skiplist_always_checkpoint to force a checkpoint of
> every skiplist database every time recovery is run (generally the  
> first
> time it gets accessed after a cyrus restart, or after any crash)
>
> The theory behind this is "repack it while it's hot" - recovery  
> already
> accesses pretty much every page in the file, so the IO cost isn't too
> bad.
>
> ===================================================================
>
>
> --- links to diffs follow ---
> http://bugzilla.andrew.cmu.edu/cgi-bin/cvsweb.cgi/src/cyrus/lib/cyrusdb_skiplist.c.diff?r1=1.64&r2=1.65
> http://bugzilla.andrew.cmu.edu/cgi-bin/cvsweb.cgi/src/cyrus/lib/imapoptions.diff?r1=1.62&r2=1.63
> http://bugzilla.andrew.cmu.edu/cgi-bin/cvsweb.cgi/src/cyrus/lib/libcyr_cfg.c.diff?r1=1.16&r2=1.17
> http://bugzilla.andrew.cmu.edu/cgi-bin/cvsweb.cgi/src/cyrus/lib/libcyr_cfg.h.diff?r1=1.12&r2=1.13
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.andrew.cmu.edu/pipermail/cyrus-devel/attachments/20090506/ff517a51/attachment.html 

From dave64 at andrew.cmu.edu  Thu May  7 19:42:01 2009
From: dave64 at andrew.cmu.edu (Dave McMurtrie)
Date: Thu, 07 May 2009 19:42:01 -0400
Subject: Performance deficiency in mboxlist.c when using flat mailboxes.db
	in a non-murder environment
Message-ID: <4A0371C9.7060608@andrew.cmu.edu>

Hi,

Ben Carter from the University of Pittsburgh submitted this patch to me 
today.  They're doing an upgrade to 2.3.14 in a non-murder environment 
with a flat mailboxes database, and he discovered a serious performance 
problem doing certain LIST commands.

The following patch fixes the problem:

https://bugzilla.andrew.cmu.edu/show_bug.cgi?id=3156

Thanks,

Dave
-- 
Dave McMurtrie, SPE
Email Systems Team Leader
Carnegie Mellon University,
Computing Services

From brong at fastmail.fm  Thu May  7 23:01:56 2009
From: brong at fastmail.fm (Bron Gondwana)
Date: Fri, 8 May 2009 13:01:56 +1000
Subject: Performance deficiency in mboxlist.c when using flat
	mailboxes.db in a non-murder environment
In-Reply-To: <4A0371C9.7060608@andrew.cmu.edu>
References: <4A0371C9.7060608@andrew.cmu.edu>
Message-ID: <20090508030156.GA11545@brong.net>

On Thu, May 07, 2009 at 07:42:01PM -0400, Dave McMurtrie wrote:
> Hi,
>
> Ben Carter from the University of Pittsburgh submitted this patch to me  
> today.  They're doing an upgrade to 2.3.14 in a non-murder environment  
> with a flat mailboxes database, and he discovered a serious performance  
> problem doing certain LIST commands.
>
> The following patch fixes the problem:
>
> https://bugzilla.andrew.cmu.edu/show_bug.cgi?id=3156

Looks fine to me.  Thanks Ben.

Bron ( um, so why are you using a flat mailboxes.db?? )

From brong at fastmail.fm  Thu May 14 02:45:23 2009
From: brong at fastmail.fm (Bron Gondwana)
Date: Thu, 14 May 2009 16:45:23 +1000
Subject: CONDSTORE and replication
Message-ID: <20090514064523.GA6462@brong.net>

We've enabled CONDSTORE on an entire "Store" here at FastMail
to get an idea of what's involved in supporting it.  (it's the
store that the FastMail staff are on, so we're testing it on
ourselves first!)

So - there's a problem where \Seen changes are not causing the
modseq information to be replicated.  I think the following
patch fixes the issue (need to run sync_client on all users
who are currently not in sync, then leave things running for
a bit to make sure), but there is a downside.

Each time you set a \Seen flag (including just viewing a
message without using .PEEK), it will cause the entire
mailbox metadata to be:

a) read from disk
b) copied across the wire

That kind of sucks, even more than just the IO cost of
writing a couple of bit64s to the index file (modseq
in the record and highestmodseq in the header).  David,
I've CC'd you on this to get your opinion on more
fine-grained replication logging for metadata operations.
sync_log_mailbox is a pretty heavy-weight flyswat for
single record changes.

I'm thinking something like

sync_log_index(mailbox->name, uidrange), where uidrange
contains a compressed range of UIDs (seen-db format)

The sync_client would then just send a the correct values
for:

a) all header values (including highestmodseq, lastmodified,
   uidnext, etc)

b) all values within just the listed index records, which
   would overwrite the records for the same UID in the
   target mailbox, unconditionally.

   (b1) modulo our "if the GUIDs don't match then log the
        issue and don't update the record at all" logic here
        at FastMail to avoid destroying delivered messages
        after a hard-failover, of course....


What do you think?  Sane?  I suspect it's a bit of work, but
it would reduce traffic a fair bit.  Probably wouldn't help
with IO though, since every index file algorithm seems to be
linear search based.

Bron.

diff --git a/imap/imapd.c b/imap/imapd.c
index acddf31..365df80 100644
--- a/imap/imapd.c
+++ b/imap/imapd.c
@@ -4564,7 +4564,7 @@ void cmd_store(char *tag, char *sequence, int usinguid)
 
 	/* We only need to log a MAILBOX event if we've changed
 	   a flag other than \Seen */
-	if (storeargs.system_flags || nflags ||
+	if (storeargs.system_flags || nflags || (imapd_mailbox->options & OPT_IMAP_CONDSTORE) ||
 	    storeargs.operation == STORE_REPLACE) {
 	    sync_log_mailbox(imapd_mailbox->name);
 	}

From murch at andrew.cmu.edu  Thu May 14 08:06:01 2009
From: murch at andrew.cmu.edu (Ken Murchison)
Date: Thu, 14 May 2009 08:06:01 -0400
Subject: CONDSTORE and replication
In-Reply-To: <20090514064523.GA6462@brong.net>
References: <20090514064523.GA6462@brong.net>
Message-ID: <4A0C0929.2060802@andrew.cmu.edu>

I knew that changing only \Seen with CONDSTORE had consequences  :)

I think your addition to the replication protocol makes sense.


Bron Gondwana wrote:
> We've enabled CONDSTORE on an entire "Store" here at FastMail
> to get an idea of what's involved in supporting it.  (it's the
> store that the FastMail staff are on, so we're testing it on
> ourselves first!)
> 
> So - there's a problem where \Seen changes are not causing the
> modseq information to be replicated.  I think the following
> patch fixes the issue (need to run sync_client on all users
> who are currently not in sync, then leave things running for
> a bit to make sure), but there is a downside.
> 
> Each time you set a \Seen flag (including just viewing a
> message without using .PEEK), it will cause the entire
> mailbox metadata to be:
> 
> a) read from disk
> b) copied across the wire
> 
> That kind of sucks, even more than just the IO cost of
> writing a couple of bit64s to the index file (modseq
> in the record and highestmodseq in the header).  David,
> I've CC'd you on this to get your opinion on more
> fine-grained replication logging for metadata operations.
> sync_log_mailbox is a pretty heavy-weight flyswat for
> single record changes.
> 
> I'm thinking something like
> 
> sync_log_index(mailbox->name, uidrange), where uidrange
> contains a compressed range of UIDs (seen-db format)
> 
> The sync_client would then just send a the correct values
> for:
> 
> a) all header values (including highestmodseq, lastmodified,
>    uidnext, etc)
> 
> b) all values within just the listed index records, which
>    would overwrite the records for the same UID in the
>    target mailbox, unconditionally.
> 
>    (b1) modulo our "if the GUIDs don't match then log the
>         issue and don't update the record at all" logic here
>         at FastMail to avoid destroying delivered messages
>         after a hard-failover, of course....
> 
> 
> What do you think?  Sane?  I suspect it's a bit of work, but
> it would reduce traffic a fair bit.  Probably wouldn't help
> with IO though, since every index file algorithm seems to be
> linear search based.
> 
> Bron.
> 
> diff --git a/imap/imapd.c b/imap/imapd.c
> index acddf31..365df80 100644
> --- a/imap/imapd.c
> +++ b/imap/imapd.c
> @@ -4564,7 +4564,7 @@ void cmd_store(char *tag, char *sequence, int usinguid)
>  
>  	/* We only need to log a MAILBOX event if we've changed
>  	   a flag other than \Seen */
> -	if (storeargs.system_flags || nflags ||
> +	if (storeargs.system_flags || nflags || (imapd_mailbox->options & OPT_IMAP_CONDSTORE) ||
>  	    storeargs.operation == STORE_REPLACE) {
>  	    sync_log_mailbox(imapd_mailbox->name);
>  	}
> 

-- 
Kenneth Murchison
Systems Programmer
Carnegie Mellon University

From ekgermann at cctec.com  Thu May 14 15:29:04 2009
From: ekgermann at cctec.com (Eric K Germann)
Date: Thu, 14 May 2009 15:29:04 -0400 (EDT)
Subject: Tracking down a db4 error
Message-ID: <4315.70.63.17.166.1242329344.squirrel@noc.semperen.com>

Long time cyrus user, new to list.

I've been running 2.3.14 of imapd for a while now and I still get periodic
fatal region errors from lmtpunix.  However, I have no idea what db is
corrupt.

In imapd.conf, I've set about everything I can to skiplist, so I would
like to find out some way to see what db file it's whining about.  The
frequency is way down, so moving to skiplist has helped, but I'd like to
nail the final files if possible.

/etc/imapd.conf

duplicate_db:           skiplist
tlscache_db:            skiplist
annotation_db:          skiplist
mboxlist_db:            skiplist
mboxkey_db:             skiplist
ptscache_db:            skiplist
seenstate_db:           skiplist
statuscache_db:         skiplist
subscription_db:        skiplist



Is there a way to have it log the filename it's choking on?

Thanks

EKG



-- 
=======================================================================
Eric K Germann
ekgermann (at) cctec (dot) com
Phone: 419 605 4026  :::: Fax:   419 567 2053

PGP/GPG Fingerprint: B047 5EFD 3851 6D79 A851  709B 07FC 9F13 7496 9066

"Every man must do two things alone; he must do his own believing and
 his own dying." -- Martin Luther
========================================================================



From bawood at umich.edu  Wed May 20 10:48:42 2009
From: bawood at umich.edu (Brian Awood)
Date: Wed, 20 May 2009 10:48:42 -0400
Subject: reconstruct changes
Message-ID: <200905201048.42809.bawood@umich.edu>

We'd like to make some changes to reconstruct, but would like to get 
some feedback on them.  Attached is a diff of the changes. 
First we'd like to make -k the default behavior if delayed expunge is 
enabled.  Second, add a -w flag to warn about changes that would be 
made to a mailbox, similar to ctl_mboxlist -w.  Third, add -e to 
enable indexing un-indexed messages to a new cyrus.expunge rather 
than putting them into the cyrus.index file.  -e can be useful when a 
expunge file isn't verifiable or corrupted and you don't want a large 
number of deleted messages to reappear in a users mailbox.  

---
Brian Awood
University of Michigan
-------------- next part --------------
A non-text attachment was scrubbed...
Name: reconstruct.diff
Type: text/x-diff
Size: 14024 bytes
Desc: not available
Url : http://lists.andrew.cmu.edu/pipermail/cyrus-devel/attachments/20090520/e4990923/attachment.bin 

From brong at fastmail.fm  Wed May 20 19:28:42 2009
From: brong at fastmail.fm (Bron Gondwana)
Date: Thu, 21 May 2009 09:28:42 +1000
Subject: reconstruct changes
In-Reply-To: <200905201048.42809.bawood@umich.edu>
References: <200905201048.42809.bawood@umich.edu>
Message-ID: <20090520232842.GA4399@brong.net>

On Wed, May 20, 2009 at 10:48:42AM -0400, Brian Awood wrote:
> We'd like to make some changes to reconstruct, but would like to get 
> some feedback on them.  Attached is a diff of the changes. 
> First we'd like to make -k the default behavior if delayed expunge is 
> enabled.  Second, add a -w flag to warn about changes that would be 
> made to a mailbox, similar to ctl_mboxlist -w.  Third, add -e to 
> enable indexing un-indexed messages to a new cyrus.expunge rather 
> than putting them into the cyrus.index file.  -e can be useful when a 
> expunge file isn't verifiable or corrupted and you don't want a large 
> number of deleted messages to reappear in a users mailbox.  

I approve of all these ideas.

> +    while ((opt = getopt(argc, argv, "C:kp:rmfsxgGwe")) != EOF) {
>  
> +    if( config_getenum(IMAPOPT_EXPUNGE_MODE) ==
> +	    IMAP_ENUM_EXPUNGE_MODE_DELAYED ) {
> +	keepflag = 1;
> +    }

The only downside here is that there's no way to say "don't keep
expunged messages" short of using a different config file.  Maybe
add '-K' to mean "don't keep expunged"?  Messy, I know.

> +    if( !warn_only ) {

(in various places) - the only downside I see here is that later
check may assume earlier "create missing stuff" ran - so you'll
get errors looking at things inside a mailbox because you just
can't open it without a cyrus.header file, and you chose not to
create that.

I don't have an easy answer for you though!

I've added some patches (I think they're all upstream now) to
syslog if reconstruct finds anything to fix, and also syslog
when it gets run on any particular mailbox (to make tracking
down changes easier), and I suspect a few more syslog
statements in there wouldn't go astray either.  But your patch
looks good as-is.

Bron.


From bawood at umich.edu  Thu May 21 00:45:41 2009
From: bawood at umich.edu (Brian Awood)
Date: Thu, 21 May 2009 00:45:41 -0400
Subject: reconstruct changes
In-Reply-To: <20090520232842.GA4399@brong.net>
References: <200905201048.42809.bawood@umich.edu>
	<20090520232842.GA4399@brong.net>
Message-ID: <200905210045.41645.bawood@umich.edu>


On Wednesday 20 May 2009 @ 19:28, Bron Gondwana wrote:
>
> I approve of all these ideas.

Great!

> > +    while ((opt = getopt(argc, argv, "C:kp:rmfsxgGwe")) != EOF)
> > {
> >
> > +    if( config_getenum(IMAPOPT_EXPUNGE_MODE) ==
> > +	    IMAP_ENUM_EXPUNGE_MODE_DELAYED ) {
> > +	keepflag = 1;
> > +    }
>
> The only downside here is that there's no way to say "don't keep
> expunged messages" short of using a different config file.  Maybe
> add '-K' to mean "don't keep expunged"?  Messy, I know.

We thought about that, but we couldn't think of a reason why you would 
want to throw out the expunged mail prematurely.  And if you really 
want to do that, using another config file or cyr_expire is a little 
cumbersome but would effectively be the sysadmin equivalent to 
asking "do you really want to delete all those files?"

> > +    if( !warn_only ) {
>
> (in various places) - the only downside I see here is that later
> check may assume earlier "create missing stuff" ran - so you'll
> get errors looking at things inside a mailbox because you just
> can't open it without a cyrus.header file, and you chose not to
> create that.

Yes, one case where this can be a problem is if you run with -f and a 
new mailbox is found.  It doesn't get added to the mboxlist, but then 
tries to reconstruct the new mailbox.  The idea behind -w is that it 
warns about changes that would likely be user visible, so if an admin 
isn't expecting anything other than a rebuild of the cache file, they 
can be sure that is all that will happen.  This will become 
especially important for us in the near future, since we are working 
on rolling out a user self-service "unexpunge" webapp and people will 
expect to be able to get back expunged mail for the past 10 days!

> I don't have an easy answer for you though!
>
> I've added some patches (I think they're all upstream now) to
> syslog if reconstruct finds anything to fix, and also syslog
> when it gets run on any particular mailbox (to make tracking
> down changes easier), and I suspect a few more syslog
> statements in there wouldn't go astray either.  But your patch
> looks good as-is.
>
> Bron.

Better logging is always welcome!  If anyone else has questions or 
comments I'll be glad to read them, I can also submit it in bugzilla 
if it seems acceptable for a future release.

Brian

From woods-cyrus at weird.com  Thu May 28 16:02:45 2009
From: woods-cyrus at weird.com (Greg A. Woods)
Date: Thu, 28 May 2009 16:02:45 -0400
Subject: The ldapdb plug-in initialization for clients is too noisy!
Message-ID: <m1M9loO-000kmyC@most.weird.com>

The ldapdb plug-in initialization for clients is too noisy!

I recently, and for the first time, have built Cyrus-SASL with the
ldapdb plug-in.  Now every client on every system _not_ using LDAP
complains on every start up with the following error:

	auxpropfunc error invalid parameter supplied

(which I've patched to be a little more detailed showing _which_ plug-in
returned the error! -- SASL error reporting is sometimes horribly
minimalistic and almost totally unhelpful!)

It seems the LDAP plug-in is a little too eager to initialize itself
even when it is not called upon or used, and a little too worried about
its inability to do this initialization when it is not actually needed.

I don't want to have to build custom versions of SASL (and, because I
need to link everything statically, custom versions of everything that
uses SASL) for just the LDAP users I support.  The question is how to
fix this in a way which would be acceptable for an upstream patch.

One simple possibility is to return some other more benign error from
ldapdb_auxprop_plug_init() instead of SASL_BADPARAM (eg. SASL_NOMECH)
and then transmute the log level flag from SASL_LOG_ERR to something
more like SASL_LOG_NOTE so that it won't spam the console and/or other
critical error log files.

More correct might be to defer configuring all of the plug-in until the
point where it is first actually called upon to do something.  I see
that some other plug-ins do try to load config parameters in their
_plug_init() routines, but almost universally they will 

Yet another possibility might be to have an explicit way to disable the
plug-in in the configuration file and thereby avoid all initialization
if it has been disabled.  Perhaps the default <none> entry for
ldapdb_uri should work this way?  (i.e. the error return should simply
be removed?)

(note all of the other plug-ins I compile into my SASL libraries run
their *_plug_init() function successfully -- though of course none of
the others I use require such extensive and crazy initialization!)

-- 
						Greg A. Woods

+1 416 218-0098                VE3TCP          RoboHack <woods at robohack.ca>
Planix, Inc. <woods at planix.com>      Secrets of the Weird <woods at weird.com>

From alexey.melnikov at isode.com  Thu May 28 17:18:24 2009
From: alexey.melnikov at isode.com (Alexey Melnikov)
Date: Thu, 28 May 2009 22:18:24 +0100
Subject: The ldapdb plug-in initialization for clients is too noisy!
In-Reply-To: <m1M9loO-000kmyC@most.weird.com>
References: <m1M9loO-000kmyC@most.weird.com>
Message-ID: <4A1EFFA0.2000109@isode.com>

Greg A. Woods wrote:

>The ldapdb plug-in initialization for clients is too noisy!
>
>I recently, and for the first time, have built Cyrus-SASL with the
>ldapdb plug-in.  Now every client on every system _not_ using LDAP
>complains on every start up with the following error:
>
>	auxpropfunc error invalid parameter supplied
>
>(which I've patched to be a little more detailed showing _which_ plug-in
>returned the error! -- SASL error reporting is sometimes horribly
>minimalistic and almost totally unhelpful!)
>
>It seems the LDAP plug-in is a little too eager to initialize itself
>even when it is not called upon or used, and a little too worried about
>its inability to do this initialization when it is not actually needed.
>
>I don't want to have to build custom versions of SASL (and, because I
>need to link everything statically, custom versions of everything that
>uses SASL) for just the LDAP users I support.  The question is how to
>fix this in a way which would be acceptable for an upstream patch.
>
>One simple possibility is to return some other more benign error from
>ldapdb_auxprop_plug_init() instead of SASL_BADPARAM (eg. SASL_NOMECH)
>and then transmute the log level flag from SASL_LOG_ERR to something
>more like SASL_LOG_NOTE so that it won't spam the console and/or other
>critical error log files.
>  
>
>More correct might be to defer configuring all of the plug-in until the
>point where it is first actually called upon to do something.  I see
>that some other plug-ins do try to load config parameters in their
>_plug_init() routines, but almost universally they will 
>
>Yet another possibility might be to have an explicit way to disable the
>plug-in in the configuration file and thereby avoid all initialization
>if it has been disabled.  Perhaps the default <none> entry for
>ldapdb_uri should work this way?  (i.e. the error return should simply
>be removed?)
>  
>
Greg, you don't need to change anything, because there is a solution to 
you problem already
Just set the "auxprop_plugin" option to exclude ldapdb.

>(note all of the other plug-ins I compile into my SASL libraries run
>their *_plug_init() function successfully -- though of course none of
>the others I use require such extensive and crazy initialization!)
>  
>


From woods-cyrus at weird.com  Thu May 28 20:04:36 2009
From: woods-cyrus at weird.com (Greg A. Woods)
Date: Thu, 28 May 2009 20:04:36 -0400
Subject: The ldapdb plug-in initialization for clients is too noisy!
In-Reply-To: <4A1EFFA0.2000109@isode.com>
References: <m1M9loO-000kmyC@most.weird.com>
	<4A1EFFA0.2000109@isode.com>
Message-ID: <m1M9paR-000kmPC@most.weird.com>

At Thu, 28 May 2009 22:18:24 +0100, Alexey Melnikov <alexey.melnikov at isode.com> wrote:
Subject: Re: The ldapdb plug-in initialization for clients is too noisy!
> 
> Greg, you don't need to change anything, because there is a solution to 
> you problem already
> Just set the "auxprop_plugin" option to exclude ldapdb.

(I really hate it that sasl options, which I assume I would put in my
imapd.conf(5) file, right?, are very poorly documented.  There are no
sasl*.5 manual pages it seems, and even the HTML junk is incomplete and
confusing to me.)

I don't see any way to exclude one plug-in from being used -- unless I
first determine the list of all valid plug-ins, then elide that one I
don't want from the list.

I'd rather fix the broken useless error reporting once and for all time!  :-)

-- 
						Greg A. Woods

+1 416 218-0098                VE3TCP          RoboHack <woods at robohack.ca>
Planix, Inc. <woods at planix.com>      Secrets of the Weird <woods at weird.com>

From ml at awinkelmann.de  Fri May 29 06:52:56 2009
From: ml at awinkelmann.de (Andreas Winkelmann)
Date: Fri, 29 May 2009 12:52:56 +0200 (CEST)
Subject: The ldapdb plug-in initialization for clients is too noisy!
In-Reply-To: <4A1EFFA0.2000109@isode.com>
References: <m1M9loO-000kmyC@most.weird.com> <4A1EFFA0.2000109@isode.com>
Message-ID: <fbcbf86738412997e74af4c33ca51a07.squirrel@a-angels.ath.cx>

> Greg, you don't need to change anything, because there is a solution to
> you problem already
> Just set the "auxprop_plugin" option to exclude ldapdb.

This will not stop the auxprop-plugins from beeing loaded/initialized. And
exactly this gives the warnings/errors.

If it is really needed to have the unused files on disk, someone can add
some dummy-options to the config file(s).

For ldabdb:

ldapdb_uri: dummy or wahtever

For sql:

sql_select: dummy or whatever

--
Andreas


From woods-cyrus at weird.com  Sat May 30 12:17:52 2009
From: woods-cyrus at weird.com (Greg A. Woods)
Date: Sat, 30 May 2009 12:17:52 -0400
Subject: The ldapdb plug-in initialization for clients is too noisy!
In-Reply-To: <fbcbf86738412997e74af4c33ca51a07.squirrel@a-angels.ath.cx>
References: <m1M9loO-000kmyC@most.weird.com> <4A1EFFA0.2000109@isode.com>
	<fbcbf86738412997e74af4c33ca51a07.squirrel@a-angels.ath.cx>
Message-ID: <m1MARFr-000koTC@most.weird.com>

At Fri, 29 May 2009 12:52:56 +0200 (CEST), "Andreas Winkelmann" <ml at awinkelmann.de> wrote:
Subject: Re: The ldapdb plug-in initialization for clients is too noisy!
> 
> > Greg, you don't need to change anything, because there is a solution to
> > you problem already
> > Just set the "auxprop_plugin" option to exclude ldapdb.
> 
> This will not stop the auxprop-plugins from beeing loaded/initialized. And
> exactly this gives the warnings/errors.
> 
> If it is really needed to have the unused files on disk, someone can add
> some dummy-options to the config file(s).
> 
> For ldabdb:
> 
> ldapdb_uri: dummy or wahtever
> 
> For sql:
> 
> sql_select: dummy or whatever

The more I look at this code, the more I think that the ldapdb (and sql)
plug-in is horribly abusing the *_*_plug_init() functions for things it
should not be doing at that point.

(there's also a whole huge botch on how plug-in version checking is done
as currently both the caller and the callee are doing it!)

I think the proper thing to do, at least the minimal proper thing
without rully re-engineering everything so that the *_plug_init() API is
better respected, is as I suggested:  return SASL_NOMECH when the
configuration isn't set (instead of the bogus SASL_BADPARAM value) and
then don't scream so loudly for the log message when it the result is
simply SASL_NOMECH (some cleanup and other changes here too, but none
for SQL as I don't and won't use it).

This code really needs a huge amount more TLC and re-factoring throughout.



Index: plugins/ldapdb.c
===================================================================
RCS file: /cvs/src/sasl/plugins/ldapdb.c,v
retrieving revision 1.4
diff -u -r1.4 ldapdb.c
--- plugins/ldapdb.c	23 Nov 2008 17:20:52 -0000	1.4
+++ plugins/ldapdb.c	29 May 2009 22:52:43 -0000
@@ -393,8 +393,7 @@
 
     if (ret != LDAP_SUCCESS) goto done;
 
-    for(msg=ldap_first_message(cp.ld, res); msg; msg=ldap_next_message(cp.ld, msg))
-    {
+    for(msg=ldap_first_message(cp.ld, res); msg; msg=ldap_next_message(cp.ld, msg)) {
     	if (ldap_msgtype(msg) != LDAP_RES_SEARCH_ENTRY) continue;
 	bvals = ldap_get_values_len(cp.ld, msg, attrs[0]);
 	if (!bvals) continue;
@@ -457,41 +456,42 @@
     const char *s;
     unsigned len;
 
-    if(p->inited) return SASL_OK;
+    if (p->inited)
+	return SASL_OK;
 
     utils->getopt(utils->getopt_context, ldapdb, "ldapdb_uri", &p->uri, NULL);
-    if(!p->uri) return SASL_BADPARAM;
+    if (!p->uri)
+	return SASL_NOMECH;
 
     utils->getopt(utils->getopt_context, ldapdb, "ldapdb_id",
-    	(const char **)&p->id.bv_val, &len);
+		  (const char **)&p->id.bv_val, &len);
     p->id.bv_len = len;
     utils->getopt(utils->getopt_context, ldapdb, "ldapdb_pw",
-    	(const char **)&p->pw.bv_val, &len);
+		  (const char **)&p->pw.bv_val, &len);
     p->pw.bv_len = len;
     utils->getopt(utils->getopt_context, ldapdb, "ldapdb_mech",
-    	(const char **)&p->mech.bv_val, &len);
+		  (const char **)&p->mech.bv_val, &len);
     p->mech.bv_len = len;
     utils->getopt(utils->getopt_context, ldapdb, "ldapdb_starttls", &s, NULL);
-    if (s)
-    {
-    	if (!strcasecmp(s, "demand")) p->use_tls = 2;
-	else if (!strcasecmp(s, "try")) p->use_tls = 1;
+    if (s) {
+    	if (!strcasecmp(s, "demand"))
+	    p->use_tls = 2;
+	else if (!strcasecmp(s, "try"))
+	    p->use_tls = 1;
     }
     utils->getopt(utils->getopt_context, ldapdb, "ldapdb_rc", &s, &len);
-    if (s)
-    {
+    if (s) {
     	char *str = utils->malloc(sizeof("LDAPRC=")+len);
 	if (!str) return SASL_NOMEM;
 	strcpy( str, "LDAPRC=" );
 	strcpy( str + sizeof("LDAPRC=")-1, s );
-	if (putenv(str))
-	{
+	if (putenv(str)) {
 	    utils->free(str);
 	    return SASL_NOMEM;
 	}
     }
     utils->getopt(utils->getopt_context, ldapdb, "ldapdb_canon_attr",
-	(const char **)&p->canon.bv_val, &len);
+		  (const char **)&p->canon.bv_val, &len);
     p->canon.bv_len = len;
     p->inited = 1;
 
Index: lib/server.c
===================================================================
RCS file: /cvs/src/sasl/lib/server.c,v
retrieving revision 1.160
diff -u -r1.160 server.c
--- lib/server.c	8 Apr 2009 19:36:20 -0000	1.160
+++ lib/server.c	29 May 2009 22:52:44 -0000
@@ -423,15 +423,17 @@
     if ((result != SASL_OK) && (result != SASL_NOUSER)
         && (result != SASL_CONTINUE)) {
 	_sasl_log(NULL, SASL_LOG_DEBUG,
-		  "server add_plugin entry_point error %z\n", result);
+		  "%s_client_plug_init() failed in sasl_server_add_plugin(): %z\n",
+		  plugname, result);
 	return result;
     }
 
     /* Make sure plugin is using the same SASL version as us */
+    /* XXX why check this again?  *_server_plug_init() functions do this check already! */
     if (version != SASL_SERVER_PLUG_VERSION)
     {
 	_sasl_log(NULL, SASL_LOG_ERR,
-		  "version mismatch on plugin");
+		  "version mismatch in sasl_server_add_plugin() for plugin '%s'", plugname);
 	return SASL_BADVERS;
     }
 
Index: lib/client.c
===================================================================
RCS file: /cvs/src/sasl/lib/client.c,v
retrieving revision 1.74
diff -u -r1.74 client.c
--- lib/client.c	8 Apr 2009 19:36:20 -0000	1.74
+++ lib/client.c	29 May 2009 22:52:44 -0000
@@ -157,11 +157,12 @@
   if (result != SASL_OK)
   {
     _sasl_log(NULL, SASL_LOG_WARN,
-	      "entry_point failed in sasl_client_add_plugin for %s",
-	      plugname);
+	      "%s_client_plug_init(): failed in sasl_client_add_plugin(): %z",
+	      plugname, result);
     return result;
   }
 
+  /* XXX why check this again?  *_client_plug_init() functions do this check already! */
   if (version != SASL_CLIENT_PLUG_VERSION)
   {
     _sasl_log(NULL, SASL_LOG_WARN,
@@ -442,6 +443,17 @@
  * Apps should be encouraged to simply use space or comma space
  * though
  */
+/*
+ * XXX Debugging client/server "No worthy mechs found" failures from the logic
+ * herein is next to impossible!
+ *
+ * Error flags should be set so that users can determine which requirements
+ * were met and which were not.
+ *
+ * In the mean time the best one can do is stop execution in this function,
+ * print *conn, *mechlist, etc., and then, if necessary, i.e. it's not
+ * blatantly obvious, step through each rule to see what happens.
+ */
 int sasl_client_start(sasl_conn_t *conn,
 		      const char *mechlist,
 		      sasl_interact_t **prompt_need,
@@ -507,7 +519,7 @@
 
 	/* foreach in client list */
 	for (m = cmechlist->mech_list; m != NULL; m = m->next) {
-	    int myflags;
+	    unsigned int myflags;
 	    
 	    /* Is this the mechanism the server is suggesting? */
 	    if (strcasecmp(m->m.plug->mech_name, name))
@@ -521,15 +533,16 @@
 	    if (minssf > m->m.plug->max_ssf)
 		break;
 
-	    /* Does it meet our security properties? */
+	    /* if there's an external layer with a better SSF then this is no
+	     * longer considered a plaintext mechanism
+	     */
 	    myflags = conn->props.security_flags;
-	    
-	    /* if there's an external layer this is no longer plaintext */
 	    if ((conn->props.min_ssf <= conn->external.ssf) && 
 		(conn->external.ssf > 1)) {
 		myflags &= ~SASL_SEC_NOPLAINTEXT;
 	    }
 
+	    /* Does it meet our security properties? */
 	    if (((myflags ^ m->m.plug->security_flags) & myflags) != 0) {
 		break;
 	    }
Index: lib/canonusr.c
===================================================================
RCS file: /cvs/src/sasl/lib/canonusr.c,v
retrieving revision 1.20
diff -u -r1.20 canonusr.c
--- lib/canonusr.c	10 Mar 2009 16:27:52 -0000	1.20
+++ lib/canonusr.c	29 May 2009 22:52:44 -0000
@@ -307,14 +307,15 @@
 			   &out_version, &plug, plugname);
 
     if(result != SASL_OK) {
-	_sasl_log(NULL, SASL_LOG_ERR, "canonuserfunc error %i\n",result);
+	_sasl_log(NULL, SASL_LOG_ERR, "%s_canonuser_plug_init() failed in sasl_canonuser_add_plugin(): %z\n",
+		  plugname, result);
 	return result;
     }
 
     if(!plug->canon_user_server && !plug->canon_user_client) {
 	/* We need at least one of these implemented */
 	_sasl_log(NULL, SASL_LOG_ERR,
-		  "canonuser plugin without either client or server side");
+		  "canonuser plugin '%s' without either client or server side", plugname);
 	return SASL_BADPROT;
     }
     
Index: lib/auxprop.c
===================================================================
RCS file: /cvs/src/sasl/lib/auxprop.c,v
retrieving revision 1.19
diff -u -r1.19 auxprop.c
--- lib/auxprop.c	28 Jan 2009 22:49:14 -0000	1.19
+++ lib/auxprop.c	29 May 2009 22:52:45 -0000
@@ -813,13 +813,15 @@
 
     /* Check if out_version is too old.
        We only support the current at the moment */
+    /* XXX why check this again?  *_auxprop_plug_init() functions do this check already! */
     if (result == SASL_OK && out_version < SASL_AUXPROP_PLUG_VERSION) {
         result = SASL_BADVERS;
     }
 
     if(result != SASL_OK) {
-	_sasl_log(NULL, SASL_LOG_ERR, "auxpropfunc error %s\n",
-		  sasl_errstring(result, NULL, NULL));
+	_sasl_log(NULL, (result != SASL_NOMECH) ? SASL_LOG_WARN : SASL_LOG_NOTE,
+		  "%s_auxprop_plug_init() failed in sasl_auxprop_add_plugin(): %z\n",
+		  plugname, result);
 	return result;
     }
 

-- 
						Greg A. Woods

+1 416 218-0098                VE3TCP          RoboHack <woods at robohack.ca>
Planix, Inc. <woods at planix.com>      Secrets of the Weird <woods at weird.com>