[GAP] Update

[Waldbieser, Carl]

Michael,

Yes, I think I understand the GAP strategy, now.  A single execution path for each directory will ensure that you don't have processes overwriting each other's work, which is more of a challenge for an LDAP directory than say an RDBMS, where transactions would come to your rescue.

I agree with the idea that an eventual consistency is good enough for most apps.  

I don't have particular concerns, but I was knocking around the idea of how one might increase the efficiency of the system by having separate worker processes for different groups within the same directory.   It sounds like the latency with a single worker per directory is not really much of an issue, though.

Thanks,
Carl

----- Original Message -----
From: "Michael R. Gettes" <gettes at cmu.edu>
To: "Carl Waldbieser" <waldbiec at lafayette.edu>
Cc: identity-services-gap at lists.andrew.cmu.edu
Sent: Thursday, November 13, 2014 11:52:17 AM
Subject: Re: [GAP] Update

Carl, I hope Rahul’s answer is sufficient.  We have similar concerns but, based on app behavior, have come to realize updates need to be timely but not immediate.  Our strategy seems to show updates can be made to multiple servers and applied sufficiently fast where applications really don’t notice any inconsistencies.  Perfect is the enemy of good enough (or the saying is something like it).  We find good enough is what apps need.  We believe our approach is very practical.  Happy to discuss any particular concerns you may have either on this list or separately.

/mrg

> On Nov 13, 2014, at 11:03 AM, Waldbieser, Carl <waldbiec at lafayette.edu> wrote:
> 
> Michael,
> 
> Is there an architectural diagram of GAP somewhere?
> I was wondering if GAP is farming out group updates to multiple LDAP-updaters.
> At Lafayette, account memberships and group members must be kept in sync in application logic (our OpenLDAP directory service does not synchronize these values).  So I was also wondering if the components of GAP that update LDAP can distribute the tasks for updating group members and updating account memberships to separate workers?
> 
> Thanks,
> Carl Waldbieser
> ITS System Programmer
> Lafayette College
> 
> ----- Original Message -----
> From: "Michael R. Gettes" <gettes at cmu.edu>
> To: identity-services-gap at lists.andrew.cmu.edu
> Sent: Thursday, November 13, 2014 10:31:34 AM
> Subject: [GAP] Update
> 
> Hi All,
> 
> I am hoping Rahul will be providing updates from this point forward.  
> 
> Based on discussions at TechX, CMU will continue to develop GAP as the grouper devs will be providing a basic capability within grouper so there will still be a need for something to provide efficient and fast LDAP provisioning (and possibly other provisioning capabilities).  Keep in mind while we have been concentrating on LDAP and AD, GAP could be extended to other capabilities if it is an appropriate mechanism to do so.  We already have about 10 institutions on this mailing list.  We do hope GAP is useful to others as we don’t want this to be a CMU only piece of software.
> 
> Rahul is concentrating on getting grouper 2.2.1 out the door but has a working prototype for batching of large changes to static group objects.  Essentially, we are doing a read-ahead into the AMQ queue and applying up to 1000 adds or deletes in a single LDAP modify.  Initial testing gets group creation or deletion down to around 3-4 min for a group of 25,000.  Keep in mind GAP already does periodic bulk updates and this change is to make normal incremental operations to static group objects even faster.  At this time, we don’t think there is much we can do to speed up isMemberOf operations.  We are getting more comfortable with the idea of sites like PSU and Indiana being able to use this successfully.  Our concern is we have made this work for CMU and code changes may be needed to be more flexible for different types of operating scenarios and DITs and so on.  So, the more who work on it, the better it will be for all.
> 
> It was great seeing folks at TechX.
> 
> /mrg
> 
> _______________________________________________
> Identity-services-gap mailing list
> Identity-services-gap at lists.andrew.cmu.edu
> https://lists.andrew.cmu.edu/mailman/listinfo/identity-services-gap