[GAP] Grouper ActiveMQ Provisioner (GAP)

Thu Oct 2 14:59:33 EDT 2014

That’s wonderful news Albert.  We look forward to working with you.  I should have been more clear, the mailing list is something you can subscribe to so we can have discussion there as well.  Please feel free to subscribe.

/mrg

On Oct 2, 2014, at 2:38 PM, Wu, Albert <albertwu at ucla.edu<mailto:albertwu at ucla.edu>> wrote:

Michael,

UCLA is interested in participating in GAP's development. We'll begin taking a look at the code and hopefully be able to participate intelligently at TechX.

albert
________________________________
From: grouper-users-request at internet2.edu<mailto:grouper-users-request at internet2.edu> <grouper-users-request at internet2.edu<mailto:grouper-users-request at internet2.edu>> on behalf of Michael R. Gettes <gettes at cmu.edu<mailto:gettes at cmu.edu>>
Sent: Thursday, October 2, 2014 11:32 AM
To: grouper-users at internet2.edu<mailto:grouper-users at internet2.edu>
Subject: [grouper-users] Re: Grouper ActiveMQ Provisioner (GAP)

Given the discussion around the future of PSP…

CMU would like to offer up the following in response to our email from back in July.  We were hoping PSU would have an opportunity to work with the code but, as unfortunately happens and we’ve all been there, the PSU priorities changed and they have not yet had a chance to work with the GAP code outlined below.

We have updated our code to include changes to better support Active Directory and the myriad of AD enabled apps and their expectations on how attributes get set.

We have one more significant update to the code we will be making around batched updates.  Right now, every grouper change is processed with a corresponding single update to a target LDAP server.  In the not too distant future, we will have support to batch grouper updates by 500 or 1000 which will dramatically improve performance for static group objects.  At this time, we believe the batch update is the only future update we intend to make for now pending the future direction of the Grouper project along with feedback provided by others.  We are happy to work with others and this is why we share this code.

If you would like to begin using our code or just give it a look, it is available at:

https://github.com/cmu-ids

If you have questions, please feel free to email identity-services-gap at lists.andrew.cmu.edu<mailto:identity-services-gap at lists.andrew.cmu.edu> - we appreciate any feedback.

Our hope is the capabilities of GAP will be useful input to the future of integrations with Grouper as noted at https://spaces.internet2.edu/display/Grouper/Post+PSP+Provisioning

and hopefully we can have some interesting discussions about all this during ACAMP at TechX in Indianapolis at the end of this month.

/mrg

On Jul 14, 2014, at 4:21 PM, Michael Gettes <gettes at cmu.edu<mailto:gettes at cmu.edu>> wrote:

Kind Grouper Community,

Carnegie Mellon University’s Identity Services Team has developed some code that may be of interest.  We weren’t satisfied with the flexibility and performance of PSP.  In our environment, we saw an average of 3 grouper change operations per second - sometimes as high as 9 or 10, using PSP.  We have a number of large groups and lots of group dependencies based on group math and group nesting for many purposes.  We use groups to provision services like GoogleApps and are in process of deploying OIM (still !!!) where we use groups to describe who gets various services.  In other words, I think we are effectively trying to use Grouper itself.  However, our 389 and AD LDAP environments weren’t being updated fast enough.

ActiveMQ can be found at activemq.apache.org<http://activemq.apache.org/>.  With AMQ we can easily monitor queue activity, usage over time, graphing usage and monitor for backlogs so we can actually see how our environment is doing.  This is not something easily achieved within Grouper itself or with PSP.  There is no real dependency on AMQ with what we have developed other than the API calls used to plug into AMQ.  We make use of JMS and the Stomp messaging protocol.  It shouldn’t be hard to substitute another messaging service if you want to do so.  Our AMQ environment is 2 servers in redundant mode running 5.9.0 and this wasn’t hard to get going.  We, identity services, run the AMQ environment.  In our testing, we have processed many many millions of messages.  AMQ is LDAP enabled for authentication and access control.  Our AMQ is only for Identity related activities - we do not run a generalized message service for applications.  We do NOT run this as an ESB even though AMQ is used to create ESBs.  Our mindset is to “keep it simple”.

Much of the code I am about to describe was written by Rahul Doshi of CMU.  The design was a team effort.  This was Rahul’s first significant development in Perl.  He’s rather used to developing in Java.  We got some input from Brown University and their grouper changelog to AMQ code, but we ended up redoing everything from scratch.

We process the grouper changelog and express the changes in simple messages to ActiveMQ.  We experienced the ability to write about 10,000 messages per minute to AMQ (that’s 167/sec we believe this number has gone up quite a bit since our original tests).  We have a dispatcher which, via a config file, will resend the grouper change messages to other queues within AMQ.  This allows us to filter changes so a queue will only see what it needs to see.  For example, we don’t want to publish to AD our course groups, so we don’t send those changes by specifying an appropriate pattern.  It’s not so much how fast the messages get passed around - at least we know with AMQ it isn’t a concern.  Of course, how fast we process messages by the consumers (described below) is the real issue.

For our 389 LDAP server (which could be any LDAP server really), we have 2 consumers.  One to process standard LDAP group objects and the other consumer processes isMemberOf attributes in the user object - similar to the AD memberOf.  A single change for add userid X to group Y would get translated into 2 messages sent to the 389.groups and 389.ismemberof queues in AMQ.  The consumers, written in perl, read a message off the queue, processes the change, and moves on to the next.  The consumers support a full-sync mode to allow for a push of all groups to be re-sync’d as well as the normal mode of handling each individual change as it comes.

Some numbers:  under load, we have seen 30 messages/sec for normal mode (not full sync) messages.  Each result in multiple LDAP operations.  Performance really depends on the LDAP server and its configuration.   We have seen LDAP ops as high as 174/sec and these are irrespective of whether MOD ops are done.  We can create or delete a group with about 30,000 members in about 5-7 min (not including LDAP replication).  A full group sync of about 5000 groups where little changes are made takes about 25 min.  For isMemberOf processing, a full sync takes about 33 minutes.  We believe we can still speed up the full-sync aspects of this - so this effort is still a work in progress.  Our hope is full-sync operations would be done no more frequently than weekly.  We found PSP getting out of sync and the need for full-sync operations.  We are working towards full-sync operations being performed on an exceptional basis.

For AD, we support bushy style LDAP structure and handle DNs within MS specs (each DN component can’t be more than 64 chars) and the samAccountName is reversed to make it usable by normal AD group UI mechanisms.

We are in process of working through licensing issues with our tech transfer office (hence my questions about licensing).  PSU is interested in using this code and we will work with them to make sure it is sufficiently flexible in configuration and works outside of CMU.  We expect to have the code available on github at some point in the coming couple of months.

Given our work, we still believe Grouper should package ActiveMQ with it so custom changelog processors no longer need to be written - and it would just be AMQ consumers for people to do whatever they want.  Consumers are so easy to write.  We have external to Identity Services various applications reading identity data from AMQ, in message formats we specify and control who can read/write queues.

The intent with this email was to give a brief overview of what we have been doing.  The numbers are complicated but look pretty good, we haven’t provided gory details because of the complexity of things.  GAP has been in production, fully replacing PSP, since June 1, 2014.

Lastly, THANK YOU SO MUCH to the Grouper team for Grouper 2.2 and the new UI.  CMU is very excited to begin work (probably tomorrow) on getting this new version of Grouper up and running and to have it deployed to production as soon as is practical.

Happy to answer questions about GAP, if you have any.

/mrg

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.andrew.cmu.edu/pipermail/identity-services-gap/attachments/20141002/0b7bc2cd/attachment-0001.html 