[GAP] isMemberOf add/removes

[Jeff McCullough]

Thank you. Splitting the queues was it. Both queues were processed in no time, 3 min, which is congruent with your experience. Both queues were emptied about the same time.

Jeff

> On Dec 16, 2015, at 8:09 PM, Michael Gettes <gettes at gmail.com> wrote:
> 
> +10000
> 
> And please don't get hung up on the notion of running an additional queue and processor is expensive.  It isn't!  Just do it.  :-)
> 
> /mrg
>    
> 
> On Dec 16, 2015 22:16, "Jeffrey Eaton via Identity-services-gap" <identity-services-gap at lists.andrew.cmu.edu <mailto:identity-services-gap at lists.andrew.cmu.edu>> wrote:
> I would strongly recommend splitting the isMemberOf and group member entries into two different ActiveMQ queues, and running two parallel copies of the GAP code - one against each queue.  The group member queue will run much faster because the gap process can look ahead in the queue to batch up multiple changes to the same group object.  The isMemberOf  queue will still be slower, but should not be unreasonably so.
> 
> -jeaton
> 
> 
> > On Dec 16, 2015, at 9:43 PM, Jeff McCullough via Identity-services-gap <identity-services-gap at lists.andrew.cmu.edu <mailto:identity-services-gap at lists.andrew.cmu.edu>> wrote:
> >
> > Hi Michael,
> >
> > Okay. Some new data. An old email you wrote:
> >
> >> Folks - we have been running with this code in production.  This matches up with prior email about getting large groups into LDAP very quickly.  We see it taking about 5 min to create a group of 25,000 members (for static group objects) if conditions are right.  LDAP is no longer the limiting factor but how fast one gets the data out of grouper is now the issue.  of course, performing 25000 mods for isMemberOf takes longer but group objects and isMemberOf are separate processes.
> >
> > I just added 17k members to a group. It took about 50 min to finish. The amq queue (includes addmember and addIsMemberOf entries) was being processed around 24 / sec to begin and around 5 / sec by the end. The limiting factor seems to be the group searches to determine if the member already exists. As the group grows the searches take longer. Alternately, doing a full sync on a group is very, very fast because the entries are batched. There may still be something we can optimize, but just want to get some clarity on your data. When you say you created a group of 25k members in 5 min, that it would need to be the fullSync batching style rather than individual adds as I saw in the logs?
> >
> > Thanks,
> > Jeff
> >
> 
> _______________________________________________
> Identity-services-gap mailing list
> Identity-services-gap at lists.andrew.cmu.edu <mailto:Identity-services-gap at lists.andrew.cmu.edu>
> https://lists.andrew.cmu.edu/mailman/listinfo/identity-services-gap <https://lists.andrew.cmu.edu/mailman/listinfo/identity-services-gap>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/identity-services-gap/attachments/20151216/da14da57/attachment-0001.html>