[GAP] isMemberOf add/removes

[Michael Gettes]

+10000

And please don't get hung up on the notion of running an additional queue
and processor is expensive.  It isn't!  Just do it.  :-)

/mrg

On Dec 16, 2015 22:16, "Jeffrey Eaton via Identity-services-gap" <
identity-services-gap at lists.andrew.cmu.edu> wrote:

> I would strongly recommend splitting the isMemberOf and group member
> entries into two different ActiveMQ queues, and running two parallel copies
> of the GAP code - one against each queue.  The group member queue will run
> much faster because the gap process can look ahead in the queue to batch up
> multiple changes to the same group object.  The isMemberOf  queue will
> still be slower, but should not be unreasonably so.
>
> -jeaton
>
>
> > On Dec 16, 2015, at 9:43 PM, Jeff McCullough via Identity-services-gap <
> identity-services-gap at lists.andrew.cmu.edu> wrote:
> >
> > Hi Michael,
> >
> > Okay. Some new data. An old email you wrote:
> >
> >> Folks - we have been running with this code in production.  This
> matches up with prior email about getting large groups into LDAP very
> quickly.  We see it taking about 5 min to create a group of 25,000 members
> (for static group objects) if conditions are right.  LDAP is no longer the
> limiting factor but how fast one gets the data out of grouper is now the
> issue.  of course, performing 25000 mods for isMemberOf takes longer but
> group objects and isMemberOf are separate processes.
> >
> > I just added 17k members to a group. It took about 50 min to finish. The
> amq queue (includes addmember and addIsMemberOf entries) was being
> processed around 24 / sec to begin and around 5 / sec by the end. The
> limiting factor seems to be the group searches to determine if the member
> already exists. As the group grows the searches take longer. Alternately,
> doing a full sync on a group is very, very fast because the entries are
> batched. There may still be something we can optimize, but just want to get
> some clarity on your data. When you say you created a group of 25k members
> in 5 min, that it would need to be the fullSync batching style rather than
> individual adds as I saw in the logs?
> >
> > Thanks,
> > Jeff
> >
>
> _______________________________________________
> Identity-services-gap mailing list
> Identity-services-gap at lists.andrew.cmu.edu
> https://lists.andrew.cmu.edu/mailman/listinfo/identity-services-gap
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/identity-services-gap/attachments/20151216/8cf52142/attachment.html>