[GAP] isMemberOf add/removes

[Michael Gettes]

Excellent! A happy customer!  :-)

/mrg

On Dec 16, 2015 23:43, "Jeff McCullough" <jeffmc at berkeley.edu> wrote:

> Thank you. Splitting the queues was it. Both queues were processed in no
> time, 3 min, which is congruent with your experience. Both queues were
> emptied about the same time.
>
> Jeff
>
> On Dec 16, 2015, at 8:09 PM, Michael Gettes <gettes at gmail.com> wrote:
>
> +10000
>
> And please don't get hung up on the notion of running an additional queue
> and processor is expensive.  It isn't!  Just do it.  :-)
>
> /mrg
>
> On Dec 16, 2015 22:16, "Jeffrey Eaton via Identity-services-gap" <
> identity-services-gap at lists.andrew.cmu.edu> wrote:
>
>> I would strongly recommend splitting the isMemberOf and group member
>> entries into two different ActiveMQ queues, and running two parallel copies
>> of the GAP code - one against each queue.  The group member queue will run
>> much faster because the gap process can look ahead in the queue to batch up
>> multiple changes to the same group object.  The isMemberOf  queue will
>> still be slower, but should not be unreasonably so.
>>
>> -jeaton
>>
>>
>> > On Dec 16, 2015, at 9:43 PM, Jeff McCullough via Identity-services-gap <
>> identity-services-gap at lists.andrew.cmu.edu> wrote:
>> >
>> > Hi Michael,
>> >
>> > Okay. Some new data. An old email you wrote:
>> >
>> >> Folks - we have been running with this code in production.  This
>> matches up with prior email about getting large groups into LDAP very
>> quickly.  We see it taking about 5 min to create a group of 25,000 members
>> (for static group objects) if conditions are right.  LDAP is no longer the
>> limiting factor but how fast one gets the data out of grouper is now the
>> issue.  of course, performing 25000 mods for isMemberOf takes longer but
>> group objects and isMemberOf are separate processes.
>> >
>> > I just added 17k members to a group. It took about 50 min to finish.
>> The amq queue (includes addmember and addIsMemberOf entries) was being
>> processed around 24 / sec to begin and around 5 / sec by the end. The
>> limiting factor seems to be the group searches to determine if the member
>> already exists. As the group grows the searches take longer. Alternately,
>> doing a full sync on a group is very, very fast because the entries are
>> batched. There may still be something we can optimize, but just want to get
>> some clarity on your data. When you say you created a group of 25k members
>> in 5 min, that it would need to be the fullSync batching style rather than
>> individual adds as I saw in the logs?
>> >
>> > Thanks,
>> > Jeff
>> >
>>
>> _______________________________________________
>> Identity-services-gap mailing list
>> Identity-services-gap at lists.andrew.cmu.edu
>> https://lists.andrew.cmu.edu/mailman/listinfo/identity-services-gap
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.andrew.cmu.edu/pipermail/identity-services-gap/attachments/20151216/f190b6d9/attachment.html>