[GAP] isMemberOf add/removes

[Jeff McCullough]

Hi Michael,

Okay. Some new data. An old email you wrote:

> Folks - we have been running with this code in production.  This matches up with prior email about getting large groups into LDAP very quickly.  We see it taking about 5 min to create a group of 25,000 members (for static group objects) if conditions are right.  LDAP is no longer the limiting factor but how fast one gets the data out of grouper is now the issue.  of course, performing 25000 mods for isMemberOf takes longer but group objects and isMemberOf are separate processes.

I just added 17k members to a group. It took about 50 min to finish. The amq queue (includes addmember and addIsMemberOf entries) was being processed around 24 / sec to begin and around 5 / sec by the end. The limiting factor seems to be the group searches to determine if the member already exists. As the group grows the searches take longer. Alternately, doing a full sync on a group is very, very fast because the entries are batched. There may still be something we can optimize, but just want to get some clarity on your data. When you say you created a group of 25k members in 5 min, that it would need to be the fullSync batching style rather than individual adds as I saw in the logs?

Thanks,
Jeff


> On Dec 15, 2015, at 12:14 PM, Michael R Gettes <gettes at gmail.com> wrote:
> 
> Jeff,
> 
> what you report doesn’t make sense.  I think you need to take a look at the ldap logs to see what is taking so long and what operations are being made.  All the work I did with Rahul at CMU to make sure the right stuff was being done was approached from a generic LDAP perspective.  Your JMeter tests are likely not doing what GAP is doing.  Maybe from the logs you will see what GAP is doing and can then repeat it JMeter.  My guess would be you are not indexing isMemberOf or something GAP is searching for is not indexed in LDAP.  I’d be happy to help further if I can.
> 
> /mrg
> 
>> On Dec 15, 2015, at 12:33 AM, Jeff McCullough via Identity-services-gap <identity-services-gap at lists.andrew.cmu.edu> wrote:
>> 
>> Hi Everyone...
>> 
>> What sort of throughput are you getting for isMemberOf add/removes. I’m seeing just over 3 per sec which seems incredibly slow. I’ve run JMeter tests against our LDAP server (OpenDJ), and it runs around 80 per sec. When the consumer and dispatcher are running, the queues move quickly so don’t think that is the bottleneck either. SoI’m just looking to see if you have similar or vastly different results?
>> 
>> Thanks,
>> Jeff
>> _______________________________________________
>> Identity-services-gap mailing list
>> Identity-services-gap at lists.andrew.cmu.edu
>> https://lists.andrew.cmu.edu/mailman/listinfo/identity-services-gap
>