Pedro, we had discussions on performance associated in querying custom attributes in the LDAP implementation. I realized that since we will have an identity cache operating in the IDM layer. The cache needs to have LRU entries (or whatever policy that gets configured) thus avoiding round trips to the Identity Store. Bolek had opined about the use of LDAP entry change notifications to update the IDM cache. This is when the admin may have used some form of LDAP browser to update the entries or update happens via software not controlled by IDM.

Regards, Anil _______________________________________________ security-dev mailing list security-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/security-dev

----- Original Message ----- > From: "Anil Saldhana" <Anil.Saldhana(a)redhat.com&gt; > To: security-dev(a)lists.jboss.org > Sent: Thursday, December 6, 2012 12:06:03 AM > Subject: [security-dev] IDM: LDAP Custom Attributes > > Pedro, > we had discussions on performance associated in querying custom > attributes in the LDAP implementation. I realized that since we will > have an identity cache operating in the IDM layer. The cache needs to > have LRU entries (or whatever policy that gets configured) thus > avoiding > round trips to the Identity Store. You're right, one of the biggest challenges is how to perform well when querying attributes that are not part of the LDAP schema. Those attributes are not searchable and we need to make most of the query logic inside the store.

> > Bolek had opined about the use of LDAP entry change notifications to > update the IDM cache. This is when the admin may have used some form > of > LDAP browser to update the entries or update happens via software not > controlled by IDM. > Ok, going to consider that too. > Regards, > Anil > _______________________________________________ > security-dev mailing list > security-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/security-dev > _______________________________________________ security-dev mailing list security-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/security-dev

----- Original Message ----- > From: "Boleslaw Dawidowicz" <bdawidow(a)redhat.com&gt; To: "Pedro Igor > Silva" <psilva(a)redhat.com&gt; Cc: "Anil Saldhana" > <Anil.Saldhana(a)redhat.com&gt;, security-dev(a)lists.jboss.org Sent: > Thursday, December 6, 2012 10:51:00 AM Subject: Re: [security-dev] > IDM: LDAP Custom Attributes > > > On Dec 6, 2012, at 12:50 PM, Pedro Igor Silva <psilva(a)redhat.com&gt; > wrote: > >> ----- Original Message ----- >>> From: "Anil Saldhana" <Anil.Saldhana(a)redhat.com&gt; To: >>> security-dev(a)lists.jboss.org Sent: Thursday, December 6, 2012 >>> 12:06:03 AM Subject: [security-dev] IDM: LDAP Custom >>> Attributes >>> >>> Pedro, we had discussions on performance associated in querying >>> custom attributes in the LDAP implementation. I realized that >>> since we will have an identity cache operating in the IDM >>> layer. The cache needs to have LRU entries (or whatever policy >>> that gets configured) thus avoiding round trips to the Identity >>> Store. >> >> You're right, one of the biggest challenges is how to perform >> well when querying attributes that are not part of the LDAP >> schema. Those attributes are not searchable and we need to make >> most of the query logic inside the store. > > In case of LDAP I would really allow only attributes mapped > previously in the store configuration. There are too many > scenarios - like some are readOnly and managed by the server > (memberOf). LDAP is also not flexibly used store because of > enforced schema so it is a valid constraint - simplifies a lot. For > custom attributes stored in serialised manner I would simply not > allow to use them in queries or ignore in such. Simplifies a lot. Totally agree, but from our last discussion Anil suggested to still support custom attributes using serializable objects. Ignore them would be much more simple, but i understood Anil's point of view.

The current query support implementation for the LDAP store is working with managed and custom attributes. If the attribute used as a query parameter is managed, a normal LDAP search will be performed. Otherwise, we'll filter the users manually. Users should be aware that when used custom attributes they lose some performance on queries. > >> >>> >>> Bolek had opined about the use of LDAP entry change >>> notifications to update the IDM cache. This is when the admin >>> may have used some form of LDAP browser to update the entries >>> or update happens via software not controlled by IDM. >>> >> >> Ok, going to consider that too. >> >>> Regards, Anil _______________________________________________ >>> security-dev mailing list security-dev(a)lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/security-dev >>> >> _______________________________________________ security-dev >> mailing list security-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/security-dev > > _______________________________________________ security-dev mailing list security-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/security-dev

On 12/06/2012 07:15 AM, Pedro Igor Silva wrote: > ----- Original Message ----- >> From: "Boleslaw Dawidowicz" <bdawidow(a)redhat.com&gt; To: "Pedro Igor >> Silva" <psilva(a)redhat.com&gt; Cc: "Anil Saldhana" >> <Anil.Saldhana(a)redhat.com&gt;, security-dev(a)lists.jboss.org Sent: >> Thursday, December 6, 2012 10:51:00 AM Subject: Re: [security-dev] >> IDM: LDAP Custom Attributes >> >> >> On Dec 6, 2012, at 12:50 PM, Pedro Igor Silva <psilva(a)redhat.com&gt; >> wrote: >> >>> ----- Original Message ----- >>>> From: "Anil Saldhana" <Anil.Saldhana(a)redhat.com&gt; To: >>>> security-dev(a)lists.jboss.org Sent: Thursday, December 6, 2012 >>>> 12:06:03 AM Subject: [security-dev] IDM: LDAP Custom >>>> Attributes >>>> >>>> Pedro, we had discussions on performance associated in querying >>>> custom attributes in the LDAP implementation. I realized that >>>> since we will have an identity cache operating in the IDM >>>> layer. The cache needs to have LRU entries (or whatever policy >>>> that gets configured) thus avoiding round trips to the Identity >>>> Store. >>> >>> You're right, one of the biggest challenges is how to perform >>> well when querying attributes that are not part of the LDAP >>> schema. Those attributes are not searchable and we need to make >>> most of the query logic inside the store. >> >> In case of LDAP I would really allow only attributes mapped >> previously in the store configuration. There are too many >> scenarios - like some are readOnly and managed by the server >> (memberOf). LDAP is also not flexibly used store because of >> enforced schema so it is a valid constraint - simplifies a lot. For >> custom attributes stored in serialised manner I would simply not >> allow to use them in queries or ignore in such. Simplifies a lot. > > Totally agree, but from our last discussion Anil suggested to still > support custom attributes using serializable objects. Ignore them > would be much more simple, but i understood Anil's point of view. I don't know, this is a pretty big can of worms. Supporting serializable objects means supporting their correct deserialization as well, which in turn means that semantics would have to be established regarding the class loaders to use for various classes. And that means that the class loading environment has to be specified, and now it's gotten much more complex than it was before!

I strongly urge you to stick with simple data types for custom attributes: Strings, primitives, that sort of thing. Data types that you *know* can be supported by any backend store.

On 06/12/12 16:11, David M. Lloyd wrote: > On 12/06/2012 07:15 AM, Pedro Igor Silva wrote: >> ----- Original Message ----- >>> From: "Boleslaw Dawidowicz" <bdawidow(a)redhat.com&gt; To: "Pedro Igor >>> Silva" <psilva(a)redhat.com&gt; Cc: "Anil Saldhana" >>> <Anil.Saldhana(a)redhat.com&gt;, security-dev(a)lists.jboss.org Sent: >>> Thursday, December 6, 2012 10:51:00 AM Subject: Re: [security-dev] >>> IDM: LDAP Custom Attributes >>> >>> >>> On Dec 6, 2012, at 12:50 PM, Pedro Igor Silva <psilva(a)redhat.com&gt; >>> wrote: >>> >>>> ----- Original Message ----- >>>>> From: "Anil Saldhana" <Anil.Saldhana(a)redhat.com&gt; To: >>>>> security-dev(a)lists.jboss.org Sent: Thursday, December 6, 2012 >>>>> 12:06:03 AM Subject: [security-dev] IDM: LDAP Custom >>>>> Attributes >>>>> >>>>> Pedro, we had discussions on performance associated in querying >>>>> custom attributes in the LDAP implementation. I realized that >>>>> since we will have an identity cache operating in the IDM >>>>> layer. The cache needs to have LRU entries (or whatever policy >>>>> that gets configured) thus avoiding round trips to the Identity >>>>> Store. >>>> You're right, one of the biggest challenges is how to perform >>>> well when querying attributes that are not part of the LDAP >>>> schema. Those attributes are not searchable and we need to make >>>> most of the query logic inside the store. >>> In case of LDAP I would really allow only attributes mapped >>> previously in the store configuration. There are too many >>> scenarios - like some are readOnly and managed by the server >>> (memberOf). LDAP is also not flexibly used store because of >>> enforced schema so it is a valid constraint - simplifies a lot. For >>> custom attributes stored in serialised manner I would simply not >>> allow to use them in queries or ignore in such. Simplifies a lot. >> Totally agree, but from our last discussion Anil suggested to still >> support custom attributes using serializable objects. Ignore them >> would be much more simple, but i understood Anil's point of view. > I don't know, this is a pretty big can of worms. Supporting > serializable objects means supporting their correct deserialization as > well, which in turn means that semantics would have to be established > regarding the class loaders to use for various classes. And that means > that the class loading environment has to be specified, and now it's > gotten much more complex than it was before! > > I strongly urge you to stick with simple data types for custom > attributes: Strings, primitives, that sort of thing. Data types that > you *know* can be supported by any backend store. I also think that we should stick with simple attribute types like Strings and primitives. In IDM 1.x we supported binary attributes, but it was quite painful process to properly support binary attributes all databases and in the end the binary attributes were never used... And those serialization/classloading issues for support of custom java types as attribute is another potentially painful thing, which would be good to avoid IMO.

Marek >> The current query support implementation for the LDAP store is >> working with managed and custom attributes. If the attribute used as >> a query parameter is managed, a normal LDAP search will be performed. >> Otherwise, we'll filter the users manually. Users should be aware >> that when used custom attributes they lose some performance on >> queries. >> >>>>> Bolek had opined about the use of LDAP entry change >>>>> notifications to update the IDM cache. This is when the admin >>>>> may have used some form of LDAP browser to update the entries >>>>> or update happens via software not controlled by IDM. >>>>> >>>> Ok, going to consider that too. >>>> >>>>

----- Original Message ----- > From: "Boleslaw Dawidowicz" <bdawidow(a)redhat.com&gt; > To: "Pedro Igor Silva" <psilva(a)redhat.com&gt; > Cc: "Anil Saldhana" <Anil.Saldhana(a)redhat.com&gt;, security-dev(a)lists.jboss.org > Sent: Thursday, December 6, 2012 10:51:00 AM > Subject: Re: [security-dev] IDM: LDAP Custom Attributes > > > On Dec 6, 2012, at 12:50 PM, Pedro Igor Silva <psilva(a)redhat.com&gt; > wrote: > >> ----- Original Message ----- >>> From: "Anil Saldhana" <Anil.Saldhana(a)redhat.com&gt; >>> To: security-dev(a)lists.jboss.org >>> Sent: Thursday, December 6, 2012 12:06:03 AM >>> Subject: [security-dev] IDM: LDAP Custom Attributes >>> >>> Pedro, >>> we had discussions on performance associated in querying custom >>> attributes in the LDAP implementation. I realized that since we >>> will >>> have an identity cache operating in the IDM layer. The cache needs >>> to >>> have LRU entries (or whatever policy that gets configured) thus >>> avoiding >>> round trips to the Identity Store. >> You're right, one of the biggest challenges is how to perform well >> when querying attributes that are not part of the LDAP schema. >> Those attributes are not searchable and we need to make most of >> the query logic inside the store. > In case of LDAP I would really allow only attributes mapped > previously in the store configuration. There are too many scenarios > - like some are readOnly and managed by the server (memberOf). LDAP > is also not flexibly used store because of enforced schema so it is > a valid constraint - simplifies a lot. For custom attributes stored > in serialised manner I would simply not allow to use them in queries > or ignore in such. Simplifies a lot. Totally agree, but from our last discussion Anil suggested to still support custom attributes using serializable objects. Ignore them would be much more simple, but i understood Anil's point of view.

The current query support implementation for the LDAP store is working with managed and custom attributes. If the attribute used as a query parameter is managed, a normal LDAP search will be performed. Otherwise, we'll filter the users manually. Users should be aware that when used custom attributes they loose some performance on queries. >>> Bolek had opined about the use of LDAP entry change notifications >>> to >>> update the IDM cache. This is when the admin may have used some >>> form >>> of >>> LDAP browser to update the entries or update happens via software >>> not >>> controlled by IDM. >>> >> Ok, going to consider that too. >> >>> Regards, >>> Anil >>> _______________________________________________ >>> security-dev mailing list >>> security-dev(a)lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/security-dev >>> >> _______________________________________________ >> security-dev mailing list >> security-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/security-dev > _______________________________________________ security-dev mailing list security-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/security-dev

On 12/06/2012 10:41 AM, Marek Posolda wrote: > On 06/12/12 14:15, Pedro Igor Silva wrote: >> ----- Original Message ----- >>> From: "Boleslaw Dawidowicz" <bdawidow(a)redhat.com&gt; >>> To: "Pedro Igor Silva" <psilva(a)redhat.com&gt; >>> Cc: "Anil Saldhana" <Anil.Saldhana(a)redhat.com&gt;, >>> security-dev(a)lists.jboss.org >>> Sent: Thursday, December 6, 2012 10:51:00 AM >>> Subject: Re: [security-dev] IDM: LDAP Custom Attributes >>> >>> >>> On Dec 6, 2012, at 12:50 PM, Pedro Igor Silva <psilva(a)redhat.com&gt; >>> wrote: >>> >>>> ----- Original Message ----- >>>>> From: "Anil Saldhana" <Anil.Saldhana(a)redhat.com&gt; >>>>> To: security-dev(a)lists.jboss.org >>>>> Sent: Thursday, December 6, 2012 12:06:03 AM >>>>> Subject: [security-dev] IDM: LDAP Custom Attributes >>>>> >>>>> Pedro, >>>>> we had discussions on performance associated in querying custom >>>>> attributes in the LDAP implementation. I realized that since we >>>>> will >>>>> have an identity cache operating in the IDM layer. The cache needs >>>>> to >>>>> have LRU entries (or whatever policy that gets configured) thus >>>>> avoiding >>>>> round trips to the Identity Store. >>>> You're right, one of the biggest challenges is how to perform well >>>> when querying attributes that are not part of the LDAP schema. >>>> Those attributes are not searchable and we need to make most of >>>> the query logic inside the store. >>> In case of LDAP I would really allow only attributes mapped >>> previously in the store configuration. There are too many scenarios >>> - like some are readOnly and managed by the server (memberOf). LDAP >>> is also not flexibly used store because of enforced schema so it is >>> a valid constraint - simplifies a lot. For custom attributes stored >>> in serialised manner I would simply not allow to use them in queries >>> or ignore in such. Simplifies a lot. >> Totally agree, but from our last discussion Anil suggested to still >> support custom attributes using serializable objects. Ignore them >> would be much more simple, but i understood Anil's point of view. > It would be nice if users have possibility to choose, so they can either: > 1) Store custom attributes in LDAP using serialization > 2) Store those attributes in different IdentityStore (for example in > database, which is quite flexible and can easily support custom > attributes). We used this in Picketlink IDM 1, where only some > standard attributes of user (firstName, lastName, email) are stored in > LDAP and all other attributes are stored in database. With respect to > membership, LDAP obviously support only plain membership (user "john" > is in group "acme") without specifying role (user "john" is "manager" > of group "acme"), so informations about custom roles also need to be > stored in database. > > We discussed this yesterday on IRC, but not all were here. > > I think that proper support of (2) will require implementation of new > IdentityStore, which will encapsulate other IdentityStores > (LDAPIdentityStore, JPAIdentityStore) and delegate operation to proper > one from those two. In reality, we may end up creating separate identity store implementations for quirky but prominent identity stores such as MS Active Directory. I am unsure MSAD follows all the LDAPv3 rules. Over time based on user feedback and our experiences, we may end up with a bunch of Identity Store implementations. :)

> > Marek > >> >> The current query support implementation for the LDAP store is >> working with managed and custom attributes. If the attribute used as >> a query parameter is managed, a normal LDAP search will be performed. >> Otherwise, we'll filter the users manually. Users should be aware >> that when used custom attributes they loose some performance on queries. >> >>>>> Bolek had opined about the use of LDAP entry change notifications >>>>> to >>>>> update the IDM cache. This is when the admin may have used some >>>>> form >>>>> of >>>>> LDAP browser to update the entries or update happens via software >>>>> not >>>>> controlled by IDM. >>>>> >>>> Ok, going to consider that too. >>>> >>>>> Regards, >>>>> Anil _______________________________________________ security-dev mailing list security-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/security-dev

All those tips are really important. Good to know. I'm sure we need to keep them in mind. You have a strong background on this :) I'm trying to start with a basic/working implementation, providing what the IDM API/SPI asks for. I liked the idea to start thinking about some extensibility for custom/server specific implementations. I'll keep that in mind. As we discussed before, it would be nice to test the default implementation with different ldap servers (msad, openldap, etc) to find any gap that might be covered by the default impl. But first, I want to support at least the basic features to get things working. Btw, I'm looking/getting somethings from 1.x. I hope you don't mind :)

Thanks. Pedro Igor ----- Original Message ----- From: "Boleslaw Dawidowicz" <bdawidow(a)redhat.com&gt; To: "Anil Saldhana" <Anil.Saldhana(a)redhat.com&gt; Cc: security-dev(a)lists.jboss.org Sent: Thursday, December 6, 2012 6:31:24 PM Subject: Re: [security-dev] IDM: LDAP Custom Attributes On Dec 6, 2012, at 5:44 PM, Anil Saldhana <Anil.Saldhana(a)redhat.com&gt; wrote: > On 12/06/2012 10:41 AM, Marek Posolda wrote: >> On 06/12/12 14:15, Pedro Igor Silva wrote: >>> ----- Original Message ----- >>>> From: "Boleslaw Dawidowicz" <bdawidow(a)redhat.com&gt; >>>> To: "Pedro Igor Silva" <psilva(a)redhat.com&gt; >>>> Cc: "Anil Saldhana" <Anil.Saldhana(a)redhat.com&gt;, >>>> security-dev(a)lists.jboss.org >>>> Sent: Thursday, December 6, 2012 10:51:00 AM >>>> Subject: Re: [security-dev] IDM: LDAP Custom Attributes >>>> >>>> >>>> On Dec 6, 2012, at 12:50 PM, Pedro Igor Silva <psilva(a)redhat.com&gt; >>>> wrote: >>>> >>>>> ----- Original Message ----- >>>>>> From: "Anil Saldhana" <Anil.Saldhana(a)redhat.com&gt; >>>>>> To: security-dev(a)lists.jboss.org >>>>>> Sent: Thursday, December 6, 2012 12:06:03 AM >>>>>> Subject: [security-dev] IDM: LDAP Custom Attributes >>>>>> >>>>>> Pedro, >>>>>> we had discussions on performance associated in querying custom >>>>>> attributes in the LDAP implementation. I realized that since we >>>>>> will >>>>>> have an identity cache operating in the IDM layer. The cache needs >>>>>> to >>>>>> have LRU entries (or whatever policy that gets configured) thus >>>>>> avoiding >>>>>> round trips to the Identity Store. >>>>> You're right, one of the biggest challenges is how to perform well >>>>> when querying attributes that are not part of the LDAP schema. >>>>> Those attributes are not searchable and we need to make most of >>>>> the query logic inside the store. >>>> In case of LDAP I would really allow only attributes mapped >>>> previously in the store configuration. There are too many scenarios >>>> - like some are readOnly and managed by the server (memberOf). LDAP >>>> is also not flexibly used store because of enforced schema so it is >>>> a valid constraint - simplifies a lot. For custom attributes stored >>>> in serialised manner I would simply not allow to use them in queries >>>> or ignore in such. Simplifies a lot. >>> Totally agree, but from our last discussion Anil suggested to still >>> support custom attributes using serializable objects. Ignore them >>> would be much more simple, but i understood Anil's point of view. >> It would be nice if users have possibility to choose, so they can either: >> 1) Store custom attributes in LDAP using serialization >> 2) Store those attributes in different IdentityStore (for example in >> database, which is quite flexible and can easily support custom >> attributes). We used this in Picketlink IDM 1, where only some >> standard attributes of user (firstName, lastName, email) are stored in >> LDAP and all other attributes are stored in database. With respect to >> membership, LDAP obviously support only plain membership (user "john" >> is in group "acme") without specifying role (user "john" is "manager" >> of group "acme"), so informations about custom roles also need to be >> stored in database. >> >> We discussed this yesterday on IRC, but not all were here. >> >> I think that proper support of (2) will require implementation of new >> IdentityStore, which will encapsulate other IdentityStores >> (LDAPIdentityStore, JPAIdentityStore) and delegate operation to proper >> one from those two. > In reality, we may end up creating separate identity store > implementations for quirky but prominent identity stores such as MS > Active Directory. I am unsure MSAD follows all the LDAPv3 rules. Over > time based on user feedback and our experiences, we may end up with a > bunch of Identity Store implementations. :) We have quite good coverage in 1.x and I don't think there will be need for separate implementations. Differences are fairly small and can be handled with proper configuration hooks. From what I remember: 1) Strict LDAP schema enforces "member" attribute to always have value (OpenLDAP) - meaning you cannot create user entry with objectClass=groupOfNames that doesn't belong to any other group. I had dummy placeholder entry option for this purpose that was then ignored during membership resolution https://github.com/picketlink/picketlink-idm/blob/1.4/picketlink-idm-ldap... Other LDAP servers (like RedHat) simply alter this part of the schema 2) MSAD doesn't support user password update on non SSL connection. Additionally it has separate specific password rules and it is non obvious to diagnose that your operation failed because of those. Additionally... password need to be enclosed with double quotes and encoded in UTF-16LE. Simple right? ;) <option> <name>enclosePasswordWith</name> <value>"</value> </option> <option> <name>passwordEncoding</name> <value>UTF-16LE</value> </option> 3) MSAD requires 2 step user entry creation as you cannot assign password during entry creation. First you create non active user account without password. Then you set password and activate the account. And thats basically only really specific part. It was covered with very few additional config options: https://github.com/picketlink/picketlink-idm/blob/1.4/picketlink-idm-ldap... Key part: <option> <name>createEntryAttributeValues</name> <value>objectClass=top</value> <value>objectClass=inetOrgPerson</value> <value>sn= </value> <value>userAccountControl=514</value> </option> <option> <name>passwordUpdateAttributeValues</name> <value>userAccountControl=512</value> </option> 4) Important controls like Sort and Pagination are not supported by all LDAP servers - however this may be better now. 5) Obviously there are several more server specific features but I don't think those are really required for the default implementation. I would more suggest to just implement LDAP IdentityStore in a way to easily extend the class and overwrite specify parts. Then there is always an easy way to cover corner cases in the future. It is more a matter of taste if you prefer single implementation and server specific configuration templates or several implementations with predefined configuration defaults. > >> >> Marek >> >>> >>> The current query support implementation for the LDAP store is >>> working with managed and custom attributes. If the attribute used as >>> a query parameter is managed, a normal LDAP search will be performed. >>> Otherwise, we'll filter the users manually. Users should be aware >>> that when used custom attributes they loose some performance on queries. >>> >>>>>> Bolek had opined about the use of LDAP entry change notifications >>>>>> to >>>>>> update the IDM cache. This is when the admin may have used some >>>>>> form >>>>>> of >>>>>> LDAP browser to update the entries or update happens via software >>>>>> not >>>>>> controlled by IDM. >>>>>> >>>>> Ok, going to consider that too. >>>>> >>>>>> Regards, >>>>>> Anil > _______________________________________________ > security-dev mailing list > security-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/security-dev _______________________________________________ security-dev mailing list security-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/security-dev