[keycloak-dev] federated pagination Re: federation commited need feedback

[Marek Posolda]

I am already on picketlink master and trying to address the issues we 
have. Right now mapping of dynamic attributes aka 
https://issues.jboss.org/browse/PLINK-533 . I don't know when exactly is 
picketlink release planned, but maybe we can ask Pedro to do at least 
timestamped release somewhen next week?

I wonder that random searches with FederationProvider might work for 
implementations, which supports offset/limit (like external database 
provided by customer) with your idea of omit local users with federation 
link? There are more tricky things though, like if user "john" available 
in in both local and LDAP store, and then directly deleted in LDAP, 
won't be returned if queries even if he is still in Keycloak database. 
Not sure if it's good or bad as he may not be able to login anyway...

However for LDAP it's tricky... I've tested with ActiveDirectory that I 
can query with same cookie to random LDAP pages (next page or any page 
which I already previously visited), but looks that just next page is 
always guaranteed to work but previous pages are not guaranteed 
according to http://www.ietf.org/rfc/rfc2696.txt sentence:
"Returning cookies from previous searchResultDone responses besides the 
last one is undefined, as the server implementation may restrict cookies 
from being reused."

Marek

On 25.7.2014 18:40, Bill Burke wrote:
> Even worse, this "paginationContext" needs to be exportable as a URL 
> parameter.  So it looks like we can't support federated pagination 
> until we get a new Picketlink release.  How should we proceed? Looks 
> like we're stuck with only doing searches on imported users for now.
>
> search(String someRandomString) could be:
>
> Do the following to get things synced up on-demand:
> ldap.getUserByUsername(someRandomString)
> ldap.getUserByEmail(someRandomString)
>
> Then just do a search on local storage:
> local.search(someRandomString)
>
> FederationProvider just doesn't support random searches.  Which is why 
> you did what you did for AuthenticationProvider.  I'm starting to feel 
> like an idiot.
>
>
> On 7/25/2014 12:18 PM, Marek Posolda wrote:
>> Just to have things a bit more complicated, the LDAP doesn't support
>> "classic" pagination model with offset/limit like JPA, but the
>> pagination model with cookie/limit (each search returns "cookie", which
>> can be used to search for next page). Some details here
>> http://www.ietf.org/rfc/rfc2696.txt...
>>
>> It is not a problem for the Sync usecase though, as sync process will
>> have full control and can process all the pages gradually. However it
>> looks that it will affect FederationProvider though? I guess "Result"
>> object you proposed would need to store cookie in some property as well.
>> For picketlink I've added paginationContext to Picketlink IdentityQuery
>> to handle this
>> https://github.com/picketlink/picketlink/blob/master/modules/idm/api/src/main/java/org/picketlink/idm/query/IdentityQuery.java#L43 
>>
>> .
>>
>> Marek
>>
>> On 25.7.2014 15:43, Bill Burke wrote:
>>>
>>>
>>> On 7/25/2014 3:20 AM, Marek Posolda wrote:
>>>>> You're right.  The API would have to change to note the provider that
>>>>> was last used and how many were consumed for that provider.
>>>>>
>>>>> class Result {
>>>>>
>>>>>    List<UserModel> results;
>>>>>
>>>>>    String lastProvider;
>>>>>    int lastIndex;
>>>>>
>>>>> }
>>>>>
>>>>> then UserProvider search would need these methods:
>>>>>
>>>>> Result search(criteria..., int maxResults);  // start from beginning
>>>>> Result search(criteria..., String lastProvider, int lastIndex, int
>>>>> maxResults);
>>>> Sorry, I still have doubts;-)
>>>>
>>>> For example there are 10 users in Keycloak and just 5 of them are 
>>>> mapped
>>>> to LDAP. In LDAP there are just those 5 users. Then you want to search
>>>> for page1 with (lastIndex 0, maxResults 10) and you will retrieve 
>>>> those
>>>> 10 Keycloak Users. Then you want page2, so you call (lastIndex 10,
>>>> maxResults 10) and now you retrieve those 5 users from LDAP, but those
>>>> are same users, which were already retrieved on page1.
>>>>
>>>
>>> Solved by searching for local users where federation link is null
>>> only?  The side effect is that the federation provider would also have
>>> to check the database to make sure the user hasn't already been
>>> imported. This could suck as 1 pagination query could turn into
>>> MAX_RESULTS local storage queries
>