<p dir="ltr">Step 1 just at the start and then steps 2-4 in concurrent threads for different users. The user base is about 40 so after a couple of cycles all user details should be retrieved from cache I guess.</p>
<p dir="ltr">We have default values for hash iterations. And I can guarantee that there isn't any other operations going on at that time.</p>
<div class="gmail_quote">On Nov 30, 2015 8:39 PM, "Bill Burke" <<a href="mailto:bburke@redhat.com">bburke@redhat.com</a>> wrote:<br type="attribution"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">You do steps 1-4 in a continuous loop? or do you just get the token once and do steps 2-4 in a continuous loop?<br>
<br>
I want to figure out why you are getting so many invalidation messages, but there are other things that can peg the cpu. If you are using a large value for password hash iterations, this can peg the CPU very quickly on a small number of threads.<br>
<br>
On 11/30/2015 9:49 AM, Lohitha Chiranjeewa wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Bill,<br>
<br>
This particular load test consists of four API calls, starts with<br>
authentication:<br>
<br>
1) Invoke {host}/auth/realms/xxxx/protocol/openid-connect/token with<br>
"|username={username}&password={password}&grant_type=password|" as the<br>
body - we get the Access Token from this and use it for subsequent API calls<br>
2) Get user by User ID<br>
3) Get Realm Roles of above user<br>
4) Get Client Roles of a single given client for above user<br>
<br>
We run this test suite with concurrent threads (same superuser for<br>
authentication, but steps 2,3,4 for different users). But with our 2<br>
highly available M3-Medium server setup, it is impossible to go beyond<br>
10-15 threads w/o the processor hitting 100% usage.<br>
<br>
During the tests, we see millions (figuratively) of invalidation logs<br>
like this:<br>
[2015-11-30 14:32:08.0823], DEBUG,<br>
org.infinispan.interceptors.InvalidationInterceptor Timer-2 - Cache<br>
[dev-idm-a1] replicating<br>
InvalidateCommand{keys=[40113545-5069-47c4-bf1c-8f58a303caf6]}<br>
<br>
<br>
Regards,<br>
Lohitha.<br>
<br>
On Mon, Nov 30, 2015 at 8:17 PM, Bill Burke <<a href="mailto:bburke@redhat.com" target="_blank">bburke@redhat.com</a><br>
<mailto:<a href="mailto:bburke@redhat.com" target="_blank">bburke@redhat.com</a>>> wrote:<br>
<br>
I just did a loop test on GET<br>
https://{host}/auth/admin/realms/xxxxx/users/xxxx. The same userid<br>
over and over as you described. I see zero invalidations. Are you<br>
sure you aren't doing any updates?<br>
<br>
On 11/30/2015 9:33 AM, Stian Thorgersen wrote:<br>
<br>
If you can try building master and enable trace logging for<br>
"org.keycloak.models.cache.infinispan.InfinispanCacheUserProviderFactory".<br>
It will print log statements when entries are<br>
removed/invalidated in the<br>
cache.<br>
<br>
On 30 November 2015 at 15:19, Bill Burke <<a href="mailto:bburke@redhat.com" target="_blank">bburke@redhat.com</a><br>
<mailto:<a href="mailto:bburke@redhat.com" target="_blank">bburke@redhat.com</a>><br>
<mailto:<a href="mailto:bburke@redhat.com" target="_blank">bburke@redhat.com</a> <mailto:<a href="mailto:bburke@redhat.com" target="_blank">bburke@redhat.com</a>>>> wrote:<br>
<br>
Can you describe your load tests please? What kind of<br>
authentication<br>
are you doing? The only thing that would cause<br>
invalidations is HOTP.<br>
<br>
On 11/30/2015 9:13 AM, Lohitha Chiranjeewa wrote:<br>
> Bill, the problem here is that our load tests cannot go<br>
beyond a paltry<br>
> 15-20 concurrent threads with the default set up<br>
(invalidation caches<br>
> with SYNC mode - two HA AWS M3-Medium servers). Hence<br>
we're trying to<br>
> find ways to improve on the performance and the avg<br>
response time under<br>
> a load.<br>
><br>
> On Mon, Nov 30, 2015 at 7:16 PM, Bill Burke<br>
<<a href="mailto:bburke@redhat.com" target="_blank">bburke@redhat.com</a> <mailto:<a href="mailto:bburke@redhat.com" target="_blank">bburke@redhat.com</a>><br>
<mailto:<a href="mailto:bburke@redhat.com" target="_blank">bburke@redhat.com</a> <mailto:<a href="mailto:bburke@redhat.com" target="_blank">bburke@redhat.com</a>>><br>
> <mailto:<a href="mailto:bburke@redhat.com" target="_blank">bburke@redhat.com</a> <mailto:<a href="mailto:bburke@redhat.com" target="_blank">bburke@redhat.com</a>><br>
<mailto:<a href="mailto:bburke@redhat.com" target="_blank">bburke@redhat.com</a> <mailto:<a href="mailto:bburke@redhat.com" target="_blank">bburke@redhat.com</a>>>>> wrote:<br>
><br>
> BTW, besides being more scalable, we invalidate<br>
rather than replicate to<br>
> avoid transmitting sensitive security data over the<br>
network.<br>
><br>
> On 11/30/2015 8:29 AM, Lohitha Chiranjeewa wrote:<br>
> > Glad if you could look into the above and check if<br>
there are app level<br>
> > issues related to invalidations, thanks.<br>
> ><br>
> > One way to get around this through config changes<br>
is to make the<br>
> > relevant cache modes 'ASYNC' and set up suitable<br>
values for 'queue-size'<br>
> > and 'queue-flush-interval' depending on your needs.<br>
Then the<br>
> > invalidations won't happen with each and every call.<br>
> ><br>
> > Obviously there would be problems if caches aren't<br>
invalidated in time<br>
> > with the above set up, it's up to the devs to come<br>
up with a suitable<br>
> > set of configs to cater to their needs.<br>
> ><br>
> ><br>
> > Regards,<br>
> > Lohitha.<br>
> ><br>
> ><br>
> ><br>
> > On Mon, Nov 30, 2015 at 1:08 PM, Stian Thorgersen<br>
<<a href="mailto:sthorger@redhat.com" target="_blank">sthorger@redhat.com</a> <mailto:<a href="mailto:sthorger@redhat.com" target="_blank">sthorger@redhat.com</a>><br>
<mailto:<a href="mailto:sthorger@redhat.com" target="_blank">sthorger@redhat.com</a> <mailto:<a href="mailto:sthorger@redhat.com" target="_blank">sthorger@redhat.com</a>>><br>
<mailto:<a href="mailto:sthorger@redhat.com" target="_blank">sthorger@redhat.com</a> <mailto:<a href="mailto:sthorger@redhat.com" target="_blank">sthorger@redhat.com</a>><br>
<mailto:<a href="mailto:sthorger@redhat.com" target="_blank">sthorger@redhat.com</a> <mailto:<a href="mailto:sthorger@redhat.com" target="_blank">sthorger@redhat.com</a>>>><br>
> > <mailto:<a href="mailto:sthorger@redhat.com" target="_blank">sthorger@redhat.com</a><br>
<mailto:<a href="mailto:sthorger@redhat.com" target="_blank">sthorger@redhat.com</a>> <mailto:<a href="mailto:sthorger@redhat.com" target="_blank">sthorger@redhat.com</a><br>
<mailto:<a href="mailto:sthorger@redhat.com" target="_blank">sthorger@redhat.com</a>>><br>
<mailto:<a href="mailto:sthorger@redhat.com" target="_blank">sthorger@redhat.com</a> <mailto:<a href="mailto:sthorger@redhat.com" target="_blank">sthorger@redhat.com</a>><br>
<mailto:<a href="mailto:sthorger@redhat.com" target="_blank">sthorger@redhat.com</a> <mailto:<a href="mailto:sthorger@redhat.com" target="_blank">sthorger@redhat.com</a>>>>>> wrote:<br>
> ><br>
> > It is not expected behavior. Users should only<br>
be invalidated if<br>
> > changes are made.<br>
> ><br>
> > On 30 November 2015 at 07:01, Lohitha<br>
Chiranjeewa <<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a> <mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a>><br>
<mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a> <mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a>>><br>
<mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a> <mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a>><br>
<mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a> <mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a>>>><br>
> > <mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a><br>
<mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a>> <mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a><br>
<mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a>>> <mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a><br>
<mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a>><br>
<mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a> <mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a>>>>>> wrote:<br>
> ><br>
> > Just tested the same flow extensively with<br>
Keycloak 1.2.0 as<br>
> > well, and it seems the behavior is the<br>
same. Lots of MySQL<br>
> > select queries getting executed with each<br>
call. Seems there's a<br>
> > number of unnecessary cache invalidations<br>
going on which causes<br>
> > Keycloak to fetch data from the DB over and<br>
over.<br>
> ><br>
> > Could you please confirm if this is the<br>
expected behavior? As<br>
> > far as I can see there's considerable<br>
performance degradation<br>
> > due to unnecessary cache invalidations.<br>
> ><br>
> ><br>
> > On Fri, Nov 27, 2015 at 8:42 PM, Lohitha<br>
Chiranjeewa<br>
> > <<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a><br>
<mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a>> <mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a><br>
<mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a>>><br>
<mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a> <mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a>><br>
<mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a> <mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a>>>><br>
<mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a> <mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a>><br>
<mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a> <mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a>>><br>
> <mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a> <mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a>><br>
<mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a> <mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a>>>>>> wrote:<br>
> ><br>
> > I'm invoking the following API call:<br>
> ><br>
https://{host}/auth/admin/realms/xxxxx/users/55ffe851-2d94-460e-88b9-bc7340531b56<br>
> > (same realm and user ID over and over<br>
again). The two HA<br>
> > server(s) are idle apart from serving<br>
those calls. And I'm<br>
> > seeing the following SQL getting logged<br>
in my MySQL log for<br>
> > each and every call:<br>
> ><br>
> > select userentity0_.ID as ID1_42_,<br>
> > userentity0_.CREATED_TIMESTAMP as<br>
CREATED_2_42_,<br>
> > userentity0_.EMAIL as EMAIL3_42_,<br>
> > userentity0_.EMAIL_CONSTRAINT as<br>
EMAIL_CO4_42_,<br>
> > userentity0_.EMAIL_VERIFIED as<br>
EMAIL_VE5_42_,<br>
> > userentity0_.ENABLED as ENABLED6_42_,<br>
> > userentity0_.federation_link as<br>
federati7_42_,<br>
> > userentity0_.FIRST_NAME as FIRST_NA8_42_,<br>
> > userentity0_.LAST_NAME as LAST_NAM9_42_,<br>
> > userentity0_.REALM_ID as REALM_I10_42_,<br>
> ><br>
userentity0_.SERVICE_ACCOUNT_CLIENT_LINK as SERVICE11_42_,<br>
> > userentity0_.TOTP as TOTP12_42_,<br>
userentity0_.USERNAME as<br>
> > USERNAM13_42_ from USER_ENTITY<br>
userentity0_ where<br>
> ><br>
userentity0_.ID='55ffe851-2d94-460e-88b9-bc7340531b56' and<br>
> > userentity0_.REALM_ID='xxxxx'<br>
> ><br>
> ><br>
> ><br>
> > On Fri, Nov 27, 2015 at 8:03 PM, Stian<br>
Thorgersen<br>
> > <<a href="mailto:sthorger@redhat.com" target="_blank">sthorger@redhat.com</a><br>
<mailto:<a href="mailto:sthorger@redhat.com" target="_blank">sthorger@redhat.com</a>> <mailto:<a href="mailto:sthorger@redhat.com" target="_blank">sthorger@redhat.com</a><br>
<mailto:<a href="mailto:sthorger@redhat.com" target="_blank">sthorger@redhat.com</a>>><br>
<mailto:<a href="mailto:sthorger@redhat.com" target="_blank">sthorger@redhat.com</a> <mailto:<a href="mailto:sthorger@redhat.com" target="_blank">sthorger@redhat.com</a>><br>
<mailto:<a href="mailto:sthorger@redhat.com" target="_blank">sthorger@redhat.com</a> <mailto:<a href="mailto:sthorger@redhat.com" target="_blank">sthorger@redhat.com</a>>>><br>
> <mailto:<a href="mailto:sthorger@redhat.com" target="_blank">sthorger@redhat.com</a><br>
<mailto:<a href="mailto:sthorger@redhat.com" target="_blank">sthorger@redhat.com</a>> <mailto:<a href="mailto:sthorger@redhat.com" target="_blank">sthorger@redhat.com</a><br>
<mailto:<a href="mailto:sthorger@redhat.com" target="_blank">sthorger@redhat.com</a>>><br>
<mailto:<a href="mailto:sthorger@redhat.com" target="_blank">sthorger@redhat.com</a> <mailto:<a href="mailto:sthorger@redhat.com" target="_blank">sthorger@redhat.com</a>><br>
<mailto:<a href="mailto:sthorger@redhat.com" target="_blank">sthorger@redhat.com</a> <mailto:<a href="mailto:sthorger@redhat.com" target="_blank">sthorger@redhat.com</a>>>>>> wrote:<br>
> ><br>
> > Strange - what are you doing and<br>
what are the SQL queries?<br>
> ><br>
> > On 27 November 2015 at 15:23,<br>
Lohitha Chiranjeewa<br>
> > <<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a><br>
<mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a>> <mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a><br>
<mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a>>><br>
<mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a> <mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a>><br>
<mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a> <mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a>>>><br>
<mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a> <mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a>><br>
<mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a> <mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a>>><br>
> <mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a> <mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a>><br>
<mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a> <mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a>>>>>> wrote:<br>
> ><br>
> > Yes Stian, that I understand.<br>
But the problem here<br>
> > is even if I execute continuous<br>
user retrieval calls<br>
> > (same user - no other<br>
functionality in between),<br>
> > still MySQL select queries get<br>
executed for each<br>
> > call. So there lies an issue<br>
isn't it?<br>
> ><br>
> ><br>
> > On Fri, Nov 27, 2015 at 6:48<br>
PM, Stian Thorgersen<br>
> > <<a href="mailto:sthorger@redhat.com" target="_blank">sthorger@redhat.com</a><br>
<mailto:<a href="mailto:sthorger@redhat.com" target="_blank">sthorger@redhat.com</a>> <mailto:<a href="mailto:sthorger@redhat.com" target="_blank">sthorger@redhat.com</a><br>
<mailto:<a href="mailto:sthorger@redhat.com" target="_blank">sthorger@redhat.com</a>>><br>
> <mailto:<a href="mailto:sthorger@redhat.com" target="_blank">sthorger@redhat.com</a><br>
<mailto:<a href="mailto:sthorger@redhat.com" target="_blank">sthorger@redhat.com</a>> <mailto:<a href="mailto:sthorger@redhat.com" target="_blank">sthorger@redhat.com</a><br>
<mailto:<a href="mailto:sthorger@redhat.com" target="_blank">sthorger@redhat.com</a>>>><br>
<mailto:<a href="mailto:sthorger@redhat.com" target="_blank">sthorger@redhat.com</a> <mailto:<a href="mailto:sthorger@redhat.com" target="_blank">sthorger@redhat.com</a>><br>
<mailto:<a href="mailto:sthorger@redhat.com" target="_blank">sthorger@redhat.com</a> <mailto:<a href="mailto:sthorger@redhat.com" target="_blank">sthorger@redhat.com</a>>><br>
> <mailto:<a href="mailto:sthorger@redhat.com" target="_blank">sthorger@redhat.com</a><br>
<mailto:<a href="mailto:sthorger@redhat.com" target="_blank">sthorger@redhat.com</a>> <mailto:<a href="mailto:sthorger@redhat.com" target="_blank">sthorger@redhat.com</a><br>
<mailto:<a href="mailto:sthorger@redhat.com" target="_blank">sthorger@redhat.com</a>>>>>><br>
> > wrote:<br>
> ><br>
> > Things are still fetched<br>
from MySQL. Realms,<br>
> > clients, users, etc.. are<br>
then kept in the<br>
> > cache, but if it changes<br>
it's re-loaded from<br>
> > MySQL. We use an<br>
invalidation cache, not a<br>
> > distributed cache.<br>
> ><br>
> > On 27 November 2015 at<br>
14:04, Lohitha<br>
> > Chiranjeewa<br>
<<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a> <mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a>><br>
<mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a> <mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a>>><br>
<mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a> <mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a>><br>
<mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a> <mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a>>>><br>
> > <mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a><br>
<mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a>> <mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a><br>
<mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a>>> <mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a><br>
<mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a>><br>
<mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a> <mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a>>>>>> wrote:<br>
> ><br>
> > What I mean is, if it<br>
were working, I<br>
> > shouldn't see mysql<br>
queries getting executed<br>
> > right? So my guess is<br>
data is still fetched<br>
> > from the db instead of<br>
the cache.<br>
> ><br>
> > On Nov 27, 2015 5:52<br>
PM, "Stian Thorgersen"<br>
> > <<a href="mailto:sthorger@redhat.com" target="_blank">sthorger@redhat.com</a><br>
<mailto:<a href="mailto:sthorger@redhat.com" target="_blank">sthorger@redhat.com</a>> <mailto:<a href="mailto:sthorger@redhat.com" target="_blank">sthorger@redhat.com</a><br>
<mailto:<a href="mailto:sthorger@redhat.com" target="_blank">sthorger@redhat.com</a>>><br>
<mailto:<a href="mailto:sthorger@redhat.com" target="_blank">sthorger@redhat.com</a> <mailto:<a href="mailto:sthorger@redhat.com" target="_blank">sthorger@redhat.com</a>><br>
<mailto:<a href="mailto:sthorger@redhat.com" target="_blank">sthorger@redhat.com</a> <mailto:<a href="mailto:sthorger@redhat.com" target="_blank">sthorger@redhat.com</a>>>><br>
> ><br>
<mailto:<a href="mailto:sthorger@redhat.com" target="_blank">sthorger@redhat.com</a> <mailto:<a href="mailto:sthorger@redhat.com" target="_blank">sthorger@redhat.com</a>><br>
<mailto:<a href="mailto:sthorger@redhat.com" target="_blank">sthorger@redhat.com</a> <mailto:<a href="mailto:sthorger@redhat.com" target="_blank">sthorger@redhat.com</a>>><br>
<mailto:<a href="mailto:sthorger@redhat.com" target="_blank">sthorger@redhat.com</a> <mailto:<a href="mailto:sthorger@redhat.com" target="_blank">sthorger@redhat.com</a>><br>
<mailto:<a href="mailto:sthorger@redhat.com" target="_blank">sthorger@redhat.com</a> <mailto:<a href="mailto:sthorger@redhat.com" target="_blank">sthorger@redhat.com</a>>>>>> wrote:<br>
> ><br>
> > Yup, so it's<br>
working now?<br>
> ><br>
> > On 27 November 2015<br>
at 13:20, Lohitha<br>
> > Chiranjeewa<br>
<<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a> <mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a>><br>
<mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a> <mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a>>><br>
<mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a> <mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a>><br>
<mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a> <mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a>>>><br>
> ><br>
<mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a> <mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a>><br>
<mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a> <mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a>>><br>
<mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a> <mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a>><br>
<mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a> <mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a>>>>>> wrote:<br>
> ><br>
> > Apologies,<br>
keycloak-server.json<br>
> > entries should<br>
change to:<br>
> ><br>
> > "realm": {<br>
> ><br>
"provider": "jpa"<br>
> > },<br>
> ><br>
> > "user": {<br>
> ><br>
"provider": "jpa"<br>
> > },<br>
> ><br>
> ><br>
"userSessionPersister": {<br>
> ><br>
"provider": "jpa"<br>
> > },<br>
> ><br>
> > On Fri, Nov 27,<br>
2015 at 5:49 PM,<br>
> > Lohitha Chiranjeewa<br>
> ><br>
<<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a> <mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a>><br>
<mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a> <mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a>>><br>
<mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a> <mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a>><br>
<mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a> <mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a>>>><br>
> ><br>
<mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a> <mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a>><br>
<mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a> <mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a>>><br>
<br>
> <mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a> <mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a>><br>
<mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a> <mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a>>>>>> wrote:<br>
> ><br>
> > Hi Stian,<br>
> ><br>
> > As per the<br>
migration guide, I<br>
> > should have<br>
Infinispan up and<br>
> > running for<br>
realms, users and<br>
> > user<br>
sessions without<br>
> doing any<br>
> > specific<br>
changes.<br>
> ><br>
keycloak-server.json was<br>
> > reverted<br>
back to<br>
have the<br>
> > following<br>
entries:<br>
> > ...<br>
> ><br>
"realm": {<br>
> ><br>
"provider":<br>
> "infinispan"<br>
> > },<br>
> ><br>
> ><br>
"user": {<br>
> ><br>
"provider":<br>
> "infinispan"<br>
> > },<br>
> ><br>
> ><br>
> "userSessionPersister": {<br>
> ><br>
"provider":<br>
> "infinispan"<br>
> > },<br>
> > ...<br>
> ><br>
> > In the<br>
Admin Console I<br>
> have both<br>
> > Realm<br>
Cache and<br>
User Cache<br>
> > enables.<br>
I see certain<br>
> ><br>
Infinispan related<br>
logs<br>
> getting<br>
> > logged as<br>
well.<br>
> ><br>
> > However,<br>
at the same<br>
> time, I see<br>
> > MySQL<br>
queries getting<br>
> executed<br>
> > for all user<br>
retrieval API<br>
> ><br>
invocations (even<br>
if the same<br>
> > user is<br>
retrieved<br>
> continuously):<br>
> > ...<br>
> > select<br>
userentity0_.ID as<br>
> > ID1_42_,<br>
> ><br>
> userentity0_.CREATED_TIMESTAMP<br>
> > as<br>
CREATED_2_42_,<br>
> ><br>
userentity0_.EMAIL as<br>
> > EMAIL3_42_,<br>
> ><br>
> userentity0_.EMAIL_CONSTRAINT as<br>
> ><br>
EMAIL_CO4_42_,<br>
> ><br>
> userentity0_.EMAIL_VERIFIED as<br>
> ><br>
EMAIL_VE5_42_,<br>
> ><br>
userentity0_.ENABLED as<br>
> > ENABLED6_42_,<br>
> ><br>
> userentity0_.federation_link as<br>
> ><br>
federati7_42_,<br>
> ><br>
userentity0_.FIRST_NAME as<br>
> ><br>
FIRST_NA8_42_,<br>
> ><br>
userentity0_.LAST_NAME as<br>
> ><br>
LAST_NAM9_42_,<br>
> ><br>
userentity0_.REALM_ID as<br>
> ><br>
REALM_I10_42_,<br>
> ><br>
> userentity0_.SERVICE_ACCOUNT_CLIENT_LINK<br>
> > as<br>
SERVICE11_42_,<br>
> ><br>
userentity0_.TOTP as<br>
> TOTP12_42_,<br>
> ><br>
userentity0_.USERNAME as<br>
> ><br>
USERNAM13_42_ from<br>
> USER_ENTITY<br>
> ><br>
userentity0_ where<br>
> ><br>
> userentity0_.ID='55ffe851-2d94-460e-88b9-bc7340531b56'<br>
> > and<br>
> userentity0_.REALM_ID='xxxxx'<br>
> > ...<br>
> ><br>
> > So it seems<br>
something is<br>
> wrong<br>
> > here.<br>
Could you<br>
point out any<br>
> > areas<br>
that I could<br>
> further look<br>
> > into?<br>
> ><br>
> ><br>
> > Regards,<br>
> > Lohitha.<br>
> ><br>
> > On Thu,<br>
Nov 26,<br>
2015 at<br>
> 7:58 PM,<br>
> > Stian<br>
Thorgersen<br>
> ><br>
<<a href="mailto:sthorger@redhat.com" target="_blank">sthorger@redhat.com</a> <mailto:<a href="mailto:sthorger@redhat.com" target="_blank">sthorger@redhat.com</a>><br>
<mailto:<a href="mailto:sthorger@redhat.com" target="_blank">sthorger@redhat.com</a> <mailto:<a href="mailto:sthorger@redhat.com" target="_blank">sthorger@redhat.com</a>>><br>
> <mailto:<a href="mailto:sthorger@redhat.com" target="_blank">sthorger@redhat.com</a><br>
<mailto:<a href="mailto:sthorger@redhat.com" target="_blank">sthorger@redhat.com</a>> <mailto:<a href="mailto:sthorger@redhat.com" target="_blank">sthorger@redhat.com</a><br>
<mailto:<a href="mailto:sthorger@redhat.com" target="_blank">sthorger@redhat.com</a>>>><br>
> ><br>
<mailto:<a href="mailto:sthorger@redhat.com" target="_blank">sthorger@redhat.com</a> <mailto:<a href="mailto:sthorger@redhat.com" target="_blank">sthorger@redhat.com</a>><br>
<mailto:<a href="mailto:sthorger@redhat.com" target="_blank">sthorger@redhat.com</a> <mailto:<a href="mailto:sthorger@redhat.com" target="_blank">sthorger@redhat.com</a>>><br>
<mailto:<a href="mailto:sthorger@redhat.com" target="_blank">sthorger@redhat.com</a> <mailto:<a href="mailto:sthorger@redhat.com" target="_blank">sthorger@redhat.com</a>><br>
<mailto:<a href="mailto:sthorger@redhat.com" target="_blank">sthorger@redhat.com</a> <mailto:<a href="mailto:sthorger@redhat.com" target="_blank">sthorger@redhat.com</a>>>>>> wrote:<br>
> ><br>
> > Please<br>
read the migration guide<br>
> ><br>
> > On 26<br>
November 2015 at<br>
> > 14:53,<br>
Lohitha Chiranjeewa<br>
> ><br>
<<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a> <mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a>><br>
<mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a> <mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a>>><br>
<mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a> <mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a>><br>
<mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a> <mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a>>>><br>
> ><br>
> <mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a> <mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a>><br>
<mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a> <mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a>>><br>
<br>
<mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a> <mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a>><br>
<mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a> <mailto:<a href="mailto:kalc04@gmail.com" target="_blank">kalc04@gmail.com</a>>>>>><br>
> > wrote:<br>
> ><br>
> > Hi,<br>
> ><br>
> ><br>
We're in the<br>
> process of<br>
> ><br>
assessing the<br>
> impact on<br>
> ><br>
upgrading from<br>
> Keycloak<br>
> ><br>
1.2.0 to<br>
1.6.1.<br>
> We came<br>
> ><br>
across an<br>
issue when<br>
> ><br>
trying to<br>
enable<br>
> ><br>
Infinispan<br>
cache<br>
> through<br>
> > the<br>
> keycloak-server.json<br>
> ><br>
file as we<br>
used<br>
> to do in<br>
> ><br>
1.2.0.<br>
> ><br>
> ><br>
We have<br>
the following<br>
> ><br>
entries in<br>
1.6.1:<br>
> ><br>
"realm": {<br>
> ><br>
"provider":<br>
> ><br>
"infinispan"<br>
> ><br>
},<br>
> ><br>
> ><br>
"user": {<br>
> ><br>
"provider":<br>
> ><br>
"infinispan"<br>
> ><br>
},<br>
> ><br>
> ><br>
> ><br>
> "userSessionPersister": {<br>
> ><br>
"provider":<br>
> ><br>
"infinispan"<br>
> ><br>
},<br>
> ><br>
.........<br>
> ><br>
> ><br>
> "connectionsInfinispan": {<br>
> ><br>
> "default" : {<br>
> ><br>
> ><br>
"cacheContainer" :<br>
> ><br>
> "java:comp/env/infinispan/Keycloak"<br>
> ><br>
}<br>
> ><br>
}<br>
> ><br>
> > All<br>
configurations in<br>
> > 1.6.1<br>
> standalone-ha.xml<br>
> ><br>
file remains<br>
> comparable<br>
> > (and<br>
correct to<br>
> the best<br>
> ><br>
of our<br>
knowledge)<br>
> with<br>
> ><br>
the ones<br>
in 1.2.0.<br>
> ><br>
> ><br>
With the above<br>
> configs,<br>
> ><br>
when we<br>
start the<br>
> ><br>
Keycloak<br>
service the<br>
> ><br>
following<br>
> error(s) get<br>
> ><br>
logged:<br>
> ><br>
> ><br>
18:03:31,610 ERROR<br>
> ><br>
> [org.jboss.msc.service.fail]<br>
> ><br>
(ServerService Thread<br>
> ><br>
Pool -- 64)<br>
> MSC000001:<br>
> ><br>
Failed to<br>
start<br>
> service<br>
> ><br>
><br>
jboss.undertow.deployment.default-server.default-host./auth:<br>
> ><br>
> org.jboss.msc.service.StartException<br>
> ><br>
in service<br>
> ><br>
><br>
jboss.undertow.deployment.default-server.default-host./auth:<br>
> ><br>
> java.lang.RuntimeException:<br>
> ><br>
Failed to<br>
construct<br>
> ><br>
public<br>
> ><br>
><br>
<br>
org.keycloak.services.resources.KeycloakApplication(javax.servlet.ServletContext,org.jboss.resteasy.core.Dispatcher)<br>
> ><br>
at<br>
> ><br>
><br>
<br>
org.wildfly.extension.undertow.deployment.UndertowDeploymentService$1.run(UndertowDeploymentService.java:85)<br>
> ><br>
at<br>
> ><br>
><br>
<br>
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)<br>
> ><br>
[rt.jar:1.7.0_45]<br>
> ><br>
at<br>
> ><br>
><br>
java.util.concurrent.FutureTask.run(FutureTask.java:262)<br>
> ><br>
[rt.jar:1.7.0_45]<br>
> ><br>
at<br>
> ><br>
><br>
<br>
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)<br>
> ><br>
[rt.jar:1.7.0_45]<br>
> ><br>
at<br>
> ><br>
><br>
<br>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)<br>
> ><br>
[rt.jar:1.7.0_45]<br>
> ><br>
at<br>
> ><br>
> java.lang.Thread.run(Thread.java:744)<br>
> ><br>
[rt.jar:1.7.0_45]<br>
> ><br>
at<br>
> ><br>
><br>
org.jboss.threads.JBossThread.run(JBossThread.java:320)<br>
> ><br>
> [jboss-threads-2.2.0.Final.jar:2.2.0.Final]<br>
> ><br>
Caused by:<br>
> ><br>
> java.lang.RuntimeException:<br>
> ><br>
Failed to<br>
construct<br>
> ><br>
public<br>
> ><br>
><br>
<br>
org.keycloak.services.resources.KeycloakApplication(javax.servlet.ServletContext,org.jboss.resteasy.core.Dispatcher)<br>
> ><br>
at<br>
> ><br>
><br>
<br>
org.jboss.resteasy.core.ConstructorInjectorImpl.construct(ConstructorInjectorImpl.java:160)<br>
> ><br>
at<br>
> ><br>
><br>
<br>
org.jboss.resteasy.spi.ResteasyProviderFactory.createProviderInstance(ResteasyProviderFactory.java:2211)<br>
> ><br>
at<br>
> ><br>
><br>
<br>
org.jboss.resteasy.spi.ResteasyDeployment.createApplication(ResteasyDeployment.java:295)<br>
> ><br>
at<br>
> ><br>
><br>
<br>
org.jboss.resteasy.spi.ResteasyDeployment.start(ResteasyDeployment.java:236)<br>
> ><br>
at<br>
> ><br>
><br>
<br>
org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.init(ServletContainerDispatcher.java:112)<br>
> ><br>
at<br>
> ><br>
><br>
<br>
org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.init(HttpServletDispatcher.java:36)<br>
> ><br>
at<br>
> ><br>
><br>
<br>
io.undertow.servlet.core.LifecyleInterceptorInvocation.proceed(LifecyleInterceptorInvocation.java:117)<br>
> ><br>
at<br>
> ><br>
><br>
<br>
org.wildfly.extension.undertow.security.RunAsLifecycleInterceptor.init(RunAsLifecycleInterceptor.java:78)<br>
> ><br>
at<br>
> ><br>
><br>
<br>
io.undertow.servlet.core.LifecyleInterceptorInvocation.proceed(LifecyleInterceptorInvocation.java:103)<br>
> ><br>
at<br>
> ><br>
><br>
<br>
io.undertow.servlet.core.ManagedServlet$DefaultInstanceStrategy.start(ManagedServlet.java:230)<br>
> ><br>
at<br>
> ><br>
><br>
<br>
io.undertow.servlet.core.ManagedServlet.createServlet(ManagedServlet.java:131)<br>
> ><br>
at<br>
> ><br>
><br>
<br>
io.undertow.servlet.core.DeploymentManagerImpl.start(DeploymentManagerImpl.java:511)<br>
> ><br>
at<br>
> ><br>
><br>
<br>
org.wildfly.extension.undertow.deployment.UndertowDeploymentService.startContext(UndertowDeploymentService.java:101)<br>
> ><br>
at<br>
> ><br>
><br>
<br>
org.wildfly.extension.undertow.deployment.UndertowDeploymentService$1.run(UndertowDeploymentService.java:82)<br>
> ><br>
... 6<br>
more<br>
> ><br>
Caused by:<br>
> ><br>
> java.lang.RuntimeException:<br>
> ><br>
Failed to find<br>
> provider<br>
> ><br>
infinispan<br>
for realm<br>
> ><br>
at<br>
> ><br>
><br>
<br>
org.keycloak.services.DefaultKeycloakSessionFactory.init(DefaultKeycloakSessionFactory.java:66)<br>
> ><br>
at<br>
> ><br>
><br>
<br>
org.keycloak.services.resources.KeycloakApplication.createSessionFactory(KeycloakApplication.java:162)<br>
> ><br>
at<br>
> ><br>
><br>
<br>
org.keycloak.services.resources.KeycloakApplication.<init>(KeycloakApplication.java:62)<br>
> ><br>
at<br>
> ><br>
><br>
sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native<br>
> ><br>
Method)<br>
> [rt.jar:1.7.0_45]<br>
> ><br>
at<br>
> ><br>
><br>
<br>
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)<br>
> ><br>
[rt.jar:1.7.0_45]<br>
> ><br>
at<br>
> ><br>
><br>
<br>
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)<br>
> ><br>
[rt.jar:1.7.0_45]<br>
> ><br>
at<br>
> ><br>
><br>
java.lang.reflect.Constructor.newInstance(Constructor.java:526)<br>
> ><br>
[rt.jar:1.7.0_45]<br>
> ><br>
at<br>
> ><br>
><br>
<br>
org.jboss.resteasy.core.ConstructorInjectorImpl.construct(ConstructorInjectorImpl.java:148)<br>
> ><br>
...<br>
19 more<br>
> ><br>
> ><br>
> ><br>
Is the new<br>
way to<br>
> enable<br>
> ><br>
Infinispan<br>
> different to<br>
> ><br>
what we had<br>
> earlier? If<br>
> ><br>
so, can<br>
someone<br>
> please<br>
> ><br>
point out the<br>
> correct way?<br>
> ><br>
> ><br>
> ><br>
Regards,<br>
> ><br>
Lohitha.<br>
> ><br>
> ><br>
> ><br>
> ><br>
> _______________________________________________<br>
> ><br>
keycloak-user<br>
> mailing list<br>
> > <a href="mailto:keycloak-user@lists.jboss.org" target="_blank">keycloak-user@lists.jboss.org</a><br>
<mailto:<a href="mailto:keycloak-user@lists.jboss.org" target="_blank">keycloak-user@lists.jboss.org</a>><br>
<mailto:<a href="mailto:keycloak-user@lists.jboss.org" target="_blank">keycloak-user@lists.jboss.org</a><br>
<mailto:<a href="mailto:keycloak-user@lists.jboss.org" target="_blank">keycloak-user@lists.jboss.org</a>>><br>
<mailto:<a href="mailto:keycloak-user@lists.jboss.org" target="_blank">keycloak-user@lists.jboss.org</a><br>
<mailto:<a href="mailto:keycloak-user@lists.jboss.org" target="_blank">keycloak-user@lists.jboss.org</a>><br>
<mailto:<a href="mailto:keycloak-user@lists.jboss.org" target="_blank">keycloak-user@lists.jboss.org</a><br>
<mailto:<a href="mailto:keycloak-user@lists.jboss.org" target="_blank">keycloak-user@lists.jboss.org</a>>>><br>
> ><br>
> <mailto:<a href="mailto:keycloak-user@lists.jboss.org" target="_blank">keycloak-user@lists.jboss.org</a><br>
<mailto:<a href="mailto:keycloak-user@lists.jboss.org" target="_blank">keycloak-user@lists.jboss.org</a>><br>
<mailto:<a href="mailto:keycloak-user@lists.jboss.org" target="_blank">keycloak-user@lists.jboss.org</a><br>
<mailto:<a href="mailto:keycloak-user@lists.jboss.org" target="_blank">keycloak-user@lists.jboss.org</a>>><br>
> <mailto:<a href="mailto:keycloak-user@lists.jboss.org" target="_blank">keycloak-user@lists.jboss.org</a><br>
<mailto:<a href="mailto:keycloak-user@lists.jboss.org" target="_blank">keycloak-user@lists.jboss.org</a>><br>
<mailto:<a href="mailto:keycloak-user@lists.jboss.org" target="_blank">keycloak-user@lists.jboss.org</a><br>
<mailto:<a href="mailto:keycloak-user@lists.jboss.org" target="_blank">keycloak-user@lists.jboss.org</a>>>>><br>
> ><a href="https://lists.jboss.org/mailman/listinfo/keycloak-user" rel="noreferrer" target="_blank">https://lists.jboss.org/mailman/listinfo/keycloak-user</a><br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> > _______________________________________________<br>
> > keycloak-user mailing list<br>
> ><a href="mailto:keycloak-user@lists.jboss.org" target="_blank">keycloak-user@lists.jboss.org</a><br>
<mailto:<a href="mailto:keycloak-user@lists.jboss.org" target="_blank">keycloak-user@lists.jboss.org</a>><br>
<mailto:<a href="mailto:keycloak-user@lists.jboss.org" target="_blank">keycloak-user@lists.jboss.org</a><br>
<mailto:<a href="mailto:keycloak-user@lists.jboss.org" target="_blank">keycloak-user@lists.jboss.org</a>>><br>
<mailto:<a href="mailto:keycloak-user@lists.jboss.org" target="_blank">keycloak-user@lists.jboss.org</a><br>
<mailto:<a href="mailto:keycloak-user@lists.jboss.org" target="_blank">keycloak-user@lists.jboss.org</a>><br>
<mailto:<a href="mailto:keycloak-user@lists.jboss.org" target="_blank">keycloak-user@lists.jboss.org</a><br>
<mailto:<a href="mailto:keycloak-user@lists.jboss.org" target="_blank">keycloak-user@lists.jboss.org</a>>>><br>
> ><a href="https://lists.jboss.org/mailman/listinfo/keycloak-user" rel="noreferrer" target="_blank">https://lists.jboss.org/mailman/listinfo/keycloak-user</a><br>
> ><br>
><br>
> --<br>
> Bill Burke<br>
> JBoss, a division of Red Hat<br>
><a href="http://bill.burkecentral.com" rel="noreferrer" target="_blank">http://bill.burkecentral.com</a><br>
> _______________________________________________<br>
> keycloak-user mailing list<br>
> <a href="mailto:keycloak-user@lists.jboss.org" target="_blank">keycloak-user@lists.jboss.org</a><br>
<mailto:<a href="mailto:keycloak-user@lists.jboss.org" target="_blank"></a></blockquote>...</blockquote></div>