You do steps 1-4 in a continuous loop? or do you just get the token
once and do steps 2-4 in a continuous loop?
I want to figure out why you are getting so many invalidation messages,
but there are other things that can peg the cpu. If you are using a
large value for password hash iterations, this can peg the CPU very
quickly on a small number of threads.
On 11/30/2015 9:49 AM, Lohitha Chiranjeewa wrote:
Bill,
This particular load test consists of four API calls, starts with
authentication:
1) Invoke {host}/auth/realms/xxxx/protocol/openid-connect/token with
"|username={username}&password={password}&grant_type=password|" as the
body - we get the Access Token from this and use it for subsequent API calls
2) Get user by User ID
3) Get Realm Roles of above user
4) Get Client Roles of a single given client for above user
We run this test suite with concurrent threads (same superuser for
authentication, but steps 2,3,4 for different users). But with our 2
highly available M3-Medium server setup, it is impossible to go beyond
10-15 threads w/o the processor hitting 100% usage.
During the tests, we see millions (figuratively) of invalidation logs
like this:
[2015-11-30 14:32:08.0823], DEBUG,
org.infinispan.interceptors.InvalidationInterceptor Timer-2 - Cache
[dev-idm-a1] replicating
InvalidateCommand{keys=[40113545-5069-47c4-bf1c-8f58a303caf6]}
Regards,
Lohitha.
On Mon, Nov 30, 2015 at 8:17 PM, Bill Burke <bburke(a)redhat.com
<mailto:bburke@redhat.com>> wrote:
I just did a loop test on GET
https://{host}/auth/admin/realms/xxxxx/users/xxxx. The same userid
over and over as you described. I see zero invalidations. Are you
sure you aren't doing any updates?
On 11/30/2015 9:33 AM, Stian Thorgersen wrote:
If you can try building master and enable trace logging for
"org.keycloak.models.cache.infinispan.InfinispanCacheUserProviderFactory".
It will print log statements when entries are
removed/invalidated in the
cache.
On 30 November 2015 at 15:19, Bill Burke <bburke(a)redhat.com
<mailto:bburke@redhat.com>
<mailto:bburke@redhat.com <mailto:bburke@redhat.com>>> wrote:
Can you describe your load tests please? What kind of
authentication
are you doing? The only thing that would cause
invalidations is HOTP.
On 11/30/2015 9:13 AM, Lohitha Chiranjeewa wrote:
> Bill, the problem here is that our load tests cannot go
beyond a paltry
> 15-20 concurrent threads with the default set up
(invalidation caches
> with SYNC mode - two HA AWS M3-Medium servers). Hence
we're trying to
> find ways to improve on the performance and the avg
response time under
> a load.
>
> On Mon, Nov 30, 2015 at 7:16 PM, Bill Burke
<bburke(a)redhat.com <mailto:bburke@redhat.com>
<mailto:bburke@redhat.com <mailto:bburke@redhat.com>>
> <mailto:bburke@redhat.com <mailto:bburke@redhat.com>
<mailto:bburke@redhat.com <mailto:bburke@redhat.com>>>> wrote:
>
> BTW, besides being more scalable, we invalidate
rather than replicate to
> avoid transmitting sensitive security data over the
network.
>
> On 11/30/2015 8:29 AM, Lohitha Chiranjeewa wrote:
> > Glad if you could look into the above and check if
there are app level
> > issues related to invalidations, thanks.
> >
> > One way to get around this through config changes
is to make the
> > relevant cache modes 'ASYNC' and set up suitable
values for 'queue-size'
> > and 'queue-flush-interval' depending on your needs.
Then the
> > invalidations won't happen with each and every call.
> >
> > Obviously there would be problems if caches aren't
invalidated in time
> > with the above set up, it's up to the devs to come
up with a suitable
> > set of configs to cater to their needs.
> >
> >
> > Regards,
> > Lohitha.
> >
> >
> >
> > On Mon, Nov 30, 2015 at 1:08 PM, Stian Thorgersen
<sthorger(a)redhat.com <mailto:sthorger@redhat.com>
<mailto:sthorger@redhat.com <mailto:sthorger@redhat.com>>
<mailto:sthorger@redhat.com <mailto:sthorger@redhat.com>
<mailto:sthorger@redhat.com <mailto:sthorger@redhat.com>>>
> > <mailto:sthorger@redhat.com
<mailto:sthorger@redhat.com> <mailto:sthorger@redhat.com
<mailto:sthorger@redhat.com>>
<mailto:sthorger@redhat.com <mailto:sthorger@redhat.com>
<mailto:sthorger@redhat.com <mailto:sthorger@redhat.com>>>>>
wrote:
> >
> > It is not expected behavior. Users should only
be invalidated if
> > changes are made.
> >
> > On 30 November 2015 at 07:01, Lohitha
Chiranjeewa <kalc04(a)gmail.com <mailto:kalc04@gmail.com>
<mailto:kalc04@gmail.com <mailto:kalc04@gmail.com>>
<mailto:kalc04@gmail.com <mailto:kalc04@gmail.com>
<mailto:kalc04@gmail.com <mailto:kalc04@gmail.com>>>
> > <mailto:kalc04@gmail.com
<mailto:kalc04@gmail.com> <mailto:kalc04@gmail.com
<mailto:kalc04@gmail.com>> <mailto:kalc04@gmail.com
<mailto:kalc04@gmail.com>
<mailto:kalc04@gmail.com <mailto:kalc04@gmail.com>>>>>
wrote:
> >
> > Just tested the same flow extensively with
Keycloak 1.2.0 as
> > well, and it seems the behavior is the
same. Lots of MySQL
> > select queries getting executed with each
call. Seems there's a
> > number of unnecessary cache invalidations
going on which causes
> > Keycloak to fetch data from the DB over and
over.
> >
> > Could you please confirm if this is the
expected behavior? As
> > far as I can see there's considerable
performance degradation
> > due to unnecessary cache invalidations.
> >
> >
> > On Fri, Nov 27, 2015 at 8:42 PM, Lohitha
Chiranjeewa
> > <kalc04(a)gmail.com
<mailto:kalc04@gmail.com> <mailto:kalc04@gmail.com
<mailto:kalc04@gmail.com>>
<mailto:kalc04@gmail.com <mailto:kalc04@gmail.com>
<mailto:kalc04@gmail.com <mailto:kalc04@gmail.com>>>
<mailto:kalc04@gmail.com <mailto:kalc04@gmail.com>
<mailto:kalc04@gmail.com <mailto:kalc04@gmail.com>>
> <mailto:kalc04@gmail.com <mailto:kalc04@gmail.com>
<mailto:kalc04@gmail.com <mailto:kalc04@gmail.com>>>>>
wrote:
> >
> > I'm invoking the following API call:
> >
https://{host}/auth/admin/realms/xxxxx/users/55ffe851-2d94-460e-88b9-bc7340531b56
> > (same realm and user ID over and over
again). The two HA
> > server(s) are idle apart from serving
those calls. And I'm
> > seeing the following SQL getting logged
in my MySQL log for
> > each and every call:
> >
> > select userentity0_.ID as ID1_42_,
> > userentity0_.CREATED_TIMESTAMP as
CREATED_2_42_,
> > userentity0_.EMAIL as EMAIL3_42_,
> > userentity0_.EMAIL_CONSTRAINT as
EMAIL_CO4_42_,
> > userentity0_.EMAIL_VERIFIED as
EMAIL_VE5_42_,
> > userentity0_.ENABLED as ENABLED6_42_,
> > userentity0_.federation_link as
federati7_42_,
> > userentity0_.FIRST_NAME as FIRST_NA8_42_,
> > userentity0_.LAST_NAME as LAST_NAM9_42_,
> > userentity0_.REALM_ID as REALM_I10_42_,
> >
userentity0_.SERVICE_ACCOUNT_CLIENT_LINK as SERVICE11_42_,
> > userentity0_.TOTP as TOTP12_42_,
userentity0_.USERNAME as
> > USERNAM13_42_ from USER_ENTITY
userentity0_ where
> >
userentity0_.ID='55ffe851-2d94-460e-88b9-bc7340531b56' and
> > userentity0_.REALM_ID='xxxxx'
> >
> >
> >
> > On Fri, Nov 27, 2015 at 8:03 PM, Stian
Thorgersen
> > <sthorger(a)redhat.com
<mailto:sthorger@redhat.com> <mailto:sthorger@redhat.com
<mailto:sthorger@redhat.com>>
<mailto:sthorger@redhat.com <mailto:sthorger@redhat.com>
<mailto:sthorger@redhat.com <mailto:sthorger@redhat.com>>>
> <mailto:sthorger@redhat.com
<mailto:sthorger@redhat.com> <mailto:sthorger@redhat.com
<mailto:sthorger@redhat.com>>
<mailto:sthorger@redhat.com <mailto:sthorger@redhat.com>
<mailto:sthorger@redhat.com <mailto:sthorger@redhat.com>>>>>
wrote:
> >
> > Strange - what are you doing and
what are the SQL queries?
> >
> > On 27 November 2015 at 15:23,
Lohitha Chiranjeewa
> > <kalc04(a)gmail.com
<mailto:kalc04@gmail.com> <mailto:kalc04@gmail.com
<mailto:kalc04@gmail.com>>
<mailto:kalc04@gmail.com <mailto:kalc04@gmail.com>
<mailto:kalc04@gmail.com <mailto:kalc04@gmail.com>>>
<mailto:kalc04@gmail.com <mailto:kalc04@gmail.com>
<mailto:kalc04@gmail.com <mailto:kalc04@gmail.com>>
> <mailto:kalc04@gmail.com <mailto:kalc04@gmail.com>
<mailto:kalc04@gmail.com <mailto:kalc04@gmail.com>>>>>
wrote:
> >
> > Yes Stian, that I understand.
But the problem here
> > is even if I execute continuous
user retrieval calls
> > (same user - no other
functionality in between),
> > still MySQL select queries get
executed for each
> > call. So there lies an issue
isn't it?
> >
> >
> > On Fri, Nov 27, 2015 at 6:48
PM, Stian Thorgersen
> > <sthorger(a)redhat.com
<mailto:sthorger@redhat.com> <mailto:sthorger@redhat.com
<mailto:sthorger@redhat.com>>
> <mailto:sthorger@redhat.com
<mailto:sthorger@redhat.com> <mailto:sthorger@redhat.com
<mailto:sthorger@redhat.com>>>
<mailto:sthorger@redhat.com <mailto:sthorger@redhat.com>
<mailto:sthorger@redhat.com <mailto:sthorger@redhat.com>>
> <mailto:sthorger@redhat.com
<mailto:sthorger@redhat.com> <mailto:sthorger@redhat.com
<mailto:sthorger@redhat.com>>>>>
> > wrote:
> >
> > Things are still fetched
from MySQL. Realms,
> > clients, users, etc.. are
then kept in the
> > cache, but if it changes
it's re-loaded from
> > MySQL. We use an
invalidation cache, not a
> > distributed cache.
> >
> > On 27 November 2015 at
14:04, Lohitha
> > Chiranjeewa
<kalc04(a)gmail.com <mailto:kalc04@gmail.com>
<mailto:kalc04@gmail.com <mailto:kalc04@gmail.com>>
<mailto:kalc04@gmail.com <mailto:kalc04@gmail.com>
<mailto:kalc04@gmail.com <mailto:kalc04@gmail.com>>>
> > <mailto:kalc04@gmail.com
<mailto:kalc04@gmail.com> <mailto:kalc04@gmail.com
<mailto:kalc04@gmail.com>> <mailto:kalc04@gmail.com
<mailto:kalc04@gmail.com>
<mailto:kalc04@gmail.com <mailto:kalc04@gmail.com>>>>>
wrote:
> >
> > What I mean is, if it
were working, I
> > shouldn't see mysql
queries getting executed
> > right? So my guess is
data is still fetched
> > from the db instead of
the cache.
> >
> > On Nov 27, 2015 5:52
PM, "Stian Thorgersen"
> > <sthorger(a)redhat.com
<mailto:sthorger@redhat.com> <mailto:sthorger@redhat.com
<mailto:sthorger@redhat.com>>
<mailto:sthorger@redhat.com <mailto:sthorger@redhat.com>
<mailto:sthorger@redhat.com <mailto:sthorger@redhat.com>>>
> >
<mailto:sthorger@redhat.com <mailto:sthorger@redhat.com>
<mailto:sthorger@redhat.com <mailto:sthorger@redhat.com>>
<mailto:sthorger@redhat.com <mailto:sthorger@redhat.com>
<mailto:sthorger@redhat.com <mailto:sthorger@redhat.com>>>>>
wrote:
> >
> > Yup, so it's
working now?
> >
> > On 27 November 2015
at 13:20, Lohitha
> > Chiranjeewa
<kalc04(a)gmail.com <mailto:kalc04@gmail.com>
<mailto:kalc04@gmail.com <mailto:kalc04@gmail.com>>
<mailto:kalc04@gmail.com <mailto:kalc04@gmail.com>
<mailto:kalc04@gmail.com <mailto:kalc04@gmail.com>>>
> >
<mailto:kalc04@gmail.com <mailto:kalc04@gmail.com>
<mailto:kalc04@gmail.com <mailto:kalc04@gmail.com>>
<mailto:kalc04@gmail.com <mailto:kalc04@gmail.com>
<mailto:kalc04@gmail.com <mailto:kalc04@gmail.com>>>>>
wrote:
> >
> > Apologies,
keycloak-server.json
> > entries should
change to:
> >
> > "realm": {
> >
"provider": "jpa"
> > },
> >
> > "user": {
> >
"provider": "jpa"
> > },
> >
> >
"userSessionPersister": {
> >
"provider": "jpa"
> > },
> >
> > On Fri, Nov 27,
2015 at 5:49 PM,
> > Lohitha Chiranjeewa
> >
<kalc04(a)gmail.com <mailto:kalc04@gmail.com>
<mailto:kalc04@gmail.com <mailto:kalc04@gmail.com>>
<mailto:kalc04@gmail.com <mailto:kalc04@gmail.com>
<mailto:kalc04@gmail.com <mailto:kalc04@gmail.com>>>
> >
<mailto:kalc04@gmail.com <mailto:kalc04@gmail.com>
<mailto:kalc04@gmail.com <mailto:kalc04@gmail.com>>
> <mailto:kalc04@gmail.com <mailto:kalc04@gmail.com>
<mailto:kalc04@gmail.com <mailto:kalc04@gmail.com>>>>>
wrote:
> >
> > Hi Stian,
> >
> > As per the
migration guide, I
> > should have
Infinispan up and
> > running for
realms, users and
> > user
sessions without
> doing any
> > specific
changes.
> >
keycloak-server.json was
> > reverted
back to
have the
> > following
entries:
> > ...
> >
"realm": {
> >
"provider":
> "infinispan"
> > },
> >
> >
"user": {
> >
"provider":
> "infinispan"
> > },
> >
> >
> "userSessionPersister": {
> >
"provider":
> "infinispan"
> > },
> > ...
> >
> > In the
Admin Console I
> have both
> > Realm
Cache and
User Cache
> > enables.
I see certain
> >
Infinispan related
logs
> getting
> > logged as
well.
> >
> > However,
at the same
> time, I see
> > MySQL
queries getting
> executed
> > for all user
retrieval API
> >
invocations (even
if the same
> > user is
retrieved
> continuously):
> > ...
> > select
userentity0_.ID as
> > ID1_42_,
> >
> userentity0_.CREATED_TIMESTAMP
> > as
CREATED_2_42_,
> >
userentity0_.EMAIL as
> > EMAIL3_42_,
> >
> userentity0_.EMAIL_CONSTRAINT as
> >
EMAIL_CO4_42_,
> >
> userentity0_.EMAIL_VERIFIED as
> >
EMAIL_VE5_42_,
> >
userentity0_.ENABLED as
> > ENABLED6_42_,
> >
> userentity0_.federation_link as
> >
federati7_42_,
> >
userentity0_.FIRST_NAME as
> >
FIRST_NA8_42_,
> >
userentity0_.LAST_NAME as
> >
LAST_NAM9_42_,
> >
userentity0_.REALM_ID as
> >
REALM_I10_42_,
> >
> userentity0_.SERVICE_ACCOUNT_CLIENT_LINK
> > as
SERVICE11_42_,
> >
userentity0_.TOTP as
> TOTP12_42_,
> >
userentity0_.USERNAME as
> >
USERNAM13_42_ from
> USER_ENTITY
> >
userentity0_ where
> >
> userentity0_.ID='55ffe851-2d94-460e-88b9-bc7340531b56'
> > and
> userentity0_.REALM_ID='xxxxx'
> > ...
> >
> > So it seems
something is
> wrong
> > here.
Could you
point out any
> > areas
that I could
> further look
> > into?
> >
> >
> > Regards,
> > Lohitha.
> >
> > On Thu,
Nov 26,
2015 at
> 7:58 PM,
> > Stian
Thorgersen
> >
<sthorger(a)redhat.com <mailto:sthorger@redhat.com>
<mailto:sthorger@redhat.com <mailto:sthorger@redhat.com>>
> <mailto:sthorger@redhat.com
<mailto:sthorger@redhat.com> <mailto:sthorger@redhat.com
<mailto:sthorger@redhat.com>>>
> >
<mailto:sthorger@redhat.com <mailto:sthorger@redhat.com>
<mailto:sthorger@redhat.com <mailto:sthorger@redhat.com>>
<mailto:sthorger@redhat.com <mailto:sthorger@redhat.com>
<mailto:sthorger@redhat.com <mailto:sthorger@redhat.com>>>>>
wrote:
> >
> > Please
read the migration guide
> >
> > On 26
November 2015 at
> > 14:53,
Lohitha Chiranjeewa
> >
<kalc04(a)gmail.com <mailto:kalc04@gmail.com>
<mailto:kalc04@gmail.com <mailto:kalc04@gmail.com>>
<mailto:kalc04@gmail.com <mailto:kalc04@gmail.com>
<mailto:kalc04@gmail.com <mailto:kalc04@gmail.com>>>
> >
> <mailto:kalc04@gmail.com <mailto:kalc04@gmail.com>
<mailto:kalc04@gmail.com <mailto:kalc04@gmail.com>>
<mailto:kalc04@gmail.com <mailto:kalc04@gmail.com>
<mailto:kalc04@gmail.com <mailto:kalc04@gmail.com>>>>>
> > wrote:
> >
> > Hi,
> >
> >
We're in the
> process of
> >
assessing the
> impact on
> >
upgrading from
> Keycloak
> >
1.2.0 to
1.6.1.
> We came
> >
across an
issue when
> >
trying to
enable
> >
Infinispan
cache
> through
> > the
> keycloak-server.json
> >
file as we
used
> to do in
> >
1.2.0.
> >
> >
We have
the following
> >
entries in
1.6.1:
> >
"realm": {
> >
"provider":
> >
"infinispan"
> >
},
> >
> >
"user": {
> >
"provider":
> >
"infinispan"
> >
},
> >
> >
> >
> "userSessionPersister": {
> >
"provider":
> >
"infinispan"
> >
},
> >
.........
> >
> >
> "connectionsInfinispan": {
> >
> "default" : {
> >
> >
"cacheContainer" :
> >
> "java:comp/env/infinispan/Keycloak"
> >
}
> >
}
> >
> > All
configurations in
> > 1.6.1
> standalone-ha.xml
> >
file remains
> comparable
> > (and
correct to
> the best
> >
of our
knowledge)
> with
> >
the ones
in 1.2.0.
> >
> >
With the above
> configs,
> >
when we
start the
> >
Keycloak
service the
> >
following
> error(s) get
> >
logged:
> >
> >
18:03:31,610 ERROR
> >
> [org.jboss.msc.service.fail]
> >
(ServerService Thread
> >
Pool -- 64)
> MSC000001:
> >
Failed to
start
> service
> >
>
jboss.undertow.deployment.default-server.default-host./auth:
> >
> org.jboss.msc.service.StartException
> >
in service
> >
>
jboss.undertow.deployment.default-server.default-host./auth:
> >
> java.lang.RuntimeException:
> >
Failed to
construct
> >
public
> >
>
org.keycloak.services.resources.KeycloakApplication(javax.servlet.ServletContext,org.jboss.resteasy.core.Dispatcher)
> >
at
> >
>
org.wildfly.extension.undertow.deployment.UndertowDeploymentService$1.run(UndertowDeploymentService.java:85)
> >
at
> >
>
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> >
[rt.jar:1.7.0_45]
> >
at
> >
>
java.util.concurrent.FutureTask.run(FutureTask.java:262)
> >
[rt.jar:1.7.0_45]
> >
at
> >
>
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> >
[rt.jar:1.7.0_45]
> >
at
> >
>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> >
[rt.jar:1.7.0_45]
> >
at
> >
> java.lang.Thread.run(Thread.java:744)
> >
[rt.jar:1.7.0_45]
> >
at
> >
>
org.jboss.threads.JBossThread.run(JBossThread.java:320)
> >
> [jboss-threads-2.2.0.Final.jar:2.2.0.Final]
> >
Caused by:
> >
> java.lang.RuntimeException:
> >
Failed to
construct
> >
public
> >
>
org.keycloak.services.resources.KeycloakApplication(javax.servlet.ServletContext,org.jboss.resteasy.core.Dispatcher)
> >
at
> >
>
org.jboss.resteasy.core.ConstructorInjectorImpl.construct(ConstructorInjectorImpl.java:160)
> >
at
> >
>
org.jboss.resteasy.spi.ResteasyProviderFactory.createProviderInstance(ResteasyProviderFactory.java:2211)
> >
at
> >
>
org.jboss.resteasy.spi.ResteasyDeployment.createApplication(ResteasyDeployment.java:295)
> >
at
> >
>
org.jboss.resteasy.spi.ResteasyDeployment.start(ResteasyDeployment.java:236)
> >
at
> >
>
org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.init(ServletContainerDispatcher.java:112)
> >
at
> >
>
org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.init(HttpServletDispatcher.java:36)
> >
at
> >
>
io.undertow.servlet.core.LifecyleInterceptorInvocation.proceed(LifecyleInterceptorInvocation.java:117)
> >
at
> >
>
org.wildfly.extension.undertow.security.RunAsLifecycleInterceptor.init(RunAsLifecycleInterceptor.java:78)
> >
at
> >
>
io.undertow.servlet.core.LifecyleInterceptorInvocation.proceed(LifecyleInterceptorInvocation.java:103)
> >
at
> >
>
io.undertow.servlet.core.ManagedServlet$DefaultInstanceStrategy.start(ManagedServlet.java:230)
> >
at
> >
>
io.undertow.servlet.core.ManagedServlet.createServlet(ManagedServlet.java:131)
> >
at
> >
>
io.undertow.servlet.core.DeploymentManagerImpl.start(DeploymentManagerImpl.java:511)
> >
at
> >
>
org.wildfly.extension.undertow.deployment.UndertowDeploymentService.startContext(UndertowDeploymentService.java:101)
> >
at
> >
>
org.wildfly.extension.undertow.deployment.UndertowDeploymentService$1.run(UndertowDeploymentService.java:82)
> >
... 6
more
> >
Caused by:
> >
> java.lang.RuntimeException:
> >
Failed to find
> provider
> >
infinispan
for realm
> >
at
> >
>
org.keycloak.services.DefaultKeycloakSessionFactory.init(DefaultKeycloakSessionFactory.java:66)
> >
at
> >
>
org.keycloak.services.resources.KeycloakApplication.createSessionFactory(KeycloakApplication.java:162)
> >
at
> >
>
org.keycloak.services.resources.KeycloakApplication.<init>(KeycloakApplication.java:62)
> >
at
> >
>
sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> >
Method)
> [rt.jar:1.7.0_45]
> >
at
> >
>
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> >
[rt.jar:1.7.0_45]
> >
at
> >
>
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> >
[rt.jar:1.7.0_45]
> >
at
> >
>
java.lang.reflect.Constructor.newInstance(Constructor.java:526)
> >
[rt.jar:1.7.0_45]
> >
at
> >
>
org.jboss.resteasy.core.ConstructorInjectorImpl.construct(ConstructorInjectorImpl.java:148)
> >
...
19 more
> >
> >
> >
Is the new
way to
> enable
> >
Infinispan
> different to
> >
what we had
> earlier? If
> >
so, can
someone
> please
> >
point out the
> correct way?
> >
> >
> >
Regards,
> >
Lohitha.
> >
> >
> >
> >
> _______________________________________________
> >
keycloak-user
> mailing list
> > keycloak-user(a)lists.jboss.org
<mailto:keycloak-user@lists.jboss.org>
<mailto:keycloak-user@lists.jboss.org
<mailto:keycloak-user@lists.jboss.org>>
<mailto:keycloak-user@lists.jboss.org
<mailto:keycloak-user@lists.jboss.org>
<mailto:keycloak-user@lists.jboss.org
<mailto:keycloak-user@lists.jboss.org>>>
> >
> <mailto:keycloak-user@lists.jboss.org
<mailto:keycloak-user@lists.jboss.org>
<mailto:keycloak-user@lists.jboss.org
<mailto:keycloak-user@lists.jboss.org>>
> <mailto:keycloak-user@lists.jboss.org
<mailto:keycloak-user@lists.jboss.org>
<mailto:keycloak-user@lists.jboss.org
<mailto:keycloak-user@lists.jboss.org>>>>
> >https://lists.jboss.org/mailman/listinfo/keycloak-user
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > _______________________________________________
> > keycloak-user mailing list
> >keycloak-user(a)lists.jboss.org
<mailto:keycloak-user@lists.jboss.org>
<mailto:keycloak-user@lists.jboss.org
<mailto:keycloak-user@lists.jboss.org>>
<mailto:keycloak-user@lists.jboss.org
<mailto:keycloak-user@lists.jboss.org>
<mailto:keycloak-user@lists.jboss.org
<mailto:keycloak-user@lists.jboss.org>>>
> >https://lists.jboss.org/mailman/listinfo/keycloak-user
> >
>
> --
> Bill Burke
> JBoss, a division of Red Hat
>http://bill.burkecentral.com
> _______________________________________________
> keycloak-user mailing list
> keycloak-user(a)lists.jboss.org
<mailto:keycloak-user@lists.jboss.org>
<mailto:keycloak-user@lists.jboss.org
<mailto:keycloak-user@lists.jboss.org>>
<mailto:keycloak-user@lists.jboss.org
<mailto:keycloak-user@lists.jboss.org>
<mailto:keycloak-user@lists.jboss.org
<mailto:keycloak-user@lists.jboss.org>>>
>
https://lists.jboss.org/mailman/listinfo/keycloak-user
>
>
--
Bill Burke
JBoss, a division of Red Hat
http://bill.burkecentral.com
_______________________________________________
keycloak-user mailing list
keycloak-user(a)lists.jboss.org
<mailto:keycloak-user@lists.jboss.org>
<mailto:keycloak-user@lists.jboss.org
<mailto:keycloak-user@lists.jboss.org>>
https://lists.jboss.org/mailman/listinfo/keycloak-user
--
Bill Burke
JBoss, a division of Red Hat
http://bill.burkecentral.com