[keycloak-user] How to enable Infinispan cache for realms, users and user sessions in Keycloak 1.6.1?

Bill Burke bburke at redhat.com
Mon Nov 30 09:47:15 EST 2015


I just did a loop test on GET 
https://{host}/auth/admin/realms/xxxxx/users/xxxx.  The same userid over 
and over as you described.  I see zero invalidations.  Are you sure you 
aren't doing any updates?

On 11/30/2015 9:33 AM, Stian Thorgersen wrote:
> If you can try building master and enable trace logging for
> "org.keycloak.models.cache.infinispan.InfinispanCacheUserProviderFactory".
> It will print log statements when entries are removed/invalidated in the
> cache.
>
> On 30 November 2015 at 15:19, Bill Burke <bburke at redhat.com
> <mailto:bburke at redhat.com>> wrote:
>
>     Can you describe your load tests please?  What kind of authentication
>     are you doing?  The only thing that would cause invalidations is HOTP.
>
>     On 11/30/2015 9:13 AM, Lohitha Chiranjeewa wrote:
>     > Bill, the problem here is that our load tests cannot go beyond a paltry
>     > 15-20 concurrent threads with the default set up (invalidation caches
>     > with SYNC mode - two HA AWS M3-Medium servers). Hence we're trying to
>     > find ways to improve on the performance and the avg response time under
>     > a load.
>     >
>     > On Mon, Nov 30, 2015 at 7:16 PM, Bill Burke <bburke at redhat.com <mailto:bburke at redhat.com>
>     > <mailto:bburke at redhat.com <mailto:bburke at redhat.com>>> wrote:
>     >
>     >     BTW, besides being more scalable, we invalidate rather than replicate to
>     >     avoid transmitting sensitive security data over the network.
>     >
>     >     On 11/30/2015 8:29 AM, Lohitha Chiranjeewa wrote:
>     >     > Glad if you could look into the above and check if there are app level
>     >     > issues related to invalidations, thanks.
>     >     >
>     >     > One way to get around this through config changes is to make the
>     >     > relevant cache modes 'ASYNC' and set up suitable values for 'queue-size'
>     >     > and 'queue-flush-interval' depending on your needs. Then the
>     >     > invalidations won't happen with each and every call.
>     >     >
>     >     > Obviously there would be problems if caches aren't invalidated in time
>     >     > with the above set up, it's up to the devs to come up with a suitable
>     >     > set of configs to cater to their needs.
>     >     >
>     >     >
>     >     > Regards,
>     >     > Lohitha.
>     >     >
>     >     >
>     >     >
>     >     > On Mon, Nov 30, 2015 at 1:08 PM, Stian Thorgersen <sthorger at redhat.com <mailto:sthorger at redhat.com>
>     <mailto:sthorger at redhat.com <mailto:sthorger at redhat.com>>
>     >     > <mailto:sthorger at redhat.com <mailto:sthorger at redhat.com>
>     <mailto:sthorger at redhat.com <mailto:sthorger at redhat.com>>>> wrote:
>     >     >
>     >     >     It is not expected behavior. Users should only be invalidated if
>     >     >     changes are made.
>     >     >
>     >     >     On 30 November 2015 at 07:01, Lohitha Chiranjeewa <kalc04 at gmail.com <mailto:kalc04 at gmail.com> <mailto:kalc04 at gmail.com
>     <mailto:kalc04 at gmail.com>>
>     >     >     <mailto:kalc04 at gmail.com <mailto:kalc04 at gmail.com> <mailto:kalc04 at gmail.com
>     <mailto:kalc04 at gmail.com>>>> wrote:
>     >     >
>     >     >         Just tested the same flow extensively with Keycloak 1.2.0 as
>     >     >         well, and it seems the behavior is the same. Lots of MySQL
>     >     >         select queries getting executed with each call. Seems there's a
>     >     >         number of unnecessary cache invalidations going on which causes
>     >     >         Keycloak to fetch data from the DB over and over.
>     >     >
>     >     >         Could you please confirm if this is the expected behavior? As
>     >     >         far as I can see there's considerable performance degradation
>     >     >         due to unnecessary cache invalidations.
>     >     >
>     >     >
>     >     >         On Fri, Nov 27, 2015 at 8:42 PM, Lohitha Chiranjeewa
>      >     >         <kalc04 at gmail.com <mailto:kalc04 at gmail.com>
>     <mailto:kalc04 at gmail.com <mailto:kalc04 at gmail.com>>
>     <mailto:kalc04 at gmail.com <mailto:kalc04 at gmail.com>
>     >     <mailto:kalc04 at gmail.com <mailto:kalc04 at gmail.com>>>> wrote:
>     >     >
>     >     >             I'm invoking the following API call:
>     >     >             https://{host}/auth/admin/realms/xxxxx/users/55ffe851-2d94-460e-88b9-bc7340531b56
>     >     >             (same realm and user ID over and over again). The two HA
>     >     >             server(s) are idle apart from serving those calls. And I'm
>     >     >             seeing the following SQL getting logged in my MySQL log for
>     >     >             each and every call:
>     >     >
>     >     >             select userentity0_.ID as ID1_42_,
>     >     >             userentity0_.CREATED_TIMESTAMP as CREATED_2_42_,
>     >     >             userentity0_.EMAIL as EMAIL3_42_,
>     >     >             userentity0_.EMAIL_CONSTRAINT as EMAIL_CO4_42_,
>     >     >             userentity0_.EMAIL_VERIFIED as EMAIL_VE5_42_,
>     >     >             userentity0_.ENABLED as ENABLED6_42_,
>     >     >             userentity0_.federation_link as federati7_42_,
>     >     >             userentity0_.FIRST_NAME as FIRST_NA8_42_,
>     >     >             userentity0_.LAST_NAME as LAST_NAM9_42_,
>     >     >             userentity0_.REALM_ID as REALM_I10_42_,
>     >     >             userentity0_.SERVICE_ACCOUNT_CLIENT_LINK as SERVICE11_42_,
>     >     >             userentity0_.TOTP as TOTP12_42_, userentity0_.USERNAME as
>     >     >             USERNAM13_42_ from USER_ENTITY userentity0_ where
>     >     >             userentity0_.ID='55ffe851-2d94-460e-88b9-bc7340531b56' and
>     >     >             userentity0_.REALM_ID='xxxxx'
>     >     >
>     >     >
>     >     >
>     >     >             On Fri, Nov 27, 2015 at 8:03 PM, Stian Thorgersen
>     >     >             <sthorger at redhat.com <mailto:sthorger at redhat.com>
>     <mailto:sthorger at redhat.com <mailto:sthorger at redhat.com>>
>     >     <mailto:sthorger at redhat.com <mailto:sthorger at redhat.com>
>     <mailto:sthorger at redhat.com <mailto:sthorger at redhat.com>>>> wrote:
>     >     >
>     >     >                 Strange - what are you doing and what are the SQL queries?
>     >     >
>     >     >                 On 27 November 2015 at 15:23, Lohitha Chiranjeewa
>      >     >                 <kalc04 at gmail.com <mailto:kalc04 at gmail.com>
>     <mailto:kalc04 at gmail.com <mailto:kalc04 at gmail.com>>
>     <mailto:kalc04 at gmail.com <mailto:kalc04 at gmail.com>
>     >     <mailto:kalc04 at gmail.com <mailto:kalc04 at gmail.com>>>> wrote:
>     >     >
>     >     >                     Yes Stian, that I understand. But the problem here
>     >     >                     is even if I execute continuous user retrieval calls
>     >     >                     (same user - no other functionality in between),
>     >     >                     still MySQL select queries get executed for each
>     >     >                     call. So there lies an issue isn't it?
>     >     >
>     >     >
>     >     >                     On Fri, Nov 27, 2015 at 6:48 PM, Stian Thorgersen
>     >      >                     <sthorger at redhat.com <mailto:sthorger at redhat.com>
>      >     <mailto:sthorger at redhat.com <mailto:sthorger at redhat.com>>
>     <mailto:sthorger at redhat.com <mailto:sthorger at redhat.com>
>     >     <mailto:sthorger at redhat.com <mailto:sthorger at redhat.com>>>>
>     >     >                     wrote:
>     >     >
>     >     >                         Things are still fetched from MySQL. Realms,
>     >     >                         clients, users, etc.. are then kept in the
>     >     >                         cache, but if it changes it's re-loaded from
>     >     >                         MySQL. We use an invalidation cache, not a
>     >     >                         distributed cache.
>     >     >
>     >     >                         On 27 November 2015 at 14:04, Lohitha
>     >     >                         Chiranjeewa <kalc04 at gmail.com <mailto:kalc04 at gmail.com> <mailto:kalc04 at gmail.com
>     <mailto:kalc04 at gmail.com>>
>     >     >                         <mailto:kalc04 at gmail.com <mailto:kalc04 at gmail.com> <mailto:kalc04 at gmail.com
>     <mailto:kalc04 at gmail.com>>>> wrote:
>     >     >
>     >     >                             What I mean is, if it were working, I
>     >     >                             shouldn't see mysql queries getting executed
>     >     >                             right? So my guess is data is still fetched
>     >     >                             from the db instead of the cache.
>     >     >
>     >     >                             On Nov 27, 2015 5:52 PM, "Stian Thorgersen"
>     >     >                             <sthorger at redhat.com <mailto:sthorger at redhat.com>
>     <mailto:sthorger at redhat.com <mailto:sthorger at redhat.com>>
>     >     >                             <mailto:sthorger at redhat.com <mailto:sthorger at redhat.com>
>     <mailto:sthorger at redhat.com <mailto:sthorger at redhat.com>>>> wrote:
>     >     >
>     >     >                                 Yup, so it's working now?
>     >     >
>     >     >                                 On 27 November 2015 at 13:20, Lohitha
>     >     >                                 Chiranjeewa <kalc04 at gmail.com <mailto:kalc04 at gmail.com> <mailto:kalc04 at gmail.com
>     <mailto:kalc04 at gmail.com>>
>     >     >                                 <mailto:kalc04 at gmail.com <mailto:kalc04 at gmail.com> <mailto:kalc04 at gmail.com
>     <mailto:kalc04 at gmail.com>>>> wrote:
>     >     >
>     >     >                                     Apologies, keycloak-server.json
>     >     >                                     entries should change to:
>     >     >
>     >     >                                          "realm": {
>     >     >                                              "provider": "jpa"
>     >     >                                          },
>     >     >
>     >     >                                          "user": {
>     >     >                                              "provider": "jpa"
>     >     >                                          },
>     >     >
>     >     >                                          "userSessionPersister": {
>     >     >                                              "provider": "jpa"
>     >     >                                          },
>     >     >
>     >     >                                     On Fri, Nov 27, 2015 at 5:49 PM,
>     >     >                                     Lohitha Chiranjeewa
>     >     >                                     <kalc04 at gmail.com <mailto:kalc04 at gmail.com> <mailto:kalc04 at gmail.com
>     <mailto:kalc04 at gmail.com>>
>      >      >
>       <mailto:kalc04 at gmail.com <mailto:kalc04 at gmail.com>
>      >     <mailto:kalc04 at gmail.com <mailto:kalc04 at gmail.com>>>> wrote:
>      >      >
>      >      >                                         Hi Stian,
>      >      >
>      >      >                                         As per the
>     migration guide, I
>      >      >                                         should have
>     Infinispan up and
>      >      >                                         running for
>     realms, users and
>      >      >                                         user sessions without
>      >     doing any
>      >      >                                         specific changes.
>      >      >
>       keycloak-server.json was
>      >      >                                         reverted back to
>     have the
>      >      >                                         following entries:
>      >      >                                         ...
>      >      >                                              "realm": {
>      >      >                                                  "provider":
>      >     "infinispan"
>      >      >                                              },
>      >      >
>      >      >                                              "user": {
>      >      >                                                  "provider":
>      >     "infinispan"
>      >      >                                              },
>      >      >
>      >      >
>      >     "userSessionPersister": {
>      >      >                                                  "provider":
>      >     "infinispan"
>      >      >                                              },
>      >      >                                         ...
>      >      >
>      >      >                                         In the Admin Console I
>      >     have both
>      >      >                                         Realm Cache and
>     User Cache
>      >      >                                         enables. I see certain
>      >      >                                         Infinispan related
>     logs
>      >     getting
>      >      >                                         logged as well.
>      >      >
>      >      >                                         However, at the same
>      >     time, I see
>      >      >                                         MySQL queries getting
>      >     executed
>      >      >                                         for all user
>     retrieval API
>      >      >                                         invocations (even
>     if the same
>      >      >                                         user is retrieved
>      >     continuously):
>      >      >                                         ...
>      >      >                                         select
>     userentity0_.ID as
>      >      >                                         ID1_42_,
>      >      >
>      >       userentity0_.CREATED_TIMESTAMP
>      >      >                                         as CREATED_2_42_,
>      >      >                                         userentity0_.EMAIL as
>      >      >                                         EMAIL3_42_,
>      >      >
>      >       userentity0_.EMAIL_CONSTRAINT as
>      >      >                                         EMAIL_CO4_42_,
>      >      >
>      >       userentity0_.EMAIL_VERIFIED as
>      >      >                                         EMAIL_VE5_42_,
>      >      >
>       userentity0_.ENABLED as
>      >      >                                         ENABLED6_42_,
>      >      >
>      >       userentity0_.federation_link as
>      >      >                                         federati7_42_,
>      >      >
>       userentity0_.FIRST_NAME as
>      >      >                                         FIRST_NA8_42_,
>      >      >
>       userentity0_.LAST_NAME as
>      >      >                                         LAST_NAM9_42_,
>      >      >
>       userentity0_.REALM_ID as
>      >      >                                         REALM_I10_42_,
>      >      >
>      >       userentity0_.SERVICE_ACCOUNT_CLIENT_LINK
>      >      >                                         as SERVICE11_42_,
>      >      >                                         userentity0_.TOTP as
>      >     TOTP12_42_,
>      >      >
>       userentity0_.USERNAME as
>      >      >                                         USERNAM13_42_ from
>      >     USER_ENTITY
>      >      >                                         userentity0_ where
>      >      >
>      >       userentity0_.ID='55ffe851-2d94-460e-88b9-bc7340531b56'
>      >      >                                         and
>      >     userentity0_.REALM_ID='xxxxx'
>      >      >                                         ...
>      >      >
>      >      >                                         So it seems
>     something is
>      >     wrong
>      >      >                                         here. Could you
>     point out any
>      >      >                                         areas that I could
>      >     further look
>      >      >                                         into?
>      >      >
>      >      >
>      >      >                                         Regards,
>      >      >                                         Lohitha.
>      >      >
>      >      >                                         On Thu, Nov 26,
>     2015 at
>      >     7:58 PM,
>      >      >                                         Stian Thorgersen
>      >      >
>       <sthorger at redhat.com <mailto:sthorger at redhat.com>
>      >     <mailto:sthorger at redhat.com <mailto:sthorger at redhat.com>>
>     >     >                                         <mailto:sthorger at redhat.com <mailto:sthorger at redhat.com>
>     <mailto:sthorger at redhat.com <mailto:sthorger at redhat.com>>>> wrote:
>     >     >
>     >     >                                             Please read the migration guide
>     >     >
>     >     >                                             On 26 November 2015 at
>     >     >                                             14:53, Lohitha Chiranjeewa
>     >     >                                             <kalc04 at gmail.com <mailto:kalc04 at gmail.com> <mailto:kalc04 at gmail.com
>     <mailto:kalc04 at gmail.com>>
>     >      >
>      >       <mailto:kalc04 at gmail.com <mailto:kalc04 at gmail.com>
>     <mailto:kalc04 at gmail.com <mailto:kalc04 at gmail.com>>>>
>      >      >                                             wrote:
>      >      >
>      >      >                                                 Hi,
>      >      >
>      >      >                                                 We're in the
>      >     process of
>      >      >                                                 assessing the
>      >     impact on
>      >      >                                                 upgrading from
>      >     Keycloak
>      >      >                                                 1.2.0 to
>     1.6.1.
>      >     We came
>      >      >                                                 across an
>     issue when
>      >      >                                                 trying to
>     enable
>      >      >                                                 Infinispan
>     cache
>      >     through
>      >      >                                                 the
>      >     keycloak-server.json
>      >      >                                                 file as we
>     used
>      >     to do in
>      >      >                                                 1.2.0.
>      >      >
>      >      >                                                 We have
>     the following
>      >      >                                                 entries in
>     1.6.1:
>      >      >
>     "realm": {
>      >      >
>     "provider":
>      >      >                                                 "infinispan"
>      >      >                                                      },
>      >      >
>      >      >                                                      "user": {
>      >      >
>     "provider":
>      >      >                                                 "infinispan"
>      >      >                                                      },
>      >      >
>      >      >
>      >      >
>      >       "userSessionPersister": {
>      >      >
>     "provider":
>      >      >                                                 "infinispan"
>      >      >                                                      },
>      >      >                                                 .........
>      >      >
>      >      >
>      >       "connectionsInfinispan": {
>      >      >
>      >     "default" : {
>      >      >
>      >      >
>       "cacheContainer" :
>      >      >
>      >       "java:comp/env/infinispan/Keycloak"
>      >      >                                                          }
>      >      >                                                      }
>      >      >
>      >      >                                                 All
>     configurations in
>      >      >                                                 1.6.1
>      >     standalone-ha.xml
>      >      >                                                 file remains
>      >     comparable
>      >      >                                                 (and
>     correct to
>      >     the best
>      >      >                                                 of our
>     knowledge)
>      >     with
>      >      >                                                 the ones
>     in 1.2.0.
>      >      >
>      >      >                                                 With the above
>      >     configs,
>      >      >                                                 when we
>     start the
>      >      >                                                 Keycloak
>     service the
>      >      >                                                 following
>      >     error(s) get
>      >      >                                                 logged:
>      >      >
>      >      >
>       18:03:31,610 ERROR
>      >      >
>      >       [org.jboss.msc.service.fail]
>      >      >
>       (ServerService Thread
>      >      >                                                 Pool -- 64)
>      >     MSC000001:
>      >      >                                                 Failed to
>     start
>      >     service
>      >      >
>      >       jboss.undertow.deployment.default-server.default-host./auth:
>      >      >
>      >       org.jboss.msc.service.StartException
>      >      >                                                 in service
>      >      >
>      >       jboss.undertow.deployment.default-server.default-host./auth:
>      >      >
>      >       java.lang.RuntimeException:
>      >      >                                                 Failed to
>     construct
>      >      >                                                 public
>      >      >
>      >
>       org.keycloak.services.resources.KeycloakApplication(javax.servlet.ServletContext,org.jboss.resteasy.core.Dispatcher)
>      >      >                                                      at
>      >      >
>      >
>       org.wildfly.extension.undertow.deployment.UndertowDeploymentService$1.run(UndertowDeploymentService.java:85)
>      >      >                                                      at
>      >      >
>      >
>       java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>      >      >
>       [rt.jar:1.7.0_45]
>      >      >                                                      at
>      >      >
>      >       java.util.concurrent.FutureTask.run(FutureTask.java:262)
>      >      >
>       [rt.jar:1.7.0_45]
>      >      >                                                      at
>      >      >
>      >
>       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>      >      >
>       [rt.jar:1.7.0_45]
>      >      >                                                      at
>      >      >
>      >
>       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>      >      >
>       [rt.jar:1.7.0_45]
>      >      >                                                      at
>      >      >
>      >       java.lang.Thread.run(Thread.java:744)
>      >      >
>       [rt.jar:1.7.0_45]
>      >      >                                                      at
>      >      >
>      >       org.jboss.threads.JBossThread.run(JBossThread.java:320)
>      >      >
>      >       [jboss-threads-2.2.0.Final.jar:2.2.0.Final]
>      >      >                                                 Caused by:
>      >      >
>      >       java.lang.RuntimeException:
>      >      >                                                 Failed to
>     construct
>      >      >                                                 public
>      >      >
>      >
>       org.keycloak.services.resources.KeycloakApplication(javax.servlet.ServletContext,org.jboss.resteasy.core.Dispatcher)
>      >      >                                                      at
>      >      >
>      >
>       org.jboss.resteasy.core.ConstructorInjectorImpl.construct(ConstructorInjectorImpl.java:160)
>      >      >                                                      at
>      >      >
>      >
>       org.jboss.resteasy.spi.ResteasyProviderFactory.createProviderInstance(ResteasyProviderFactory.java:2211)
>      >      >                                                      at
>      >      >
>      >
>       org.jboss.resteasy.spi.ResteasyDeployment.createApplication(ResteasyDeployment.java:295)
>      >      >                                                      at
>      >      >
>      >
>       org.jboss.resteasy.spi.ResteasyDeployment.start(ResteasyDeployment.java:236)
>      >      >                                                      at
>      >      >
>      >
>       org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.init(ServletContainerDispatcher.java:112)
>      >      >                                                      at
>      >      >
>      >
>       org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.init(HttpServletDispatcher.java:36)
>      >      >                                                      at
>      >      >
>      >
>       io.undertow.servlet.core.LifecyleInterceptorInvocation.proceed(LifecyleInterceptorInvocation.java:117)
>      >      >                                                      at
>      >      >
>      >
>       org.wildfly.extension.undertow.security.RunAsLifecycleInterceptor.init(RunAsLifecycleInterceptor.java:78)
>      >      >                                                      at
>      >      >
>      >
>       io.undertow.servlet.core.LifecyleInterceptorInvocation.proceed(LifecyleInterceptorInvocation.java:103)
>      >      >                                                      at
>      >      >
>      >
>       io.undertow.servlet.core.ManagedServlet$DefaultInstanceStrategy.start(ManagedServlet.java:230)
>      >      >                                                      at
>      >      >
>      >
>       io.undertow.servlet.core.ManagedServlet.createServlet(ManagedServlet.java:131)
>      >      >                                                      at
>      >      >
>      >
>       io.undertow.servlet.core.DeploymentManagerImpl.start(DeploymentManagerImpl.java:511)
>      >      >                                                      at
>      >      >
>      >
>       org.wildfly.extension.undertow.deployment.UndertowDeploymentService.startContext(UndertowDeploymentService.java:101)
>      >      >                                                      at
>      >      >
>      >
>       org.wildfly.extension.undertow.deployment.UndertowDeploymentService$1.run(UndertowDeploymentService.java:82)
>      >      >                                                      ... 6
>     more
>      >      >                                                 Caused by:
>      >      >
>      >       java.lang.RuntimeException:
>      >      >                                                 Failed to find
>      >     provider
>      >      >                                                 infinispan
>     for realm
>      >      >                                                      at
>      >      >
>      >
>       org.keycloak.services.DefaultKeycloakSessionFactory.init(DefaultKeycloakSessionFactory.java:66)
>      >      >                                                      at
>      >      >
>      >
>       org.keycloak.services.resources.KeycloakApplication.createSessionFactory(KeycloakApplication.java:162)
>      >      >                                                      at
>      >      >
>      >
>       org.keycloak.services.resources.KeycloakApplication.<init>(KeycloakApplication.java:62)
>      >      >                                                      at
>      >      >
>      >       sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
>      >      >                                                 Method)
>      >     [rt.jar:1.7.0_45]
>      >      >                                                      at
>      >      >
>      >
>       sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
>      >      >
>       [rt.jar:1.7.0_45]
>      >      >                                                      at
>      >      >
>      >
>       sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>      >      >
>       [rt.jar:1.7.0_45]
>      >      >                                                      at
>      >      >
>      >       java.lang.reflect.Constructor.newInstance(Constructor.java:526)
>      >      >
>       [rt.jar:1.7.0_45]
>      >      >                                                      at
>      >      >
>      >
>       org.jboss.resteasy.core.ConstructorInjectorImpl.construct(ConstructorInjectorImpl.java:148)
>      >      >                                                      ...
>     19 more
>      >      >
>      >      >
>      >      >                                                 Is the new
>     way to
>      >     enable
>      >      >                                                 Infinispan
>      >     different to
>      >      >                                                 what we had
>      >     earlier? If
>      >      >                                                 so, can
>     someone
>      >     please
>      >      >                                                 point out the
>      >     correct way?
>      >      >
>      >      >
>      >      >                                                 Regards,
>      >      >                                                 Lohitha.
>      >      >
>      >      >
>      >      >
>      >      >
>      >       _______________________________________________
>      >      >                                                 keycloak-user
>      >     mailing list
>      >      > keycloak-user at lists.jboss.org
>     <mailto:keycloak-user at lists.jboss.org>
>     <mailto:keycloak-user at lists.jboss.org
>     <mailto:keycloak-user at lists.jboss.org>>
>      >      >
>      >       <mailto:keycloak-user at lists.jboss.org
>     <mailto:keycloak-user at lists.jboss.org>
>     >     <mailto:keycloak-user at lists.jboss.org <mailto:keycloak-user at lists.jboss.org>>>
>     >      >https://lists.jboss.org/mailman/listinfo/keycloak-user
>     >     >
>     >     >
>     >     >
>     >     >
>     >     >
>     >     >
>     >     >
>     >     >
>     >     >
>     >     >
>     >     >
>     >     >
>     >     >
>     >     >
>     >     > _______________________________________________
>     >     > keycloak-user mailing list
>      >     >keycloak-user at lists.jboss.org
>     <mailto:keycloak-user at lists.jboss.org>
>     <mailto:keycloak-user at lists.jboss.org
>     <mailto:keycloak-user at lists.jboss.org>>
>     >     >https://lists.jboss.org/mailman/listinfo/keycloak-user
>     >     >
>     >
>     >     --
>     >     Bill Burke
>     >     JBoss, a division of Red Hat
>     >http://bill.burkecentral.com
>     >     _______________________________________________
>     >     keycloak-user mailing list
>      > keycloak-user at lists.jboss.org
>     <mailto:keycloak-user at lists.jboss.org>
>     <mailto:keycloak-user at lists.jboss.org
>     <mailto:keycloak-user at lists.jboss.org>>
>      > https://lists.jboss.org/mailman/listinfo/keycloak-user
>      >
>      >
>
>     --
>     Bill Burke
>     JBoss, a division of Red Hat
>     http://bill.burkecentral.com
>     _______________________________________________
>     keycloak-user mailing list
>     keycloak-user at lists.jboss.org <mailto:keycloak-user at lists.jboss.org>
>     https://lists.jboss.org/mailman/listinfo/keycloak-user
>
>

-- 
Bill Burke
JBoss, a division of Red Hat
http://bill.burkecentral.com


More information about the keycloak-user mailing list