Re: [keycloak-dev] Operational monitoring of Keycloak server

From: "Stian Thorgersen" <stian(a)redhat.com&gt; To: "Vlastimil Elias" <velias(a)redhat.com&gt; Cc: keycloak-dev(a)lists.jboss.org Sent: Thursday, July 16, 2015 2:49:56 AM Subject: Re: [keycloak-dev] Operational monitoring of Keycloak server ----- Original Message ----- > From: "Vlastimil Elias" <velias(a)redhat.com&gt; > To: "Stian Thorgersen" <stian(a)redhat.com&gt; > Cc: keycloak-dev(a)lists.jboss.org > Sent: Thursday, 16 July, 2015 8:15:29 AM > Subject: Re: [keycloak-dev] Operational monitoring of Keycloak server > > Hi, > > OK, if DB is "realm" and "users" then it have to be well documented as > it is not obvious for people without deeper knowledge of KC. We could always call it realm-database or something, but we have 3 different stores so should be separate status for them > > Do we have JIRA issue for this or should I create one? Nope, so please create one > > Vl. > > > On 14.7.2015 10:06, Stian Thorgersen wrote: > > > > ----- Original Message ----- > >> From: "Vlastimil Elias" <velias(a)redhat.com&gt; > >> To: "Stian Thorgersen" <stian(a)redhat.com&gt; > >> Cc: "Scott Rossillo" <srossillo(a)smartling.com&gt;, > >> keycloak-dev(a)lists.jboss.org > >> Sent: Tuesday, 14 July, 2015 9:58:36 AM > >> Subject: Re: [keycloak-dev] Operational monitoring of Keycloak server > >> > >> On 14.7.2015 09:14, Stian Thorgersen wrote: > >>> ----- Original Message ----- > >>>> From: "Vlastimil Elias" <velias(a)redhat.com&gt; > >>>> To: "Stian Thorgersen" <stian(a)redhat.com&gt; > >>>> Cc: "Scott Rossillo" <srossillo(a)smartling.com&gt;, > >>>> keycloak-dev(a)lists.jboss.org > >>>> Sent: Tuesday, 14 July, 2015 8:56:43 AM > >>>> Subject: Re: [keycloak-dev] Operational monitoring of Keycloak server > >>>> > >>>> > >>>> > >>>> On 14.7.2015 08:42, Stian Thorgersen wrote: > >>>>> Ok so we add one that just returns OK or ERROR which doesn't require > >>>>> authentication, but then we add another that can return more details, > >>>>> including stats and what the error is. > >>>> Yep. Simple http endpoinds for now, later all this info may be expose > >>>> over DMR or some other mechanism if necessary. > >>>>> For full health we should check all external services that are used > >>>>> (db, > >>>>> identity providers and user federation providers). Cluster sync will > >>>>> depend on what Infinispan exposes, but I imagine they'll have the > >>>>> required > >>>>> details avail. > >>>> Beside endpoint for full health check (which should summarize all > >>>> services into one flag if I understand correctly, which is good for > >>>> loadbalancer) we should expose separate endpoint for each service > >>>> health, as it is better for operational monitoring systems like > >>>> Nagios. > >>>> Admins are better informed what is going wrong and they can resolve > >>>> situation more quickly without necessity to search cause of general > >>>> failure flag somewhere else. > >>> We could have a single health endpoint that returns a 200 or 500 and > >>> also > >>> returns a json doc with details. Something like: > >>> > >>> { > >>> realm: ok, > >>> users: ok, > >>> realmCache: error, > >>> userCache: error, > >>> identity-providers: { > >>> google: ok, > >>> facebook: error > >>> }, > >>> user-federation: { > >>> ldap: ok > >>> } > >>> } > >>> > >>> I wonder if that's to much details to expose without authentication. > >> This should work, but I miss info about DB connection here. It is > >> crucial from operational point of view as this allows sysadmins quickly > >> detect that connection to DB is broken. > >> > >> I believe this kind of info can be safely exposed without > >> authentication. If the endpoint is clearly documented then access can be > >> always restricted on some network infra (webserver/proxy) ahead of KC. > > DB is "realm" and "users". > > > >>>> Vl. > >>>> > >>>>> ----- Original Message ----- > >>>>>> From: "Vlastimil Elias" <velias(a)redhat.com&gt; > >>>>>> To: "Scott Rossillo" <srossillo(a)smartling.com&gt;, "Stian Thorgersen" > >>>>>> <stian(a)redhat.com&gt; > >>>>>> Cc: keycloak-dev(a)lists.jboss.org > >>>>>> Sent: Tuesday, 14 July, 2015 8:34:29 AM > >>>>>> Subject: Re: [keycloak-dev] Operational monitoring of Keycloak > >>>>>> server > >>>>>> > >>>>>> Yap, beside server-stat endpoint Stian talked about we should add > >>>>>> some > >>>>>> server-health endpoint also to be used for remote health monitoring > >>>>>> by > >>>>>> load balancers or systems like Nagios. > >>>>>> Simple http endpoint without authentication (as it doesn't leak any > >>>>>> sensitive information) is typically simplest way how to do this > >>>>>> monitoring. > >>>>>> > >>>>>> Vl. > >>>>>> > >>>>>> On 14.7.2015 05:13, Scott Rossillo wrote: > >>>>>>> Some type of health page would be great too for load balancers to > >>>>>>> monitor. Something that doesn't leak internal information but > >>>>>>> checks > >>>>>>> behind the scenes that: > >>>>>>> 1. Server can reach its databas(es) > >>>>>>> 2. Server cluster sync is working > >>>>>>> 3. Server can reach federation providers, etc. > >>>>>>> Endpoint should respond to get requests and return an http status > >>>>>>> reflective of server state. > >>>>>>> > >>>>>>> On Mon, Jul 13, 2015 at 11:18 AM Stian Thorgersen <stian(a)redhat.com > >>>>>>> <mailto:stian@redhat.com>> wrote: > >>>>>>> > >>>>>>> So looks like we're at agreement to add the additional info > >>>>>>> you > >>>>>>> wanted to server info page. > >>>>>>> > >>>>>>> How about we add an additional endpoint server-stat that can > >>>>>>> collect some stats about the server? > >>>>>>> > >>>>>>> ----- Original Message ----- > >>>>>>> > From: "Vlastimil Elias" <velias(a)redhat.com > >>>>>>> <mailto:velias@redhat.com>> > >>>>>>> > To: "Stian Thorgersen" <stian(a)redhat.com > >>>>>>> > <mailto:stian@redhat.com>> > >>>>>>> > Cc: keycloak-dev(a)lists.jboss.org > >>>>>>> <mailto:keycloak-dev@lists.jboss.org> > >>>>>>> > Sent: Monday, 13 July, 2015 5:06:34 PM > >>>>>>> > Subject: Re: [keycloak-dev] Operational monitoring of > >>>>>>> > Keycloak > >>>>>>> server > >>>>>>> > > >>>>>>> > Looks like I have to look at WildFly/EAP DMR to see what > >>>>>>> > is > >>>>>>> possible to > >>>>>>> > do with it, as I'm not sure if it is about remote > >>>>>>> > monitoring > >>>>>>> also and > >>>>>>> > if/how it can be use from monitoring systems like Splunk. > >>>>>>> > > >>>>>>> > Vl. > >>>>>>> > > >>>>>>> > On 13.7.2015 15:26, Stian Thorgersen wrote: > >>>>>>> > > In WildFly/EAP that's DMR right? We're planning to make > >>>>>>> Keycloak managable > >>>>>>> > > through that as well. For example everything that goes > >>>>>>> > > into > >>>>>>> > > keycloak-server.json will eventually be moved to > >>>>>>> standalone.xml. Same with > >>>>>>> > > admin endpoints, everything you can do there you'll > >>>>>>> > > eventually > >>>>>>> be able to > >>>>>>> > > do through DMR and jboss-cli as well. > >>>>>>> > > > >>>>>>> > > However, IMO it would make sense to at least expose > >>>>>>> > > Keycloak > >>>>>>> specific > >>>>>>> > > information through the admin endpoints and console as > >>>>>>> > > well. > >>>>>>> Such number > >>>>>>> > > of sessions, etc.. > >>>>>>> > > > >>>>>>> > > ----- Original Message ----- > >>>>>>> > >> From: "Vlastimil Elias" <velias(a)redhat.com > >>>>>>> <mailto:velias@redhat.com>> > >>>>>>> > >> To: keycloak-dev(a)lists.jboss.org > >>>>>>> <mailto:keycloak-dev@lists.jboss.org> > >>>>>>> > >> Sent: Monday, 13 July, 2015 3:17:16 PM > >>>>>>> > >> Subject: [keycloak-dev] Operational monitoring of > >>>>>>> > >> Keycloak > >>>>>>> > >> server > >>>>>>> > >> > >>>>>>> > >> Hi, > >>>>>>> > >> > >>>>>>> > >> as we deployed KC to production mode for > >>>>>>> https://developers.redhat.com > >>>>>>> > >> we started to think about operational monitoring, for > >>>>>>> > >> example > >>>>>>> from > >>>>>>> > >> Nagios or other systems of this type. > >>>>>>> > >> > >>>>>>> > >> KC user guide doesn't contain any chapter covering this > >>>>>>> topic, also no > >>>>>>> > >> any success over google search, so looks like KC > >>>>>>> > >> doesn't > >>>>>>> > >> have > >>>>>>> > >> any > >>>>>>> > >> solution for this yet. > >>>>>>> > >> But I believe this is an important area which must be > >>>>>>> > >> solved > >>>>>>> when KC is > >>>>>>> > >> used for production. > >>>>>>> > >> > >>>>>>> > >> I can imagine monitoring of JDBC connection if JPA is > >>>>>>> > >> used, > >>>>>>> monitoring > >>>>>>> > >> of Mongo connection if used as store, monitoring of > >>>>>>> > >> LDAP > >>>>>>> connection if > >>>>>>> > >> LDAP federation is used etc. > >>>>>>> > >> Also some statistics like numbers of active sso > >>>>>>> > >> session, > >>>>>>> number of > >>>>>>> > >> logins per minute etc should be provided there. > >>>>>>> > >> > >>>>>>> > >> Monitoring is not about Keycloak core itself, it should > >>>>>>> > >> be > >>>>>>> available for > >>>>>>> > >> extension developers also. For example we implemented > >>>>>>> > >> own > >>>>>>> > >> UserFederationProvider which calls backend REST > >>>>>>> > >> services. > >>>>>>> > >> We should be able to add info about this integration > >>>>>>> > >> into > >>>>>>> monitoring > >>>>>>> > >> endpoint to be able to catch problems with this REST > >>>>>>> > >> API. > >>>>>>> > >> > >>>>>>> > >> It should be probably implemented same way as used by > >>>>>>> > >> underlying > >>>>>>> > >> WildFly/EAP (JPA/JDBC is probably available for > >>>>>>> > >> monitoring > >>>>>>> there). I'm > >>>>>>> > >> not sure if JMX is used there still or if some new > >>>>>>> > >> framework > >>>>>>> > >> is > >>>>>>> > >> available for it. > >>>>>>> > >> Or KC should use some form of KC REST API for this, > >>>>>>> > >> which > >>>>>>> should be > >>>>>>> > >> extended by additional info from KC extensions? > >>>>>>> > >> > >>>>>>> > >> What do you think? > >>>>>>> > >> > >>>>>>> > >> Vlastimil > >>>>>>> > >> > >>>>>>> > >> P.S we have https://issues.jboss.org/browse/RHD-552 for > >>>>>>> > >> Red > >>>>>>> > >> Hat > >>>>>>> > >> Developer instance of KC > >>>>>>> > >> > >>>>>>> > >> -- > >>>>>>> > >> Vlastimil Elias > >>>>>>> > >> Principal Software Engineer > >>>>>>> > >> jboss.org <http://jboss.org> Development Team > >>>>>>> > >> > >>>>>>> > >> _______________________________________________ > >>>>>>> > >> keycloak-dev mailing list > >>>>>>> > >> keycloak-dev(a)lists.jboss.org > >>>>>>> <mailto:keycloak-dev@lists.jboss.org> > >>>>>>> > >> https://lists.jboss.org/mailman/listinfo/keycloak-dev > >>>>>>> > >> > >>>>>>> > > >>>>>>> > -- > >>>>>>> > Vlastimil Elias > >>>>>>> > Principal Software Engineer > >>>>>>> > jboss.org <http://jboss.org> Development Team > >>>>>>> > > >>>>>>> > > >>>>>>> _______________________________________________ > >>>>>>> keycloak-dev mailing list > >>>>>>> keycloak-dev(a)lists.jboss.org > >>>>>>> <mailto:keycloak-dev@lists.jboss.org> > >>>>>>> https://lists.jboss.org/mailman/listinfo/keycloak-dev > >>>>>>> > >>>>>> -- > >>>>>> Vlastimil Elias > >>>>>> Principal Software Engineer > >>>>>> jboss.org Development Team > >>>>>> > >>>>>> > >>>> -- > >>>> Vlastimil Elias > >>>> Principal Software Engineer > >>>> jboss.org Development Team > >>>> > >>>> > >> -- > >> Vlastimil Elias > >> Principal Software Engineer > >> jboss.org Development Team > >> > >> > > -- > Vlastimil Elias > Principal Software Engineer > jboss.org Development Team > > _______________________________________________ keycloak-dev mailing list keycloak-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/keycloak-dev