Re: [keycloak-dev] Operational monitoring of Keycloak server

----- Original Message ----- > From: "Vlastimil Elias" <velias(a)redhat.com&gt; > To: "Stian Thorgersen" <stian(a)redhat.com&gt; > Cc: "Scott Rossillo" <srossillo(a)smartling.com&gt;, keycloak-dev(a)lists.jboss.org > Sent: Tuesday, 14 July, 2015 9:58:36 AM > Subject: Re: [keycloak-dev] Operational monitoring of Keycloak server > > On 14.7.2015 09:14, Stian Thorgersen wrote: >> ----- Original Message ----- >>> From: "Vlastimil Elias" <velias(a)redhat.com&gt; >>> To: "Stian Thorgersen" <stian(a)redhat.com&gt; >>> Cc: "Scott Rossillo" <srossillo(a)smartling.com&gt;, >>> keycloak-dev(a)lists.jboss.org >>> Sent: Tuesday, 14 July, 2015 8:56:43 AM >>> Subject: Re: [keycloak-dev] Operational monitoring of Keycloak server >>> >>> >>> >>> On 14.7.2015 08:42, Stian Thorgersen wrote: >>>> Ok so we add one that just returns OK or ERROR which doesn't require >>>> authentication, but then we add another that can return more details, >>>> including stats and what the error is. >>> Yep. Simple http endpoinds for now, later all this info may be expose >>> over DMR or some other mechanism if necessary. >>>> For full health we should check all external services that are used (db, >>>> identity providers and user federation providers). Cluster sync will >>>> depend on what Infinispan exposes, but I imagine they'll have the >>>> required >>>> details avail. >>> Beside endpoint for full health check (which should summarize all >>> services into one flag if I understand correctly, which is good for >>> loadbalancer) we should expose separate endpoint for each service >>> health, as it is better for operational monitoring systems like Nagios. >>> Admins are better informed what is going wrong and they can resolve >>> situation more quickly without necessity to search cause of general >>> failure flag somewhere else. >> We could have a single health endpoint that returns a 200 or 500 and also >> returns a json doc with details. Something like: >> >> { >> realm: ok, >> users: ok, >> realmCache: error, >> userCache: error, >> identity-providers: { >> google: ok, >> facebook: error >> }, >> user-federation: { >> ldap: ok >> } >> } >> >> I wonder if that's to much details to expose without authentication. > This should work, but I miss info about DB connection here. It is > crucial from operational point of view as this allows sysadmins quickly > detect that connection to DB is broken. > > I believe this kind of info can be safely exposed without > authentication. If the endpoint is clearly documented then access can be > always restricted on some network infra (webserver/proxy) ahead of KC. DB is "realm" and "users". >>> Vl. >>> >>>> ----- Original Message ----- >>>>> From: "Vlastimil Elias" <velias(a)redhat.com&gt; >>>>> To: "Scott Rossillo" <srossillo(a)smartling.com&gt;, "Stian Thorgersen" >>>>> <stian(a)redhat.com&gt; >>>>> Cc: keycloak-dev(a)lists.jboss.org >>>>> Sent: Tuesday, 14 July, 2015 8:34:29 AM >>>>> Subject: Re: [keycloak-dev] Operational monitoring of Keycloak server >>>>> >>>>> Yap, beside server-stat endpoint Stian talked about we should add some >>>>> server-health endpoint also to be used for remote health monitoring by >>>>> load balancers or systems like Nagios. >>>>> Simple http endpoint without authentication (as it doesn't leak any >>>>> sensitive information) is typically simplest way how to do this >>>>> monitoring. >>>>> >>>>> Vl. >>>>> >>>>> On 14.7.2015 05:13, Scott Rossillo wrote: >>>>>> Some type of health page would be great too for load balancers to >>>>>> monitor. Something that doesn't leak internal information but checks >>>>>> behind the scenes that: >>>>>> 1. Server can reach its databas(es) >>>>>> 2. Server cluster sync is working >>>>>> 3. Server can reach federation providers, etc. >>>>>> Endpoint should respond to get requests and return an http status >>>>>> reflective of server state. >>>>>> >>>>>> On Mon, Jul 13, 2015 at 11:18 AM Stian Thorgersen <stian(a)redhat.com >>>>>> <mailto:stian@redhat.com>> wrote: >>>>>> >>>>>> So looks like we're at agreement to add the additional info you >>>>>> wanted to server info page. >>>>>> >>>>>> How about we add an additional endpoint server-stat that can >>>>>> collect some stats about the server? >>>>>> >>>>>> ----- Original Message ----- >>>>>> > From: "Vlastimil Elias" <velias(a)redhat.com >>>>>> <mailto:velias@redhat.com>> >>>>>> > To: "Stian Thorgersen" <stian(a)redhat.com >>>>>> > <mailto:stian@redhat.com>> >>>>>> > Cc: keycloak-dev(a)lists.jboss.org >>>>>> <mailto:keycloak-dev@lists.jboss.org> >>>>>> > Sent: Monday, 13 July, 2015 5:06:34 PM >>>>>> > Subject: Re: [keycloak-dev] Operational monitoring of Keycloak >>>>>> server >>>>>> > >>>>>> > Looks like I have to look at WildFly/EAP DMR to see what is >>>>>> possible to >>>>>> > do with it, as I'm not sure if it is about remote monitoring >>>>>> also and >>>>>> > if/how it can be use from monitoring systems like Splunk. >>>>>> > >>>>>> > Vl. >>>>>> > >>>>>> > On 13.7.2015 15:26, Stian Thorgersen wrote: >>>>>> > > In WildFly/EAP that's DMR right? We're planning to make >>>>>> Keycloak managable >>>>>> > > through that as well. For example everything that goes into >>>>>> > > keycloak-server.json will eventually be moved to >>>>>> standalone.xml. Same with >>>>>> > > admin endpoints, everything you can do there you'll >>>>>> > > eventually >>>>>> be able to >>>>>> > > do through DMR and jboss-cli as well. >>>>>> > > >>>>>> > > However, IMO it would make sense to at least expose Keycloak >>>>>> specific >>>>>> > > information through the admin endpoints and console as well. >>>>>> Such number >>>>>> > > of sessions, etc.. >>>>>> > > >>>>>> > > ----- Original Message ----- >>>>>> > >> From: "Vlastimil Elias" <velias(a)redhat.com >>>>>> <mailto:velias@redhat.com>> >>>>>> > >> To: keycloak-dev(a)lists.jboss.org >>>>>> <mailto:keycloak-dev@lists.jboss.org> >>>>>> > >> Sent: Monday, 13 July, 2015 3:17:16 PM >>>>>> > >> Subject: [keycloak-dev] Operational monitoring of Keycloak >>>>>> > >> server >>>>>> > >> >>>>>> > >> Hi, >>>>>> > >> >>>>>> > >> as we deployed KC to production mode for >>>>>> https://developers.redhat.com >>>>>> > >> we started to think about operational monitoring, for >>>>>> > >> example >>>>>> from >>>>>> > >> Nagios or other systems of this type. >>>>>> > >> >>>>>> > >> KC user guide doesn't contain any chapter covering this >>>>>> topic, also no >>>>>> > >> any success over google search, so looks like KC doesn't >>>>>> > >> have >>>>>> > >> any >>>>>> > >> solution for this yet. >>>>>> > >> But I believe this is an important area which must be solved >>>>>> when KC is >>>>>> > >> used for production. >>>>>> > >> >>>>>> > >> I can imagine monitoring of JDBC connection if JPA is used, >>>>>> monitoring >>>>>> > >> of Mongo connection if used as store, monitoring of LDAP >>>>>> connection if >>>>>> > >> LDAP federation is used etc. >>>>>> > >> Also some statistics like numbers of active sso session, >>>>>> number of >>>>>> > >> logins per minute etc should be provided there. >>>>>> > >> >>>>>> > >> Monitoring is not about Keycloak core itself, it should be >>>>>> available for >>>>>> > >> extension developers also. For example we implemented own >>>>>> > >> UserFederationProvider which calls backend REST services. >>>>>> > >> We should be able to add info about this integration into >>>>>> monitoring >>>>>> > >> endpoint to be able to catch problems with this REST API. >>>>>> > >> >>>>>> > >> It should be probably implemented same way as used by >>>>>> > >> underlying >>>>>> > >> WildFly/EAP (JPA/JDBC is probably available for monitoring >>>>>> there). I'm >>>>>> > >> not sure if JMX is used there still or if some new framework >>>>>> > >> is >>>>>> > >> available for it. >>>>>> > >> Or KC should use some form of KC REST API for this, which >>>>>> should be >>>>>> > >> extended by additional info from KC extensions? >>>>>> > >> >>>>>> > >> What do you think? >>>>>> > >> >>>>>> > >> Vlastimil >>>>>> > >> >>>>>> > >> P.S we have https://issues.jboss.org/browse/RHD-552 for Red >>>>>> > >> Hat >>>>>> > >> Developer instance of KC >>>>>> > >> >>>>>> > >> -- >>>>>> > >> Vlastimil Elias >>>>>> > >> Principal Software Engineer >>>>>> > >> jboss.org <http://jboss.org> Development Team >>>>>> > >> >>>>>> > >> _______________________________________________ >>>>>> > >> keycloak-dev mailing list >>>>>> > >> keycloak-dev(a)lists.jboss.org >>>>>> <mailto:keycloak-dev@lists.jboss.org> >>>>>> > >> https://lists.jboss.org/mailman/listinfo/keycloak-dev >>>>>> > >> >>>>>> > >>>>>> > -- >>>>>> > Vlastimil Elias >>>>>> > Principal Software Engineer >>>>>> > jboss.org <http://jboss.org> Development Team >>>>>> > >>>>>> > >>>>>> _______________________________________________ >>>>>> keycloak-dev mailing list >>>>>> keycloak-dev(a)lists.jboss.org >>>>>> <mailto:keycloak-dev@lists.jboss.org> >>>>>> https://lists.jboss.org/mailman/listinfo/keycloak-dev >>>>>> >>>>> -- >>>>> Vlastimil Elias >>>>> Principal Software Engineer >>>>> jboss.org Development Team >>>>> >>>>> >>> -- >>> Vlastimil Elias >>> Principal Software Engineer >>> jboss.org Development Team >>> >>> > -- > Vlastimil Elias > Principal Software Engineer > jboss.org Development Team > >