From: "Vlastimil Elias" <velias(a)redhat.com>
To: "Stian Thorgersen" <stian(a)redhat.com>
Cc: keycloak-dev(a)lists.jboss.org
Sent: Thursday, 16 July, 2015 8:15:29 AM
Subject: Re: [keycloak-dev] Operational monitoring of Keycloak server
Hi,
OK, if DB is "realm" and "users" then it have to be well documented
as
it is not obvious for people without deeper knowledge of KC.
We could always call it realm-database or something, but we have 3 different stores so
should be separate status for them
Vl.
On 14.7.2015 10:06, Stian Thorgersen wrote:
>
> ----- Original Message -----
>> From: "Vlastimil Elias" <velias(a)redhat.com>
>> To: "Stian Thorgersen" <stian(a)redhat.com>
>> Cc: "Scott Rossillo" <srossillo(a)smartling.com>,
>> keycloak-dev(a)lists.jboss.org
>> Sent: Tuesday, 14 July, 2015 9:58:36 AM
>> Subject: Re: [keycloak-dev] Operational monitoring of Keycloak server
>>
>> On 14.7.2015 09:14, Stian Thorgersen wrote:
>>> ----- Original Message -----
>>>> From: "Vlastimil Elias" <velias(a)redhat.com>
>>>> To: "Stian Thorgersen" <stian(a)redhat.com>
>>>> Cc: "Scott Rossillo" <srossillo(a)smartling.com>,
>>>> keycloak-dev(a)lists.jboss.org
>>>> Sent: Tuesday, 14 July, 2015 8:56:43 AM
>>>> Subject: Re: [keycloak-dev] Operational monitoring of Keycloak server
>>>>
>>>>
>>>>
>>>> On 14.7.2015 08:42, Stian Thorgersen wrote:
>>>>> Ok so we add one that just returns OK or ERROR which doesn't
require
>>>>> authentication, but then we add another that can return more
details,
>>>>> including stats and what the error is.
>>>> Yep. Simple http endpoinds for now, later all this info may be expose
>>>> over DMR or some other mechanism if necessary.
>>>>> For full health we should check all external services that are used
>>>>> (db,
>>>>> identity providers and user federation providers). Cluster sync
will
>>>>> depend on what Infinispan exposes, but I imagine they'll have
the
>>>>> required
>>>>> details avail.
>>>> Beside endpoint for full health check (which should summarize all
>>>> services into one flag if I understand correctly, which is good for
>>>> loadbalancer) we should expose separate endpoint for each service
>>>> health, as it is better for operational monitoring systems like Nagios.
>>>> Admins are better informed what is going wrong and they can resolve
>>>> situation more quickly without necessity to search cause of general
>>>> failure flag somewhere else.
>>> We could have a single health endpoint that returns a 200 or 500 and also
>>> returns a json doc with details. Something like:
>>>
>>> {
>>> realm: ok,
>>> users: ok,
>>> realmCache: error,
>>> userCache: error,
>>> identity-providers: {
>>> google: ok,
>>> facebook: error
>>> },
>>> user-federation: {
>>> ldap: ok
>>> }
>>> }
>>>
>>> I wonder if that's to much details to expose without authentication.
>> This should work, but I miss info about DB connection here. It is
>> crucial from operational point of view as this allows sysadmins quickly
>> detect that connection to DB is broken.
>>
>> I believe this kind of info can be safely exposed without
>> authentication. If the endpoint is clearly documented then access can be
>> always restricted on some network infra (webserver/proxy) ahead of KC.
> DB is "realm" and "users".
>
>>>> Vl.
>>>>
>>>>> ----- Original Message -----
>>>>>> From: "Vlastimil Elias" <velias(a)redhat.com>
>>>>>> To: "Scott Rossillo" <srossillo(a)smartling.com>,
"Stian Thorgersen"
>>>>>> <stian(a)redhat.com>
>>>>>> Cc: keycloak-dev(a)lists.jboss.org
>>>>>> Sent: Tuesday, 14 July, 2015 8:34:29 AM
>>>>>> Subject: Re: [keycloak-dev] Operational monitoring of Keycloak
server
>>>>>>
>>>>>> Yap, beside server-stat endpoint Stian talked about we should
add some
>>>>>> server-health endpoint also to be used for remote health
monitoring by
>>>>>> load balancers or systems like Nagios.
>>>>>> Simple http endpoint without authentication (as it doesn't
leak any
>>>>>> sensitive information) is typically simplest way how to do this
>>>>>> monitoring.
>>>>>>
>>>>>> Vl.
>>>>>>
>>>>>> On 14.7.2015 05:13, Scott Rossillo wrote:
>>>>>>> Some type of health page would be great too for load
balancers to
>>>>>>> monitor. Something that doesn't leak internal
information but checks
>>>>>>> behind the scenes that:
>>>>>>> 1. Server can reach its databas(es)
>>>>>>> 2. Server cluster sync is working
>>>>>>> 3. Server can reach federation providers, etc.
>>>>>>> Endpoint should respond to get requests and return an http
status
>>>>>>> reflective of server state.
>>>>>>>
>>>>>>> On Mon, Jul 13, 2015 at 11:18 AM Stian Thorgersen
<stian(a)redhat.com
>>>>>>> <mailto:stian@redhat.com>> wrote:
>>>>>>>
>>>>>>> So looks like we're at agreement to add the
additional info
>>>>>>> you
>>>>>>> wanted to server info page.
>>>>>>>
>>>>>>> How about we add an additional endpoint server-stat
that can
>>>>>>> collect some stats about the server?
>>>>>>>
>>>>>>> ----- Original Message -----
>>>>>>> > From: "Vlastimil Elias"
<velias(a)redhat.com
>>>>>>> <mailto:velias@redhat.com>>
>>>>>>> > To: "Stian Thorgersen"
<stian(a)redhat.com
>>>>>>> > <mailto:stian@redhat.com>>
>>>>>>> > Cc: keycloak-dev(a)lists.jboss.org
>>>>>>> <mailto:keycloak-dev@lists.jboss.org>
>>>>>>> > Sent: Monday, 13 July, 2015 5:06:34 PM
>>>>>>> > Subject: Re: [keycloak-dev] Operational
monitoring of
>>>>>>> > Keycloak
>>>>>>> server
>>>>>>> >
>>>>>>> > Looks like I have to look at WildFly/EAP DMR to
see what is
>>>>>>> possible to
>>>>>>> > do with it, as I'm not sure if it is about
remote monitoring
>>>>>>> also and
>>>>>>> > if/how it can be use from monitoring systems
like Splunk.
>>>>>>> >
>>>>>>> > Vl.
>>>>>>> >
>>>>>>> > On 13.7.2015 15:26, Stian Thorgersen wrote:
>>>>>>> > > In WildFly/EAP that's DMR right?
We're planning to make
>>>>>>> Keycloak managable
>>>>>>> > > through that as well. For example
everything that goes
>>>>>>> > > into
>>>>>>> > > keycloak-server.json will eventually be
moved to
>>>>>>> standalone.xml. Same with
>>>>>>> > > admin endpoints, everything you can do
there you'll
>>>>>>> > > eventually
>>>>>>> be able to
>>>>>>> > > do through DMR and jboss-cli as well.
>>>>>>> > >
>>>>>>> > > However, IMO it would make sense to at
least expose
>>>>>>> > > Keycloak
>>>>>>> specific
>>>>>>> > > information through the admin endpoints and
console as
>>>>>>> > > well.
>>>>>>> Such number
>>>>>>> > > of sessions, etc..
>>>>>>> > >
>>>>>>> > > ----- Original Message -----
>>>>>>> > >> From: "Vlastimil Elias"
<velias(a)redhat.com
>>>>>>> <mailto:velias@redhat.com>>
>>>>>>> > >> To: keycloak-dev(a)lists.jboss.org
>>>>>>> <mailto:keycloak-dev@lists.jboss.org>
>>>>>>> > >> Sent: Monday, 13 July, 2015 3:17:16 PM
>>>>>>> > >> Subject: [keycloak-dev] Operational
monitoring of
>>>>>>> > >> Keycloak
>>>>>>> > >> server
>>>>>>> > >>
>>>>>>> > >> Hi,
>>>>>>> > >>
>>>>>>> > >> as we deployed KC to production mode
for
>>>>>>>
https://developers.redhat.com
>>>>>>> > >> we started to think about operational
monitoring, for
>>>>>>> > >> example
>>>>>>> from
>>>>>>> > >> Nagios or other systems of this type.
>>>>>>> > >>
>>>>>>> > >> KC user guide doesn't contain any
chapter covering this
>>>>>>> topic, also no
>>>>>>> > >> any success over google search, so
looks like KC doesn't
>>>>>>> > >> have
>>>>>>> > >> any
>>>>>>> > >> solution for this yet.
>>>>>>> > >> But I believe this is an important area
which must be
>>>>>>> > >> solved
>>>>>>> when KC is
>>>>>>> > >> used for production.
>>>>>>> > >>
>>>>>>> > >> I can imagine monitoring of JDBC
connection if JPA is
>>>>>>> > >> used,
>>>>>>> monitoring
>>>>>>> > >> of Mongo connection if used as store,
monitoring of LDAP
>>>>>>> connection if
>>>>>>> > >> LDAP federation is used etc.
>>>>>>> > >> Also some statistics like numbers of
active sso session,
>>>>>>> number of
>>>>>>> > >> logins per minute etc should be
provided there.
>>>>>>> > >>
>>>>>>> > >> Monitoring is not about Keycloak core
itself, it should
>>>>>>> > >> be
>>>>>>> available for
>>>>>>> > >> extension developers also. For example
we implemented own
>>>>>>> > >> UserFederationProvider which calls
backend REST services.
>>>>>>> > >> We should be able to add info about
this integration into
>>>>>>> monitoring
>>>>>>> > >> endpoint to be able to catch problems
with this REST API.
>>>>>>> > >>
>>>>>>> > >> It should be probably implemented same
way as used by
>>>>>>> > >> underlying
>>>>>>> > >> WildFly/EAP (JPA/JDBC is probably
available for
>>>>>>> > >> monitoring
>>>>>>> there). I'm
>>>>>>> > >> not sure if JMX is used there still or
if some new
>>>>>>> > >> framework
>>>>>>> > >> is
>>>>>>> > >> available for it.
>>>>>>> > >> Or KC should use some form of KC REST
API for this, which
>>>>>>> should be
>>>>>>> > >> extended by additional info from KC
extensions?
>>>>>>> > >>
>>>>>>> > >> What do you think?
>>>>>>> > >>
>>>>>>> > >> Vlastimil
>>>>>>> > >>
>>>>>>> > >> P.S we have
https://issues.jboss.org/browse/RHD-552 for
>>>>>>> > >> Red
>>>>>>> > >> Hat
>>>>>>> > >> Developer instance of KC
>>>>>>> > >>
>>>>>>> > >> --
>>>>>>> > >> Vlastimil Elias
>>>>>>> > >> Principal Software Engineer
>>>>>>> > >>
jboss.org <
http://jboss.org>
Development Team
>>>>>>> > >>
>>>>>>> > >>
_______________________________________________
>>>>>>> > >> keycloak-dev mailing list
>>>>>>> > >> keycloak-dev(a)lists.jboss.org
>>>>>>> <mailto:keycloak-dev@lists.jboss.org>
>>>>>>> > >>
https://lists.jboss.org/mailman/listinfo/keycloak-dev
>>>>>>> > >>
>>>>>>> >
>>>>>>> > --
>>>>>>> > Vlastimil Elias
>>>>>>> > Principal Software Engineer
>>>>>>> >
jboss.org <
http://jboss.org> Development
Team
>>>>>>> >
>>>>>>> >
>>>>>>> _______________________________________________
>>>>>>> keycloak-dev mailing list
>>>>>>> keycloak-dev(a)lists.jboss.org
>>>>>>> <mailto:keycloak-dev@lists.jboss.org>
>>>>>>>
https://lists.jboss.org/mailman/listinfo/keycloak-dev
>>>>>>>
>>>>>> --
>>>>>> Vlastimil Elias
>>>>>> Principal Software Engineer
>>>>>>
jboss.org Development Team
>>>>>>
>>>>>>
>>>> --
>>>> Vlastimil Elias
>>>> Principal Software Engineer
>>>>
jboss.org Development Team
>>>>
>>>>
>> --
>> Vlastimil Elias
>> Principal Software Engineer
>>
jboss.org Development Team
>>
>>
--
Vlastimil Elias
Principal Software Engineer
jboss.org Development Team