[keycloak-user] Architectural Blueprint/Recommendations

[Dmitry Telegin]

Hi David,

Please see the answers and remarks inline.

On Mon, 2018-06-18 at 14:40 +0000, Everson, David (MNIT) wrote:
(skipped)
> 15. Keycloak should be clustered for high availability.
> 16. Keycloak environment would be hosted on AWS, more than likely EC2
> instances.
> 17. Client applications also hosted in AWS.
> 18. Keycloak's database would be PostgreSQL hosted in AWS RDS.

Speaking of Keycloak on AWS, this is absolutely doable, but not that
trivial. Please be sure to have read the document [1], especially the
"Troubleshooting AWS specifics" part, and relevant ML postings [2].
Long story short, AWS doesn't allow for IP multicast between the nodes,
which is the default node discovery mode in JGroups (the backbone of
Keycloak clustering). You should use S3_PING or JDBC_PING instead.

> 
> A few questions/concerns of the working group:
> 
> A. Is there any information available on the maximum size of an
> Keycloak installation? Will Keycloak be scalable and performant given
> the above assumptions and constraints.

AFAIK, nobody has performed actual Keycloak benchmarking yet
(publicly). There's however a Keycloak benchmarking suite based on
Gatling [3]. It hasn't been updated for about two years, so first we'll
need to make sure it works with recent Keycloak versions.

> 
> B. What's the best recommendation for distributing the Keycloak
> instances and realms.  Right now the group has three options on the
> table:  1) A single Keycloak install per application (i.e.
> client);  2) A single Keycloak install per organizational unit (i.e.
> realm); or 3) A single Keycloak install per organization (i.e.
> serving all realms and clients).

The pros for A and B is obviously that you get some degree of
separation/isolation, which might be good from the security and
availability POV. However, this comes at a price of complexity; you'll
have to deploy, monitor & maintain each separate instance / group of
instances, each having different configs and dedicated database.

Another big issue is load distribution. I doubt that your
clients/realms all have equal, uniform load patterns. Given that each
Keycloak instance will have its hardware limitations (CPU, RAM), you
potentially end up with some nodes overloaded and others idle. The C
scenario is obviously free from this issue.

> 
> C. A major concern the group has with a single Keycloak install (#3
> in previous bullet) is the high-availability in terms of performance
> and concerns of a rouge client affecting other applications
> negatively.  What is the community's recommendation for addressing
> this concern?

As you will necessarily have a load balancer / reverse proxy in front
of your Keycloak cluster, you can enforce rate limiting / throttling on
your load balancer. For example, haproxy implements rate limiting based
on IP addresses, URLs and HTTP headers [4].

> 
> D. Another major concern the group has with a single Keycloak install
> is the restarts that are necessary when an organization unit deploys
> a new or updated template.  The concern is that all applications
> would be unavailable during the restart.   We would be operating in a
> clustered environment, is the best solution to this concern
> restarting individual members of the cluster rather than the entire
> cluster?

Could you please elaborate on template deployment? In Keycloak
parlance, "templates" can be understood either as "client templates" or
"HTML templates" (within custom GUI themes).

Client templates surely can be created/updated via GUI or REST API,
without the need for restart. For GUI themes, they can be deployed a)
as Wildfly modules, b) via "themes" directory. While the former option
indeed requires restart, the latter does not. Keycloak 4.x also adds c)
hot deployment of themes by dropping theme JARs into the "deployments"
directory.

> 
> E. For reporting and governance processes, the Keycloak API performs
> quite poorly when we execute use cases such as "Report all Users of
> an Application".  Given the version we are currently on, to
> accomplish this we need to query all users in the realm and then
> filter the users if they have the client/role combination.  We
> understand that a future release addresses this use case, but in the
> meantime the concern is such a query will negatively affect all other
> clients using Keycloak.  Any recommendations on handling this use
> case prior to Keycloak 4.x?

Is this indeed addressed by Keycloak 4.x? (just wondering, couldn't
find any info)

Keycloak admin REST API has an endpoint called "Return List of Users
that have the specified role name", see [5] (identical for KC 3.x and
4.x). You could use this endpoint, however you will have to iterate
over client roles and then merge and de-duplicate the results. Anyway,
this should be much more efficient than your current approach.

In general, this looks like a classical use case for Realm Resource
Provider [6]. The query you described easily maps to a single SQL/JPQL
statement, so you could implement a custom REST resource that would
execute exactly that query and return results.

Unfortunately, custom REST resources in Keycloak are public by default
(protected resources should become a part of the hypothetical Admin
Resource SPI somewhere in the future). However, you can implement that
(relatively) easily with the techniques demonstrated in Beercloak [7].

> 
> F. Upgrading Versions of Keycloak.  We have experienced some
> difficulty of upgrading versions on server-side (we need to export,
> import vs a simple DB backup and deployment).  What is the
> recommendations for handling the upgrade of Keycloak from one version
> to the next given the size of our user base?

Could you please elaborate a bit on the problems that you're facing?
The export/import scenario is relevant for database upgrades (e.g.
PostgreSQL 9 -> 10), but Keycloak does ship migration scripts that
should upgrade the data+metadata automatically. Why doesn't that work
in your case? Let us know, probably this could be fixed.

> 
> I'm sorry for the long post, hopefully folks get to this point.  Any
> insight that we could receive would be greatly appreciated. We are at
> a critical cross-roads in our Keycloak adoption and want to ensure we
> do this correctly.

Sorry it took so long to reply. Keycloak is a great product, I hope it
fulfills your needs. Good luck!

Dmitry Telegin
CTO, Acutus s.r.o.
Keycloak Consulting and Training

[1] https://blog.keycloak.org/2018/01/keycloak-cross-data-center-setup-
in-aws.html
[2] https://www.keycloak.org/search.html?q=aws
[3] https://github.com/rvansa/keycloak-benchmark
[4] https://blog.codecentric.de/en/2014/12/haproxy-http-header-rate-lim
iting/
[5] https://www.keycloak.org/docs-api/3.4/rest-api/index.html
[6] https://www.keycloak.org/docs/latest/server_development/index.html#
_extensions_rest
[7] https://github.com/dteleguin/beercloak

> 
> Thanks!
> Dave
> 
> 
> Dave Everson
> Application Development Team Lead | Environmental Health
> Minnesota IT Services | Partners in Minnesota Department of Health
> 625 Robert Street North
> St. Paul, MN 55155
> O: 651-201-5146
> Information Technology for Minnesota Government |
>  mn.gov/mnit<http://mn.gov/mnit>
> [Minnesota IT Services Logo]
> [Facebook logo]<https://www.facebook.com/MN.ITServices>[LinkedIn
> logo]<https://www.linkedin.com/company/mn-it-services>[Twitter
> logo]<https://twitter.com/mnit_services>
> 
> _______________________________________________
> keycloak-user mailing list
> keycloak-user at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/keycloak-user