[keycloak-user] How to gracefully delete /clean up key-cloak with large number of realms
Hi,
I am using keycloak 4.5.0.Final and 4.7.0.Final.I have about 600+ realms, and i am looking
for a graceful way to delete realms from al ive system (without bringing down keycloak
nodes)
I have a cluster set up(standalone-ha.xml), with 3 or 4 nodes and i use jdbc ping for
cluster discovery.
I need to know, whats the safest way to delete/clean up realms in such a setup.
I tried deleting the tenants using a shell script, which invokes
/opt/softwareag/keycloak-4.7.0.Final/bin/kcadm.sh delete realms/$realm_name in a loop.
The realm deletion is slow ( which is ok),,but mostly i see that the cluster node becomes
un responsive after running this command, i see large number of
" Uncaught server error: javax.persistence.OptimisticLockException:
org.hibernate.exception.LockAcquisitionException: could not execute statement"
exceptions.
The worst part of the problem is that the node does not go down completely and is still
part of the cluster but un-responsive. So any info cached in the node becomes inaccessible
(user, realm token info mostly) and impacts the logon/login to a set of realms owned by
this node :(
If i gracefully shutdown the node (manually) using jboss-cli , the node goes down and
allows the other cluster nodes to rebalance.
But until i take manual action, this sick node remains part of the cluster and makes a
part of realm/users totally un usable..
I tried doing the same with REST APIs instead of kcadm and the effect is same (node
becomes unresponsive and but does not leave cluster)
Any idea, how can i gracefully delete realms from a live system ,without bring down
keycloak??
I am thinking of :a) bringing up a temporary node to cluster , run the delete command from
there, and shutdown this node, but what i am not sure is , when i add another node, will
rebalance cause a part of data which is already stored in existing cluster nodes to be
transferred to this node, if yes, then clearly this solution will not work..
b) is there a way to bring another node in standalone mode and delete but that may cause
dead lock, as the cluster is unware of this new node and does not coordinate (compete with
it).
c) can i delete the unwanted realms directly from database and clean up the cache in all
cluster nodes? will that impact live traffic, if so how??
Regards,Madhu