[keycloak-user] How to gracefully delete /clean up key-cloak with large number of realms

[Madhu]

 Will be awsome if some one can throw light on this, especially from cache rebalancing point of view, i tried using the infispan cli , jmx , widlfy management console, but not able to look into the content of cache.. 
Will one of us have already tried something like this/similar and can share their knowledge it will be super great!
RegardsMadhu
    On Tuesday, 19 March, 2019, 3:50:13 pm IST, Madhu <kkcmadhu at yahoo.com> wrote:  
 
 Hi,
I am using keycloak 4.5.0.Final and 4.7.0.Final.I have about 600+ realms, and i am looking for a graceful way to delete realms from al ive system (without bringing down keycloak nodes)
I have a cluster set up(standalone-ha.xml), with 3 or 4 nodes and i use jdbc ping for cluster discovery.
I need to know, whats the safest way to delete/clean up realms in such a setup.
I tried deleting the tenants using a shell script, which  invokes  /opt/softwareag/keycloak-4.7.0.Final/bin/kcadm.sh delete realms/$realm_name in a loop.
The realm deletion is slow ( which is ok),,but mostly i see that the cluster node becomes un responsive after running this command, i see large  number of 
" Uncaught server error: javax.persistence.OptimisticLockException: org.hibernate.exception.LockAcquisitionException: could not execute statement"
exceptions.

The worst part of the problem is that the node does not go down completely and  is still part of the cluster but un-responsive. So any info cached in the node becomes inaccessible (user, realm token info mostly) and impacts the logon/login to a set of realms owned by this node  :(  
If i gracefully shutdown the node (manually) using jboss-cli , the node goes down and allows the other cluster nodes to rebalance.
But until i take manual action, this sick node remains part of the cluster and makes a part of realm/users totally un usable..
I tried doing the same with REST APIs instead of kcadm  and the effect is same (node becomes unresponsive and but does not leave cluster)
Any idea, how can i gracefully delete realms from a live system ,without bring down keycloak??

I am thinking of :a) bringing up a temporary node to cluster , run the delete command from there, and shutdown this node, but what i am not sure is , when i add another node, will rebalance cause a part of data which is already stored in existing cluster nodes to be transferred to this node, if yes, then clearly this solution will not work..
b) is there a way to bring another node in standalone mode and delete but that may cause dead lock, as the cluster is unware of this new node and does not coordinate (compete with it).
c) can i delete the unwanted realms directly from database and clean up the cache in all cluster nodes? will that  impact live traffic, if so how??

Regards,Madhu