[keycloak-user] start up of keycloak nodes roughly increases two folds for every 100 tenants.

[Madhu]

Hi I am using keycloak 4.5. i created about 600+ tenants with 50 users each for a performance testing.

Upon creating tenants the start up time of keycloak increases drastically. This seems to be due to pretty much all entities at start up.. 
I tried disabling realm cache, user cache and did not help.. can you suggest how to bring down the start up time?

Is it absolutely necessary for keycloak to load every thing at start up??

This is an extract from hibernate stat i got on a c4 xlarge ec2 instance ( 4 core 8 gig), keycloak configured with xms=xmx=5g.

018-11-24 10:33:19,998 INFO [org.hibernate.envers.boot.internal.EnversServiceImpl] (ServerService Thread Pool – 61) Envers integration enabled? : true
2018-11-24 10:33:20,499 INFO [org.hibernate.validator.internal.util.Version] (ServerService Thread Pool – 61) HV000001: Hibernate Validator 5.3.6.Final
2018-11-24 10:33:21,296 INFO [org.hibernate.hql.internal.QueryTranslatorFactoryInitiator] (ServerService Thread Pool – 61) HHH000397: Using ASTQueryTranslatorFactory
^C
[centos at ip-172-31-45-199 log]$ 11:10:45,750 INFO [org.hibernate.engi ne.internal.StatisticalLoggingSessionEventListener] (ServerService Th read Pool – 61) Session Metrics {
 669457663 nanoseconds spent acquiring 92974 JDBC connections;
 148185664 nanoseconds spent releasing 92974 JDBC connections;
 1852958902 nanoseconds spent preparing 92974 JDBC statements;
 35866600579 nanoseconds spent executing 92974 JDBC statements;
 0 nanoseconds spent executing 0 JDBC batches;
 0 nanoseconds spent performing 0 L2C puts;
 0 nanoseconds spent performing 0 L2C hits;
 0 nanoseconds spent performing 0 L2C misses;
 543461113 nanoseconds spent executing 2 flushes (flushing a total of 227216 entities and 158902 collections);
 2197548626817 nanoseconds spent executing 14139 partial-flushes ( flushing a total of* 1042012050 entities and 1042012050 collections*)
}
11:10:45,780 INFO [org.hibernate.engine.internal.StatisticalLoggingS essionEventListener] (ServerService Thread Pool – 61) Session Metric s 
{ 7689387 nanoseconds spent acquiring 1 JDBC connections; 34263 nanoseconds spent releasing 1 JDBC connections; 8025969 nanoseconds spent preparing 1 JDBC statements; 909784 nanoseconds spent executing 1 JDBC statements; 0 nanoseconds spent executing 0 JDBC batches; 0 nanoseconds spent performing 0 L2C puts; 0 nanoseconds spent performing 0 L2C hits; 0 nanoseconds spent performing 0 L2C misses; 3525215 nanoseconds spent executing 3 flushes (flushing a total o f 3 entities and 0 collections); 0 nanoseconds spent executing 0 partial-flushes (flushing a total  of 0 entities and 0 collections)}
11:10:45,795 INFO [org.hibernate.engine.internal.StatisticalLoggingS essionEventListener] (ServerService Thread Pool – 61) Session Metric s {
 437680 nanoseconds spent acquiring 1 JDBC connections;
 10539 nanoseconds spent releasing 1 JDBC connections;
 465001 nanoseconds spent preparing 1 JDBC statements;
 719260 nanoseconds spent executing 1 JDBC statements;
 0 nanoseconds spent executing 0 JDBC batches;
 0 nanoseconds spent performing 0 L2C puts;
 0 nanoseconds spent performing 0 L2C hits;
 0 nanoseconds spent performing 0 L2C misses;
 0 nanoseconds spent executing 0 flushes (flushing a total of 0 en tities and 0 collections);
 17455 nanoseconds spent executing 1 partial-flushes (flushing a total of 0 entities and 0 collections)

All My 600 +realms are pretty much same i.e. each realm has a client scope, a java script mapper (to get all the realm roles into resouce role),couple of attribute mappers, 2 users groups ( 1 for admins) and 1 for other users. i have about 50 users in each realm and all the user belongs to one of the 2 user groups ( no custom roles though).. 

Also, I bench marked the start up time after creating 50 or 100 realms and the start up time increases as the number of realms increases .

I am able to manage as i have disabled the admin console and use rest endpoints.. but still the start up time and loading pretty much every thing seems little wiered.

Please correct my understanding if i am wrong here..

| No of Realms | Start up time in mins |
| 0 realms | 0.22 mins |
| 100 realms | 2.34 mins |
| 200 realms | 2.53 mins |
| 300 realms | 5.34 mins |
| 400 realms | 9.42 mins |
| 500 realms | 14.6 mins |
| 650 realms | 37 mins |


Like wise the time taken to create tenants too gradually increases ( i use import to create realms)

from about 3 seconds for first few realms to about 30 sec for 600th realm..

Any advise /help will be appreciated.