Hi Marek I'm working with Fabricio on the federation performance issues with Keycloak. In answer to your question we are using the latest KC 1.9.7 version (we upgraded this week from 1.9.2). To give you some indication of the running a gatling direct access login test (results below). As you can see below in (1) using KC out of the box. Great performance - we saw 110 tx per sec on a 4 core system. In scenario (2) using a stubbed federator (simply an echo plugin not connecting to any back end services), performance is unacceptable. 1) Not using the federator - Stub federator (disabled) - while 29 tx per second we could easily get to a stable 110 tx per second. 300 Users (hitting single server) ---- Global Information -------------------------------------------------------- > request count 9185 (OK=9185 KO=0 ) > min response time 18 (OK=18 KO=- ) > max response time 723 (OK=723 KO=- ) > mean response time 27 (OK=27 KO=- ) > std deviation 44 (OK=44 KO=- ) > response time 50th percentile 20 (OK=20 KO=- ) > response time 75th percentile 21 (OK=21 KO=- ) > mean requests/sec 29.626 (OK=29.626 KO=- ) ---- Response Time Distribution ------------------------------------------------ > t < 800 ms 9185 (100%) > 800 ms < t < 1200 ms 0 ( 0%) > t > 1200 ms 0 ( 0%) > failed 0 ( 0%) 2) Stub federator (enabled)- if we brought test down to 12 tx per second (about 90 users) the response times dropped to < 1200 ms response times, however not even close to meeting out acceptance creteria. 300 Users (hitting single server) ---- Global Information -------------------------------------------------------- > request count 8496 (OK=8496 KO=0 ) > min response time 511 (OK=511 KO=- ) > max response time 11191 (OK=11191 KO=- ) > mean response time 6832 (OK=6832 KO=- ) > std deviation 2329 (OK=2329 KO=- ) > response time 50th percentile 7194 (OK=7194 KO=- ) > response time 75th percentile 8690 (OK=8690 KO=- ) > mean requests/sec 27.404 (OK=27.404 KO=- ) ---- Response Time Distribution ------------------------------------------------ > t < 800 ms 154 ( 2%) > 800 ms < t < 1200 ms 85 ( 1%) > t > 1200 ms 8257 ( 97%) > failed 0 ( 0%) This is currently a show stopper for us and is blocking our path to production. Do you run similar tests and how can we help you optimise the performance? Regards Tom. Date: Wed, 8 Jun 2016 12:28:19 +0200 From: Marek Posolda <mposolda(a)redhat.com&gt; Subject: Re: [keycloak-user] Performance issues with Federation provider enabled To: Fabricio Milone <fabricio.milone(a)shinetech.com&gt;, keycloak-user <keycloak-user(a)lists.jboss.org&gt; Message-ID: <5757F343.1040803(a)redhat.com&gt; Content-Type: text/plain; charset="windows-1252" Hi, what's the keycloak version used? Could you try latest keycloak and check if performance is still the issue? Marek On 08/06/16 01:30, Fabricio Milone wrote: > Hi all, > > I sent this email yesterday with 5 or more attachments, so I think it > was blocked or something... here I go again :) > > I've been running load tests on our application during the last few > weeks, and having some performance issues when my custom federator is > enabled. > > The performance issue does not exist when the federator is disabled. > *Configuration*: > > I have a cluster of 2 instances of Keycloak, with a standalone DB, > we've verified the DB isn't an issue when the federator is disabled. > Both instances have a quad core CPU and they are in the same network. > We?ve left the memory at 512MB. The test script, database and API that > connects to the federator are in separate machines. > *Federator*: > > We have a simple custom federator that makes calls to a very > performant api, which has been tested and is ok. Additionally, we've > tested stubbing the API so the performance is not a problem there. > This federator is using a jaxb marshaller to create a request, again > tested in isolation and is performing well. > > As the federator is doing a lot of calls to the API (3 per login > request), I've implemented a httpclient that uses a > PoolingHttpClientConnectionManager with 1000 connections available to > use, instead of using the standard apache httpclient from http > components. That hasn't improved a bit the performance of the system. > *Tests*: > It is a gatling scala script that could generate around ~300 (or more) > requests/second to the direct grants login endpoint using random > usernames from a list (all of them already registered using KC). The > script is doing a round robin across both instances of Keycloak with > an even distribution to each KC instance. > The idea is simulate a load of 300 to 1500 concurrent users trying to > login into our systems. > *Problem*: > > If I run the tests without using a federation I can see a very good > performance, but when I try to run the tests with the custom > federation code, the performance drops from ~150 requests/second to 22 > req/sec using both instances. > Memory wise, it seems to be ok. I've never seen an error related to > memory with this configuration, also if you take a look at the > attached visualVM screenshot you'll see that memory is not a problem > or it seems not to be. > CPU utilisation is very low to my mind, I'd expect more than 80% of > usage or something like that. > There is a method that is leading the CPU samples on VisualVM called > Semaphore.tryAcquire(). Not quite sure what's that for, still > investigating. > > I can see that a lot of new threads are being created when the test > starts, as it creates around 60requests/second to the direct grants > login call, but it seems to be a bottleneck at some point. > > So I'm wondering if there is some configuration I'm missing on > Keycloak side that could be affecting the cluster performance when a > federator is enabled. Maybe something related to jpa connections, > infinispan configuration or even wildfly. > > I'd really appreciate your help on this one as I'm out of ideas. > > I've attached some screenshots of visualVM and tests results from my > last run today. > > > Sorry for the long email and please let me know if you need further > information. > > Thank you in advance, > > Regards, > Fab > > -- > *Fabricio Milone* > Developer _______________________________________________ keycloak-user mailing list keycloak-user(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/keycloak-user

Hi Marek > Thanks, AFAIK we didn't tried much performance testing with federationProviders enabled. It's on todo list though. Also we plan some refactoring of userStorage + userFederation, so we will likely go into it later. Yes and we've found it is a major bottleneck in our system testing (using stub to remove internal back end dependencies). Can you suggest any short term measures to improve performance, we're blocked from pushing this to production at the moment? This is a major feature of the system I'm guessing this affects the LDAP / AD integration / federator performance too. Do you have any timeframe around the priority to address this? > For your case, the performance bottleneck can be in your federationProvider implementation, so I am not sure if it's the issue in Keycloak or rather issue in your implementation. As indicated we've created a stub implementation, code included below, to demonstrate there is an issue calling a federator in KC. /** Code snippet **/ public class StubFederationProvider implements UserFederationProvider { private static final Logger logger = Logger.getLogger(StubFederationProvider.class); protected KeycloakSession session; protected UserFederationProviderModel model; public StubFederationProvider(KeycloakSession session, UserFederationProviderModel model){ this.session = session; this.model = model; } public UserFederationProviderModel getModel() { return model; } @Override public UserModel getUserByUsername(RealmModel realm, String username) { UserModel userModel = addUserModelToUserStorage(realm, username); userModel.setEnabled(true); userModel.setFederationLink(model.getId()); return userModel; } protected UserModel addUserModelToUserStorage(RealmModel realm, String username) { return session.userStorage().addUser(realm, username); } @Override public UserModel getUserByEmail(RealmModel realm, String email) { return null; } @Override public List<UserModel> searchByAttributes(Map<String, String> attributes, RealmModel realm, int maxResults) { return Collections.emptyList(); } @Override public List<UserModel> getGroupMembers(RealmModel realm, GroupModel group, int firstResult, int maxResults) { return null; } @Override public void preRemove(RealmModel realm) { // complete We don't care about the realm being removed } @Override public void preRemove(RealmModel realm, RoleModel role) { // complete we dont'care if a role is removed } @Override public void preRemove(RealmModel realmModel, GroupModel groupModel) { // complete we dont'care if a role is removed } @Override public boolean isValid(RealmModel realm, UserModel local) { return userExists(local.getUsername()); } /** * Returns supported credentials by this federator. PASSWORD is always supported but TOTP is optional for each user. * * @param user * @return */ @Override public Set<String> getSupportedCredentialTypes(UserModel user) { Set<String> supportedCredentialTypes = new HashSet<>(); supportedCredentialTypes.add(UserCredentialModel.PASSWORD); // check for any otp configured on this user if (user.isOtpEnabled()) { supportedCredentialTypes.add(UserCredentialModel.TOTP); supportedCredentialTypes.add(UserCredentialModel.HOTP); } return supportedCredentialTypes; } @Override public boolean validCredentials(RealmModel realm, UserModel user, List<UserCredentialModel> input) { for (UserCredentialModel cred : input) { if (cred.getType().equals(UserCredentialModel.PASSWORD)) { return validate(user, cred.getValue()); } else if (cred.getType().equals(UserCredentialModel.TOTP)) { return CredentialValidation.validTOTP(realm, user, cred.getValue()); } else if (cred.getType().equals(UserCredentialModel.HOTP)) { return CredentialValidation.validHOTP(realm, user, cred.getValue()); } } return false; } @Override public boolean validCredentials(RealmModel realm, UserModel user, UserCredentialModel... input) { for (UserCredentialModel cred : input) { if (cred.getType().equals(UserCredentialModel.PASSWORD)) { return validate(user, cred.getValue()); } else if (cred.getType().equals(UserCredentialModel.TOTP)) { return CredentialValidation.validTOTP(realm, user, cred.getValue()); } else if (cred.getType().equals(UserCredentialModel.HOTP)) { return CredentialValidation.validHOTP(realm, user, cred.getValue()); } } return false; } @Override public void close() { } /** * Keycloak will call this method if it finds an imported UserModel. Here we proxy the UserModel with * a Readonly proxy which will barf if password is updated. * * @param local * @return */ @Override public UserModel validateAndProxy(RealmModel realm, UserModel local) { if (isValid(realm, local)) { getUserDetails(local); return new StubUserModelProxy(local, this); } else { return null; } } @Override public boolean synchronizeRegistrations() { return true; } /** * Called if this federation provider has priority and supports synchronized registrations. * * @param realm * @param user * @return */ @Override public UserModel register(RealmModel realm, UserModel user) { user.setSingleAttribute("status", "OK"); return user; } @Override public boolean removeUser(RealmModel realm, UserModel user) { // Not supported. Used as a part of the Workaround to https://issues.jboss.org/browse/KEYCLOAK-1075 return true; } /** * Supported credentials by this federator. PASSWORD is a supported type. TOTP depends on the user. * * @return supportedCredentialTypes */ @Override public Set<String> getSupportedCredentialTypes() { Set<String> supportedCredentialTypes = new HashSet<>(); supportedCredentialTypes.add(UserCredentialModel.PASSWORD); supportedCredentialTypes.add(UserCredentialModel.TOTP); supportedCredentialTypes.add(UserCredentialModel.HOTP); return supportedCredentialTypes; } @Override public CredentialValidationOutput validCredentials(RealmModel realm, UserCredentialModel credential) { throw new IllegalStateException("validCredentials not supported"); } private boolean userExists(String username) { return true; } private void getUserDetails(UserModel user) { user.setFirstName("first name"); user.setLastName("last name"); } public boolean validate(UserModel user, String password) { return true; } } /** End Snippet **/ Regards Tom Connolly. ------------------------------ *From:* Marek Posolda <mposolda(a)redhat.com&gt; *To:* Thomas Connolly <thomas_connolly(a)yahoo.com>; " keycloak-user(a)lists.jboss.org&quot; <keycloak-user(a)lists.jboss.org&gt; *Sent:* Monday, June 13, 2016 5:47 PM *Subject:* Re: [keycloak-user] Performance issues with Federation provider enabled Thanks, AFAIK we didn't tried much performance testing with federationProviders enabled. It's on todo list though. Also we plan some refactoring of userStorage + userFederation, so we will likely go into it later. For your case, the performance bottleneck can be in your federationProvider implementation, so I am not sure if it's the issue in Keycloak or rather issue in your implementation. One thing to note (maybe it's not an issue in your case, but just adding it to be sure you're aware): UserFederationProvider.close is currently not called. So if you are rely on this method to free any important resources related to your implementation, you shouldn't as it doesn't work right now. We are working on improving this for next version. Marek On 13/06/16 07:57, Thomas Connolly wrote: Hi Marek I'm working with Fabricio on the federation performance issues with Keycloak. In answer to your question we are using the latest KC 1.9.7 version (we upgraded this week from 1.9.2). To give you some indication of the running a gatling direct access login test (results below). As you can see below in (1) using KC out of the box. Great performance - we saw 110 tx per sec on a 4 core system. In scenario (2) using a stubbed federator (simply an echo plugin not connecting to any back end services), performance is unacceptable. 1) Not using the federator - Stub federator (disabled) - while 29 tx per second we could easily get to a stable 110 tx per second. 300 Users (hitting single server) ---- Global Information -------------------------------------------------------- > request count 9185 (OK=9185 KO=0 ) > min response time 18 (OK=18 KO=- ) > max response time 723 (OK=723 KO=- ) > mean response time 27 (OK=27 KO=- ) > std deviation 44 (OK=44 KO=- ) > response time 50th percentile 20 (OK=20 KO=- ) > response time 75th percentile 21 (OK=21 KO=- ) > mean requests/sec 29.626 (OK=29.626 KO=- ) ---- Response Time Distribution ------------------------------------------------ > t < 800 ms 9185 (100%) > 800 ms < t < 1200 ms 0 ( 0%) > t > 1200 ms 0 ( 0%) > failed 0 ( 0%) 2) Stub federator (enabled)- if we brought test down to 12 tx per second (about 90 users) the response times dropped to < 1200 ms response times, however not even close to meeting out acceptance creteria. 300 Users (hitting single server) ---- Global Information -------------------------------------------------------- > request count 8496 (OK=8496 KO=0 ) > min response time 511 (OK=511 KO=- ) > max response time 11191 (OK=11191 KO=- ) > mean response time 6832 (OK=6832 KO=- ) > std deviation 2329 (OK=2329 KO=- ) > response time 50th percentile 7194 (OK=7194 KO=- ) > response time 75th percentile 8690 (OK=8690 KO=- ) > mean requests/sec 27.404 (OK=27.404 KO=- ) ---- Response Time Distribution ------------------------------------------------ > t < 800 ms 154 ( 2%) > 800 ms < t < 1200 ms 85 ( 1%) > t > 1200 ms 8257 ( 97%) > failed 0 ( 0%) This is currently a show stopper for us and is blocking our path to production. Do you run similar tests and how can we help you optimise the performance? Regards Tom. Date: Wed, 8 Jun 2016 12:28:19 +0200 From: Marek Posolda <mposolda(a)redhat.com&gt; <mposolda(a)redhat.com&gt; Subject: Re: [keycloak-user] Performance issues with Federation provider enabled To: Fabricio Milone <fabricio.milone(a)shinetech.com&gt; <fabricio.milone(a)shinetech.com&gt;, keycloak-user <keycloak-user(a)lists.jboss.org&gt; <keycloak-user(a)lists.jboss.org&gt; Message-ID: <5757F343.1040803(a)redhat.com&gt; <5757F343.1040803(a)redhat.com&gt; Content-Type: text/plain; charset="windows-1252" Hi, what's the keycloak version used? Could you try latest keycloak and check if performance is still the issue? Marek On 08/06/16 01:30, Fabricio Milone wrote: > Hi all, > > I sent this email yesterday with 5 or more attachments, so I think it > was blocked or something... here I go again :) > > I've been running load tests on our application during the last few > weeks, and having some performance issues when my custom federator is > enabled. > > The performance issue does not exist when the federator is disabled. > *Configuration*: > > I have a cluster of 2 instances of Keycloak, with a standalone DB, > we've verified the DB isn't an issue when the federator is disabled. > Both instances have a quad core CPU and they are in the same network. > We?ve left the memory at 512MB. The test script, database and API that > connects to the federator are in separate machines. > *Federator*: > > We have a simple custom federator that makes calls to a very > performant api, which has been tested and is ok. Additionally, we've > tested stubbing the API so the performance is not a problem there. > This federator is using a jaxb marshaller to create a request, again > tested in isolation and is performing well. > > As the federator is doing a lot of calls to the API (3 per login > request), I've implemented a httpclient that uses a > PoolingHttpClientConnectionManager with 1000 connections available to > use, instead of using the standard apache httpclient from http > components. That hasn't improved a bit the performance of the system. > *Tests*: > It is a gatling scala script that could generate around ~300 (or more) > requests/second to the direct grants login endpoint using random > usernames from a list (all of them already registered using KC). The > script is doing a round robin across both instances of Keycloak with > an even distribution to each KC instance. > The idea is simulate a load of 300 to 1500 concurrent users trying to > login into our systems. > *Problem*: > > If I run the tests without using a federation I can see a very good > performance, but when I try to run the tests with the custom > federation code, the performance drops from ~150 requests/second to 22 > req/sec using both instances. > Memory wise, it seems to be ok. I've never seen an error related to > memory with this configuration, also if you take a look at the > attached visualVM screenshot you'll see that memory is not a problem > or it seems not to be. > CPU utilisation is very low to my mind, I'd expect more than 80% of > usage or something like that. > There is a method that is leading the CPU samples on VisualVM called > Semaphore.tryAcquire(). Not quite sure what's that for, still > investigating. > > I can see that a lot of new threads are being created when the test > starts, as it creates around 60requests/second to the direct grants > login call, but it seems to be a bottleneck at some point. > > So I'm wondering if there is some configuration I'm missing on > Keycloak side that could be affecting the cluster performance when a > federator is enabled. Maybe something related to jpa connections, > infinispan configuration or even wildfly. > > I'd really appreciate your help on this one as I'm out of ideas. > > I've attached some screenshots of visualVM and tests results from my > last run today. > > > Sorry for the long email and please let me know if you need further > information. > > Thank you in advance, > > Regards, > Fab > > -- > *Fabricio Milone* > Developer _______________________________________________ keycloak-user mailing listkeycloak-user@lists.jboss.orghttps://lists.jboss.org/mailman/listinfo/keycloak-user _______________________________________________ keycloak-user mailing list keycloak-user(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/keycloak-user