[keycloak-user] Performance issues with Federation provider enabled
Bill Burke
bburke at redhat.com
Mon Jun 13 15:47:53 EDT 2016
In your validateAndProxy() method you call getUserDetails() which is
calling user.setFirstName, setLastName. So, basically every time the
user is queried via getUserByUsername(), you are doing
1) A database update
2) Invalidating the user cache across the cluster
On 6/13/16 6:35 AM, Thomas Connolly wrote:
> Hi Marek
>
> > Thanks, AFAIK we didn't tried much performance testing with
> federationProviders enabled. It's on todo list though. Also we plan
> some refactoring of userStorage + userFederation, so we will likely go
> into it later.
>
> Yes and we've found it is a major bottleneck in our system testing
> (using stub to remove internal back end dependencies).
> Can you suggest any short term measures to improve performance, we're
> blocked from pushing this to production at the moment?
> This is a major feature of the system I'm guessing this affects the
> LDAP / AD integration / federator performance too.
> Do you have any timeframe around the priority to address this?
>
> > For your case, the performance bottleneck can be in your federationProvider
> implementation, so I am not sure if it's the issue in Keycloak or
> rather issue in your implementation.
>
> As indicated we've created a stub implementation, code included below,
> to demonstrate there is an issue calling a federator in KC.
>
> /** Code snippet **/
> public class StubFederationProvider implements UserFederationProvider {
> private static final Logger logger =
> Logger.getLogger(StubFederationProvider.class);
> protected KeycloakSession session;
> protected UserFederationProviderModel model;
>
> public StubFederationProvider(KeycloakSession session,
> UserFederationProviderModel model){
> this.session = session;
> this.model = model;
> }
>
> public UserFederationProviderModel getModel() {
> return model;
> }
>
> @Override
> public UserModel getUserByUsername(RealmModel realm, String username) {
>
> UserModel userModel = addUserModelToUserStorage(realm, username);
> userModel.setEnabled(true);
> userModel.setFederationLink(model.getId());
>
> returnuserModel;
> }
>
> protected UserModel addUserModelToUserStorage(RealmModel realm, String
> username) {
> return session.userStorage().addUser(realm, username);
> }
>
> @Override
> public UserModel getUserByEmail(RealmModel realm, String email) {
> return null;
> }
>
> @Override
> public List<UserModel> searchByAttributes(Map<String, String>
> attributes, RealmModel realm, int maxResults) {
> return Collections.emptyList();
> }
>
> @Override
> public List<UserModel> getGroupMembers(RealmModel realm, GroupModel
> group, int firstResult, int maxResults) {
> returnnull;
> }
>
> @Override
> public void preRemove(RealmModel realm) {
> // complete We don't care about the realm being removed
> }
>
> @Override
> public void preRemove(RealmModel realm, RoleModel role) {
> // complete we dont'care if a role is removed
>
> }
>
> @Override
> public void preRemove(RealmModel realmModel, GroupModel groupModel) {
> // complete we dont'care if a role is removed
> }
>
> @Override
> public boolean isValid(RealmModel realm, UserModel local) {
>
> return userExists(local.getUsername());
> }
> /**
> * Returns supported credentials by this federator. PASSWORD is
> always supported but TOTP is optional for each user.
> *
> * @param user
> * @return
> */
> @Override
> public Set<String> getSupportedCredentialTypes(UserModel user) {
> Set<String> supportedCredentialTypes = new HashSet<>();
>
> supportedCredentialTypes.add(UserCredentialModel.PASSWORD);
>
> // check for any otp configured on this user
> if (user.isOtpEnabled()) {
> supportedCredentialTypes.add(UserCredentialModel.TOTP);
> supportedCredentialTypes.add(UserCredentialModel.HOTP);
> }
>
> returnsupportedCredentialTypes;
> }
>
> @Override
> public boolean validCredentials(RealmModel realm, UserModel user,
> List<UserCredentialModel> input) {
> for (UserCredentialModel cred : input) {
> if (cred.getType().equals(UserCredentialModel.PASSWORD)) {
> return validate(user, cred.getValue());
> } else if (cred.getType().equals(UserCredentialModel.TOTP)) {
> return CredentialValidation.validTOTP(realm, user, cred.getValue());
> } else if (cred.getType().equals(UserCredentialModel.HOTP)) {
> return CredentialValidation.validHOTP(realm, user, cred.getValue());
> }
> }
> returnfalse;
> }
>
> @Override
> public boolean validCredentials(RealmModel realm, UserModel user,
> UserCredentialModel... input) {
> for (UserCredentialModel cred : input) {
> if (cred.getType().equals(UserCredentialModel.PASSWORD)) {
> return validate(user, cred.getValue());
> } else if (cred.getType().equals(UserCredentialModel.TOTP)) {
> return CredentialValidation.validTOTP(realm, user, cred.getValue());
> } else if (cred.getType().equals(UserCredentialModel.HOTP)) {
> return CredentialValidation.validHOTP(realm, user, cred.getValue());
> }
> }
> returnfalse;
> }
>
> @Override
> public void close() {
>
> }
> /**
> * Keycloak will call this method if it finds an imported
> UserModel. Here we proxy the UserModel with
> * a Readonly proxy which will barf if password is updated.
> *
> * @param local
> * @return
> */
> @Override
> public UserModel validateAndProxy(RealmModel realm, UserModel local) {
> if (isValid(realm, local)) {
> getUserDetails(local);
> return new StubUserModelProxy(local, this);
> } else {
> returnnull;
> }
> }
>
> @Override
> public boolean synchronizeRegistrations() {
> return true;
> }
>
> /**
> * Called if this federation provider has priority and supports
> synchronized registrations.
> *
> * @param realm
> * @param user
> * @return
> */
> @Override
> public UserModel register(RealmModel realm, UserModel user) {
>
> user.setSingleAttribute("status", "OK");
> returnuser;
> }
>
> @Override
> public boolean removeUser(RealmModel realm, UserModel user) {
> // Not supported. Used as a part of the Workaround to
> https://issues.jboss.org/browse/KEYCLOAK-1075
> returntrue;
> }
>
> /**
> * Supported credentials by this federator. PASSWORD is a supported
> type. TOTP depends on the user.
> *
> * @return supportedCredentialTypes
> */
> @Override
> public Set<String> getSupportedCredentialTypes() {
> Set<String> supportedCredentialTypes = new HashSet<>();
> supportedCredentialTypes.add(UserCredentialModel.PASSWORD);
> supportedCredentialTypes.add(UserCredentialModel.TOTP);
> supportedCredentialTypes.add(UserCredentialModel.HOTP);
> returnsupportedCredentialTypes;
> }
>
> @Override
> public CredentialValidationOutput validCredentials(RealmModel realm,
> UserCredentialModel credential) {
> throw new IllegalStateException("validCredentials not supported");
> }
>
> private boolean userExists(String username){
> returntrue;
> }
> private void getUserDetails(UserModel user) {
> user.setFirstName("first name");
> user.setLastName("last name");
> }
>
> public boolean validate(UserModel user, String password) {
> returntrue;
> }
>
> }
>
> /** End Snippet **/
>
> Regards Tom Connolly.
>
>
> ------------------------------------------------------------------------
> *From:* Marek Posolda <mposolda at redhat.com>
> *To:* Thomas Connolly <thomas_connolly at yahoo.com>;
> "keycloak-user at lists.jboss.org" <keycloak-user at lists.jboss.org>
> *Sent:* Monday, June 13, 2016 5:47 PM
> *Subject:* Re: [keycloak-user] Performance issues with Federation
> provider enabled
>
> Thanks, AFAIK we didn't tried much performance testing with
> federationProviders enabled. It's on todo list though. Also we plan
> some refactoring of userStorage + userFederation, so we will likely go
> into it later.
>
> For your case, the performance bottleneck can be in your
> federationProvider implementation, so I am not sure if it's the issue
> in Keycloak or rather issue in your implementation.
>
> One thing to note (maybe it's not an issue in your case, but just
> adding it to be sure you're aware): UserFederationProvider.close is
> currently not called. So if you are rely on this method to free any
> important resources related to your implementation, you shouldn't as
> it doesn't work right now. We are working on improving this for next
> version.
>
> Marek
>
> On 13/06/16 07:57, Thomas Connolly wrote:
>> Hi Marek
>>
>> I'm working with Fabricio on the federation performance issues with
>> Keycloak.
>>
>> In answer to your question we are using the latest KC 1.9.7 version
>> (we upgraded this week from 1.9.2).
>>
>> To give you some indication of the running a gatling direct access
>> login test (results below).
>>
>> As you can see below in (1) using KC out of the box. Great
>> performance - we saw 110 tx per sec on a 4 core system.
>> In scenario (2) using a stubbed federator (simply an echo plugin not
>> connecting to any back end services), performance is unacceptable.
>>
>> 1) Not using the federator - Stub federator (disabled) - while 29 tx
>> per second we could easily get to a stable 110 tx per second.
>> 300 Users (hitting single server)
>> ---- Global Information
>> --------------------------------------------------------
>> > request count 9185 (OK=9185 KO=0 )
>> > min response time 18
>> (OK=18 KO=- )
>> > max response time 723
>> (OK=723 KO=- )
>> > mean response time 27
>> (OK=27 KO=- )
>> > std deviation 44 (OK=44 KO=- )
>> > response time 50th percentile 20
>> (OK=20 KO=- )
>> > response time 75th percentile 21
>> (OK=21 KO=- )
>> > mean requests/sec 29.626 (OK=29.626 KO=- )
>> ---- Response Time Distribution
>> ------------------------------------------------
>> > t < 800 ms 9185 (100%)
>> > 800 ms < t < 1200 ms 0 ( 0%)
>> > t > 1200 ms 0 ( 0%)
>> > failed 0 ( 0%)
>>
>> 2) Stub federator (enabled)- if we brought test down to 12 tx per
>> second (about 90 users) the response times dropped to < 1200 ms
>> response times, however not even close to meeting out acceptance
>> creteria.
>> 300 Users (hitting single server)
>> ---- Global Information
>> --------------------------------------------------------
>> > request count 8496 (OK=8496 KO=0 )
>> > min response time 511
>> (OK=511 KO=- )
>> > max response time 11191
>> (OK=11191 KO=- )
>> > mean response time 6832
>> (OK=6832 KO=- )
>> > std deviation 2329 (OK=2329 KO=- )
>> > response time 50th percentile 7194
>> (OK=7194 KO=- )
>> > response time 75th percentile 8690
>> (OK=8690 KO=- )
>> > mean requests/sec 27.404 (OK=27.404 KO=- )
>> ---- Response Time Distribution
>> ------------------------------------------------
>> > t < 800 ms 154 ( 2%)
>> > 800 ms < t < 1200 ms 85 ( 1%)
>> > t > 1200 ms 8257 ( 97%)
>> > failed 0 ( 0%)
>>
>> This is currently a show stopper for us and is blocking our path to
>> production.
>> Do you run similar tests and how can we help you optimise the
>> performance?
>>
>> Regards
>> Tom.
>>
>>
>> Date: Wed, 8 Jun 2016 12:28:19 +0200
>> From: Marek Posolda <mposolda at redhat.com> <mailto:mposolda at redhat.com>
>> Subject: Re: [keycloak-user] Performance issues with Federation
>> provider enabled
>> To: Fabricio Milone <fabricio.milone at shinetech.com>
>> <mailto:fabricio.milone at shinetech.com>, keycloak-user
>> <keycloak-user at lists.jboss.org> <mailto:keycloak-user at lists.jboss.org>
>> Message-ID: <5757F343.1040803 at redhat.com>
>> <mailto:5757F343.1040803 at redhat.com>
>> Content-Type: text/plain; charset="windows-1252"
>>
>> Hi,
>>
>> what's the keycloak version used? Could you try latest keycloak and
>> check if performance is still the issue?
>>
>> Marek
>>
>> On 08/06/16 01:30, Fabricio Milone wrote:
>> > Hi all,
>> >
>> > I sent this email yesterday with 5 or more attachments, so I think it
>> > was blocked or something... here I go again :)
>> >
>> > I've been running load tests on our application during the last few
>> > weeks, and having some performance issues when my custom federator is
>> > enabled.
>> >
>> > The performance issue does not exist when the federator is disabled.
>> > *Configuration*:
>> >
>> > I have a cluster of 2 instances of Keycloak, with a standalone DB,
>> > we've verified the DB isn't an issue when the federator is disabled.
>> > Both instances have a quad core CPU and they are in the same network.
>> > We?ve left the memory at 512MB. The test script, database and API that
>> > connects to the federator are in separate machines.
>> > *Federator*:
>> >
>> > We have a simple custom federator that makes calls to a very
>> > performant api, which has been tested and is ok. Additionally, we've
>> > tested stubbing the API so the performance is not a problem there.
>> > This federator is using a jaxb marshaller to create a request, again
>> > tested in isolation and is performing well.
>> >
>> > As the federator is doing a lot of calls to the API (3 per login
>> > request), I've implemented a httpclient that uses a
>> > PoolingHttpClientConnectionManager with 1000 connections available to
>> > use, instead of using the standard apache httpclient from http
>> > components. That hasn't improved a bit the performance of the system.
>> > *Tests*:
>> > It is a gatling scala script that could generate around ~300 (or more)
>> > requests/second to the direct grants login endpoint using random
>> > usernames from a list (all of them already registered using KC). The
>> > script is doing a round robin across both instances of Keycloak with
>> > an even distribution to each KC instance.
>> > The idea is simulate a load of 300 to 1500 concurrent users trying to
>> > login into our systems.
>> > *Problem*:
>> >
>> > If I run the tests without using a federation I can see a very good
>> > performance, but when I try to run the tests with the custom
>> > federation code, the performance drops from ~150 requests/second to 22
>> > req/sec using both instances.
>> > Memory wise, it seems to be ok. I've never seen an error related to
>> > memory with this configuration, also if you take a look at the
>> > attached visualVM screenshot you'll see that memory is not a problem
>> > or it seems not to be.
>> > CPU utilisation is very low to my mind, I'd expect more than 80% of
>> > usage or something like that.
>> > There is a method that is leading the CPU samples on VisualVM called
>> > Semaphore.tryAcquire(). Not quite sure what's that for, still
>> > investigating.
>> >
>> > I can see that a lot of new threads are being created when the test
>> > starts, as it creates around 60requests/second to the direct grants
>> > login call, but it seems to be a bottleneck at some point.
>> >
>> > So I'm wondering if there is some configuration I'm missing on
>> > Keycloak side that could be affecting the cluster performance when a
>> > federator is enabled. Maybe something related to jpa connections,
>> > infinispan configuration or even wildfly.
>> >
>> > I'd really appreciate your help on this one as I'm out of ideas.
>> >
>> > I've attached some screenshots of visualVM and tests results from my
>> > last run today.
>> >
>> >
>> > Sorry for the long email and please let me know if you need further
>> > information.
>> >
>> > Thank you in advance,
>> >
>> > Regards,
>> > Fab
>> >
>> > --
>> > *Fabricio Milone*
>> > Developer
>>
>>
>> _______________________________________________
>> keycloak-user mailing list
>> keycloak-user at lists.jboss.org <mailto:keycloak-user at lists.jboss.org>
>> https://lists.jboss.org/mailman/listinfo/keycloak-user
>
>
>
>
>
> _______________________________________________
> keycloak-user mailing list
> keycloak-user at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/keycloak-user
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/keycloak-user/attachments/20160613/e13afa3f/attachment-0001.html
More information about the keycloak-user
mailing list