[keycloak-user] Keycloak OAuth High CPU usage

Wed May 25 03:09:38 EDT 2016

I did some tests with Linux VM when investigating how Keycloak scales. I
had Keycloak running on a VM that was permitted 50% of a single core and
had a throughput of 50 scenarios. Where a scenario includes a login
request, a code to token request and a logout request. In our performance
lab with a single node and a not particularly beefy machine we're seeing
150+ scenarios/second.

On 24 May 2016 at 16:05, Vaibhav Naldurgkar <
vaibhav_naldurgkar at persistent.com> wrote:

> Hello,
>
>
>
> What are the tests results on a Linux VM ? I just done same jmeter tests
> on AWS m4.xlarge instance; however far behind than the laptop tests results.
>
> @Stian – have you done tests using Linux VM ?
>
>
>
>
>
> Thanks, Vaibhav
>
>
>
> *From:* Herzberg, Manuel [mailto:manuel.herzberg at atos.net]
> *Sent:* Tuesday, May 24, 2016 5:52 PM
> *To:* stian at redhat.com; Vaibhav Naldurgkar
> *Cc:* keycloak-user at lists.jboss.org
> *Subject:* RE: [keycloak-user] Keycloak OAuth High CPU usage
>
>
>
> Hello,
>
> I am evaluating the Keycloak performance. Here my practical experience. My
> scenario is the same as Vaibhav’s:
>
>
>
> ·         Large amount of token have to be generated. This is done by
> requesting the Keycloak token REST endpoint via http. The different realms
> I am using have 1k 2k 3k and 4k keys for signing the tokens. (RSA) Longer
> keys result to longer runtime to generate these tokens.
>
>
>
> ·         I have more than 10k user each realm. Each request includes a
> new user.
> Requests look like this:
> host1:8080/auth/realms/demo-3072/protocol/openid-connect/token/
> with data:
>
> username=testuser1&password=password&client_id=customer-portal&grant_type=password
>
>
>
> ·         The response includes 3 tokens(access, refresh and id). In
> total more than 30 000 token have to be generated and signed.
>
>
>
> @Stian. You wrote you are able to invoke 10000 token refreshes in under 60
> seconds. A token refresh includes access, refresh and id token right? Can
> you explain us your scenario? How do you get such a high number?
>
> Some more results: just signing 3000 Token (800 Byte each) with a 2k key
> takes me 20 seconds (laptop i5-4310U, 12gb ram). I am doing this outside
> Keycloak with my own java program, but with the same implementation
> Keycloak is using. (sign() method in RSAProvider).
>
> The Keycloak implementation is signing tokens with RSA. HMAC and ECC are
> implemented as well as I saw in the code. Changing from RSA to HMAC or ECC
> is not possible in current release as i experienced. Are there plans to
> provide this in future? Defining this in a configuration file or via
> parameters would be nice.
>
> Best regards, Manuel Herzberg
>
>
>
>
>
> *From:* keycloak-user-bounces at lists.jboss.org [
> mailto:keycloak-user-bounces at lists.jboss.org
> <keycloak-user-bounces at lists.jboss.org>] *On Behalf Of *Stian Thorgersen
> *Sent:* Tuesday, May 24, 2016 8:31 AM
> *To:* Vaibhav Naldurgkar
> *Cc:* keycloak-user at lists.jboss.org
> *Subject:* Re: [keycloak-user] Keycloak OAuth High CPU usage
>
>
>
>
>
>
>
> On 23 May 2016 at 10:02, Vaibhav Naldurgkar <
> vaibhav_naldurgkar at persistent.com> wrote:
>
> Yes, the direct access grant is ON for this client. I am trying to
> understand what you mean by “not planning on using web based flow?” Could
> you provide more clarification on this.
>
>
>
> If you are planning to do the web based flow (authorization code grant
> flow) you should test with that rather than direct grant. That being said
> the direct grant should still perform as well.
>
>
>
>
>
> This is what the scenario I am trying to execute and still have high CPU
> usages for KeyCloak Java process.
>
>
>
> ·         The end point URL
> /auth/realms/master/protocol/openid-connect/token has been called by Jmeter
> for 20 concurrent users per seconds to generate the tokens.
>
> ·         Even if used with crul command like “*curl -X POST -d
> "=admin&password=admin&password&client_id=HelloTest&grant_type=password"
> http://localhost:8080/auth/realms/master/protocol/openid-connect/token
> <http://localhost:8080/auth/realms/master/protocol/openid-connect/token>*”
> , in this case also the CPU utilizations goes around 100%.
>
> ·          After around 3 seconds of the test, in the output of top
> command on the KeyCloak server the CPU% for keycloak java process goes
> beyond 100%.
>
>
>
> Would it be possible for you to have a quick call for faster fix of this
> issue. This performance issue is holding to move KeyCloak to use as OAuth
> provider. If any other way is convenient for you please let me know for
> further discussion.
>
>
>
> Your JMeter test is using 20 concurrent threads to send as many requests
> to the direct grant api as it can. This will obviously cause Keycloak to
> consume a high percentage of the CPU. Especially if you are running
> everything on localhost as the network isn't going to be a bottleneck.
> Neither will the database as Keycloak caches everything in memory. The
> bottleneck will be the CPU.
>
>
>
> Authenticating users and obtaining a token requires password hashing as
> well as signing tokens, both are mainly CPU intensive. As you are using the
> direct grant api there's also less network traffic.
>
>
>
> You need to add some reports to your JMeter test so you can see how many
> requests Keycloak can handle. That way you can find out how many users can
> be authenticated per-second on your machine.
>
>
>
> If you only have 500 users remember they won't all login at the same time
> (seconds). Even if they all login at 9am sharp they will be spread out over
> 10 minutes or so, which would only be 1.2 logins/second.
>
>
>
>
>
> Thanks, Vaibhav
>
>
>
>
>
>
>
>
>
> *From:* Stian Thorgersen [mailto:sthorger at redhat.com]
> *Sent:* Monday, May 23, 2016 12:01 PM
>
>
> *To:* Vaibhav Naldurgkar
> *Cc:* keycloak-user at lists.jboss.org
> *Subject:* Re: [keycloak-user] Keycloak OAuth High CPU usage
>
>
>
> You are using direct grant to authenticate a user and obtain a token in
> the example above. This authenticates and creates a new session for each
> request. Are you not planning on using web based flow?
>
>
>
> What do you have password hashing intervals set to? Verifying password is
> CPU intensive, more than signing tokens.
>
>
>
> It shouldn't matter that user is stored in RedHat IdM as the user would be
> cached in Keycloak after first authentication, but it may be an idea to
> just double check by trying to authenticate to a user in Keycloak and not
> RH IdM.
>
>
>
> What results are you actually getting?
>
>
>
>
>
>
>
> On 20 May 2016 at 11:27, Vaibhav Naldurgkar <
> vaibhav_naldurgkar at persistent.com> wrote:
>
> Hi Stian,
>
>
>
> After reading your tests results of 10000 token refreshes in under 60
> seconds on your laptop, I am sure I am not following correct configuration
> and the documents are missing for reference.
>
>
>
> Could you please verify the below steps along with the screen-shots for
> the steps which I am following for the adding client and testing the Load
> performance using Jmeter. Please suggest if any changes are needed in the
> client configuration. In this case we are obtaining the token for user from
> KeyCloak.
>
>
>
> In my case the user have been stored on RedHat IdM which has been
> federated using KeyCloak.
>
>
>
>
>
> Step 1. Create new client called “LoadTest” , use the Client Protocol as
> “Openid-connect”.
>
> Used all defaults values post save of the client action.
>
>
>
> Step 2. Start the load tests using Jmeter and using the path as
> *“/auth/realms/master/protocol/openid-connect/token”* . Used 20 Number of
> Threads and used Post method.
>
>
>
>
>
> Below is the screen-shot for the step 1 related to Add Client.
>
>
>
>
>
>
>
>
>
> Below is the screen shot for the load test using Jmeter. In this case the
> Client ID was used as HelloTest.
>
>
>
>
>
>
>
> Http requests.
>
>
>
>
>
>
>
> Thanks, Vaibhav
>
>
>
>
>
> *From:* Stian Thorgersen [mailto:sthorger at redhat.com]
> *Sent:* Friday, May 20, 2016 1:01 PM
>
>
> *To:* Vaibhav Naldurgkar
> *Cc:* keycloak-user at lists.jboss.org
> *Subject:* Re: [keycloak-user] Keycloak OAuth High CPU usage
>
>
>
> Can you please elaborate a bit more on how your are testing scenario is?
> I'm a bit confused to what you are testing when you are talking about
> generating new tokens. Are you using OIDC or SAML? Are you talking about
> code->token exchanges, refresh token requests, or what?
>
>
>
> To test if your hardware is capable to deal with the load you need to test
> logins (verifying passwords are CPU intensive) as well as obtaining tokens
> (both code->token, done after login, and refreshing token, done ~1 min or
> so by active users, but most users won't continuously use the application).
>
>
>
> 500 users should be no problem at all. As an example with a single thread
> (which will use a single core) I could invoke 10000 token refreshes in
> under 60 seconds on my laptop. So a single core on my laptop should be able
> to handle 500 users.
>
>
>
> On 20 May 2016 at 08:00, Vaibhav Naldurgkar <
> vaibhav_naldurgkar at persistent.com> wrote:
>
> Hi Stian,
>
> Thank you for your reply.
>
>
>
> The new tokens needs to be generated for each user, which is needed from
> security point of view. The performance tests were also conducted using
> single Admin user and token for admin user; however in that case the
> performance was not good. In between 15th to 20th admin token access
> requests – the CPU usage of keycloak Java process was crossing 90 to 120%
> mark.
>
>
>
>
>
> As you have mentioned, Creating tokes are expected to be a bit CPU
> intensive – what should be the server configuration in terms of CPU to deal
> with more than 500 users to use keycloak as OAuth provider.
>
>
>
>
>
> Thanks, Vaibhav
>
>
>
>
>
>
>
> *From:* Stian Thorgersen [mailto:sthorger at redhat.com]
> *Sent:* Thursday, May 19, 2016 6:28 PM
> *To:* Vaibhav Naldurgkar
> *Cc:* keycloak-user at lists.jboss.org
> *Subject:* Re: [keycloak-user] Keycloak OAuth High CPU usage
>
>
>
> Creating tokes are expected to be a bit CPU intensive as they need to be
> signed. When you say you try to generate tokens for 10-20 users are you
> doing performance tests and having 10-20 threads generating tokens? It
> shouldn't make any difference if you have 10 or if you have 200 users, it's
> the total number of tokens that can be generated that's an issue. Having
> 200 concurrent users with a access token timeout of 60 seconds should mean
> that you need to be able to generate roughly 200/60 tokens = 3.3 tokens/sec.
>
>
>
> On 19 May 2016 at 13:24, Vaibhav Naldurgkar <
> vaibhav_naldurgkar at persistent.com> wrote:
>
> Hi All,
>
>
>
> I am using Keycloak 1.9.3 with default configuration. Keycloak server is
> installed on RHEL 6.5 virtual image with 4 CPU , 8 GB RAM and java version
> is jdk1.8.0_73 We are trying to use keycloak as a OAuth provider. But when
> we try and generate token(
> http:///auth/realms/master/protocol/openid-connect/token
> <http://auth/realms/master/protocol/openid-connect/token>) for more than
> 10-20 users the server gets too slow and cpu usage goes over 100%.
>
> Any pointers on how to improve performance of keycloak OAuth provider. We
> need to support at least 200 concurrent users.
>
>
>
>
>
> Thanks, Vaibhav
>
> DISCLAIMER ========== This e-mail may contain privileged and confidential
> information which is the property of Persistent Systems Ltd. It is intended
> only for the use of the individual or entity to which it is addressed. If
> you are not the intended recipient, you are not authorized to read, retain,
> copy, print, distribute or use this message. If you have received this
> communication in error, please notify the sender and delete all copies of
> this message. Persistent Systems Ltd. does not accept any liability for
> virus infected mails.
>
>
> _______________________________________________
> keycloak-user mailing list
> keycloak-user at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/keycloak-user
>
>
>
> DISCLAIMER ========== This e-mail may contain privileged and confidential
> information which is the property of Persistent Systems Ltd. It is intended
> only for the use of the individual or entity to which it is addressed. If
> you are not the intended recipient, you are not authorized to read, retain,
> copy, print, distribute or use this message. If you have received this
> communication in error, please notify the sender and delete all copies of
> this message. Persistent Systems Ltd. does not accept any liability for
> virus infected mails.
>
>
>
> DISCLAIMER ========== This e-mail may contain privileged and confidential
> information which is the property of Persistent Systems Ltd. It is intended
> only for the use of the individual or entity to which it is addressed. If
> you are not the intended recipient, you are not authorized to read, retain,
> copy, print, distribute or use this message. If you have received this
> communication in error, please notify the sender and delete all copies of
> this message. Persistent Systems Ltd. does not accept any liability for
> virus infected mails.
>
>
>
> DISCLAIMER ========== This e-mail may contain privileged and confidential
> information which is the property of Persistent Systems Ltd. It is intended
> only for the use of the individual or entity to which it is addressed. If
> you are not the intended recipient, you are not authorized to read, retain,
> copy, print, distribute or use this message. If you have received this
> communication in error, please notify the sender and delete all copies of
> this message. Persistent Systems Ltd. does not accept any liability for
> virus infected mails.
>
>
>
> DISCLAIMER ========== This e-mail may contain privileged and confidential
> information which is the property of Persistent Systems Ltd. It is intended
> only for the use of the individual or entity to which it is addressed. If
> you are not the intended recipient, you are not authorized to read, retain,
> copy, print, distribute or use this message. If you have received this
> communication in error, please notify the sender and delete all copies of
> this message. Persistent Systems Ltd. does not accept any liability for
> virus infected mails.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/keycloak-user/attachments/20160525/890a3187/attachment-0001.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image003.png
Type: image/png
Size: 18447 bytes
Desc: not available
Url : http://lists.jboss.org/pipermail/keycloak-user/attachments/20160525/890a3187/attachment-0003.png 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image002.png
Type: image/png
Size: 11865 bytes
Desc: not available
Url : http://lists.jboss.org/pipermail/keycloak-user/attachments/20160525/890a3187/attachment-0004.png 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 101405 bytes
Desc: not available
Url : http://lists.jboss.org/pipermail/keycloak-user/attachments/20160525/890a3187/attachment-0005.png 