[keycloak-user] Keycloak OAuth High CPU usage

Mon Jun 13 06:29:20 EDT 2016

As I stated several times before the server should be more than capable to
handle 200 concurrent users.

I don't understand what your issues are as we have no problems with that
type of load when we're benchmarking. We also have plenty of users of
Keycloak that has higher loads than you have. So I'm not sure what your
actually issues are. KEYCLOAK-3057 will not necessarily fix your issues,
it's not a high priority to add and won't be the default (RSA is the
expected signature format for JWTs).

On 13 June 2016 at 11:48, Vaibhav Naldurgkar <
vaibhav_naldurgkar at persistent.com> wrote:

> Stian,
>
>
>
> I noticed that there is an Enhancement added for similar performance
> issues through 3057 https://issues.jboss.org/browse/KEYCLOAK-3057 . I am
> looking forward for the release of 2.0.X, could you let me know if this
> helps to improve performance issue which I am facing.
>
>
>
> Thanks, Vaibhav
>
>
>
> *From:* Stian Thorgersen [mailto:sthorger at redhat.com]
> *Sent:* Thursday, May 26, 2016 11:17 AM
>
> *To:* Vaibhav Naldurgkar
> *Cc:* keycloak-user at lists.jboss.org
> *Subject:* Re: [keycloak-user] Keycloak OAuth High CPU usage
>
>
>
> Again, CPU load is expected to be high while having 20 threads send as
> many requests as they can. It's the total throughput that matters here.
>
>
>
> There are loads of tuning you can do, but you should be able to get decent
> numbers without any tuning.
>
>
>
> On 26 May 2016 at 07:09, Vaibhav Naldurgkar <
> vaibhav_naldurgkar at persistent.com> wrote:
>
> I still wondering what odd configuration I am following on my RHEL VM
> which is not sustaining few user request when checked from the output of
> top command. Could you please suggest if there are any Java specific
> parameters needs to be tuned for performance improvement. If needed I will
> share my configuration files for reference.
>
>
>
> Below is the screenshot of top output during one of the load test.
>
>
>
>
>
>
>
>
>
> *Thanks, Vaibhav*
>
>
>
>
>
>
>
>
>
>
>
> *From:* Stian Thorgersen [mailto:sthorger at redhat.com]
> *Sent:* Wednesday, May 25, 2016 12:40 PM
> *To:* Vaibhav Naldurgkar
> *Cc:* Herzberg, Manuel; keycloak-user at lists.jboss.org
>
>
> *Subject:* Re: [keycloak-user] Keycloak OAuth High CPU usage
>
>
>
> I did some tests with Linux VM when investigating how Keycloak scales. I
> had Keycloak running on a VM that was permitted 50% of a single core and
> had a throughput of 50 scenarios. Where a scenario includes a login
> request, a code to token request and a logout request. In our performance
> lab with a single node and a not particularly beefy machine we're seeing
> 150+ scenarios/second.
>
>
>
> On 24 May 2016 at 16:05, Vaibhav Naldurgkar <
> vaibhav_naldurgkar at persistent.com> wrote:
>
> Hello,
>
>
>
> What are the tests results on a Linux VM ? I just done same jmeter tests
> on AWS m4.xlarge instance; however far behind than the laptop tests results.
>
> @Stian – have you done tests using Linux VM ?
>
>
>
>
>
> Thanks, Vaibhav
>
>
>
> *From:* Herzberg, Manuel [mailto:manuel.herzberg at atos.net]
> *Sent:* Tuesday, May 24, 2016 5:52 PM
> *To:* stian at redhat.com; Vaibhav Naldurgkar
> *Cc:* keycloak-user at lists.jboss.org
> *Subject:* RE: [keycloak-user] Keycloak OAuth High CPU usage
>
>
>
> Hello,
>
> I am evaluating the Keycloak performance. Here my practical experience. My
> scenario is the same as Vaibhav’s:
>
>
>
> ·         Large amount of token have to be generated. This is done by
> requesting the Keycloak token REST endpoint via http. The different realms
> I am using have 1k 2k 3k and 4k keys for signing the tokens. (RSA) Longer
> keys result to longer runtime to generate these tokens.
>
>
>
> ·         I have more than 10k user each realm. Each request includes a
> new user.
> Requests look like this:
> host1:8080/auth/realms/demo-3072/protocol/openid-connect/token/
> with data:
>
> username=testuser1&password=password&client_id=customer-portal&grant_type=password
>
>
>
> ·         The response includes 3 tokens(access, refresh and id). In
> total more than 30 000 token have to be generated and signed.
>
>
>
> @Stian. You wrote you are able to invoke 10000 token refreshes in under 60
> seconds. A token refresh includes access, refresh and id token right? Can
> you explain us your scenario? How do you get such a high number?
>
> Some more results: just signing 3000 Token (800 Byte each) with a 2k key
> takes me 20 seconds (laptop i5-4310U, 12gb ram). I am doing this outside
> Keycloak with my own java program, but with the same implementation
> Keycloak is using. (sign() method in RSAProvider).
>
> The Keycloak implementation is signing tokens with RSA. HMAC and ECC are
> implemented as well as I saw in the code. Changing from RSA to HMAC or ECC
> is not possible in current release as i experienced. Are there plans to
> provide this in future? Defining this in a configuration file or via
> parameters would be nice.
>
> Best regards, Manuel Herzberg
>
>
>
>
>
> *From:* keycloak-user-bounces at lists.jboss.org [
> mailto:keycloak-user-bounces at lists.jboss.org
> <keycloak-user-bounces at lists.jboss.org>] *On Behalf Of *Stian Thorgersen
> *Sent:* Tuesday, May 24, 2016 8:31 AM
> *To:* Vaibhav Naldurgkar
> *Cc:* keycloak-user at lists.jboss.org
> *Subject:* Re: [keycloak-user] Keycloak OAuth High CPU usage
>
>
>
>
>
>
>
> On 23 May 2016 at 10:02, Vaibhav Naldurgkar <
> vaibhav_naldurgkar at persistent.com> wrote:
>
> Yes, the direct access grant is ON for this client. I am trying to
> understand what you mean by “not planning on using web based flow?” Could
> you provide more clarification on this.
>
>
>
> If you are planning to do the web based flow (authorization code grant
> flow) you should test with that rather than direct grant. That being said
> the direct grant should still perform as well.
>
>
>
>
>
> This is what the scenario I am trying to execute and still have high CPU
> usages for KeyCloak Java process.
>
>
>
> ·         The end point URL
> /auth/realms/master/protocol/openid-connect/token has been called by Jmeter
> for 20 concurrent users per seconds to generate the tokens.
>
> ·         Even if used with crul command like “*curl -X POST -d
> "=admin&password=admin&password&client_id=HelloTest&grant_type=password"
> http://localhost:8080/auth/realms/master/protocol/openid-connect/token
> <http://localhost:8080/auth/realms/master/protocol/openid-connect/token>*”
> , in this case also the CPU utilizations goes around 100%.
>
> ·          After around 3 seconds of the test, in the output of top
> command on the KeyCloak server the CPU% for keycloak java process goes
> beyond 100%.
>
>
>
> Would it be possible for you to have a quick call for faster fix of this
> issue. This performance issue is holding to move KeyCloak to use as OAuth
> provider. If any other way is convenient for you please let me know for
> further discussion.
>
>
>
> Your JMeter test is using 20 concurrent threads to send as many requests
> to the direct grant api as it can. This will obviously cause Keycloak to
> consume a high percentage of the CPU. Especially if you are running
> everything on localhost as the network isn't going to be a bottleneck.
> Neither will the database as Keycloak caches everything in memory. The
> bottleneck will be the CPU.
>
>
>
> Authenticating users and obtaining a token requires password hashing as
> well as signing tokens, both are mainly CPU intensive. As you are using the
> direct grant api there's also less network traffic.
>
>
>
> You need to add some reports to your JMeter test so you can see how many
> requests Keycloak can handle. That way you can find out how many users can
> be authenticated per-second on your machine.
>
>
>
> If you only have 500 users remember they won't all login at the same time
> (seconds). Even if they all login at 9am sharp they will be spread out over
> 10 minutes or so, which would only be 1.2 logins/second.
>
>
>
>
>
> Thanks, Vaibhav
>
>
>
>
>
>
>
>
>
> *From:* Stian Thorgersen [mailto:sthorger at redhat.com]
> *Sent:* Monday, May 23, 2016 12:01 PM
>
>
> *To:* Vaibhav Naldurgkar
> *Cc:* keycloak-user at lists.jboss.org
> *Subject:* Re: [keycloak-user] Keycloak OAuth High CPU usage
>
>
>
> You are using direct grant to authenticate a user and obtain a token in
> the example above. This authenticates and creates a new session for each
> request. Are you not planning on using web based flow?
>
>
>
> What do you have password hashing intervals set to? Verifying password is
> CPU intensive, more than signing tokens.
>
>
>
> It shouldn't matter that user is stored in RedHat IdM as the user would be
> cached in Keycloak after first authentication, but it may be an idea to
> just double check by trying to authenticate to a user in Keycloak and not
> RH IdM.
>
>
>
> What results are you actually getting?
>
>
>
>
>
>
>
> On 20 May 2016 at 11:27, Vaibhav Naldurgkar <
> vaibhav_naldurgkar at persistent.com> wrote:
>
> Hi Stian,
>
>
>
> After reading your tests results of 10000 token refreshes in under 60
> seconds on your laptop, I am sure I am not following correct configuration
> and the documents are missing for reference.
>
>
>
> Could you please verify the below steps along with the screen-shots for
> the steps which I am following for the adding client and testing the Load
> performance using Jmeter. Please suggest if any changes are needed in the
> client configuration. In this case we are obtaining the token for user from
> KeyCloak.
>
>
>
> In my case the user have been stored on RedHat IdM which has been
> federated using KeyCloak.
>
>
>
>
>
> Step 1. Create new client called “LoadTest” , use the Client Protocol as
> “Openid-connect”.
>
> Used all defaults values post save of the client action.
>
>
>
> Step 2. Start the load tests using Jmeter and using the path as
> *“/auth/realms/master/protocol/openid-connect/token”* . Used 20 Number of
> Threads and used Post method.
>
>
>
>
>
> Below is the screen-shot for the step 1 related to Add Client.
>
>
>
>
>
>
>
>
>
> Below is the screen shot for the load test using Jmeter. In this case the
> Client ID was used as HelloTest.
>
>
>
>
>
>
>
> Http requests.
>
>
>
>
>
>
>
> Thanks, Vaibhav
>
>
>
>
>
> *From:* Stian Thorgersen [mailto:sthorger at redhat.com]
> *Sent:* Friday, May 20, 2016 1:01 PM
>
>
> *To:* Vaibhav Naldurgkar
> *Cc:* keycloak-user at lists.jboss.org
> *Subject:* Re: [keycloak-user] Keycloak OAuth High CPU usage
>
>
>
> Can you please elaborate a bit more on how your are testing scenario is?
> I'm a bit confused to what you are testing when you are talking about
> generating new tokens. Are you using OIDC or SAML? Are you talking about
> code->token exchanges, refresh token requests, or what?
>
>
>
> To test if your hardware is capable to deal with the load you need to test
> logins (verifying passwords are CPU intensive) as well as obtaining tokens
> (both code->token, done after login, and refreshing token, done ~1 min or
> so by active users, but most users won't continuously use the application).
>
>
>
> 500 users should be no problem at all. As an example with a single thread
> (which will use a single core) I could invoke 10000 token refreshes in
> under 60 seconds on my laptop. So a single core on my laptop should be able
> to handle 500 users.
>
>
>
> On 20 May 2016 at 08:00, Vaibhav Naldurgkar <
> vaibhav_naldurgkar at persistent.com> wrote:
>
> Hi Stian,
>
> Thank you for your reply.
>
>
>
> The new tokens needs to be generated for each user, which is needed from
> security point of view. The performance tests were also conducted using
> single Admin user and token for admin user; however in that case the
> performance was not good. In between 15th to 20th admin token access
> requests – the CPU usage of keycloak Java process was crossing 90 to 120%
> mark.
>
>
>
>
>
> As you have mentioned, Creating tokes are expected to be a bit CPU
> intensive – what should be the server configuration in terms of CPU to deal
> with more than 500 users to use keycloak as OAuth provider.
>
>
>
>
>
> Thanks, Vaibhav
>
>
>
>
>
>
>
> *From:* Stian Thorgersen [mailto:sthorger at redhat.com]
> *Sent:* Thursday, May 19, 2016 6:28 PM
> *To:* Vaibhav Naldurgkar
> *Cc:* keycloak-user at lists.jboss.org
> *Subject:* Re: [keycloak-user] Keycloak OAuth High CPU usage
>
>
>
> Creating tokes are expected to be a bit CPU intensive as they need to be
> signed. When you say you try to generate tokens for 10-20 users are you
> doing performance tests and having 10-20 threads generating tokens? It
> shouldn't make any difference if you have 10 or if you have 200 users, it's
> the total number of tokens that can be generated that's an issue. Having
> 200 concurrent users with a access token timeout of 60 seconds should mean
> that you need to be able to generate roughly 200/60 tokens = 3.3 tokens/sec.
>
>
>
> On 19 May 2016 at 13:24, Vaibhav Naldurgkar <
> vaibhav_naldurgkar at persistent.com> wrote:
>
> Hi All,
>
>
>
> I am using Keycloak 1.9.3 with default configuration. Keycloak server is
> installed on RHEL 6.5 virtual image with 4 CPU , 8 GB RAM and java version
> is jdk1.8.0_73 We are trying to use keycloak as a OAuth provider. But when
> we try and generate token(
> http:///auth/realms/master/protocol/openid-connect/token
> <http://auth/realms/master/protocol/openid-connect/token>) for more than
> 10-20 users the server gets too slow and cpu usage goes over 100%.
>
> Any pointers on how to improve performance of keycloak OAuth provider. We
> need to support at least 200 concurrent users.
>
>
>
>
>
> Thanks, Vaibhav
>
> DISCLAIMER ========== This e-mail may contain privileged and confidential
> information which is the property of Persistent Systems Ltd. It is intended
> only for the use of the individual or entity to which it is addressed. If
> you are not the intended recipient, you are not authorized to read, retain,
> copy, print, distribute or use this message. If you have received this
> communication in error, please notify the sender and delete all copies of
> this message. Persistent Systems Ltd. does not accept any liability for
> virus infected mails.
>
>
> _______________________________________________
> keycloak-user mailing list
> keycloak-user at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/keycloak-user
>
>
>
> DISCLAIMER ========== This e-mail may contain privileged and confidential
> information which is the property of Persistent Systems Ltd. It is intended
> only for the use of the individual or entity to which it is addressed. If
> you are not the intended recipient, you are not authorized to read, retain,
> copy, print, distribute or use this message. If you have received this
> communication in error, please notify the sender and delete all copies of
> this message. Persistent Systems Ltd. does not accept any liability for
> virus infected mails.
>
>
>
> DISCLAIMER ========== This e-mail may contain privileged and confidential
> information which is the property of Persistent Systems Ltd. It is intended
> only for the use of the individual or entity to which it is addressed. If
> you are not the intended recipient, you are not authorized to read, retain,
> copy, print, distribute or use this message. If you have received this
> communication in error, please notify the sender and delete all copies of
> this message. Persistent Systems Ltd. does not accept any liability for
> virus infected mails.
>
>
>
> DISCLAIMER ========== This e-mail may contain privileged and confidential
> information which is the property of Persistent Systems Ltd. It is intended
> only for the use of the individual or entity to which it is addressed. If
> you are not the intended recipient, you are not authorized to read, retain,
> copy, print, distribute or use this message. If you have received this
> communication in error, please notify the sender and delete all copies of
> this message. Persistent Systems Ltd. does not accept any liability for
> virus infected mails.
>
>
>
> DISCLAIMER ========== This e-mail may contain privileged and confidential
> information which is the property of Persistent Systems Ltd. It is intended
> only for the use of the individual or entity to which it is addressed. If
> you are not the intended recipient, you are not authorized to read, retain,
> copy, print, distribute or use this message. If you have received this
> communication in error, please notify the sender and delete all copies of
> this message. Persistent Systems Ltd. does not accept any liability for
> virus infected mails.
>
>
>
> DISCLAIMER ========== This e-mail may contain privileged and confidential
> information which is the property of Persistent Systems Ltd. It is intended
> only for the use of the individual or entity to which it is addressed. If
> you are not the intended recipient, you are not authorized to read, retain,
> copy, print, distribute or use this message. If you have received this
> communication in error, please notify the sender and delete all copies of
> this message. Persistent Systems Ltd. does not accept any liability for
> virus infected mails.
>
>
>
> DISCLAIMER ========== This e-mail may contain privileged and confidential
> information which is the property of Persistent Systems Ltd. It is intended
> only for the use of the individual or entity to which it is addressed. If
> you are not the intended recipient, you are not authorized to read, retain,
> copy, print, distribute or use this message. If you have received this
> communication in error, please notify the sender and delete all copies of
> this message. Persistent Systems Ltd. does not accept any liability for
> virus infected mails.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/keycloak-user/attachments/20160613/f2036b74/attachment-0001.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 28486 bytes
Desc: not available
Url : http://lists.jboss.org/pipermail/keycloak-user/attachments/20160613/f2036b74/attachment-0004.png 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image004.png
Type: image/png
Size: 18447 bytes
Desc: not available
Url : http://lists.jboss.org/pipermail/keycloak-user/attachments/20160613/f2036b74/attachment-0005.png 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image003.png
Type: image/png
Size: 11865 bytes
Desc: not available
Url : http://lists.jboss.org/pipermail/keycloak-user/attachments/20160613/f2036b74/attachment-0006.png 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image002.png
Type: image/png
Size: 101405 bytes
Desc: not available
Url : http://lists.jboss.org/pipermail/keycloak-user/attachments/20160613/f2036b74/attachment-0007.png 