New subject: Keycloak OAuth High CPU usage

Thursday, 26 May 2016

Again, CPU load is expected to be high while having 20 threads send as many
requests as they can. It's the total throughput that matters here.

There are loads of tuning you can do, but you should be able to get decent
numbers without any tuning.

On 26 May 2016 at 07:09, Vaibhav Naldurgkar <
vaibhav_naldurgkar(a)persistent.com&gt; wrote:

...
 I still wondering what odd configuration I am following on my RHEL
VM
 which is not sustaining few user request when checked from the output of
 top command. Could you please suggest if there are any Java specific
 parameters needs to be tuned for performance improvement. If needed I will
 share my configuration files for reference.

 Below is the screenshot of top output during one of the load test.

 *Thanks, Vaibhav*

 *From:* Stian Thorgersen [mailto:sthorger@redhat.com]
 *Sent:* Wednesday, May 25, 2016 12:40 PM
 *To:* Vaibhav Naldurgkar
 *Cc:* Herzberg, Manuel; keycloak-user(a)lists.jboss.org

 *Subject:* Re: [keycloak-user] Keycloak OAuth High CPU usage

 I did some tests with Linux VM when investigating how Keycloak scales. I
 had Keycloak running on a VM that was permitted 50% of a single core and
 had a throughput of 50 scenarios. Where a scenario includes a login
 request, a code to token request and a logout request. In our performance
 lab with a single node and a not particularly beefy machine we're seeing
 150+ scenarios/second.

 On 24 May 2016 at 16:05, Vaibhav Naldurgkar <
 vaibhav_naldurgkar(a)persistent.com&gt; wrote:

 Hello,

 What are the tests results on a Linux VM ? I just done same jmeter tests
 on AWS m4.xlarge instance; however far behind than the laptop tests results.

 @Stian – have you done tests using Linux VM ?

 Thanks, Vaibhav

 *From:* Herzberg, Manuel [mailto:manuel.herzberg@atos.net]
 *Sent:* Tuesday, May 24, 2016 5:52 PM
 *To:* stian(a)redhat.com; Vaibhav Naldurgkar
 *Cc:* keycloak-user(a)lists.jboss.org
 *Subject:* RE: [keycloak-user] Keycloak OAuth High CPU usage

 Hello,

 I am evaluating the Keycloak performance. Here my practical experience. My
 scenario is the same as Vaibhav’s:

 ·         Large amount of token have to be generated. This is done by
 requesting the Keycloak token REST endpoint via http. The different realms
 I am using have 1k 2k 3k and 4k keys for signing the tokens. (RSA) Longer
 keys result to longer runtime to generate these tokens.

 ·         I have more than 10k user each realm. Each request includes a
 new user.
 Requests look like this:
 host1:8080/auth/realms/demo-3072/protocol/openid-connect/token/
 with data:

username=testuser1&password=password&client_id=customer-portal&grant_type=password

 ·         The response includes 3 tokens(access, refresh and id). In
 total more than 30 000 token have to be generated and signed.

 @Stian. You wrote you are able to invoke 10000 token refreshes in under 60
 seconds. A token refresh includes access, refresh and id token right? Can
 you explain us your scenario? How do you get such a high number?

 Some more results: just signing 3000 Token (800 Byte each) with a 2k key
 takes me 20 seconds (laptop i5-4310U, 12gb ram). I am doing this outside
 Keycloak with my own java program, but with the same implementation
 Keycloak is using. (sign() method in RSAProvider).

 The Keycloak implementation is signing tokens with RSA. HMAC and ECC are
 implemented as well as I saw in the code. Changing from RSA to HMAC or ECC
 is not possible in current release as i experienced. Are there plans to
 provide this in future? Defining this in a configuration file or via
 parameters would be nice.

 Best regards, Manuel Herzberg

 *From:* keycloak-user-bounces(a)lists.jboss.org [
 mailto:keycloak-user-bounces@lists.jboss.org
 <keycloak-user-bounces(a)lists.jboss.org>] *On Behalf Of *Stian Thorgersen
 *Sent:* Tuesday, May 24, 2016 8:31 AM
 *To:* Vaibhav Naldurgkar
 *Cc:* keycloak-user(a)lists.jboss.org
 *Subject:* Re: [keycloak-user] Keycloak OAuth High CPU usage

 On 23 May 2016 at 10:02, Vaibhav Naldurgkar <
 vaibhav_naldurgkar(a)persistent.com&gt; wrote:

 Yes, the direct access grant is ON for this client. I am trying to
 understand what you mean by “not planning on using web based flow?” Could
 you provide more clarification on this.

 If you are planning to do the web based flow (authorization code grant
 flow) you should test with that rather than direct grant. That being said
 the direct grant should still perform as well.

 This is what the scenario I am trying to execute and still have high CPU
 usages for KeyCloak Java process.

 ·         The end point URL
 /auth/realms/master/protocol/openid-connect/token has been called by Jmeter
 for 20 concurrent users per seconds to generate the tokens.

 ·         Even if used with crul command like “*curl -X POST -d

"=admin&password=admin&password&client_id=HelloTest&grant_type=password"
 http://localhost:8080/auth/realms/master/protocol/openid-connect/token
 <http://localhost:8080/auth/realms/master/protocol/openid-connect/token>*”
 , in this case also the CPU utilizations goes around 100%.

 ·          After around 3 seconds of the test, in the output of top
 command on the KeyCloak server the CPU% for keycloak java process goes
 beyond 100%.

 Would it be possible for you to have a quick call for faster fix of this
 issue. This performance issue is holding to move KeyCloak to use as OAuth
 provider. If any other way is convenient for you please let me know for
 further discussion.

 Your JMeter test is using 20 concurrent threads to send as many requests
 to the direct grant api as it can. This will obviously cause Keycloak to
 consume a high percentage of the CPU. Especially if you are running
 everything on localhost as the network isn't going to be a bottleneck.
 Neither will the database as Keycloak caches everything in memory. The
 bottleneck will be the CPU.

 Authenticating users and obtaining a token requires password hashing as
 well as signing tokens, both are mainly CPU intensive. As you are using the
 direct grant api there's also less network traffic.

 You need to add some reports to your JMeter test so you can see how many
 requests Keycloak can handle. That way you can find out how many users can
 be authenticated per-second on your machine.

 If you only have 500 users remember they won't all login at the same time
 (seconds). Even if they all login at 9am sharp they will be spread out over
 10 minutes or so, which would only be 1.2 logins/second.

 Thanks, Vaibhav

 *From:* Stian Thorgersen [mailto:sthorger@redhat.com]
 *Sent:* Monday, May 23, 2016 12:01 PM

 *To:* Vaibhav Naldurgkar
 *Cc:* keycloak-user(a)lists.jboss.org
 *Subject:* Re: [keycloak-user] Keycloak OAuth High CPU usage

 You are using direct grant to authenticate a user and obtain a token in
 the example above. This authenticates and creates a new session for each
 request. Are you not planning on using web based flow?

 What do you have password hashing intervals set to? Verifying password is
 CPU intensive, more than signing tokens.

 It shouldn't matter that user is stored in RedHat IdM as the user would be
 cached in Keycloak after first authentication, but it may be an idea to
 just double check by trying to authenticate to a user in Keycloak and not
 RH IdM.

 What results are you actually getting?

 On 20 May 2016 at 11:27, Vaibhav Naldurgkar <
 vaibhav_naldurgkar(a)persistent.com&gt; wrote:

 Hi Stian,

 After reading your tests results of 10000 token refreshes in under 60
 seconds on your laptop, I am sure I am not following correct configuration
 and the documents are missing for reference.

 Could you please verify the below steps along with the screen-shots for
 the steps which I am following for the adding client and testing the Load
 performance using Jmeter. Please suggest if any changes are needed in the
 client configuration. In this case we are obtaining the token for user from
 KeyCloak.

 In my case the user have been stored on RedHat IdM which has been
 federated using KeyCloak.

 Step 1. Create new client called “LoadTest” , use the Client Protocol as
 “Openid-connect”.

 Used all defaults values post save of the client action.

 Step 2. Start the load tests using Jmeter and using the path as
 *“/auth/realms/master/protocol/openid-connect/token”* . Used 20 Number of
 Threads and used Post method.

 Below is the screen-shot for the step 1 related to Add Client.

 Below is the screen shot for the load test using Jmeter. In this case the
 Client ID was used as HelloTest.

 Http requests.

 Thanks, Vaibhav

 *From:* Stian Thorgersen [mailto:sthorger@redhat.com]
 *Sent:* Friday, May 20, 2016 1:01 PM

 *To:* Vaibhav Naldurgkar
 *Cc:* keycloak-user(a)lists.jboss.org
 *Subject:* Re: [keycloak-user] Keycloak OAuth High CPU usage

 Can you please elaborate a bit more on how your are testing scenario is?
 I'm a bit confused to what you are testing when you are talking about
 generating new tokens. Are you using OIDC or SAML? Are you talking about
 code->token exchanges, refresh token requests, or what?

 To test if your hardware is capable to deal with the load you need to test
 logins (verifying passwords are CPU intensive) as well as obtaining tokens
 (both code->token, done after login, and refreshing token, done ~1 min or
 so by active users, but most users won't continuously use the application).

 500 users should be no problem at all. As an example with a single thread
 (which will use a single core) I could invoke 10000 token refreshes in
 under 60 seconds on my laptop. So a single core on my laptop should be able
 to handle 500 users.

 On 20 May 2016 at 08:00, Vaibhav Naldurgkar <
 vaibhav_naldurgkar(a)persistent.com&gt; wrote:

 Hi Stian,

 Thank you for your reply.

 The new tokens needs to be generated for each user, which is needed from
 security point of view. The performance tests were also conducted using
 single Admin user and token for admin user; however in that case the
 performance was not good. In between 15th to 20th admin token access
 requests – the CPU usage of keycloak Java process was crossing 90 to 120%
 mark.

 As you have mentioned, Creating tokes are expected to be a bit CPU
 intensive – what should be the server configuration in terms of CPU to deal
 with more than 500 users to use keycloak as OAuth provider.

 Thanks, Vaibhav

 *From:* Stian Thorgersen [mailto:sthorger@redhat.com]
 *Sent:* Thursday, May 19, 2016 6:28 PM
 *To:* Vaibhav Naldurgkar
 *Cc:* keycloak-user(a)lists.jboss.org
 *Subject:* Re: [keycloak-user] Keycloak OAuth High CPU usage

 Creating tokes are expected to be a bit CPU intensive as they need to be
 signed. When you say you try to generate tokens for 10-20 users are you
 doing performance tests and having 10-20 threads generating tokens? It
 shouldn't make any difference if you have 10 or if you have 200 users, it's
 the total number of tokens that can be generated that's an issue. Having
 200 concurrent users with a access token timeout of 60 seconds should mean
 that you need to be able to generate roughly 200/60 tokens = 3.3 tokens/sec.

 On 19 May 2016 at 13:24, Vaibhav Naldurgkar <
 vaibhav_naldurgkar(a)persistent.com&gt; wrote:

 Hi All,

 I am using Keycloak 1.9.3 with default configuration. Keycloak server is
 installed on RHEL 6.5 virtual image with 4 CPU , 8 GB RAM and java version
 is jdk1.8.0_73 We are trying to use keycloak as a OAuth provider. But when
 we try and generate token(
 http:///auth/realms/master/protocol/openid-connect/token
 <http://auth/realms/master/protocol/openid-connect/token>) for more than
 10-20 users the server gets too slow and cpu usage goes over 100%.

 Any pointers on how to improve performance of keycloak OAuth provider. We
 need to support at least 200 concurrent users.

 Thanks, Vaibhav

 DISCLAIMER ========== This e-mail may contain privileged and confidential
 information which is the property of Persistent Systems Ltd. It is intended
 only for the use of the individual or entity to which it is addressed. If
 you are not the intended recipient, you are not authorized to read, retain,
 copy, print, distribute or use this message. If you have received this
 communication in error, please notify the sender and delete all copies of
 this message. Persistent Systems Ltd. does not accept any liability for
 virus infected mails.

 _______________________________________________
 keycloak-user mailing list
 keycloak-user(a)lists.jboss.org
 https://lists.jboss.org/mailman/listinfo/keycloak-user

 DISCLAIMER ========== This e-mail may contain privileged and confidential
 information which is the property of Persistent Systems Ltd. It is intended
 only for the use of the individual or entity to which it is addressed. If
 you are not the intended recipient, you are not authorized to read, retain,
 copy, print, distribute or use this message. If you have received this
 communication in error, please notify the sender and delete all copies of
 this message. Persistent Systems Ltd. does not accept any liability for
 virus infected mails.

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

Re: [keycloak-user] Keycloak OAuth High CPU usage