Re: [keycloak-user] Keycloak OAuth High CPU usage

Stian, I noticed that there is an Enhancement added for similar performance issues through 3057 https://issues.jboss.org/browse/KEYCLOAK-3057 . I am looking forward for the release of 2.0.X, could you let me know if this helps to improve performance issue which I am facing. Thanks, Vaibhav *From:* Stian Thorgersen [mailto:sthorger@redhat.com] *Sent:* Thursday, May 26, 2016 11:17 AM *To:* Vaibhav Naldurgkar *Cc:* keycloak-user(a)lists.jboss.org *Subject:* Re: [keycloak-user] Keycloak OAuth High CPU usage Again, CPU load is expected to be high while having 20 threads send as many requests as they can. It's the total throughput that matters here. There are loads of tuning you can do, but you should be able to get decent numbers without any tuning. On 26 May 2016 at 07:09, Vaibhav Naldurgkar < vaibhav_naldurgkar(a)persistent.com&gt; wrote: I still wondering what odd configuration I am following on my RHEL VM which is not sustaining few user request when checked from the output of top command. Could you please suggest if there are any Java specific parameters needs to be tuned for performance improvement. If needed I will share my configuration files for reference. Below is the screenshot of top output during one of the load test. *Thanks, Vaibhav* *From:* Stian Thorgersen [mailto:sthorger@redhat.com] *Sent:* Wednesday, May 25, 2016 12:40 PM *To:* Vaibhav Naldurgkar *Cc:* Herzberg, Manuel; keycloak-user(a)lists.jboss.org *Subject:* Re: [keycloak-user] Keycloak OAuth High CPU usage I did some tests with Linux VM when investigating how Keycloak scales. I had Keycloak running on a VM that was permitted 50% of a single core and had a throughput of 50 scenarios. Where a scenario includes a login request, a code to token request and a logout request. In our performance lab with a single node and a not particularly beefy machine we're seeing 150+ scenarios/second. On 24 May 2016 at 16:05, Vaibhav Naldurgkar < vaibhav_naldurgkar(a)persistent.com&gt; wrote: Hello, What are the tests results on a Linux VM ? I just done same jmeter tests on AWS m4.xlarge instance; however far behind than the laptop tests results. @Stian – have you done tests using Linux VM ? Thanks, Vaibhav *From:* Herzberg, Manuel [mailto:manuel.herzberg@atos.net] *Sent:* Tuesday, May 24, 2016 5:52 PM *To:* stian(a)redhat.com; Vaibhav Naldurgkar *Cc:* keycloak-user(a)lists.jboss.org *Subject:* RE: [keycloak-user] Keycloak OAuth High CPU usage Hello, I am evaluating the Keycloak performance. Here my practical experience. My scenario is the same as Vaibhav’s: · Large amount of token have to be generated. This is done by requesting the Keycloak token REST endpoint via http. The different realms I am using have 1k 2k 3k and 4k keys for signing the tokens. (RSA) Longer keys result to longer runtime to generate these tokens. · I have more than 10k user each realm. Each request includes a new user. Requests look like this: host1:8080/auth/realms/demo-3072/protocol/openid-connect/token/ with data: username=testuser1&password=password&client_id=customer-portal&grant_type=password · The response includes 3 tokens(access, refresh and id). In total more than 30 000 token have to be generated and signed. @Stian. You wrote you are able to invoke 10000 token refreshes in under 60 seconds. A token refresh includes access, refresh and id token right? Can you explain us your scenario? How do you get such a high number? Some more results: just signing 3000 Token (800 Byte each) with a 2k key takes me 20 seconds (laptop i5-4310U, 12gb ram). I am doing this outside Keycloak with my own java program, but with the same implementation Keycloak is using. (sign() method in RSAProvider). The Keycloak implementation is signing tokens with RSA. HMAC and ECC are implemented as well as I saw in the code. Changing from RSA to HMAC or ECC is not possible in current release as i experienced. Are there plans to provide this in future? Defining this in a configuration file or via parameters would be nice. Best regards, Manuel Herzberg *From:* keycloak-user-bounces(a)lists.jboss.org [ mailto:keycloak-user-bounces@lists.jboss.org <keycloak-user-bounces(a)lists.jboss.org>] *On Behalf Of *Stian Thorgersen *Sent:* Tuesday, May 24, 2016 8:31 AM *To:* Vaibhav Naldurgkar *Cc:* keycloak-user(a)lists.jboss.org *Subject:* Re: [keycloak-user] Keycloak OAuth High CPU usage On 23 May 2016 at 10:02, Vaibhav Naldurgkar < vaibhav_naldurgkar(a)persistent.com&gt; wrote: Yes, the direct access grant is ON for this client. I am trying to understand what you mean by “not planning on using web based flow?” Could you provide more clarification on this. If you are planning to do the web based flow (authorization code grant flow) you should test with that rather than direct grant. That being said the direct grant should still perform as well. This is what the scenario I am trying to execute and still have high CPU usages for KeyCloak Java process. · The end point URL /auth/realms/master/protocol/openid-connect/token has been called by Jmeter for 20 concurrent users per seconds to generate the tokens. · Even if used with crul command like “*curl -X POST -d "=admin&password=admin&password&client_id=HelloTest&grant_type=password" http://localhost:8080/auth/realms/master/protocol/openid-connect/token <http://localhost:8080/auth/realms/master/protocol/openid-connect/token>*” , in this case also the CPU utilizations goes around 100%. · After around 3 seconds of the test, in the output of top command on the KeyCloak server the CPU% for keycloak java process goes beyond 100%. Would it be possible for you to have a quick call for faster fix of this issue. This performance issue is holding to move KeyCloak to use as OAuth provider. If any other way is convenient for you please let me know for further discussion. Your JMeter test is using 20 concurrent threads to send as many requests to the direct grant api as it can. This will obviously cause Keycloak to consume a high percentage of the CPU. Especially if you are running everything on localhost as the network isn't going to be a bottleneck. Neither will the database as Keycloak caches everything in memory. The bottleneck will be the CPU. Authenticating users and obtaining a token requires password hashing as well as signing tokens, both are mainly CPU intensive. As you are using the direct grant api there's also less network traffic. You need to add some reports to your JMeter test so you can see how many requests Keycloak can handle. That way you can find out how many users can be authenticated per-second on your machine. If you only have 500 users remember they won't all login at the same time (seconds). Even if they all login at 9am sharp they will be spread out over 10 minutes or so, which would only be 1.2 logins/second. Thanks, Vaibhav *From:* Stian Thorgersen [mailto:sthorger@redhat.com] *Sent:* Monday, May 23, 2016 12:01 PM *To:* Vaibhav Naldurgkar *Cc:* keycloak-user(a)lists.jboss.org *Subject:* Re: [keycloak-user] Keycloak OAuth High CPU usage You are using direct grant to authenticate a user and obtain a token in the example above. This authenticates and creates a new session for each request. Are you not planning on using web based flow? What do you have password hashing intervals set to? Verifying password is CPU intensive, more than signing tokens. It shouldn't matter that user is stored in RedHat IdM as the user would be cached in Keycloak after first authentication, but it may be an idea to just double check by trying to authenticate to a user in Keycloak and not RH IdM. What results are you actually getting? On 20 May 2016 at 11:27, Vaibhav Naldurgkar < vaibhav_naldurgkar(a)persistent.com&gt; wrote: Hi Stian, After reading your tests results of 10000 token refreshes in under 60 seconds on your laptop, I am sure I am not following correct configuration and the documents are missing for reference. Could you please verify the below steps along with the screen-shots for the steps which I am following for the adding client and testing the Load performance using Jmeter. Please suggest if any changes are needed in the client configuration. In this case we are obtaining the token for user from KeyCloak. In my case the user have been stored on RedHat IdM which has been federated using KeyCloak. Step 1. Create new client called “LoadTest” , use the Client Protocol as “Openid-connect”. Used all defaults values post save of the client action. Step 2. Start the load tests using Jmeter and using the path as *“/auth/realms/master/protocol/openid-connect/token”* . Used 20 Number of Threads and used Post method. Below is the screen-shot for the step 1 related to Add Client. Below is the screen shot for the load test using Jmeter. In this case the Client ID was used as HelloTest. Http requests. Thanks, Vaibhav *From:* Stian Thorgersen [mailto:sthorger@redhat.com] *Sent:* Friday, May 20, 2016 1:01 PM *To:* Vaibhav Naldurgkar *Cc:* keycloak-user(a)lists.jboss.org *Subject:* Re: [keycloak-user] Keycloak OAuth High CPU usage Can you please elaborate a bit more on how your are testing scenario is? I'm a bit confused to what you are testing when you are talking about generating new tokens. Are you using OIDC or SAML? Are you talking about code->token exchanges, refresh token requests, or what? To test if your hardware is capable to deal with the load you need to test logins (verifying passwords are CPU intensive) as well as obtaining tokens (both code->token, done after login, and refreshing token, done ~1 min or so by active users, but most users won't continuously use the application). 500 users should be no problem at all. As an example with a single thread (which will use a single core) I could invoke 10000 token refreshes in under 60 seconds on my laptop. So a single core on my laptop should be able to handle 500 users. On 20 May 2016 at 08:00, Vaibhav Naldurgkar < vaibhav_naldurgkar(a)persistent.com&gt; wrote: Hi Stian, Thank you for your reply. The new tokens needs to be generated for each user, which is needed from security point of view. The performance tests were also conducted using single Admin user and token for admin user; however in that case the performance was not good. In between 15th to 20th admin token access requests – the CPU usage of keycloak Java process was crossing 90 to 120% mark. As you have mentioned, Creating tokes are expected to be a bit CPU intensive – what should be the server configuration in terms of CPU to deal with more than 500 users to use keycloak as OAuth provider. Thanks, Vaibhav *From:* Stian Thorgersen [mailto:sthorger@redhat.com] *Sent:* Thursday, May 19, 2016 6:28 PM *To:* Vaibhav Naldurgkar *Cc:* keycloak-user(a)lists.jboss.org *Subject:* Re: [keycloak-user] Keycloak OAuth High CPU usage Creating tokes are expected to be a bit CPU intensive as they need to be signed. When you say you try to generate tokens for 10-20 users are you doing performance tests and having 10-20 threads generating tokens? It shouldn't make any difference if you have 10 or if you have 200 users, it's the total number of tokens that can be generated that's an issue. Having 200 concurrent users with a access token timeout of 60 seconds should mean that you need to be able to generate roughly 200/60 tokens = 3.3 tokens/sec. On 19 May 2016 at 13:24, Vaibhav Naldurgkar < vaibhav_naldurgkar(a)persistent.com&gt; wrote: Hi All, I am using Keycloak 1.9.3 with default configuration. Keycloak server is installed on RHEL 6.5 virtual image with 4 CPU , 8 GB RAM and java version is jdk1.8.0_73 We are trying to use keycloak as a OAuth provider. But when we try and generate token( http:///auth/realms/master/protocol/openid-connect/token <http://auth/realms/master/protocol/openid-connect/token>) for more than 10-20 users the server gets too slow and cpu usage goes over 100%. Any pointers on how to improve performance of keycloak OAuth provider. We need to support at least 200 concurrent users. Thanks, Vaibhav DISCLAIMER ========== This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails. _______________________________________________ keycloak-user mailing list keycloak-user(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/keycloak-user DISCLAIMER ========== This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails. DISCLAIMER ========== This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails. DISCLAIMER ========== This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails. DISCLAIMER ========== This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails. DISCLAIMER ========== This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails. DISCLAIMER ========== This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails.