Re: [keycloak-user] Keycloak OAuth High CPU usage

Monday, 13 June 2016

Stian,

I noticed that there is an Enhancement added for similar performance issues through 3057
https://issues.jboss.org/browse/KEYCLOAK-3057 . I am looking forward for the release of
2.0.X, could you let me know if this helps to improve performance issue which I am
facing.

Thanks, Vaibhav

From: Stian Thorgersen [mailto:sthorger@redhat.com]
Sent: Thursday, May 26, 2016 11:17 AM
To: Vaibhav Naldurgkar
Cc: keycloak-user(a)lists.jboss.org
Subject: Re: [keycloak-user] Keycloak OAuth High CPU usage

Again, CPU load is expected to be high while having 20 threads send as many requests as
they can. It's the total throughput that matters here.

There are loads of tuning you can do, but you should be able to get decent numbers without
any tuning.

On 26 May 2016 at 07:09, Vaibhav Naldurgkar
<vaibhav_naldurgkar@persistent.com<mailto:vaibhav_naldurgkar@persistent.com>>
wrote:
I still wondering what odd configuration I am following on my RHEL VM which is not
sustaining few user request when checked from the output of top command. Could you please
suggest if there are any Java specific parameters needs to be tuned for performance
improvement. If needed I will share my configuration files for reference.

Below is the screenshot of top output during one of the load test.

[cid:image001.png@01D1C586.773ED0F0]

Thanks, Vaibhav

From: Stian Thorgersen [mailto:sthorger@redhat.com<mailto:sthorger@redhat.com>]
Sent: Wednesday, May 25, 2016 12:40 PM
To: Vaibhav Naldurgkar
Cc: Herzberg, Manuel;
keycloak-user@lists.jboss.org<mailto:keycloak-user@lists.jboss.org>

Subject: Re: [keycloak-user] Keycloak OAuth High CPU usage

I did some tests with Linux VM when investigating how Keycloak scales. I had Keycloak
running on a VM that was permitted 50% of a single core and had a throughput of 50
scenarios. Where a scenario includes a login request, a code to token request and a logout
request. In our performance lab with a single node and a not particularly beefy machine
we're seeing 150+ scenarios/second.

On 24 May 2016 at 16:05, Vaibhav Naldurgkar
<vaibhav_naldurgkar@persistent.com<mailto:vaibhav_naldurgkar@persistent.com>>
wrote:
Hello,

What are the tests results on a Linux VM ? I just done same jmeter tests on AWS m4.xlarge
instance; however far behind than the laptop tests results.
@Stian – have you done tests using Linux VM ?

Thanks, Vaibhav

From: Herzberg, Manuel
[mailto:manuel.herzberg@atos.net<mailto:manuel.herzberg@atos.net>]
Sent: Tuesday, May 24, 2016 5:52 PM
To: stian@redhat.com<mailto:stian@redhat.com>; Vaibhav Naldurgkar
Cc: keycloak-user@lists.jboss.org<mailto:keycloak-user@lists.jboss.org>
Subject: RE: [keycloak-user] Keycloak OAuth High CPU usage

Hello,

I am evaluating the Keycloak performance. Here my practical experience. My scenario is the
same as Vaibhav’s:

•         Large amount of token have to be generated. This is done by requesting the
Keycloak token REST endpoint via http. The different realms I am using have 1k 2k 3k and
4k keys for signing the tokens. (RSA) Longer keys result to longer runtime to generate
these tokens.

•         I have more than 10k user each realm. Each request includes a new user.
Requests look like this:
host1:8080/auth/realms/demo-3072/protocol/openid-connect/token/
with data:
username=testuser1&password=password&client_id=customer-portal&grant_type=password

•         The response includes 3 tokens(access, refresh and id). In total more than 30
000 token have to be generated and signed.

@Stian. You wrote you are able to invoke 10000 token refreshes in under 60 seconds. A
token refresh includes access, refresh and id token right? Can you explain us your
scenario? How do you get such a high number?

Some more results: just signing 3000 Token (800 Byte each) with a 2k key takes me 20
seconds (laptop i5-4310U, 12gb ram). I am doing this outside Keycloak with my own java
program, but with the same implementation Keycloak is using. (sign() method in
RSAProvider).
The Keycloak implementation is signing tokens with RSA. HMAC and ECC are implemented as
well as I saw in the code. Changing from RSA to HMAC or ECC is not possible in current
release as i experienced. Are there plans to provide this in future? Defining this in a
configuration file or via parameters would be nice.
Best regards, Manuel Herzberg

From:
keycloak-user-bounces@lists.jboss.org<mailto:keycloak-user-bounces@lists.jboss.org>
[mailto:keycloak-user-bounces@lists.jboss.org] On Behalf Of Stian Thorgersen
Sent: Tuesday, May 24, 2016 8:31 AM
To: Vaibhav Naldurgkar
Cc: keycloak-user@lists.jboss.org<mailto:keycloak-user@lists.jboss.org>
Subject: Re: [keycloak-user] Keycloak OAuth High CPU usage

On 23 May 2016 at 10:02, Vaibhav Naldurgkar
<vaibhav_naldurgkar@persistent.com<mailto:vaibhav_naldurgkar@persistent.com>>
wrote:
Yes, the direct access grant is ON for this client. I am trying to understand what you
mean by “not planning on using web based flow?” Could you provide more clarification on
this.

If you are planning to do the web based flow (authorization code grant flow) you should
test with that rather than direct grant. That being said the direct grant should still
perform as well.

This is what the scenario I am trying to execute and still have high CPU usages for
KeyCloak Java process.

•         The end point URL /auth/realms/master/protocol/openid-connect/token has been
called by Jmeter for 20 concurrent users per seconds to generate the tokens.

•         Even if used with crul command like “curl -X POST -d
"=admin&password=admin&password&client_id=HelloTest&grant_type=password"
 http://localhost:8080/auth/realms/master/protocol/openid-connect/token” , in this case
also the CPU utilizations goes around 100%.

•          After around 3 seconds of the test, in the output of top command on the
KeyCloak server the CPU% for keycloak java process goes beyond 100%.

Would it be possible for you to have a quick call for faster fix of this issue. This
performance issue is holding to move KeyCloak to use as OAuth provider. If any other way
is convenient for you please let me know for further discussion.

Your JMeter test is using 20 concurrent threads to send as many requests to the direct
grant api as it can. This will obviously cause Keycloak to consume a high percentage of
the CPU. Especially if you are running everything on localhost as the network isn't
going to be a bottleneck. Neither will the database as Keycloak caches everything in
memory. The bottleneck will be the CPU.

Authenticating users and obtaining a token requires password hashing as well as signing
tokens, both are mainly CPU intensive. As you are using the direct grant api there's
also less network traffic.

You need to add some reports to your JMeter test so you can see how many requests Keycloak
can handle. That way you can find out how many users can be authenticated per-second on
your machine.

If you only have 500 users remember they won't all login at the same time (seconds).
Even if they all login at 9am sharp they will be spread out over 10 minutes or so, which
would only be 1.2 logins/second.

Thanks, Vaibhav

From: Stian Thorgersen [mailto:sthorger@redhat.com<mailto:sthorger@redhat.com>]
Sent: Monday, May 23, 2016 12:01 PM

To: Vaibhav Naldurgkar
Cc: keycloak-user@lists.jboss.org<mailto:keycloak-user@lists.jboss.org>
Subject: Re: [keycloak-user] Keycloak OAuth High CPU usage

You are using direct grant to authenticate a user and obtain a token in the example above.
This authenticates and creates a new session for each request. Are you not planning on
using web based flow?

What do you have password hashing intervals set to? Verifying password is CPU intensive,
more than signing tokens.

It shouldn't matter that user is stored in RedHat IdM as the user would be cached in
Keycloak after first authentication, but it may be an idea to just double check by trying
to authenticate to a user in Keycloak and not RH IdM.

What results are you actually getting?

On 20 May 2016 at 11:27, Vaibhav Naldurgkar
<vaibhav_naldurgkar@persistent.com<mailto:vaibhav_naldurgkar@persistent.com>>
wrote:
Hi Stian,

After reading your tests results of 10000 token refreshes in under 60 seconds on your
laptop, I am sure I am not following correct configuration and the documents are missing
for reference.

Could you please verify the below steps along with the screen-shots for the steps which I
am following for the adding client and testing the Load performance using Jmeter. Please
suggest if any changes are needed in the client configuration. In this case we are
obtaining the token for user from KeyCloak.

In my case the user have been stored on RedHat IdM which has been federated using
KeyCloak.

Step 1. Create new client called “LoadTest” , use the Client Protocol as
“Openid-connect”.
Used all defaults values post save of the client action.

Step 2. Start the load tests using Jmeter and using the path as
“/auth/realms/master/protocol/openid-connect/token” . Used 20 Number of Threads and used
Post method.

Below is the screen-shot for the step 1 related to Add Client.

[cid:image002.png@01D1C586.773ED0F0]

Below is the screen shot for the load test using Jmeter. In this case the Client ID was
used as HelloTest.

[cid:image003.png@01D1C586.773ED0F0]

Http requests.

[cid:image004.png@01D1C586.773ED0F0]

Thanks, Vaibhav

From: Stian Thorgersen [mailto:sthorger@redhat.com<mailto:sthorger@redhat.com>]
Sent: Friday, May 20, 2016 1:01 PM

To: Vaibhav Naldurgkar
Cc: keycloak-user@lists.jboss.org<mailto:keycloak-user@lists.jboss.org>
Subject: Re: [keycloak-user] Keycloak OAuth High CPU usage

Can you please elaborate a bit more on how your are testing scenario is? I'm a bit
confused to what you are testing when you are talking about generating new tokens. Are you
using OIDC or SAML? Are you talking about code->token exchanges, refresh token
requests, or what?

To test if your hardware is capable to deal with the load you need to test logins
(verifying passwords are CPU intensive) as well as obtaining tokens (both code->token,
done after login, and refreshing token, done ~1 min or so by active users, but most users
won't continuously use the application).

500 users should be no problem at all. As an example with a single thread (which will use
a single core) I could invoke 10000 token refreshes in under 60 seconds on my laptop. So a
single core on my laptop should be able to handle 500 users.

On 20 May 2016 at 08:00, Vaibhav Naldurgkar
<vaibhav_naldurgkar@persistent.com<mailto:vaibhav_naldurgkar@persistent.com>>
wrote:
Hi Stian,
Thank you for your reply.

The new tokens needs to be generated for each user, which is needed from security point of
view. The performance tests were also conducted using single Admin user and token for
admin user; however in that case the performance was not good. In between 15th to 20th
admin token access requests – the CPU usage of keycloak Java process was crossing 90 to
120% mark.

As you have mentioned, Creating tokes are expected to be a bit CPU intensive – what should
be the server configuration in terms of CPU to deal with more than 500 users to use
keycloak as OAuth provider.

Thanks, Vaibhav

From: Stian Thorgersen [mailto:sthorger@redhat.com<mailto:sthorger@redhat.com>]
Sent: Thursday, May 19, 2016 6:28 PM
To: Vaibhav Naldurgkar
Cc: keycloak-user@lists.jboss.org<mailto:keycloak-user@lists.jboss.org>
Subject: Re: [keycloak-user] Keycloak OAuth High CPU usage

Creating tokes are expected to be a bit CPU intensive as they need to be signed. When you
say you try to generate tokens for 10-20 users are you doing performance tests and having
10-20 threads generating tokens? It shouldn't make any difference if you have 10 or if
you have 200 users, it's the total number of tokens that can be generated that's
an issue. Having 200 concurrent users with a access token timeout of 60 seconds should
mean that you need to be able to generate roughly 200/60 tokens = 3.3 tokens/sec.

On 19 May 2016 at 13:24, Vaibhav Naldurgkar
<vaibhav_naldurgkar@persistent.com<mailto:vaibhav_naldurgkar@persistent.com>>
wrote:
Hi All,

I am using Keycloak 1.9.3 with default configuration. Keycloak server is installed on RHEL
6.5 virtual image with 4 CPU , 8 GB RAM and java version is jdk1.8.0_73 We are trying to
use keycloak as a OAuth provider. But when we try and generate
token(http:///auth/realms/master/protocol/openid-connect/token<http://auth/realms/master/protocol/openid-connect/token>)
for more than 10-20 users the server gets too slow and cpu usage goes over 100%.
Any pointers on how to improve performance of keycloak OAuth provider. We need to support
at least 200 concurrent users.

Thanks, Vaibhav

DISCLAIMER ========== This e-mail may contain privileged and confidential information
which is the property of Persistent Systems Ltd. It is intended only for the use of the
individual or entity to which it is addressed. If you are not the intended recipient, you
are not authorized to read, retain, copy, print, distribute or use this message. If you
have received this communication in error, please notify the sender and delete all copies
of this message. Persistent Systems Ltd. does not accept any liability for virus infected
mails.

_______________________________________________
keycloak-user mailing list
keycloak-user@lists.jboss.org<mailto:keycloak-user@lists.jboss.org>
https://lists.jboss.org/mailman/listinfo/keycloak-user

DISCLAIMER ========== This e-mail may contain privileged and confidential information
which is the property of Persistent Systems Ltd. It is intended only for the use of the
individual or entity to which it is addressed. If you are not the intended recipient, you
are not authorized to read, retain, copy, print, distribute or use this message. If you
have received this communication in error, please notify the sender and delete all copies
of this message. Persistent Systems Ltd. does not accept any liability for virus infected
mails.

DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the property of
Persistent Systems Ltd. It is intended only for the use of the individual or entity to
which it is addressed. If you are not the intended recipient, you are not authorized to
read, retain, copy, print, distribute or use this message. If you have received this
communication in error, please notify the sender and delete all copies of this message.
Persistent Systems Ltd. does not accept any liability for virus infected mails.

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

Re: [keycloak-user] Keycloak OAuth High CPU usage