3:08 a.m.
I've sent PR https://github.com/keycloak/keycloak/pull/490 with first
prototype of Keycloak performance test with JMeter. For now it's using
embedded Undertow (reused KeycloakServer stuff from testsuite) also with
performance-tools, so it's easy to add new users. Most of the stuff is
configurable through System properties, which allows to easily switch
different databases and models without any needed changes in
configuration files (For example PostgreSQL, MySQL, Mongo,
enabled/disabled simple cache etc), which is also good for easy setup of
performance test in Jenkins to have it in CI. Some details in README:
testsuite/performance-web/README.md .
If you like the approach, I can continue and possibly setup CI. For now,
I've tried some testing on my laptop with 10.000 users in database and
each user having 1 realm role (available through scope to tested web
application) and 0 application roles... (I guess having configurable
number of roles, scopes per user might be useful as well and I can
possibly add it? )
Each test is using 50 clients (each client having different username, so
using users like user-0, user-1, user-49 ) and each client doing 50 test
iterations. Each iteration is doing:
- Redirect to KC login screen
- Login into KC
- Exchange code for token
- Refresh token 2 times
- Logout
Some results from test on my laptop:
PostgreSQL with simple cache disabled:
uri count total min
average max
"Login request" 2500 381151 10 152 1168
"Confirm Login request" 2500 1202384 39 480 1505
"Exchange code request" 2500 1162489 28 464 1677
"RefreshToken request" 5000 1992961 22 398 1630
"Logout request" 2500 786237 21 314 1383
PostgreSQL with simple cache enabled:
uri count total min average
max
"Login request" 2500 168819 6 67 857
"Confirm Login request" 2500 412464 18 164 1205
"Exchange code request" 2500 670047 21 268 1121
"RefreshToken request" 5000 997389 10 199 976
"Logout request" 2500 475672 17 190 1328
Mongo model with simple cache disabled:
uri count total min
average max
"Login request" 2500 171976 6 68 1014
"Confirm Login request" 2500 412258 11 164 981
"Exchange code request" 2500 512125 12 204 1144
"RefreshToken request" 5000 923467 11 184 1073
"Logout request" 2500 367173 10 146 1108
Mongo model with simple cache enabled:
uri count total min average
max
"Login request" 2500 135489 5 54 1085
"Confirm Login request" 2500 224540 7 89 1318
"Exchange code request" 2500 343649 9 137 905
"RefreshToken request" 5000 556202 8 111 750
"Logout request" 2500 292986 10 117 1574
Marek
3:51 a.m.
Merged
One quick question can you easily point the performance tests to an external server? So
that we can test with the WildFly dist rather than embedded Undertow.
----- Original Message -----
From: "Marek Posolda" <mposolda(a)redhat.com>
To: keycloak-dev(a)lists.jboss.org
Sent: Monday, 30 June, 2014 9:08:48 AM
Subject: [keycloak-dev] JMeter Performance test
I've sent PR https://github.com/keycloak/keycloak/pull/490 with first
prototype of Keycloak performance test with JMeter. For now it's using
embedded Undertow (reused KeycloakServer stuff from testsuite) also with
performance-tools, so it's easy to add new users. Most of the stuff is
configurable through System properties, which allows to easily switch
different databases and models without any needed changes in
configuration files (For example PostgreSQL, MySQL, Mongo,
enabled/disabled simple cache etc), which is also good for easy setup of
performance test in Jenkins to have it in CI. Some details in README:
testsuite/performance-web/README.md .
If you like the approach, I can continue and possibly setup CI. For now,
I've tried some testing on my laptop with 10.000 users in database and
each user having 1 realm role (available through scope to tested web
application) and 0 application roles... (I guess having configurable
number of roles, scopes per user might be useful as well and I can
possibly add it? )
Each test is using 50 clients (each client having different username, so
using users like user-0, user-1, user-49 ) and each client doing 50 test
iterations. Each iteration is doing:
- Redirect to KC login screen
- Login into KC
- Exchange code for token
- Refresh token 2 times
- Logout
Some results from test on my laptop:
PostgreSQL with simple cache disabled:
uri count total min
average max
"Login request" 2500 381151 10 152 1168
"Confirm Login request" 2500 1202384 39 480 1505
"Exchange code request" 2500 1162489 28 464 1677
"RefreshToken request" 5000 1992961 22 398 1630
"Logout request" 2500 786237 21 314 1383
PostgreSQL with simple cache enabled:
uri count total min average
max
"Login request" 2500 168819 6 67 857
"Confirm Login request" 2500 412464 18 164 1205
"Exchange code request" 2500 670047 21 268 1121
"RefreshToken request" 5000 997389 10 199 976
"Logout request" 2500 475672 17 190 1328
Mongo model with simple cache disabled:
uri count total min
average max
"Login request" 2500 171976 6 68 1014
"Confirm Login request" 2500 412258 11 164 981
"Exchange code request" 2500 512125 12 204 1144
"RefreshToken request" 5000 923467 11 184 1073
"Logout request" 2500 367173 10 146 1108
Mongo model with simple cache enabled:
uri count total min average
max
"Login request" 2500 135489 5 54 1085
"Confirm Login request" 2500 224540 7 89 1318
"Exchange code request" 2500 343649 9 137 905
"RefreshToken request" 5000 556202 8 111 750
"Logout request" 2500 292986 10 117 1574
Marek
_______________________________________________
keycloak-dev mailing list
keycloak-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/keycloak-dev
4:42 a.m.
On 1.7.2014 10:51, Stian Thorgersen wrote:
Merged
One quick question can you easily point the performance tests to an external server? So
that we can test with the WildFly dist rather than embedded Undertow.
Yes, you can
point it to any server you want. There are just 2 things
though:
1) Performance test is using different users right now (IMO much more
close to "real" environment than using same username in all sessions),
so it needs users to be created before running jmeter test. So
KeycloakPerfServer (embedded undertow) now uses perf-tools and users are
created with it. So the Wildfly/AS7 would also need perf-tools deployed
and users created with that.
2) It uses simple web application for performance testing. So the
application needs to be deployed as well.
I was also thinking about using Keycloak TokenService endpoints
directly,however in the web application there is some pre/post
processing and parsing of tokens etc, which is much easier to do in Java
then in JMeter itself. which allows to check for example that correct
user is logged and so it's easier to verify that performance test is
doing what is is supposed to do and using different user in each session
etc.
Do you think it makes sense to have also test for TokenService endpoints
directly? Was also wondering about adding tests for AccountService as
this might be used by any user, so this stuff can be used quite often?
Not sure about admin console as this won't be hardly used by many users
at the same time (load won't be so big to admin console endpoints).
Marek
----- Original Message -----
> From: "Marek Posolda" <mposolda(a)redhat.com>
> To: keycloak-dev(a)lists.jboss.org
> Sent: Monday, 30 June, 2014 9:08:48 AM
> Subject: [keycloak-dev] JMeter Performance test
>
> I've sent PR https://github.com/keycloak/keycloak/pull/490 with first
> prototype of Keycloak performance test with JMeter. For now it's using
> embedded Undertow (reused KeycloakServer stuff from testsuite) also with
> performance-tools, so it's easy to add new users. Most of the stuff is
> configurable through System properties, which allows to easily switch
> different databases and models without any needed changes in
> configuration files (For example PostgreSQL, MySQL, Mongo,
> enabled/disabled simple cache etc), which is also good for easy setup of
> performance test in Jenkins to have it in CI. Some details in README:
> testsuite/performance-web/README.md .
>
> If you like the approach, I can continue and possibly setup CI. For now,
> I've tried some testing on my laptop with 10.000 users in database and
> each user having 1 realm role (available through scope to tested web
> application) and 0 application roles... (I guess having configurable
> number of roles, scopes per user might be useful as well and I can
> possibly add it? )
>
> Each test is using 50 clients (each client having different username, so
> using users like user-0, user-1, user-49 ) and each client doing 50 test
> iterations. Each iteration is doing:
> - Redirect to KC login screen
> - Login into KC
> - Exchange code for token
> - Refresh token 2 times
> - Logout
>
> Some results from test on my laptop:
>
> PostgreSQL with simple cache disabled:
>
> uri count total min
> average max
> "Login request" 2500 381151 10 152 1168
> "Confirm Login request" 2500 1202384 39 480 1505
> "Exchange code request" 2500 1162489 28 464 1677
> "RefreshToken request" 5000 1992961 22 398 1630
> "Logout request" 2500 786237 21 314 1383
>
>
> PostgreSQL with simple cache enabled:
>
> uri count total min average
> max
> "Login request" 2500 168819 6 67 857
> "Confirm Login request" 2500 412464 18 164 1205
> "Exchange code request" 2500 670047 21 268 1121
> "RefreshToken request" 5000 997389 10 199 976
> "Logout request" 2500 475672 17 190 1328
>
>
> Mongo model with simple cache disabled:
>
> uri count total min
> average max
> "Login request" 2500 171976 6 68 1014
> "Confirm Login request" 2500 412258 11 164 981
> "Exchange code request" 2500 512125 12 204 1144
> "RefreshToken request" 5000 923467 11 184 1073
> "Logout request" 2500 367173 10 146 1108
>
>
> Mongo model with simple cache enabled:
>
> uri count total min average
> max
> "Login request" 2500 135489 5 54 1085
> "Confirm Login request" 2500 224540 7 89 1318
> "Exchange code request" 2500 343649 9 137 905
> "RefreshToken request" 5000 556202 8 111 750
> "Logout request" 2500 292986 10 117 1574
>
> Marek
> _______________________________________________
> keycloak-dev mailing list
> keycloak-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/keycloak-dev
>
3831
days inactive
3832
days old
2 comments
2 participants
participants (2)
-
Marek Posolda -
Stian Thorgersen