[keycloak-dev] Performance test

[Marek Posolda]

Hi,

I've added support for JMeter based performance testing into testsuite 
and I did some initial performance tests to compare Picketlink based 
store with MongoDB store. As underlying DB for Picketlink, I used MySQL. 
For now, the testing scenarios I did are those:

1) Test for creating 10 realms. Each test iteration creates one new 
realm and adds some applications, roles and requiredCredentials to it 
(namely 5 applications, 7 roles, 3 roles per each created application)

Results: average 662 ms for Picketlink, 135 ms for Mongo (NOTE: 
Average=average time of one test iteration, which means creating of one 
realm)


2) Test for creating 20000 users. This test is using 10 worker threads. 
Each worker will create 2000 users in one of the realms created in step 
1. Each user is added in separate iteration and also some infos are 
added to this user (Password, default roles of realm and attributes like 
firstName, lastName, email etc)

Results: average 103 ms for Picketlink, 63 ms for Mongo (NOTE: 
Average=average time of one test iteration, which means creating of one 
user and his info. Both Picketlink and Mongo have worse and worse times 
during test progress as it seems that time could depend on total number 
of already existing objects in DB, which is bigger and bigger. 
Unfortunately this difference is more visible for Mongo. I believe that 
more investigation of performance and adding some DB indexes will help. 
I am attaching charts.)


3) Test for reading users. This test is using 10 worker threads. Each 
thread will read realm first and then it will read 5 users from this 
realm (Also roles and password are read for each user). I used 500 
iterations for this test (So 2500 read users for each worker and 25000 
read users in total. DB is prepared with 20000 users from test2, so some 
users are read twice)
Results: average 4586 ms for Picketlink, 518 ms for Mongo (NOTE: 
Average=average time of one test iteration, which means reading of 5 
users and their roles + verifying their password. I am attaching charts)


I will try to do more performance tuning and possibly improve times. Let 
me know if you have comments or ideas how to improve testing. I will 
send PR soon and share my MongoDB based storage implementation and 
performance tests.

Marek
-------------- next part --------------
A non-text attachment was scrubbed...
Name: results-createUsers-mongo-10thread-2000loops-2.png
Type: image/png
Size: 32890 bytes
Desc: not available
Url : http://lists.jboss.org/pipermail/keycloak-dev/attachments/20130918/dc97f0af/attachment-0004.png 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: results-createUsers-picketlink-10thread-2000loops-2.png
Type: image/png
Size: 46248 bytes
Desc: not available
Url : http://lists.jboss.org/pipermail/keycloak-dev/attachments/20130918/dc97f0af/attachment-0005.png 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: results-readUsers-mongo-10thread-500loops-2.png
Type: image/png
Size: 42170 bytes
Desc: not available
Url : http://lists.jboss.org/pipermail/keycloak-dev/attachments/20130918/dc97f0af/attachment-0006.png 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: results-readUsers-picketlink-10thread-500loops-2.png
Type: image/png
Size: 40691 bytes
Desc: not available
Url : http://lists.jboss.org/pipermail/keycloak-dev/attachments/20130918/dc97f0af/attachment-0007.png