[seam-dev] Initial performance findings and baselines

[Jay Balunas]

Hello All,

I have been looking at Seam 2.1 performance and begun profiling the wiki
example.  Below are my initial findings.  More will follow but I wanted to
provide some base information that we look at and compare with changes that
we make.

Environment
------------------------------
All tests were executed on my laptop (specs provide below).

* Lenovo T61
  - 2GB Ram
  - Intel(R) Core(TM)2 Duo CPU     T7500  @ 2.20GHz
  - 7200RPM 200GB Hard drive
  - Fedora 8 with all updates as of 9-22-09
  - MySQL 5.0.45
  - Sun jdk1.5.0_14
  - JBoss AS 4.2.3.GA

* Test tools
  - jmeter 2.3.2
  - jprofiler 5.12

Test Description
-----------------------------
* Tests were all done with the wiki example
  - svn trunk (rev 9017)
  - deployed following the production deployment instructions
    - plus addition mysql instructions from Christian
  - MySQL primed with the live seamframework.org data as of July 30th.
* Between all tests
  - AS server and test applications were restarted
  - 3 page hits were conducted prior to load tests to prime system
* The tests described here were all focused on the first page of the users
forum
  - http://localhost:8080/Community/SeamUsers

Multi User Load Test Results
----------------------------
* These tests were primarily to provide a base line of performance for
comparison.
  - profiling was not done as part of these tests
  - result times are averages of the runs completed

JBoss AS 4.2.3 as is
---------------------
25 users x 25 requests each
 - Average request time 8.85 sec
 - Median request time   8.22 sec
 - Throughput                 131.4 per min
 - notes:
    - server.log was 1.2 GB
    - See jboss_423_base_25u.png

50 users x 25 requests each
 - Average request time 22.72 sec
 - Median request time   18.33 sec
 - Throughput                 96 per min
 - notes:
    - server.log is 2.4 GB
    - this pushed my machine to its limit
    - the final 10-15 threads would appear to freeze
       - but would finish fine (except for 1 of the runs)
    - See jboss_423_base_50u.png

Notes: Obviously the biggest issue here was logging I reduced all logging to
INFO by modifying the jboss-log4j.xml file and setting root logging level
and adjusting one other category.  The results were dramatic as seen below.

JBoss AS 4.2.3 with reduced logging
------------------------------------
25 users x 25 requests each
 - Average request time 3.08 sec
 - Median request time   2.94 sec
 - Throughput                 283.5 per min
 - notes:
    - server.log was 87.6 KB
    - See jboss_423_rd_log_25u.png

50 users x 25 requests each
 - Average request time 10.10 sec
 - Median request time   9.30 sec
 - Throughput                 237.6 per min
 - notes:
    - server.log is 96 KB
    - all thread completed correctly and without issue.
    - See jboss_423_rd_log_50u.png

1 user x 1000 requests
 - wanted to get sample with just one user and many requests
 - Average request time 264 ms
 - Median request time   255 ms
 - Throughput                 255.5 per min
 - notes:
    - server.log is 79 KB
    - Very tight deviation only 42 ms as expected

Notes: Obviously this is the configuration that I will use for the profiling
tests to follow, and for comparison runs based on changes made to the code.


I have begun profiling runs of the 25 user tests through JProfiler to
identify where threads are blocking and tracking down areas for
improvements.  These tests with the profiler truly push my machine to the
limit, but I am see some results.  I will be writing a follow up email with
at least some of that information soon - possibly today.

Regards,
Jay

-- 
blog: http://in.relation.to/Bloggers/Jay
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/seam-dev/attachments/20080924/a1dbc257/attachment.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: jboss_423_base_25u.png
Type: image/png
Size: 33337 bytes
Desc: not available
Url : http://lists.jboss.org/pipermail/seam-dev/attachments/20080924/a1dbc257/attachment.png 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: jboss_423_base_50u.png
Type: image/png
Size: 37783 bytes
Desc: not available
Url : http://lists.jboss.org/pipermail/seam-dev/attachments/20080924/a1dbc257/attachment-0001.png 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: jboss_423_rd_log_25u.png
Type: image/png
Size: 33730 bytes
Desc: not available
Url : http://lists.jboss.org/pipermail/seam-dev/attachments/20080924/a1dbc257/attachment-0002.png 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: jboss_423_rd_log_50u.png
Type: image/png
Size: 38277 bytes
Desc: not available
Url : http://lists.jboss.org/pipermail/seam-dev/attachments/20080924/a1dbc257/attachment-0003.png