[jboss-jira] [JBoss JIRA] (AS7-2903) AS 7.1 Clustering performs 10x slower than AS 5.x
Paul Ferraro (JIRA)
jira-events at lists.jboss.org
Tue Jan 17 13:09:18 EST 2012
[ https://issues.jboss.org/browse/AS7-2903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12659922#comment-12659922 ]
Paul Ferraro commented on AS7-2903:
-----------------------------------
FYI, the 5.1.0.CR4 upgrade should be getting pulled into master momentarily:
https://github.com/jbossas/jboss-as/pull/1129
> AS 7.1 Clustering performs 10x slower than AS 5.x
> -------------------------------------------------
>
> Key: AS7-2903
> URL: https://issues.jboss.org/browse/AS7-2903
> Project: Application Server 7
> Issue Type: Bug
> Components: Clustering
> Affects Versions: No Release, 7.1.0.Beta1
> Environment: 2b3ea9b6bc31e8b72d3e2a5aac6176067ef9da5e
> Reporter: Radoslav Husar
> Assignee: Paul Ferraro
> Priority: Blocker
> Fix For: 7.1.0.Final
>
>
> Update:
> This was a suspicion first, now I have compared results for AS5. *Clustering performs 10x slower* with the same use case in 2 node cluster*. Also this has been tested on revision with updated Infinispan to CR1. CPU usage seems relatively high, network is not used to its maximum, most likely looks like blocking in threading.
> AS7
> {noformat}
> Nodes: 2, Sessions: 100, active: 100, samples: 8363, throughput 418.1 samples/s, 0.0 MB/s, mean response: 138 ms, sampling errors: 0, invalid samples: 0, valid samples: 8363 (100%)
> Nodes: 2, Sessions: 200, active: 200, samples: 9607, throughput 480.3 samples/s, 0.0 MB/s, mean response: 298 ms, sampling errors: 0, invalid samples: 0, valid samples: 9607 (100%)
> Nodes: 2, Sessions: 300, active: 300, samples: 12779, throughput 555.6 samples/s, 0.0 MB/s, mean response: 550 ms, sampling errors: 0, invalid samples: 151, valid samples: 12628 (98%)
> Nodes: 2, Sessions: 400, active: 400, samples: 13336, throughput 666.7 samples/s, 0.0 MB/s, mean response: 498 ms, sampling errors: 0, invalid samples: 1, valid samples: 13335 (99%)
> Nodes: 2, Sessions: 500, active: 500, samples: 11852, throughput 592.5 samples/s, 0.0 MB/s, mean response: 441 ms, sampling errors: 0, invalid samples: 1, valid samples: 11851 (99%)
> Nodes: 2, Sessions: 600, active: 600, samples: 16012, throughput 800.5 samples/s, 0.0 MB/s, mean response: 634 ms, sampling errors: 0, invalid samples: 0, valid samples: 16012 (100%)
> Nodes: 2, Sessions: 700, active: 700, samples: 14436, throughput 721.7 samples/s, 0.0 MB/s, mean response: 888 ms, sampling errors: 0, invalid samples: 350, valid samples: 14086 (97%)
> Nodes: 2, Sessions: 800, active: 800, samples: 14479, throughput 629.5 samples/s, 0.0 MB/s, mean response: 1080 ms, sampling errors: 0, invalid samples: 0, valid samples: 14479 (100%)
> Nodes: 2, Sessions: 900, active: 900, samples: 14473, throughput 723.6 samples/s, 0.0 MB/s, mean response: 1345 ms, sampling errors: 0, invalid samples: 450, valid samples: 14023 (96%)
> Nodes: 2, Sessions: 1000, active: 1000, samples: 14570, throughput 728.4 samples/s, 0.0 MB/s, mean response: 1274 ms, sampling errors: 0, invalid samples: 1, valid samples: 14569 (99%)
> Nodes: 2, Sessions: 1100, active: 1100, samples: 13808, throughput 690.3 samples/s, 0.0 MB/s, mean response: 1933 ms, sampling errors: 0, invalid samples: 1100, valid samples: 12708 (92%)
> Nodes: 2, Sessions: 1200, active: 1200, samples: 13252, throughput 662.5 samples/s, 0.0 MB/s, mean response: 1833 ms, sampling errors: 0, invalid samples: 600, valid samples: 12652 (95%)
> Nodes: 2, Sessions: 1300, active: 1300, samples: 17815, throughput 754.8 samples/s, 0.0 MB/s, mean response: 1581 ms, sampling errors: 0, invalid samples: 1300, valid samples: 16515 (92%)
> Nodes: 2, Sessions: 1400, active: 1400, samples: 17558, throughput 877.8 samples/s, 0.0 MB/s, mean response: 1509 ms, sampling errors: 0, invalid samples: 1, valid samples: 17557 (99%)
> {noformat}
> AS 5
> {noformat}
> Nodes: 2, Sessions: 100, active: 100, samples: 19240, throughput 961.2 samples/s, 0.0 MB/s, mean response: 4 ms, sampling errors: 0, invalid samples: 0, valid samples: 19240 (100%)
> Nodes: 2, Sessions: 200, active: 200, samples: 37887, throughput 1,894.1 samples/s, 0.0 MB/s, mean response: 5 ms, sampling errors: 0, invalid samples: 0, valid samples: 37887 (100%)
> Nodes: 2, Sessions: 300, active: 300, samples: 69732, throughput 2,843.5 samples/s, 0.0 MB/s, mean response: 4 ms, sampling errors: 0, invalid samples: 0, valid samples: 69732 (100%)
> Nodes: 2, Sessions: 400, active: 400, samples: 75148, throughput 3,755.1 samples/s, 0.0 MB/s, mean response: 6 ms, sampling errors: 0, invalid samples: 0, valid samples: 75148 (100%)
> Nodes: 2, Sessions: 500, active: 500, samples: 92979, throughput 4,648.4 samples/s, 0.0 MB/s, mean response: 6 ms, sampling errors: 0, invalid samples: 3, valid samples: 92976 (99%)
> Nodes: 2, Sessions: 600, active: 600, samples: 108989, throughput 5,449.0 samples/s, 0.0 MB/s, mean response: 10 ms, sampling errors: 0, invalid samples: 1, valid samples: 108988 (99%)
> Nodes: 2, Sessions: 700, active: 700, samples: 112413, throughput 5,508.6 samples/s, 0.0 MB/s, mean response: 26 ms, sampling errors: 0, invalid samples: 0, valid samples: 112413 (100%)
> Nodes: 2, Sessions: 800, active: 800, samples: 88810, throughput 4,435.9 samples/s, 0.0 MB/s, mean response: 78 ms, sampling errors: 0, invalid samples: 0, valid samples: 88810 (100%)
> Nodes: 2, Sessions: 900, active: 900, samples: 63417, throughput 3,170.6 samples/s, 0.0 MB/s, mean response: 188 ms, sampling errors: 0, invalid samples: 487, valid samples: 62930 (99%)
> Nodes: 2, Sessions: 1000, active: 1000, samples: 85718, throughput 4,285.4 samples/s, 0.0 MB/s, mean response: 129 ms, sampling errors: 0, invalid samples: 0, valid samples: 85718 (100%)
> Nodes: 2, Sessions: 1100, active: 1100, samples: 67710, throughput 3,385.1 samples/s, 0.0 MB/s, mean response: 232 ms, sampling errors: 0, invalid samples: 0, valid samples: 67710 (100%)
> Nodes: 2, Sessions: 1200, active: 1200, samples: 88369, throughput 4,417.9 samples/s, 0.0 MB/s, mean response: 165 ms, sampling errors: 0, invalid samples: 2, valid samples: 88367 (99%)
> Nodes: 2, Sessions: 1300, active: 1300, samples: 113719, throughput 5,685.4 samples/s, 0.0 MB/s, mean response: 124 ms, sampling errors: 0, invalid samples: 0, valid samples: 113719 (100%)
> Nodes: 2, Sessions: 1400, active: 1400, samples: 119857, throughput 5,210.8 samples/s, 0.0 MB/s, mean response: 163 ms, sampling errors: 0, invalid samples: 0, valid samples: 119857 (100%)
> Nodes: 2, Sessions: 1500, active: 1500, samples: 102037, throughput 5,099.0 samples/s, 0.0 MB/s, mean response: 197 ms, sampling errors: 0, invalid samples: 0, valid samples: 102037 (100%)
> Nodes: 2, Sessions: 1600, active: 1600, samples: 175482, throughput 4,535.1 samples/s, 0.0 MB/s, mean response: 244 ms, sampling errors: 0, invalid samples: 0, valid samples: 175482 (100%)
> Nodes: 2, Sessions: 1700, active: 1700, samples: 110511, throughput 5,524.7 samples/s, 0.0 MB/s, mean response: 236 ms, sampling errors: 0, invalid samples: 1992, valid samples: 108519 (98%)
> Nodes: 2, Sessions: 1800, active: 1800, samples: 120824, throughput 6,009.3 samples/s, 0.0 MB/s, mean response: 201 ms, sampling errors: 0, invalid samples: 0, valid samples: 120824 (100%)
> Nodes: 2, Sessions: 1900, active: 1900, samples: 159295, throughput 7,875.4 samples/s, 0.0 MB/s, mean response: 140 ms, sampling errors: 0, invalid samples: 0, valid samples: 159295 (100%)
> Nodes: 2, Sessions: 2000, active: 2000, samples: 75813, throughput 3,769.3 samples/s, 0.0 MB/s, mean response: 430 ms, sampling errors: 0, invalid samples: 2000, valid samples: 73813 (97%)
> Nodes: 2, Sessions: 2100, active: 2100, samples: 134139, throughput 5,831.5 samples/s, 0.0 MB/s, mean response: 252 ms, sampling errors: 0, invalid samples: 0, valid samples: 134139 (100%)
> {noformat}
> ----------------------------------------------------------------------------
> This was suspected.
> Summary: Cluster suspected to perform 100x slower than unclustered
> Cluster performs 100x slower than unclustered when using HTTP session replication benchmark in our perf lab. Throughput with 200 clients drops from 49,851.0 r/s to 425.3 samples/s when using clustering in standalone-ha.xml server configuration. I cannot guarantee the runs, this might be a result of a hidden networking issue in the lab.
> Here are some sample runs:
> *Unclustered*
> {noformat}
> Nodes: 2, Sessions: 100, active: 100, samples: 680317, throughput 45,346.9 samples/s, 0.2 MB/s, mean response: 2 ms, sampling errors: 0, invalid samples: 0, valid samples: 680317 (100%)
> Nodes: 2, Sessions: 200, active: 200, samples: 747890, throughput 49,851.0 samples/s, 0.2 MB/s, mean response: 3 ms, sampling errors: 0, invalid samples: 0, valid samples: 747890 (100%)
> Nodes: 2, Sessions: 300, active: 300, samples: 722338, throughput 48,149.4 samples/s, 0.2 MB/s, mean response: 6 ms, sampling errors: 0, invalid samples: 0, valid samples: 722338 (100%)
> Nodes: 2, Sessions: 400, active: 400, samples: 702323, throughput 46,813.7 samples/s, 0.2 MB/s, mean response: 8 ms, sampling errors: 0, invalid samples: 0, valid samples: 702323 (100%)
> Nodes: 2, Sessions: 500, active: 500, samples: 690571, throughput 46,028.9 samples/s, 0.2 MB/s, mean response: 10 ms, sampling errors: 0, invalid samples: 0, valid samples: 690571 (100%)
> Nodes: 2, Sessions: 600, active: 600, samples: 689405, throughput 45,949.6 samples/s, 0.2 MB/s, mean response: 13 ms, sampling errors: 0, invalid samples: 0, valid samples: 689405 (100%)
> Nodes: 2, Sessions: 700, active: 700, samples: 681118, throughput 45,400.3 samples/s, 0.2 MB/s, mean response: 15 ms, sampling errors: 0, invalid samples: 0, valid samples: 681118 (100%)
> Nodes: 2, Sessions: 800, active: 800, samples: 685431, throughput 45,684.7 samples/s, 0.2 MB/s, mean response: 16 ms, sampling errors: 0, invalid samples: 0, valid samples: 685431 (100%)
> Nodes: 2, Sessions: 900, active: 900, samples: 668869, throughput 44,580.9 samples/s, 0.2 MB/s, mean response: 20 ms, sampling errors: 0, invalid samples: 0, valid samples: 668869 (100%)
> Nodes: 2, Sessions: 1000, active: 1000, samples: 675849, throughput 45,046.1 samples/s, 0.2 MB/s, mean response: 21 ms, sampling errors: 0, invalid samples: 0, valid samples: 675849 (100%)
> {noformat}
> *Clustered*
> {noformat}
> Nodes: 2, Sessions: 100, active: 100, samples: 7350, throughput 489.9 samples/s, 0.0 MB/s, mean response: 196 ms, sampling errors: 0, invalid samples: 49, valid samples: 7301 (99%)
> Nodes: 2, Sessions: 200, active: 200, samples: 6380, throughput 425.3 samples/s, 0.0 MB/s, mean response: 458 ms, sampling errors: 0, invalid samples: 362, valid samples: 6018 (94%)
> Nodes: 2, Sessions: 300, active: 300, samples: 5821, throughput 382.4 samples/s, 0.0 MB/s, mean response: 748 ms, sampling errors: 0, invalid samples: 484, valid samples: 5337 (91%)
> Nodes: 2, Sessions: 400, active: 262, samples: 3846, throughput 256.3 samples/s, 0.0 MB/s, mean response: 1056 ms, sampling errors: 0, invalid samples: 333, valid samples: 3513 (91%)
> Nodes: 2, Sessions: 500, active: 452, samples: 6389, throughput 425.6 samples/s, 0.0 MB/s, mean response: 1104 ms, sampling errors: 0, invalid samples: 565, valid samples: 5824 (91%)
> Nodes: 2, Sessions: 600, active: 55, samples: 985, throughput 65.7 samples/s, 0.0 MB/s, mean response: 1590 ms, sampling errors: 457, invalid samples: 55, valid samples: 473 (48%)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
More information about the jboss-jira
mailing list