[ https://issues.jboss.org/browse/AS7-2903?page=com.atlassian.jira.plugin.s... ] Radoslav Husar commented on AS7-2903: ------------------------------------- Hi Scott, you are more than welcome to comment on the issue and chip in. Let me give you more perspective. {quote}I'm curious as to why 100x slower. Do you have a similar (unclustered versus clustered) comparison for throughput numbers for a previous AS?{quote} Its comparing clustered to unclustered that's why the number will be relatively big anyway, but 100x does not seem right in either case, thus the issue. I will have the numbers shortly and provide more feedback on this. However it is important to know about these issues ASAP since there are multiple teams/projects participating in the results (mainly JGroups, Infinispan and clustering layer and others in AS7 ). {quote}Are there errors in the server.log, that aren't there when unclustered?{quote} There were no errors in the log. But please note that there are more issues in clustering from the functional side, I am keeping a list here JBPAPP-7577. {quote}We did see the "pause" effect in earlier releases, as we made AS faster.{quote} You are probably referring to issue discovered with clustered Seam application where all of us were participating, yes I do remember. {quote}unclustered as5 throughput to the AS7 unclustered throughput. {quote} Agreed, on the way but its taking its time. {quote}The FC settings can be jacked up to the sky,{quote} Currently its important to look at the default, we can get to tuning as we go and make them the default. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/AS7-2903?page=com.atlassian.jira.plugin.s... ] Radoslav Husar commented on AS7-2903: ------------------------------------- Also note there is a known issue with Infinispan 'Brahma' 5.1.0.BETA5 used directly in AS7 that there is a 75% performance degradation. See https://bugzilla.redhat.com/show_bug.cgi?id=758759 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/AS7-2903?page=com.atlassian.jira.plugin.s... ] Radoslav Husar updated AS7-2903: -------------------------------- Summary: AS 7.1 Clustering performs 10x slower than AS 5.x (was: Cluster suspected to perform 100x slower than unclustered) Affects Version/s: 7.1.0.Beta1 Environment: 2b3ea9b6bc31e8b72d3e2a5aac6176067ef9da5e (was: eaebc8a0041e2c7bd9b7de93ea2d2bf87701abff) Priority: Blocker (was: Critical) Description: Update: This was a suspicion first, now I have compared results for AS5. *Clustering performs 10x slower* with the same use case in 2 node cluster*. Also this has been tested on revision with updated Infinispan to CR1. CPU usage seems relatively high, network is not used to its maximum, most likely looks like blocking in threading. AS7 {noformat} Nodes: 2, Sessions: 100, active: 100, samples: 8363, throughput 418.1 samples/s, 0.0 MB/s, mean response: 138 ms, sampling errors: 0, invalid samples: 0, valid samples: 8363 (100%) Nodes: 2, Sessions: 200, active: 200, samples: 9607, throughput 480.3 samples/s, 0.0 MB/s, mean response: 298 ms, sampling errors: 0, invalid samples: 0, valid samples: 9607 (100%) Nodes: 2, Sessions: 300, active: 300, samples: 12779, throughput 555.6 samples/s, 0.0 MB/s, mean response: 550 ms, sampling errors: 0, invalid samples: 151, valid samples: 12628 (98%) Nodes: 2, Sessions: 400, active: 400, samples: 13336, throughput 666.7 samples/s, 0.0 MB/s, mean response: 498 ms, sampling errors: 0, invalid samples: 1, valid samples: 13335 (99%) Nodes: 2, Sessions: 500, active: 500, samples: 11852, throughput 592.5 samples/s, 0.0 MB/s, mean response: 441 ms, sampling errors: 0, invalid samples: 1, valid samples: 11851 (99%) Nodes: 2, Sessions: 600, active: 600, samples: 16012, throughput 800.5 samples/s, 0.0 MB/s, mean response: 634 ms, sampling errors: 0, invalid samples: 0, valid samples: 16012 (100%) Nodes: 2, Sessions: 700, active: 700, samples: 14436, throughput 721.7 samples/s, 0.0 MB/s, mean response: 888 ms, sampling errors: 0, invalid samples: 350, valid samples: 14086 (97%) Nodes: 2, Sessions: 800, active: 800, samples: 14479, throughput 629.5 samples/s, 0.0 MB/s, mean response: 1080 ms, sampling errors: 0, invalid samples: 0, valid samples: 14479 (100%) Nodes: 2, Sessions: 900, active: 900, samples: 14473, throughput 723.6 samples/s, 0.0 MB/s, mean response: 1345 ms, sampling errors: 0, invalid samples: 450, valid samples: 14023 (96%) Nodes: 2, Sessions: 1000, active: 1000, samples: 14570, throughput 728.4 samples/s, 0.0 MB/s, mean response: 1274 ms, sampling errors: 0, invalid samples: 1, valid samples: 14569 (99%) Nodes: 2, Sessions: 1100, active: 1100, samples: 13808, throughput 690.3 samples/s, 0.0 MB/s, mean response: 1933 ms, sampling errors: 0, invalid samples: 1100, valid samples: 12708 (92%) Nodes: 2, Sessions: 1200, active: 1200, samples: 13252, throughput 662.5 samples/s, 0.0 MB/s, mean response: 1833 ms, sampling errors: 0, invalid samples: 600, valid samples: 12652 (95%) Nodes: 2, Sessions: 1300, active: 1300, samples: 17815, throughput 754.8 samples/s, 0.0 MB/s, mean response: 1581 ms, sampling errors: 0, invalid samples: 1300, valid samples: 16515 (92%) Nodes: 2, Sessions: 1400, active: 1400, samples: 17558, throughput 877.8 samples/s, 0.0 MB/s, mean response: 1509 ms, sampling errors: 0, invalid samples: 1, valid samples: 17557 (99%) {noformat} AS 5 {noformat} Nodes: 2, Sessions: 100, active: 100, samples: 19240, throughput 961.2 samples/s, 0.0 MB/s, mean response: 4 ms, sampling errors: 0, invalid samples: 0, valid samples: 19240 (100%) Nodes: 2, Sessions: 200, active: 200, samples: 37887, throughput 1,894.1 samples/s, 0.0 MB/s, mean response: 5 ms, sampling errors: 0, invalid samples: 0, valid samples: 37887 (100%) Nodes: 2, Sessions: 300, active: 300, samples: 69732, throughput 2,843.5 samples/s, 0.0 MB/s, mean response: 4 ms, sampling errors: 0, invalid samples: 0, valid samples: 69732 (100%) Nodes: 2, Sessions: 400, active: 400, samples: 75148, throughput 3,755.1 samples/s, 0.0 MB/s, mean response: 6 ms, sampling errors: 0, invalid samples: 0, valid samples: 75148 (100%) Nodes: 2, Sessions: 500, active: 500, samples: 92979, throughput 4,648.4 samples/s, 0.0 MB/s, mean response: 6 ms, sampling errors: 0, invalid samples: 3, valid samples: 92976 (99%) Nodes: 2, Sessions: 600, active: 600, samples: 108989, throughput 5,449.0 samples/s, 0.0 MB/s, mean response: 10 ms, sampling errors: 0, invalid samples: 1, valid samples: 108988 (99%) Nodes: 2, Sessions: 700, active: 700, samples: 112413, throughput 5,508.6 samples/s, 0.0 MB/s, mean response: 26 ms, sampling errors: 0, invalid samples: 0, valid samples: 112413 (100%) Nodes: 2, Sessions: 800, active: 800, samples: 88810, throughput 4,435.9 samples/s, 0.0 MB/s, mean response: 78 ms, sampling errors: 0, invalid samples: 0, valid samples: 88810 (100%) Nodes: 2, Sessions: 900, active: 900, samples: 63417, throughput 3,170.6 samples/s, 0.0 MB/s, mean response: 188 ms, sampling errors: 0, invalid samples: 487, valid samples: 62930 (99%) Nodes: 2, Sessions: 1000, active: 1000, samples: 85718, throughput 4,285.4 samples/s, 0.0 MB/s, mean response: 129 ms, sampling errors: 0, invalid samples: 0, valid samples: 85718 (100%) Nodes: 2, Sessions: 1100, active: 1100, samples: 67710, throughput 3,385.1 samples/s, 0.0 MB/s, mean response: 232 ms, sampling errors: 0, invalid samples: 0, valid samples: 67710 (100%) Nodes: 2, Sessions: 1200, active: 1200, samples: 88369, throughput 4,417.9 samples/s, 0.0 MB/s, mean response: 165 ms, sampling errors: 0, invalid samples: 2, valid samples: 88367 (99%) Nodes: 2, Sessions: 1300, active: 1300, samples: 113719, throughput 5,685.4 samples/s, 0.0 MB/s, mean response: 124 ms, sampling errors: 0, invalid samples: 0, valid samples: 113719 (100%) Nodes: 2, Sessions: 1400, active: 1400, samples: 119857, throughput 5,210.8 samples/s, 0.0 MB/s, mean response: 163 ms, sampling errors: 0, invalid samples: 0, valid samples: 119857 (100%) Nodes: 2, Sessions: 1500, active: 1500, samples: 102037, throughput 5,099.0 samples/s, 0.0 MB/s, mean response: 197 ms, sampling errors: 0, invalid samples: 0, valid samples: 102037 (100%) Nodes: 2, Sessions: 1600, active: 1600, samples: 175482, throughput 4,535.1 samples/s, 0.0 MB/s, mean response: 244 ms, sampling errors: 0, invalid samples: 0, valid samples: 175482 (100%) Nodes: 2, Sessions: 1700, active: 1700, samples: 110511, throughput 5,524.7 samples/s, 0.0 MB/s, mean response: 236 ms, sampling errors: 0, invalid samples: 1992, valid samples: 108519 (98%) Nodes: 2, Sessions: 1800, active: 1800, samples: 120824, throughput 6,009.3 samples/s, 0.0 MB/s, mean response: 201 ms, sampling errors: 0, invalid samples: 0, valid samples: 120824 (100%) Nodes: 2, Sessions: 1900, active: 1900, samples: 159295, throughput 7,875.4 samples/s, 0.0 MB/s, mean response: 140 ms, sampling errors: 0, invalid samples: 0, valid samples: 159295 (100%) Nodes: 2, Sessions: 2000, active: 2000, samples: 75813, throughput 3,769.3 samples/s, 0.0 MB/s, mean response: 430 ms, sampling errors: 0, invalid samples: 2000, valid samples: 73813 (97%) Nodes: 2, Sessions: 2100, active: 2100, samples: 134139, throughput 5,831.5 samples/s, 0.0 MB/s, mean response: 252 ms, sampling errors: 0, invalid samples: 0, valid samples: 134139 (100%) {noformat} ---------------------------------------------------------------------------- This was suspected. Summary: Cluster suspected to perform 100x slower than unclustered Cluster performs 100x slower than unclustered when using HTTP session replication benchmark in our perf lab. Throughput with 200 clients drops from 49,851.0 r/s to 425.3 samples/s when using clustering in standalone-ha.xml server configuration. I cannot guarantee the runs, this might be a result of a hidden networking issue in the lab. Here are some sample runs: *Unclustered* {noformat} Nodes: 2, Sessions: 100, active: 100, samples: 680317, throughput 45,346.9 samples/s, 0.2 MB/s, mean response: 2 ms, sampling errors: 0, invalid samples: 0, valid samples: 680317 (100%) Nodes: 2, Sessions: 200, active: 200, samples: 747890, throughput 49,851.0 samples/s, 0.2 MB/s, mean response: 3 ms, sampling errors: 0, invalid samples: 0, valid samples: 747890 (100%) Nodes: 2, Sessions: 300, active: 300, samples: 722338, throughput 48,149.4 samples/s, 0.2 MB/s, mean response: 6 ms, sampling errors: 0, invalid samples: 0, valid samples: 722338 (100%) Nodes: 2, Sessions: 400, active: 400, samples: 702323, throughput 46,813.7 samples/s, 0.2 MB/s, mean response: 8 ms, sampling errors: 0, invalid samples: 0, valid samples: 702323 (100%) Nodes: 2, Sessions: 500, active: 500, samples: 690571, throughput 46,028.9 samples/s, 0.2 MB/s, mean response: 10 ms, sampling errors: 0, invalid samples: 0, valid samples: 690571 (100%) Nodes: 2, Sessions: 600, active: 600, samples: 689405, throughput 45,949.6 samples/s, 0.2 MB/s, mean response: 13 ms, sampling errors: 0, invalid samples: 0, valid samples: 689405 (100%) Nodes: 2, Sessions: 700, active: 700, samples: 681118, throughput 45,400.3 samples/s, 0.2 MB/s, mean response: 15 ms, sampling errors: 0, invalid samples: 0, valid samples: 681118 (100%) Nodes: 2, Sessions: 800, active: 800, samples: 685431, throughput 45,684.7 samples/s, 0.2 MB/s, mean response: 16 ms, sampling errors: 0, invalid samples: 0, valid samples: 685431 (100%) Nodes: 2, Sessions: 900, active: 900, samples: 668869, throughput 44,580.9 samples/s, 0.2 MB/s, mean response: 20 ms, sampling errors: 0, invalid samples: 0, valid samples: 668869 (100%) Nodes: 2, Sessions: 1000, active: 1000, samples: 675849, throughput 45,046.1 samples/s, 0.2 MB/s, mean response: 21 ms, sampling errors: 0, invalid samples: 0, valid samples: 675849 (100%) {noformat} *Clustered* {noformat} Nodes: 2, Sessions: 100, active: 100, samples: 7350, throughput 489.9 samples/s, 0.0 MB/s, mean response: 196 ms, sampling errors: 0, invalid samples: 49, valid samples: 7301 (99%) Nodes: 2, Sessions: 200, active: 200, samples: 6380, throughput 425.3 samples/s, 0.0 MB/s, mean response: 458 ms, sampling errors: 0, invalid samples: 362, valid samples: 6018 (94%) Nodes: 2, Sessions: 300, active: 300, samples: 5821, throughput 382.4 samples/s, 0.0 MB/s, mean response: 748 ms, sampling errors: 0, invalid samples: 484, valid samples: 5337 (91%) Nodes: 2, Sessions: 400, active: 262, samples: 3846, throughput 256.3 samples/s, 0.0 MB/s, mean response: 1056 ms, sampling errors: 0, invalid samples: 333, valid samples: 3513 (91%) Nodes: 2, Sessions: 500, active: 452, samples: 6389, throughput 425.6 samples/s, 0.0 MB/s, mean response: 1104 ms, sampling errors: 0, invalid samples: 565, valid samples: 5824 (91%) Nodes: 2, Sessions: 600, active: 55, samples: 985, throughput 65.7 samples/s, 0.0 MB/s, mean response: 1590 ms, sampling errors: 457, invalid samples: 55, valid samples: 473 (48%) {noformat} was: Cluster performs 100x slower than unclustered when using HTTP session replication benchmark in our perf lab. Throughput with 200 clients drops from 49,851.0 r/s to 425.3 samples/s when using clustering in standalone-ha.xml server configuration. I cannot guarantee the runs, this might be a result of a hidden networking issue in the lab. Here are some sample runs: *Unclustered* {noformat} Nodes: 2, Sessions: 100, active: 100, samples: 680317, throughput 45,346.9 samples/s, 0.2 MB/s, mean response: 2 ms, sampling errors: 0, invalid samples: 0, valid samples: 680317 (100%) Nodes: 2, Sessions: 200, active: 200, samples: 747890, throughput 49,851.0 samples/s, 0.2 MB/s, mean response: 3 ms, sampling errors: 0, invalid samples: 0, valid samples: 747890 (100%) Nodes: 2, Sessions: 300, active: 300, samples: 722338, throughput 48,149.4 samples/s, 0.2 MB/s, mean response: 6 ms, sampling errors: 0, invalid samples: 0, valid samples: 722338 (100%) Nodes: 2, Sessions: 400, active: 400, samples: 702323, throughput 46,813.7 samples/s, 0.2 MB/s, mean response: 8 ms, sampling errors: 0, invalid samples: 0, valid samples: 702323 (100%) Nodes: 2, Sessions: 500, active: 500, samples: 690571, throughput 46,028.9 samples/s, 0.2 MB/s, mean response: 10 ms, sampling errors: 0, invalid samples: 0, valid samples: 690571 (100%) Nodes: 2, Sessions: 600, active: 600, samples: 689405, throughput 45,949.6 samples/s, 0.2 MB/s, mean response: 13 ms, sampling errors: 0, invalid samples: 0, valid samples: 689405 (100%) Nodes: 2, Sessions: 700, active: 700, samples: 681118, throughput 45,400.3 samples/s, 0.2 MB/s, mean response: 15 ms, sampling errors: 0, invalid samples: 0, valid samples: 681118 (100%) Nodes: 2, Sessions: 800, active: 800, samples: 685431, throughput 45,684.7 samples/s, 0.2 MB/s, mean response: 16 ms, sampling errors: 0, invalid samples: 0, valid samples: 685431 (100%) Nodes: 2, Sessions: 900, active: 900, samples: 668869, throughput 44,580.9 samples/s, 0.2 MB/s, mean response: 20 ms, sampling errors: 0, invalid samples: 0, valid samples: 668869 (100%) Nodes: 2, Sessions: 1000, active: 1000, samples: 675849, throughput 45,046.1 samples/s, 0.2 MB/s, mean response: 21 ms, sampling errors: 0, invalid samples: 0, valid samples: 675849 (100%) {noformat} *Clustered* {noformat} Nodes: 2, Sessions: 100, active: 100, samples: 7350, throughput 489.9 samples/s, 0.0 MB/s, mean response: 196 ms, sampling errors: 0, invalid samples: 49, valid samples: 7301 (99%) Nodes: 2, Sessions: 200, active: 200, samples: 6380, throughput 425.3 samples/s, 0.0 MB/s, mean response: 458 ms, sampling errors: 0, invalid samples: 362, valid samples: 6018 (94%) Nodes: 2, Sessions: 300, active: 300, samples: 5821, throughput 382.4 samples/s, 0.0 MB/s, mean response: 748 ms, sampling errors: 0, invalid samples: 484, valid samples: 5337 (91%) Nodes: 2, Sessions: 400, active: 262, samples: 3846, throughput 256.3 samples/s, 0.0 MB/s, mean response: 1056 ms, sampling errors: 0, invalid samples: 333, valid samples: 3513 (91%) Nodes: 2, Sessions: 500, active: 452, samples: 6389, throughput 425.6 samples/s, 0.0 MB/s, mean response: 1104 ms, sampling errors: 0, invalid samples: 565, valid samples: 5824 (91%) Nodes: 2, Sessions: 600, active: 55, samples: 985, throughput 65.7 samples/s, 0.0 MB/s, mean response: 1590 ms, sampling errors: 457, invalid samples: 55, valid samples: 473 (48%) {noformat} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/AS7-2903?page=com.atlassian.jira.plugin.s... ] Jason Greene updated AS7-2903: ------------------------------ Fix Version/s: 7.1.0.Final (was: 7.1.0.CR1) Should block final not cr -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/AS7-2903?page=com.atlassian.jira.plugin.s... ] Radoslav Husar commented on AS7-2903: ------------------------------------- {quote}Unclustered cannot be 100x faster !{quote} As we discussed, comparing clustered to unclustered makes no sense (there is no baseline). The *application is clustering isolated test*, thus the results will be different. Since replication is processed in the background, the analysis of the issue gets easier. Numbers that do make sense is comparing AS5 to AS7 which yields 10x difference. {quote}Where's your test ? {quote} Open jobs in Hudson -> Configure and reproduce the steps. The environment is too specific, for this to be easy it would require more work for me but there is no time allocated for that ATM. {quote}Would you like to run our test from 4 years ago ?{quote} Well, this is it. This is the same session replication test I worked on with Brian 3 years ago. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/AS7-2903?page=com.atlassian.jira.plugin.s... ] Manik Surtani commented on AS7-2903: ------------------------------------ @Rado, could you try with AS7 + Infinispan SNAPSHOT that contains this commit: https://github.com/infinispan/infinispan/commit/fef55a7eb8ca99cd94e431210... -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/AS7-2903?page=com.atlassian.jira.plugin.s... ] Manik Surtani commented on AS7-2903: ------------------------------------ @Prasad Infinispan 5.1.0.CR4 is now out with most of these fixes. See http://infinispan.blogspot.com/2012/01/infinispan-510cr4-is-out.html -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/AS7-2903?page=com.atlassian.jira.plugin.s... ] Paul Ferraro commented on AS7-2903: ----------------------------------- FYI, the 5.1.0.CR4 upgrade should be getting pulled into master momentarily: https://github.com/jbossas/jboss-as/pull/1129 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/AS7-2903?page=com.atlassian.jira.plugin.s... ] Radoslav Husar commented on AS7-2903: ------------------------------------- Manik, the results with CR4 are the same as with the snapshot I reported. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira