[jboss-jira] [JBoss JIRA] (AS7-2903) AS 7.1 Clustering performs 10x slower than AS 5.x

AS 7.1 Clustering performs 10x slower than AS 5.x ------------------------------------------------- Key: AS7-2903 URL: https://issues.jboss.org/browse/AS7-2903 Project: Application Server 7 Issue Type: Bug Components: Clustering Affects Versions: No Release, 7.1.0.Beta1 Environment: 2b3ea9b6bc31e8b72d3e2a5aac6176067ef9da5e Reporter: Radoslav Husar Assignee: Paul Ferraro Priority: Blocker Fix For: 7.1.0.Final Update: This was a suspicion first, now I have compared results for AS5. *Clustering performs 10x slower* with the same use case in 2 node cluster*. Also this has been tested on revision with updated Infinispan to CR1. CPU usage seems relatively high, network is not used to its maximum, most likely looks like blocking in threading. AS7 {noformat} Nodes: 2, Sessions: 100, active: 100, samples: 8363, throughput 418.1 samples/s, 0.0 MB/s, mean response: 138 ms, sampling errors: 0, invalid samples: 0, valid samples: 8363 (100%) Nodes: 2, Sessions: 200, active: 200, samples: 9607, throughput 480.3 samples/s, 0.0 MB/s, mean response: 298 ms, sampling errors: 0, invalid samples: 0, valid samples: 9607 (100%) Nodes: 2, Sessions: 300, active: 300, samples: 12779, throughput 555.6 samples/s, 0.0 MB/s, mean response: 550 ms, sampling errors: 0, invalid samples: 151, valid samples: 12628 (98%) Nodes: 2, Sessions: 400, active: 400, samples: 13336, throughput 666.7 samples/s, 0.0 MB/s, mean response: 498 ms, sampling errors: 0, invalid samples: 1, valid samples: 13335 (99%) Nodes: 2, Sessions: 500, active: 500, samples: 11852, throughput 592.5 samples/s, 0.0 MB/s, mean response: 441 ms, sampling errors: 0, invalid samples: 1, valid samples: 11851 (99%) Nodes: 2, Sessions: 600, active: 600, samples: 16012, throughput 800.5 samples/s, 0.0 MB/s, mean response: 634 ms, sampling errors: 0, invalid samples: 0, valid samples: 16012 (100%) Nodes: 2, Sessions: 700, active: 700, samples: 14436, throughput 721.7 samples/s, 0.0 MB/s, mean response: 888 ms, sampling errors: 0, invalid samples: 350, valid samples: 14086 (97%) Nodes: 2, Sessions: 800, active: 800, samples: 14479, throughput 629.5 samples/s, 0.0 MB/s, mean response: 1080 ms, sampling errors: 0, invalid samples: 0, valid samples: 14479 (100%) Nodes: 2, Sessions: 900, active: 900, samples: 14473, throughput 723.6 samples/s, 0.0 MB/s, mean response: 1345 ms, sampling errors: 0, invalid samples: 450, valid samples: 14023 (96%) Nodes: 2, Sessions: 1000, active: 1000, samples: 14570, throughput 728.4 samples/s, 0.0 MB/s, mean response: 1274 ms, sampling errors: 0, invalid samples: 1, valid samples: 14569 (99%) Nodes: 2, Sessions: 1100, active: 1100, samples: 13808, throughput 690.3 samples/s, 0.0 MB/s, mean response: 1933 ms, sampling errors: 0, invalid samples: 1100, valid samples: 12708 (92%) Nodes: 2, Sessions: 1200, active: 1200, samples: 13252, throughput 662.5 samples/s, 0.0 MB/s, mean response: 1833 ms, sampling errors: 0, invalid samples: 600, valid samples: 12652 (95%) Nodes: 2, Sessions: 1300, active: 1300, samples: 17815, throughput 754.8 samples/s, 0.0 MB/s, mean response: 1581 ms, sampling errors: 0, invalid samples: 1300, valid samples: 16515 (92%) Nodes: 2, Sessions: 1400, active: 1400, samples: 17558, throughput 877.8 samples/s, 0.0 MB/s, mean response: 1509 ms, sampling errors: 0, invalid samples: 1, valid samples: 17557 (99%) {noformat} AS 5 {noformat} Nodes: 2, Sessions: 100, active: 100, samples: 19240, throughput 961.2 samples/s, 0.0 MB/s, mean response: 4 ms, sampling errors: 0, invalid samples: 0, valid samples: 19240 (100%) Nodes: 2, Sessions: 200, active: 200, samples: 37887, throughput 1,894.1 samples/s, 0.0 MB/s, mean response: 5 ms, sampling errors: 0, invalid samples: 0, valid samples: 37887 (100%) Nodes: 2, Sessions: 300, active: 300, samples: 69732, throughput 2,843.5 samples/s, 0.0 MB/s, mean response: 4 ms, sampling errors: 0, invalid samples: 0, valid samples: 69732 (100%) Nodes: 2, Sessions: 400, active: 400, samples: 75148, throughput 3,755.1 samples/s, 0.0 MB/s, mean response: 6 ms, sampling errors: 0, invalid samples: 0, valid samples: 75148 (100%) Nodes: 2, Sessions: 500, active: 500, samples: 92979, throughput 4,648.4 samples/s, 0.0 MB/s, mean response: 6 ms, sampling errors: 0, invalid samples: 3, valid samples: 92976 (99%) Nodes: 2, Sessions: 600, active: 600, samples: 108989, throughput 5,449.0 samples/s, 0.0 MB/s, mean response: 10 ms, sampling errors: 0, invalid samples: 1, valid samples: 108988 (99%) Nodes: 2, Sessions: 700, active: 700, samples: 112413, throughput 5,508.6 samples/s, 0.0 MB/s, mean response: 26 ms, sampling errors: 0, invalid samples: 0, valid samples: 112413 (100%) Nodes: 2, Sessions: 800, active: 800, samples: 88810, throughput 4,435.9 samples/s, 0.0 MB/s, mean response: 78 ms, sampling errors: 0, invalid samples: 0, valid samples: 88810 (100%) Nodes: 2, Sessions: 900, active: 900, samples: 63417, throughput 3,170.6 samples/s, 0.0 MB/s, mean response: 188 ms, sampling errors: 0, invalid samples: 487, valid samples: 62930 (99%) Nodes: 2, Sessions: 1000, active: 1000, samples: 85718, throughput 4,285.4 samples/s, 0.0 MB/s, mean response: 129 ms, sampling errors: 0, invalid samples: 0, valid samples: 85718 (100%) Nodes: 2, Sessions: 1100, active: 1100, samples: 67710, throughput 3,385.1 samples/s, 0.0 MB/s, mean response: 232 ms, sampling errors: 0, invalid samples: 0, valid samples: 67710 (100%) Nodes: 2, Sessions: 1200, active: 1200, samples: 88369, throughput 4,417.9 samples/s, 0.0 MB/s, mean response: 165 ms, sampling errors: 0, invalid samples: 2, valid samples: 88367 (99%) Nodes: 2, Sessions: 1300, active: 1300, samples: 113719, throughput 5,685.4 samples/s, 0.0 MB/s, mean response: 124 ms, sampling errors: 0, invalid samples: 0, valid samples: 113719 (100%) Nodes: 2, Sessions: 1400, active: 1400, samples: 119857, throughput 5,210.8 samples/s, 0.0 MB/s, mean response: 163 ms, sampling errors: 0, invalid samples: 0, valid samples: 119857 (100%) Nodes: 2, Sessions: 1500, active: 1500, samples: 102037, throughput 5,099.0 samples/s, 0.0 MB/s, mean response: 197 ms, sampling errors: 0, invalid samples: 0, valid samples: 102037 (100%) Nodes: 2, Sessions: 1600, active: 1600, samples: 175482, throughput 4,535.1 samples/s, 0.0 MB/s, mean response: 244 ms, sampling errors: 0, invalid samples: 0, valid samples: 175482 (100%) Nodes: 2, Sessions: 1700, active: 1700, samples: 110511, throughput 5,524.7 samples/s, 0.0 MB/s, mean response: 236 ms, sampling errors: 0, invalid samples: 1992, valid samples: 108519 (98%) Nodes: 2, Sessions: 1800, active: 1800, samples: 120824, throughput 6,009.3 samples/s, 0.0 MB/s, mean response: 201 ms, sampling errors: 0, invalid samples: 0, valid samples: 120824 (100%) Nodes: 2, Sessions: 1900, active: 1900, samples: 159295, throughput 7,875.4 samples/s, 0.0 MB/s, mean response: 140 ms, sampling errors: 0, invalid samples: 0, valid samples: 159295 (100%) Nodes: 2, Sessions: 2000, active: 2000, samples: 75813, throughput 3,769.3 samples/s, 0.0 MB/s, mean response: 430 ms, sampling errors: 0, invalid samples: 2000, valid samples: 73813 (97%) Nodes: 2, Sessions: 2100, active: 2100, samples: 134139, throughput 5,831.5 samples/s, 0.0 MB/s, mean response: 252 ms, sampling errors: 0, invalid samples: 0, valid samples: 134139 (100%) {noformat} ---------------------------------------------------------------------------- This was suspected. Summary: Cluster suspected to perform 100x slower than unclustered Cluster performs 100x slower than unclustered when using HTTP session replication benchmark in our perf lab. Throughput with 200 clients drops from 49,851.0 r/s to 425.3 samples/s when using clustering in standalone-ha.xml server configuration. I cannot guarantee the runs, this might be a result of a hidden networking issue in the lab. Here are some sample runs: *Unclustered* {noformat} Nodes: 2, Sessions: 100, active: 100, samples: 680317, throughput 45,346.9 samples/s, 0.2 MB/s, mean response: 2 ms, sampling errors: 0, invalid samples: 0, valid samples: 680317 (100%) Nodes: 2, Sessions: 200, active: 200, samples: 747890, throughput 49,851.0 samples/s, 0.2 MB/s, mean response: 3 ms, sampling errors: 0, invalid samples: 0, valid samples: 747890 (100%) Nodes: 2, Sessions: 300, active: 300, samples: 722338, throughput 48,149.4 samples/s, 0.2 MB/s, mean response: 6 ms, sampling errors: 0, invalid samples: 0, valid samples: 722338 (100%) Nodes: 2, Sessions: 400, active: 400, samples: 702323, throughput 46,813.7 samples/s, 0.2 MB/s, mean response: 8 ms, sampling errors: 0, invalid samples: 0, valid samples: 702323 (100%) Nodes: 2, Sessions: 500, active: 500, samples: 690571, throughput 46,028.9 samples/s, 0.2 MB/s, mean response: 10 ms, sampling errors: 0, invalid samples: 0, valid samples: 690571 (100%) Nodes: 2, Sessions: 600, active: 600, samples: 689405, throughput 45,949.6 samples/s, 0.2 MB/s, mean response: 13 ms, sampling errors: 0, invalid samples: 0, valid samples: 689405 (100%) Nodes: 2, Sessions: 700, active: 700, samples: 681118, throughput 45,400.3 samples/s, 0.2 MB/s, mean response: 15 ms, sampling errors: 0, invalid samples: 0, valid samples: 681118 (100%) Nodes: 2, Sessions: 800, active: 800, samples: 685431, throughput 45,684.7 samples/s, 0.2 MB/s, mean response: 16 ms, sampling errors: 0, invalid samples: 0, valid samples: 685431 (100%) Nodes: 2, Sessions: 900, active: 900, samples: 668869, throughput 44,580.9 samples/s, 0.2 MB/s, mean response: 20 ms, sampling errors: 0, invalid samples: 0, valid samples: 668869 (100%) Nodes: 2, Sessions: 1000, active: 1000, samples: 675849, throughput 45,046.1 samples/s, 0.2 MB/s, mean response: 21 ms, sampling errors: 0, invalid samples: 0, valid samples: 675849 (100%) {noformat} *Clustered* {noformat} Nodes: 2, Sessions: 100, active: 100, samples: 7350, throughput 489.9 samples/s, 0.0 MB/s, mean response: 196 ms, sampling errors: 0, invalid samples: 49, valid samples: 7301 (99%) Nodes: 2, Sessions: 200, active: 200, samples: 6380, throughput 425.3 samples/s, 0.0 MB/s, mean response: 458 ms, sampling errors: 0, invalid samples: 362, valid samples: 6018 (94%) Nodes: 2, Sessions: 300, active: 300, samples: 5821, throughput 382.4 samples/s, 0.0 MB/s, mean response: 748 ms, sampling errors: 0, invalid samples: 484, valid samples: 5337 (91%) Nodes: 2, Sessions: 400, active: 262, samples: 3846, throughput 256.3 samples/s, 0.0 MB/s, mean response: 1056 ms, sampling errors: 0, invalid samples: 333, valid samples: 3513 (91%) Nodes: 2, Sessions: 500, active: 452, samples: 6389, throughput 425.6 samples/s, 0.0 MB/s, mean response: 1104 ms, sampling errors: 0, invalid samples: 565, valid samples: 5824 (91%) Nodes: 2, Sessions: 600, active: 55, samples: 985, throughput 65.7 samples/s, 0.0 MB/s, mean response: 1590 ms, sampling errors: 457, invalid samples: 55, valid samples: 473 (48%) {noformat}