[jboss-jira] [JBoss JIRA] (WFLY-5198) Clustering performance regression

[Michal Vinkler (JIRA)]

Michal Vinkler created WFLY-5198:
------------------------------------

             Summary: Clustering performance regression
                 Key: WFLY-5198
                 URL: https://issues.jboss.org/browse/WFLY-5198
             Project: WildFly
          Issue Type: Bug
          Components: Clustering
            Reporter: Michal Vinkler
            Assignee: Paul Ferraro
            Priority: Blocker


Our clustering stress tests show huge performance drop of 7.0.0.DR8 bits compared to 6.4.0 bits.

Compare these results:

stress-heavyread-session-repl-sync
7.0.0.DR8: [throughput|http://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-7x-stress-heavyread-session-repl-sync_noperf21/2/artifact/report/graph-throughput.png] + [response times|http://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-7x-stress-heavyread-session-repl-sync_noperf21/2/artifact/report/graph-reponse-times.png]
6.4.0.GA:  [throughput|http://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-stress-heavyread-session-repl-sync_noperf21/1/artifact/report/graph-throughput.png] + [response times|http://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-stress-heavyread-session-repl-sync_noperf21/1/artifact/report/graph-reponse-times.png]

Jenkins runs (server configs + logs can be found here):
7.0.0.DR8: http://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-7x-stress-heavyread-session-repl-sync_noperf21/2/
6.4.0.GA: http://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-stress-heavyread-session-repl-sync_noperf21/1/

One can observe that when using 7.0.0.DR8 bits, throughput reaches a threshold of approximately 6000 TPS. This happens in each test using *REPL-SYNC* cache, for example:

stress-session-repl-sync
7.0.0.DR8: [throughput|http://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-7x-stress-session-repl-sync_noperf21/2/artifact/report/graph-throughput.png] + [response times|http://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-7x-stress-session-repl-sync_noperf21/2/artifact/report/graph-reponse-times.png]
6.4.0.GA:  [throughput|http://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-stress-session-repl-sync_noperf21/1/artifact/report/graph-throughput.png] + [response times|http://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-stress-session-repl-sync_noperf21/1/artifact/report/graph-reponse-times.png]

All tests use a 4-node EAP cluster and 5 nodes that generate load. Adding another node to the EAP cluster, the threshold is lowered to 5200 TPS and the max. response time raises, see:

stress-heavyread-session-repl-sync with 5-node cluster
7.0.0.DR8: [throughput|http://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/mvinkler_eap-7x-stress-heavyread-session-repl-sync_noperf21/1/artifact/report/graph-throughput.png] + [response times|http://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/mvinkler_eap-7x-stress-heavyread-session-repl-sync_noperf21/1/artifact/report/graph-reponse-times.png]


*Most of the other scenarios indicate performance regression as well*.

stress-session-dist-sync
7.0.0.DR8: [throughput|http://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-7x-stress-session-dist-sync_noperf21/1/artifact/report/graph-throughput.png] + [response times|http://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-7x-stress-session-dist-sync_noperf21/1/artifact/report/graph-reponse-times.png]
6.4.0.GA: [throughput|http://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-stress-session-dist-sync_noperf21/1/artifact/report/graph-throughput.png] + [response times|http://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-stress-session-dist-sync_noperf21/1/artifact/report/graph-reponse-times.png]

stress-session-repl-async
7.0.0.DR8: [throughput|http://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-7x-stress-session-repl-async_noperf21/2/artifact/report/graph-throughput.png] + [response times|http://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-7x-stress-session-repl-async_noperf21/2/artifact/report/graph-reponse-times.png]
6.4.0.GA: [throughput|https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-stress-session-repl-async_noperf21/1/artifact/report/graph-throughput.png] + [response times|http://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-stress-session-repl-async_noperf21/1/artifact/report/graph-reponse-times.png]

This issue is related to https://issues.jboss.org/browse/JBEAP-768



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)