[jboss-jira] [JBoss JIRA] (WFLY-5822) Clustering performance regression in ejbremote-dist-sync scenario

[Ladislav Thon (JIRA)]

    [ https://issues.jboss.org/browse/WFLY-5822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13142850#comment-13142850 ] 

Ladislav Thon commented on WFLY-5822:
-------------------------------------

[~rachmato] The steps for adding Byteman to a SmartFrog job are different for client and for servers. In any case, you have to put the Byteman JAR and your Byteman script somewhere where all the machines in the test can find it (in my case, I believe that both {{/home/lthon/...}} and {{/home/hudson/users-tmp/lthon/...}} would work). And then:

For server:

For 4-node cluster, the {{sfConfigTemplate}} contains something like

{code}
   CLUSTER_JVM_ARGS [
      ["-Xmx5g -XX:+UseLargePages -Djava.net.preferIPv4Stack=true"],
      ["-Xmx5g -XX:+UseLargePages -Djava.net.preferIPv4Stack=true"],
      ["-Xmx5g -XX:+UseLargePages -Djava.net.preferIPv4Stack=true"],
      ["-Xmx5g -XX:+UseLargePages -Djava.net.preferIPv4Stack=true"]
   ];
{code}

You can override it in {{sfConfig}} in the Jenkins job under _Deployment configuration_ to contain system properties needed for Byteman. For example:

{code}
   CLUSTER_JVM_ARGS [
      ["-Xmx5g -XX:+UseLargePages -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Dorg.jboss.byteman.verbose -Dorg.jboss.byteman.transform.all -Dorg.jboss.byteman.compile.to.bytecode -javaagent:/home/lthon/byteman.jar=script:/home/lthon/script-server.btm,boot:/home/lthon/byteman.jar"],
      ["-Xmx5g -XX:+UseLargePages -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Dorg.jboss.byteman.verbose -Dorg.jboss.byteman.transform.all -Dorg.jboss.byteman.compile.to.bytecode -javaagent:/home/lthon/byteman.jar=script:/home/lthon/script-server.btm,boot:/home/lthon/byteman.jar"],
      ["-Xmx5g -XX:+UseLargePages -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Dorg.jboss.byteman.verbose -Dorg.jboss.byteman.transform.all -Dorg.jboss.byteman.compile.to.bytecode -javaagent:/home/lthon/byteman.jar=script:/home/lthon/script-server.btm,boot:/home/lthon/byteman.jar"],
      ["-Xmx5g -XX:+UseLargePages -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Dorg.jboss.byteman.verbose -Dorg.jboss.byteman.transform.all -Dorg.jboss.byteman.compile.to.bytecode -javaagent:/home/lthon/byteman.jar=script:/home/lthon/script-server.btm,boot:/home/lthon/byteman.jar"]
   ];
{code}

For client:

The client runs in the SmartFrog process, so it's enough to pass the Byteman arguments to that. In the Jenkins job, you can do that by modifying a configuration variable {{SF_OPTS}} under _SmartFrog environment_.

For example, the {{eap-7x-failover-ejb-ejbremote-jvmkill-repl-async}} job contains this settings there:

{code}
-Xmx5g -XX:+UseLargePages -DXlog4j.configuration=clustering/client/log4j-ejbclient-trace.properties
{code}

To add Byteman, it would look like this:

{code}
-Xmx5g -XX:+UseLargePages -DXlog4j.configuration=clustering/client/log4j-ejbclient-trace.properties -Dorg.jboss.byteman.verbose -Dorg.jboss.byteman.transform.all -Dorg.jboss.byteman.compile.to.bytecode -javaagent:/home/lthon/byteman.jar=script:/home/lthon/script-client.btm,boot:/home/lthon/byteman.jar"
{code}

Here's a job I created to demonstrate both client and server: https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/lthon___byteman___eap-7x-failover-ejb-ejbremote-jvmkill-repl-async/ Look for a string of "!!!!!" in all the workers logs (perf17 to perf21).

I used Byteman 3.0.3 (i.e., the latest). The scripts I used were:

{code:title=script-client.btm|borderStyle=solid}
RULE test
CLASS org.jboss.ejb.client.EJBClient
METHOD createSession
AT ENTRY
IF TRUE
DO traceln("!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!")
ENDRULE
{code}

{code:title=script-server.btm|borderStyle=solid}
RULE test
CLASS org.jgroups.protocols.UDP
METHOD createSockets
AT ENTRY
IF TRUE
DO traceln("!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!")
ENDRULE
{code}

If you understand Byteman, you know that you can remove a lot of options that I used (and how to put additional options if you need them). Also, if your Byteman script for the client contains a syntax error, you won't get an error message and you will kill the Jenkins slave process (learned the hard way!), so beware :-)

> Clustering performance regression in ejbremote-dist-sync scenario 
> ------------------------------------------------------------------
>
>                 Key: WFLY-5822
>                 URL: https://issues.jboss.org/browse/WFLY-5822
>             Project: WildFly
>          Issue Type: Bug
>          Components: Clustering, EJB
>    Affects Versions: 10.0.0.CR5
>            Reporter: Michal Vinkler
>            Assignee: Richard Achmatowicz
>            Priority: Critical
>
> Compared to EAP 6, all SYNC scenarios have the same/better performance except of this one, wonder why?
> Compare these results:
> stress-ejbremote-dist-sync
> 7.0.0.ER2: [throughput|http://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-7x-stress-ejbremote-dist-sync/4/artifact/report/graph-throughput.png]
> 6.4.0.GA: [throughput|http://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-stress-ejbremote-dist-sync_noperf21/1/artifact/report/graph-throughput.png]
> ---------------------------------------
> Just for comparison: ejbremote REPL_SYNC scenario *performs well* on the other hand:
> stress-ejbremote-repl-sync
> 7.0.0.ER2: [throughput|http://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-7x-stress-ejbremote-repl-sync/3/artifact/report/graph-throughput.png]
> 6.4.0.GA: [throughput|http://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-stress-ejbremote-repl-sync_noperf21/2/artifact/report/graph-throughput.png]



--
This message was sent by Atlassian JIRA
(v6.4.11#64026)