[jbossts-issues] [JBoss JIRA] (JBTM-1369) Benchmark performance difference between JacORB and JDK ORB

[Michael Musgrove (JIRA)]

    [ https://issues.jboss.org/browse/JBTM-1369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12945658#comment-12945658 ] 

Michael Musgrove edited comment on JBTM-1369 at 2/19/14 10:10 AM:
------------------------------------------------------------------

We have four JTS performance unit tests - they all run 1000 empty transactions with each test using a different method to begin the txn. I got the following figures from our CI cluster but if you want to try them on your own machine do the following:

Build narayana with JDK orb support enabled:

{code}./build.sh -Didlj-enabled=true clean install -DskipTests{code}

To run just a specific test, such as Performance2 for example:

{code}./build.sh -f ArjunaJTS/jts/pom.xml -Didlj-enabled=true -Dtest=com.hp.mwtests.ts.jts.orbspecific.local.performance.Performance2{code}

View the results in the surefire report directories:

{code}JacORB: ArjunaJTS/jts/target/surefire-reports/com.hp.mwtests.ts.jts.orbspecific.local.performance.Performance1-output.txt  

JDK ORB: ArjunaJTS/jts/target/idlj-surefire-reports/com.hp.mwtests.ts.jts.orbspecific.local.performance.Performance1-output.txt{code}

The results generally show jacorb performance as being significantly better on our CI cluster but, interestingly, on my laptop JDK orb seemed to do better: 

{code:title=com.hp.mwtests.ts.jts.orbspecific.local.performance.Performance1}
1000 txns (OTSImpleManager.current), no participants (with warm up of 10 txns):
idlj
    Average time for empty transaction = 1.284000039100647
    Transactions per second = 778.8161756602676
jacorb
    Average time for empty transaction = 1.3229999542236328
    Transactions per second = 755.8579248680498
{code}

{code:title=com.hp.mwtests.ts.jts.orbspecific.local.performance.Performance2}
1000 txns (TSImpleManager.factory().create()), no participants (with warm up of 10 txns):
idlj
    Average time for empty transaction = 3.48799991607666
    Transactions per second = 286.6972546045273
jacorb
    Average time for empty transaction = 2.115999937057495
    Transactions per second = 472.5898061181409
{code}

{code:title=Test: com.hp.mwtests.ts.jts.orbspecific.local.performance.Performance3}
1000 txns (ArjunaTransactionImple), no participants (with warm up of 10 txns):
idlj
    Average time for empty transaction = 0.05400000140070915
    Transactions per second = 18518.5180381656
jacorb
    Average time for empty transaction = 0.052000001072883606
    Transactions per second = 19230.76883399276
{code}
{code: title=Test: com.hp.mwtests.ts.jts.local.synchronizations.Performance}
1000 txns (org.omg.CosTransactions.Current) each registering a single synchronization
idlj
    TPS: 191.20458891013382
jacorb
    TPS: 309.2145949288807
{code}
                
      was (Author: mmusgrov):
    We have four JTS performance unit tests - they all run 1000 txns, no participants and use different methods to begin a transaction. The results generally show jacorb performance as being significantly better: 

com.hp.mwtests.ts.jts.orbspecific.local.performance.Performance1
1000 txns (OTSImpleManager.current), no participants (with warm up of 10 txns):
idlj
    Average time for empty transaction = 1.284000039100647
    Transactions per second = 778.8161756602676
jacorb
    Average time for empty transaction = 1.3229999542236328
    Transactions per second = 755.8579248680498

com.hp.mwtests.ts.jts.orbspecific.local.performance.Performance2
1000 txns (TSImpleManager.factory().create()), no participants (with warm up of 10 txns):
idlj
    Average time for empty transaction = 3.48799991607666
    Transactions per second = 286.6972546045273
jacorb
    Average time for empty transaction = 2.115999937057495
    Transactions per second = 472.5898061181409

com.hp.mwtests.ts.jts.orbspecific.local.performance.Performance3
1000 txns (ArjunaTransactionImple), no participants (with warm up of 10 txns):
idlj
    Average time for empty transaction = 0.05400000140070915
    Transactions per second = 18518.5180381656
jacorb
    Average time for empty transaction = 0.052000001072883606
    Transactions per second = 19230.76883399276

com.hp.mwtests.ts.jts.local.synchronizations.Performance
1000 txns (org.omg.CosTransactions.Current) each registering a single synchronization
idlj
    TPS: 191.20458891013382
jacorb
    TPS: 309.2145949288807

                  
> Benchmark performance difference between JacORB and JDK ORB
> -----------------------------------------------------------
>
>                 Key: JBTM-1369
>                 URL: https://issues.jboss.org/browse/JBTM-1369
>             Project: JBoss Transaction Manager
>          Issue Type: Task
>      Security Level: Public(Everyone can see) 
>          Components: JTS, Performance Testing
>    Affects Versions: 4.17.0
>            Reporter: Michael Musgrove
>            Assignee: Gytis Trikleris
>             Fix For: 5.0.2.Final
>
>
> JBTM-934 added support for the JDK orb to JTS. IOR sizes can range in size from 512 bytes to entirely unbounded, with the ORB being able to add arbitrary data within specific portions. Since the IOR is packed into transaction logs and if the IOR sizes are significantly different we may find that transaction throughput has been improved or degraded. We should run some benchmarks to see what the impact is.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira