[jboss-jira] [JBoss JIRA] (DROOLS-1387) PHREAK is slower than ReteOO with accumulate and exists
Mario Fusco (JIRA)
issues at jboss.org
Wed Jan 4 05:01:00 EST 2017
[ https://issues.jboss.org/browse/DROOLS-1387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13342928#comment-13342928 ]
Mario Fusco commented on DROOLS-1387:
-------------------------------------
[~tkobayashi], there are 2 other aspects that are not immediately clear of your benchmark suite.
1. I don't think you should run the benchmark in a multithreaded environments, unless you don't want to check how drools performs in this specific situation. However benchmarking in a concurrent use case is a totally different matter and I believe that in this case thread contention can negatively affect the reliability of your benchmark.
2. KieSession creation and dispose shouldn't be part of the elapsed benchmark time. I saw you're using a pool of KieSession that you put back into a concurrent data structures after having removed all facts from there, but also these operations are part of the elapsed time of your benchmark and I believe that this is wrong.
More in general writing a reliable benchmark is a tricky task and that's why you should always use a well established framework like JMH. As said by [~tzimanyi] feel free to join us on IRC in case you want to discuss this.
Regarding the rules you have in your benchmark, I can confirm that RETEOO is a bit faster than PHREAK, but the difference is below the 10% and however not even close to the 30% you claimed. More important I believe that the difference is quite unrelated by the type of nodes but caused by the fact that in your use case all segments and path are linked (there are facts satisfying all rules) and then the lazy nature of phreak is not an advantage in this case.
I'm now profiling the latest benchmarks developed by [~tzimanyi] but at this moment I'm skeptical that I will be able to find any low hanging fruit to improve PHREAK performances in this specific case.
> PHREAK is slower than ReteOO with accumulate and exists
> -------------------------------------------------------
>
> Key: DROOLS-1387
> URL: https://issues.jboss.org/browse/DROOLS-1387
> Project: Drools
> Issue Type: Enhancement
> Components: core engine
> Affects Versions: 6.5.0.Final
> Reporter: Toshiya Kobayashi
> Assignee: Mario Fusco
> Labels: support
> Attachments: brms-perf-comarison-02.zip, brms-perf-comarison.zip
>
>
> PHREAK is slower than ReteOO in case of:
> 1. with accumulate
> 2. with exists
> Attached test case brms-perf-comarison.zip contains 4 tests to compare the performance:
> ReteOO_Accumulate vs Phreak_Accumulate
> -> ReteOO is 30% faster than PHREAK in average
> ReteOO_Exists vs Phreak_Exists
> -> ReteOO is 30% faster than PHREAK in average
> You can run the test with
> {noformat}
> mvn -P BRMS640GA clean test
> {noformat}
> The example output:
> {noformat}
> Running org.mk300.brms.perf.Phreak_Accumulate
> 2016-12-27 16:51:40,280 INFO (main) [RuleBase] ##################### RuleBase start
> 2016-12-27 16:51:42,322 INFO (main) [KieRepositoryImpl] KieModule was added: MemoryKieModule[releaseId=org.default:artifact:1.0.0-SNAPSHOT]
> 2016-12-27 16:51:42,519 INFO (pool-1-thread-1) [PerfCounter] Phreak_Accumulate : 0 tx/sec
> 2016-12-27 16:51:43,529 INFO (pool-1-thread-1) [PerfCounter] Phreak_Accumulate : 29,020 tx/sec
> 2016-12-27 16:51:44,533 INFO (pool-1-thread-1) [PerfCounter] Phreak_Accumulate : 159,216 tx/sec
> 2016-12-27 16:51:45,539 INFO (pool-1-thread-1) [PerfCounter] Phreak_Accumulate : 203,391 tx/sec
> 2016-12-27 16:51:46,539 INFO (pool-1-thread-1) [PerfCounter] Phreak_Accumulate : 202,453 tx/sec
> 2016-12-27 16:51:47,553 INFO (pool-1-thread-1) [PerfCounter] Phreak_Accumulate : 204,160 tx/sec
> 2016-12-27 16:51:48,555 INFO (pool-1-thread-1) [PerfCounter] Phreak_Accumulate : 199,914 tx/sec
> 2016-12-27 16:51:49,556 INFO (pool-1-thread-1) [PerfCounter] Phreak_Accumulate : 193,496 tx/sec
> 2016-12-27 16:51:50,557 INFO (pool-1-thread-1) [PerfCounter] Phreak_Accumulate : 200,377 tx/sec
> 2016-12-27 16:51:51,557 INFO (pool-1-thread-1) [PerfCounter] Phreak_Accumulate : 202,060 tx/sec
> 2016-12-27 16:51:52,522 INFO (main) [RuleBase] ##################### disposeAll end
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 12.394 sec - in org.mk300.brms.perf.Phreak_Accumulate
> Running org.mk300.brms.perf.Phreak_Exists
> 2016-12-27 16:51:52,860 INFO (main) [RuleBase] ##################### RuleBase start
> 2016-12-27 16:51:54,521 INFO (main) [KieRepositoryImpl] KieModule was added: MemoryKieModule[releaseId=org.default:artifact:1.0.0-SNAPSHOT]
> 2016-12-27 16:51:54,716 INFO (pool-1-thread-1) [PerfCounter] Phreak_Exists : 0 tx/sec
> 2016-12-27 16:51:55,737 INFO (pool-1-thread-1) [PerfCounter] Phreak_Exists : 72,411 tx/sec
> 2016-12-27 16:51:56,737 INFO (pool-1-thread-1) [PerfCounter] Phreak_Exists : 257,065 tx/sec
> 2016-12-27 16:51:57,741 INFO (pool-1-thread-1) [PerfCounter] Phreak_Exists : 292,136 tx/sec
> 2016-12-27 16:51:58,746 INFO (pool-1-thread-1) [PerfCounter] Phreak_Exists : 283,862 tx/sec
> 2016-12-27 16:51:59,753 INFO (pool-1-thread-1) [PerfCounter] Phreak_Exists : 285,201 tx/sec
> 2016-12-27 16:52:00,753 INFO (pool-1-thread-1) [PerfCounter] Phreak_Exists : 290,762 tx/sec
> 2016-12-27 16:52:01,760 INFO (pool-1-thread-1) [PerfCounter] Phreak_Exists : 291,661 tx/sec
> 2016-12-27 16:52:02,760 INFO (pool-1-thread-1) [PerfCounter] Phreak_Exists : 296,011 tx/sec
> 2016-12-27 16:52:03,767 INFO (pool-1-thread-1) [PerfCounter] Phreak_Exists : 296,844 tx/sec
> 2016-12-27 16:52:04,719 INFO (main) [RuleBase] ##################### disposeAll end
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 12.004 sec - in org.mk300.brms.perf.Phreak_Exists
> Running org.mk300.brms.perf.ReteOO_Accumulate
> 2016-12-27 16:52:05,068 INFO (main) [RuleBase] ##################### RuleBase start
> 2016-12-27 16:52:07,139 INFO (main) [KieRepositoryImpl] KieModule was added: MemoryKieModule[releaseId=org.default:artifact:1.0.0-SNAPSHOT]
> 2016-12-27 16:52:07,349 INFO (pool-1-thread-1) [PerfCounter] ReteOO_Accumulate : 0 tx/sec
> 2016-12-27 16:52:08,368 INFO (pool-1-thread-1) [PerfCounter] ReteOO_Accumulate : 50,996 tx/sec
> 2016-12-27 16:52:09,380 INFO (pool-1-thread-1) [PerfCounter] ReteOO_Accumulate : 217,269 tx/sec
> 2016-12-27 16:52:10,391 INFO (pool-1-thread-1) [PerfCounter] ReteOO_Accumulate : 252,807 tx/sec
> 2016-12-27 16:52:11,453 INFO (pool-1-thread-1) [PerfCounter] ReteOO_Accumulate : 245,798 tx/sec
> 2016-12-27 16:52:12,457 INFO (pool-1-thread-1) [PerfCounter] ReteOO_Accumulate : 225,747 tx/sec
> 2016-12-27 16:52:13,458 INFO (pool-1-thread-1) [PerfCounter] ReteOO_Accumulate : 254,959 tx/sec
> 2016-12-27 16:52:14,487 INFO (pool-1-thread-1) [PerfCounter] ReteOO_Accumulate : 249,093 tx/sec
> 2016-12-27 16:52:15,496 INFO (pool-1-thread-1) [PerfCounter] ReteOO_Accumulate : 237,899 tx/sec
> 2016-12-27 16:52:16,517 INFO (pool-1-thread-1) [PerfCounter] ReteOO_Accumulate : 237,790 tx/sec
> 2016-12-27 16:52:17,375 INFO (main) [RuleBase] ##################### disposeAll end
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 12.455 sec - in org.mk300.brms.perf.ReteOO_Accumulate
> Running org.mk300.brms.perf.ReteOO_Exists
> 2016-12-27 16:52:17,751 INFO (main) [RuleBase] ##################### RuleBase start
> 2016-12-27 16:52:19,431 INFO (main) [KieRepositoryImpl] KieModule was added: MemoryKieModule[releaseId=org.default:artifact:1.0.0-SNAPSHOT]
> 2016-12-27 16:52:19,643 INFO (pool-1-thread-1) [PerfCounter] ReteOO_Exists : 0 tx/sec
> 2016-12-27 16:52:20,656 INFO (pool-1-thread-1) [PerfCounter] ReteOO_Exists : 99,934 tx/sec
> 2016-12-27 16:52:21,679 INFO (pool-1-thread-1) [PerfCounter] ReteOO_Exists : 308,091 tx/sec
> 2016-12-27 16:52:22,700 INFO (pool-1-thread-1) [PerfCounter] ReteOO_Exists : 331,749 tx/sec
> 2016-12-27 16:52:23,721 INFO (pool-1-thread-1) [PerfCounter] ReteOO_Exists : 351,276 tx/sec
> 2016-12-27 16:52:24,721 INFO (pool-1-thread-1) [PerfCounter] ReteOO_Exists : 351,285 tx/sec
> 2016-12-27 16:52:25,732 INFO (pool-1-thread-1) [PerfCounter] ReteOO_Exists : 349,222 tx/sec
> 2016-12-27 16:52:26,746 INFO (pool-1-thread-1) [PerfCounter] ReteOO_Exists : 339,559 tx/sec
> 2016-12-27 16:52:27,754 INFO (pool-1-thread-1) [PerfCounter] ReteOO_Exists : 328,000 tx/sec
> 2016-12-27 16:52:28,756 INFO (pool-1-thread-1) [PerfCounter] ReteOO_Exists : 328,937 tx/sec
> 2016-12-27 16:52:29,647 INFO (main) [RuleBase] ##################### disposeAll end
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 12.056 sec - in org.mk300.brms.perf.ReteOO_Exists
> {noformat}
> NOTE: These tests run in different JVMs (reuseForks = false in pom.xml) so the order of test execution doesn't affect the performance.
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
More information about the jboss-jira
mailing list