10:52 p.m.
I was running some performance analysis on my rules so I could tune in hopes of scaling up
from hundreds of thousands of datapoints to millions of entries and noticed some alarming
performance scaling issues when rules are firing and inserting dependent objects. At
first I thought it was one of my more complex rules but when I broke down my examples to
the simplest possible test cases I found that the central problem was that performance was
not linear with number of facts inserted -- it was polynominal (roughly n^2.5).
My simplest test example was basically each fact inserted initially would trigger a new
logical relationship fact to be inserted. No fancy logic. Also it could not trigger
itself recursively. I tried three approaches : logical inserts of new facts, manual
inserts/retraction, and update of the current fact with new data.
Updates on current fact were roughly linear assuming I prevented re-firing.
Manual Inserts and retracts scaled with O(n^2.5)
LogicalInserts were O(n^3)
The documentation suggest insertion of dependent facts to simplify logic. But when
dealing with tens of thousands of events triggering it did not scale sufficiently. Can
anyone tell me what I may have done wrong? I have included rules and data below.
This was on 5.3 using fireAllRules. The time measured was rule execution only and
excluded times due to my testing framework.
// SIMPLIFIED RULES FILE
declare DataPoint
anomaly : boolean
entityId : String
average : double
predict : double
predictDev : double
timestamp : long
end
declare AnomalyFact
entityId : String
anomalyType : AnomalyType
ruleName : String
timestamp : long
end
rule "Traffic Below 4 Deviations"
no-loop
when
$datapoint : DataPoint (
anomaly==false,
average < (predict - (predictDev * 4))
)
then
// update self only. This line was always active.
modify($datapoint) {setAnomaly(true)};
// on logicalInsert test I commented in the following line
// insertLogical( new AnomalyFact ( $datapoint.getEntityId(), AnomalyEnum.LOW,
drools.getRule().getName(), $datapoint.getTimestamp()));
// on insert test I commented in the following line and added another rule to retract any
anomaly with no datapoint
// insert( new AnomalyFact ( $datapoint.getEntityId(), AnomalyEnum.LOW,
drools.getRule().getName(), $datapoint.getTimestamp()));
end
/* TEST RESULT CONTAINED BELOW
All data inserted triggered the rule. I was load testing for worst case behavior.
logicalInsert
n run1 run2 run3
5000 1293 1232 1767
10000 9458 8210 10079
15000 24389 31050 24311
30000 134936
manual insert/retract
n run1 run2 run3
5000 1137 1094 1169
10000 8090 5862 5616
15000 16741 16728 15874
30000 76034
update on current object only
n run1 run2 run3
5000 417 444 413
10000 600
15000 712 697 688
30000 999 970 1041 1016 1163
60000 1420
100000 1930
Thanks for any help!
*/
7:57 a.m.
You realize that the modify negating part of the condition must result in
an immediate retraction of the logical insertion?
During the runs: were there any other rules besides the one you have shown,
especially rules with patterns using AnomalyFact or DataPoint?
AnomalyType =/= AnomalyEnum
-W
On 08/02/2012, St. Lawrence, Zachary <zstlawre(a)akamai.com> wrote:
I was running some performance analysis on my rules so I could tune
in hopes
of scaling up from hundreds of thousands of datapoints to millions of
entries and noticed some alarming performance scaling issues when rules are
firing and inserting dependent objects. At first I thought it was one of my
more complex rules but when I broke down my examples to the simplest
possible test cases I found that the central problem was that performance
was not linear with number of facts inserted -- it was polynominal (roughly
n^2.5).
My simplest test example was basically each fact inserted initially would
trigger a new logical relationship fact to be inserted. No fancy logic.
Also it could not trigger itself recursively. I tried three approaches :
logical inserts of new facts, manual inserts/retraction, and update of the
current fact with new data.
Updates on current fact were roughly linear assuming I prevented re-firing.
Manual Inserts and retracts scaled with O(n^2.5)
LogicalInserts were O(n^3)
The documentation suggest insertion of dependent facts to simplify logic.
But when dealing with tens of thousands of events triggering it did not
scale sufficiently. Can anyone tell me what I may have done wrong? I have
included rules and data below.
This was on 5.3 using fireAllRules. The time measured was rule execution
only and excluded times due to my testing framework.
// SIMPLIFIED RULES FILE
declare DataPoint
anomaly : boolean
entityId : String
average : double
predict : double
predictDev : double
timestamp : long
end
declare AnomalyFact
entityId : String
anomalyType : AnomalyType
ruleName : String
timestamp : long
end
rule "Traffic Below 4 Deviations"
no-loop
when
$datapoint : DataPoint (
anomaly==false,
average < (predict - (predictDev * 4))
)
then
// update self only. This line was always active.
modify($datapoint) {setAnomaly(true)};
// on logicalInsert test I commented in the following line
// insertLogical( new AnomalyFact ( $datapoint.getEntityId(),
AnomalyEnum.LOW, drools.getRule().getName(), $datapoint.getTimestamp()));
// on insert test I commented in the following line and added another rule
to retract any anomaly with no datapoint
// insert( new AnomalyFact ( $datapoint.getEntityId(), AnomalyEnum.LOW,
drools.getRule().getName(), $datapoint.getTimestamp()));
end
/* TEST RESULT CONTAINED BELOW
All data inserted triggered the rule. I was load testing for worst case
behavior.
logicalInsert
n run1 run2 run3
5000 1293 1232 1767
10000 9458 8210 10079
15000 24389 31050 24311
30000 134936
manual insert/retract
n run1 run2 run3
5000 1137 1094 1169
10000 8090 5862 5616
15000 16741 16728 15874
30000 76034
update on current object only
n run1 run2 run3
5000 417 444 413
10000 600
15000 712 697 688
30000 999 970 1041 1016 1163
60000 1420
100000 1930
Thanks for any help!
*/
4:40 p.m.
laune wrote
You realize that the modify negating part of the condition must result in
an immediate retraction of the logical insertion?
Yes. I did realize that the logical insert would retrigger. sorry, I had
started with logical inserts without that clause. I only added the negation
portion when I wrote the manual insert test (i didn't include the manual
retract logic in my post either). Finally I changed to modify only approach
for a huge speedup. (Modify was initially the trivial case I posted but
later I used wrapper objects and was able to mirror the complexity of the
other approaches without a comparable slowdown. I used the trivial modify
example since the final approach required a drastically different datamodel
and made the post rather long)
laune wrote
During the runs: were there any other rules besides the one you have
shown,
especially rules with patterns using AnomalyFact or DataPoint?
My original rule file had many more rules but the numbers collected were for
performance testing with only a single rule once I identified that even the
simplest case was not scaling properly. So yes the rule file was only a
single rule (plus an additional retraction rule for the non-logical insert
test) the rule while useless on its own proved sufficient to show that
scaling to millions of datapoints was impossible using logical insertions or
even manual insert/retract. Luckily I was able to restructure my data to
use a modify based processing model but was dismayed that logical insert
suggestion that was documented as preferable scaled so poorly. It made me
suspect that I was misusing some part of the system.
But the case is simple to test. If performance is linear than doubling the
number of objects should result in double the time. Anything worse is not
scalable for an extremely busy systems. (thousands of points per second) My
post was largely to determine if others had also reproduced simular scaling
issues. If so then we should modify the documentation to be more
appropriate in it's suggestions.
--
View this message in context:
http://drools.46999.n3.nabble.com/rules-users-Performance-scaling-with-fa...
Sent from the Drools: User forum mailing list archive at Nabble.com.
6:36 p.m.
Also, I should have mentioned that inserting increasing numbers of
identical (!) facts against a single rule is not a realistic scenario,
especially if the Engine is not permitted to work on it between insertions.
-W
On 10 February 2012 16:40, zstlaw <zstlawre(a)akamai.com> wrote:
laune wrote
>
> You realize that the modify negating part of the condition must result in
> an immediate retraction of the logical insertion?
Yes. I did realize that the logical insert would retrigger. sorry, I had
started with logical inserts without that clause. I only added the
negation
portion when I wrote the manual insert test (i didn't include the manual
retract logic in my post either). Finally I changed to modify only
approach
for a huge speedup. (Modify was initially the trivial case I posted but
later I used wrapper objects and was able to mirror the complexity of the
other approaches without a comparable slowdown. I used the trivial modify
example since the final approach required a drastically different datamodel
and made the post rather long)
laune wrote
> During the runs: were there any other rules besides the one you have
> shown,
> especially rules with patterns using AnomalyFact or DataPoint?
My original rule file had many more rules but the numbers collected were
for
performance testing with only a single rule once I identified that even the
simplest case was not scaling properly. So yes the rule file was only a
single rule (plus an additional retraction rule for the non-logical insert
test) the rule while useless on its own proved sufficient to show that
scaling to millions of datapoints was impossible using logical insertions
or
even manual insert/retract. Luckily I was able to restructure my data to
use a modify based processing model but was dismayed that logical insert
suggestion that was documented as preferable scaled so poorly. It made me
suspect that I was misusing some part of the system.
But the case is simple to test. If performance is linear than doubling the
number of objects should result in double the time. Anything worse is not
scalable for an extremely busy systems. (thousands of points per second)
My
post was largely to determine if others had also reproduced simular scaling
issues. If so then we should modify the documentation to be more
appropriate in it's suggestions.
--
View this message in context:
http://drools.46999.n3.nabble.com/rules-users-Performance-scaling-with-fa...
Sent from the Drools: User forum mailing list archive at Nabble.com.
_______________________________________________
rules-users mailing list
rules-users(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/rules-users
7:44 p.m.
I just ran the test with non-Identical insertions and performance was as I
stated before. (The identical insertions were due to me trying to simplify
the example to fit better in a forum post. My initial tests were more robust
than the examples I posted. (Alerts were external objects having unique ids
and db representation, db persistance and other non-rule times were excluded
from my statistics. I just tried to make objects simple for the forum
discussion of performance)
I currently run rules in batches via runAllRules because my previous
performance tests had runAllRules running in a separate thread outperforming
runUntilHalt. I have not repeated that test recently and it may no longer
be true. I will try to check that again if time allows. Either way the
unit tests use runAllRules because it allows for testing from a known state
to another known state with detailed tracking of time involved required by
the rule processing.
--
View this message in context:
http://drools.46999.n3.nabble.com/rules-users-Performance-scaling-with-fa...
Sent from the Drools: User forum mailing list archive at Nabble.com.
9:43 p.m.
I just made a simple self contained example to demonstrate the problem
better. It entirely depends on the two included files. This was run on
5.3.1.Final.
Results of test runs :
Setup with 10000 in 2524
LogicalInsertPerformance with 10000 in 17049
ManualInsertPerformance with 10000 in 17841
ModifyPerformance with 10000 in 9679
Setup with 20000 in 13156
LogicalInsertPerformance with 20000 in 84740
ManualInsertPerformance with 20000 in 100079
ModifyPerformance with 20000 in 25267
You can see how doubling the number of objects increases time by a factor of
4 in all but the modify example.
//
----------------------------------------------------------------------------------------
// START RULE FILE PerfTestRules.drl
//
----------------------------------------------------------------------------------------
package com.rules;
dialect "java"
declare SimpleDataPoint
value : int
id : int
alert : boolean // for modify and manualInsert tests
alertRule : String // for modify test only
end
declare SimpleAlert
datapoint : SimpleDataPoint
ruleName : String
end
rule "SetUp"
when
$amtData : Integer()
then
for (int x=0; x< $amtData.intValue(); x++) {
insert(new SimpleDataPoint(11, x, false, null));
}
end
rule "LogicalInsertPerformance"
when
$datapoint : SimpleDataPoint (
value > 0
)
then
insertLogical(new SimpleAlert($datapoint, drools.getRule().getName()));
end
rule "ManualInsertPerformance"
no-loop
when
$datapoint : SimpleDataPoint (
value > 0,
!alert
)
then
insert(new SimpleAlert($datapoint, drools.getRule().getName()));
modify($datapoint) {setAlert(true)};
end
rule "ModifyPerformance"
no-loop
when
$datapoint : SimpleDataPoint (
value > 0,
!alert
)
then
modify($datapoint) {setAlert(true),
setAlertRule(drools.getRule().getName())};
end
//
----------------------------------------------------------------------------------------
// JUNIT file SimplePerfTest.java used to setup and run tests.
//
----------------------------------------------------------------------------------------
package com.rules;
import org.drools.KnowledgeBase;
import org.drools.KnowledgeBaseConfiguration;
import org.drools.KnowledgeBaseFactory;
import org.drools.builder.KnowledgeBuilder;
import org.drools.builder.KnowledgeBuilderFactory;
import org.drools.builder.ResourceType;
import org.drools.conf.AssertBehaviorOption;
import org.drools.io.ResourceFactory;
import org.drools.runtime.StatefulKnowledgeSession;
import org.drools.runtime.rule.Activation;
import org.drools.runtime.rule.AgendaFilter;
import org.junit.After;
import org.junit.Before;
import org.junit.Test;
public class SimplePerfTest {
private StatefulKnowledgeSession rulesSession = null;
java.util.ArrayList<String> allowList = new java.util.ArrayList<String>();
@Before
public void setUp() {
// this will parse and compile in one step
final KnowledgeBuilder kbuilder =
KnowledgeBuilderFactory.newKnowledgeBuilder();
kbuilder.add( ResourceFactory.newClassPathResource("PerfTestRules.drl",
SimplePerfTest.class ),
ResourceType.DRL );
if ( kbuilder.hasErrors() ) {
throw new IllegalStateException( kbuilder.getErrors().toString() );
}
allowList.add("SetUp");
// do not allow identical objects to be asserted separately
KnowledgeBaseConfiguration kbConf =
KnowledgeBaseFactory.newKnowledgeBaseConfiguration();
kbConf.setOption(AssertBehaviorOption.EQUALITY);
KnowledgeBase knowledgeBase =
KnowledgeBaseFactory.newKnowledgeBase(kbConf);
knowledgeBase.addKnowledgePackages(kbuilder.getKnowledgePackages());
rulesSession = knowledgeBase.newStatefulKnowledgeSession();
}
/** clean up test environment
* @throws RuleManagerException */
@After
public void tearDown() {
allowList.clear();
if (rulesSession!=null) {
rulesSession.dispose();
}
rulesSession = null;
}
@Test public void testSetup10k() {
testPerf(10000);
}
@Test public void testLogicalInsert10k() {
allowList.add("LogicalInsertPerformance");
testPerf(10000);
}
@Test public void testManualInsert10k() {
allowList.add("ManualInsertPerformance");
testPerf(10000);
}
@Test public void testModify10k() {
allowList.add("ModifyPerformance");
testPerf(10000);
}
@Test public void testSetup20k() {
testPerf(20000);
}
@Test public void testLogicalInsert20k() {
allowList.add("LogicalInsertPerformance");
testPerf(20000);
}
@Test public void testManualInsert20k() {
allowList.add("ManualInsertPerformance");
testPerf(20000);
}
@Test public void testModify20k() {
allowList.add("ModifyPerformance");
testPerf(20000);
}
private void testPerf(final Integer AMT_DATA) {
rulesSession.insert(AMT_DATA);
long start = System.currentTimeMillis();
rulesSession.fireAllRules(new AgendaFilter() {
@Override
public boolean accept(Activation activation) {
return allowList.contains(activation.getRule().getName());
}
});
System.out.println(allowList.get(allowList.size()-1) +" with
"+AMT_DATA+"
in " + (System.currentTimeMillis()-start));
}
}
--
View this message in context:
http://drools.46999.n3.nabble.com/rules-users-Performance-scaling-with-fa...
Sent from the Drools: User forum mailing list archive at Nabble.com.
6 p.m.
There's several things to be noted with the way you've set up these tests.
(1) You run all tests (Logical, Manual, Modify) with all three rules being
in the KnowledgeBase. This means that all insertions result in the creation
of activations for all three rules while firings are restricted using the
agenda filter.
(2) I suppose that you ran the tests, one after the other, as one usually
does using Eclipse/JUnit. It's cleaner to do one test in a single run,
preferably not inside Eclipse.
(3) You should be able to control -Xms and -Xmx from the command line, to
see how heap size increments affect execution times.
(4) Insert facts from Java: This lets you distinguish between the time
required for inserting (i.e., evaluating the conditions and creatign the
agenda) and firing.
Below are the times I've measured using -Xms2048M -Xmx2048M and three DRLs
containing just one rule each.
Test com/rules/PerfTestManual.drl
100000 inserted = 0.245
100000 fired = 0.282
Test com/rules/PerfTestManual.drl
200000 inserted = 0.434
200000 fired = 0.546
Test com/rules/PerfTestManual.drl
300000 inserted = 0.607
300000 fired = 0.863
Test com/rules/PerfTestManual.drl
400000 inserted = 0.828
400000 fired = 1.557
Test com/rules/PerfTestManual.drl
500000 inserted = 1.017
500000 fired = 1.838
Test com/rules/PerfTestLogical.drl
100000 inserted = 0.229
100000 fired = 0.300
Test com/rules/PerfTestLogical.drl
200000 inserted = 0.435
200000 fired = 0.558
Test com/rules/PerfTestLogical.drl
300000 inserted = 0.560
300000 fired = 0.716
Test com/rules/PerfTestLogical.drl
400000 inserted = 0.835
400000 fired = 1.699
Test com/rules/PerfTestLogical.drl
500000 inserted = 0.950
500000 fired = 1.985
Test com/rules/PerfTestModify.drl
100000 inserted = 0.286
100000 fired = 0.155
Test com/rules/PerfTestModify.drl
200000 inserted = 0.512
200000 fired = 0.291
Test com/rules/PerfTestModify.drl
300000 inserted = 0.633
300000 fired = 0.434
Test com/rules/PerfTestModify.drl
400000 inserted = 0.926
400000 fired = 0.562
Test com/rules/PerfTestModify.drl
500000 inserted = 1.026
500000 fired = 0.662
-W
On 10 February 2012 21:43, zstlaw <zstlawre(a)akamai.com> wrote:
I just made a simple self contained example to demonstrate the
problem
better. It entirely depends on the two included files. This was run on
5.3.1.Final.
Results of test runs :
Setup with 10000 in 2524
LogicalInsertPerformance with 10000 in 17049
ManualInsertPerformance with 10000 in 17841
ModifyPerformance with 10000 in 9679
Setup with 20000 in 13156
LogicalInsertPerformance with 20000 in 84740
ManualInsertPerformance with 20000 in 100079
ModifyPerformance with 20000 in 25267
You can see how doubling the number of objects increases time by a factor
of
4 in all but the modify example.
//
----------------------------------------------------------------------------------------
// START RULE FILE PerfTestRules.drl
//
----------------------------------------------------------------------------------------
package com.rules;
dialect "java"
declare SimpleDataPoint
value : int
id : int
alert : boolean // for modify and manualInsert tests
alertRule : String // for modify test only
end
declare SimpleAlert
datapoint : SimpleDataPoint
ruleName : String
end
rule "SetUp"
when
$amtData : Integer()
then
for (int x=0; x< $amtData.intValue(); x++) {
insert(new SimpleDataPoint(11, x, false, null));
}
end
rule "LogicalInsertPerformance"
when
$datapoint : SimpleDataPoint (
value > 0
)
then
insertLogical(new SimpleAlert($datapoint,
drools.getRule().getName()));
end
rule "ManualInsertPerformance"
no-loop
when
$datapoint : SimpleDataPoint (
value > 0,
!alert
)
then
insert(new SimpleAlert($datapoint, drools.getRule().getName()));
modify($datapoint) {setAlert(true)};
end
rule "ModifyPerformance"
no-loop
when
$datapoint : SimpleDataPoint (
value > 0,
!alert
)
then
modify($datapoint) {setAlert(true),
setAlertRule(drools.getRule().getName())};
end
//
----------------------------------------------------------------------------------------
// JUNIT file SimplePerfTest.java used to setup and run tests.
//
----------------------------------------------------------------------------------------
package com.rules;
import org.drools.KnowledgeBase;
import org.drools.KnowledgeBaseConfiguration;
import org.drools.KnowledgeBaseFactory;
import org.drools.builder.KnowledgeBuilder;
import org.drools.builder.KnowledgeBuilderFactory;
import org.drools.builder.ResourceType;
import org.drools.conf.AssertBehaviorOption;
import org.drools.io.ResourceFactory;
import org.drools.runtime.StatefulKnowledgeSession;
import org.drools.runtime.rule.Activation;
import org.drools.runtime.rule.AgendaFilter;
import org.junit.After;
import org.junit.Before;
import org.junit.Test;
public class SimplePerfTest {
private StatefulKnowledgeSession rulesSession = null;
java.util.ArrayList<String> allowList = new
java.util.ArrayList<String>();
@Before
public void setUp() {
// this will parse and compile in one step
final KnowledgeBuilder kbuilder =
KnowledgeBuilderFactory.newKnowledgeBuilder();
kbuilder.add(
ResourceFactory.newClassPathResource("PerfTestRules.drl",
SimplePerfTest.class ),
ResourceType.DRL );
if ( kbuilder.hasErrors() ) {
throw new IllegalStateException(
kbuilder.getErrors().toString() );
}
allowList.add("SetUp");
// do not allow identical objects to be asserted separately
KnowledgeBaseConfiguration kbConf =
KnowledgeBaseFactory.newKnowledgeBaseConfiguration();
kbConf.setOption(AssertBehaviorOption.EQUALITY);
KnowledgeBase knowledgeBase =
KnowledgeBaseFactory.newKnowledgeBase(kbConf);
knowledgeBase.addKnowledgePackages(kbuilder.getKnowledgePackages());
rulesSession = knowledgeBase.newStatefulKnowledgeSession();
}
/** clean up test environment
* @throws RuleManagerException */
@After
public void tearDown() {
allowList.clear();
if (rulesSession!=null) {
rulesSession.dispose();
}
rulesSession = null;
}
@Test public void testSetup10k() {
testPerf(10000);
}
@Test public void testLogicalInsert10k() {
allowList.add("LogicalInsertPerformance");
testPerf(10000);
}
@Test public void testManualInsert10k() {
allowList.add("ManualInsertPerformance");
testPerf(10000);
}
@Test public void testModify10k() {
allowList.add("ModifyPerformance");
testPerf(10000);
}
@Test public void testSetup20k() {
testPerf(20000);
}
@Test public void testLogicalInsert20k() {
allowList.add("LogicalInsertPerformance");
testPerf(20000);
}
@Test public void testManualInsert20k() {
allowList.add("ManualInsertPerformance");
testPerf(20000);
}
@Test public void testModify20k() {
allowList.add("ModifyPerformance");
testPerf(20000);
}
private void testPerf(final Integer AMT_DATA) {
rulesSession.insert(AMT_DATA);
long start = System.currentTimeMillis();
rulesSession.fireAllRules(new AgendaFilter() {
@Override
public boolean accept(Activation activation) {
return
allowList.contains(activation.getRule().getName());
}
});
System.out.println(allowList.get(allowList.size()-1) +"
with "+AMT_DATA+"
in " + (System.currentTimeMillis()-start));
}
}
--
View this message in context:
http://drools.46999.n3.nabble.com/rules-users-Performance-scaling-with-fa...
Sent from the Drools: User forum mailing list archive at Nabble.com.
_______________________________________________
rules-users mailing list
rules-users(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/rules-users
4457
days inactive
4462
days old
6 comments
3 participants
participants (3)
-
St. Lawrence, Zachary -
Wolfgang Laun -
zstlaw