Performance scaling with fact insertion
by St. Lawrence, Zachary
I was running some performance analysis on my rules so I could tune in hopes of scaling up from hundreds of thousands of datapoints to millions of entries and noticed some alarming performance scaling issues when rules are firing and inserting dependent objects. At first I thought it was one of my more complex rules but when I broke down my examples to the simplest possible test cases I found that the central problem was that performance was not linear with number of facts inserted -- it was polynominal (roughly n^2.5).
My simplest test example was basically each fact inserted initially would trigger a new logical relationship fact to be inserted. No fancy logic. Also it could not trigger itself recursively. I tried three approaches : logical inserts of new facts, manual inserts/retraction, and update of the current fact with new data.
Updates on current fact were roughly linear assuming I prevented re-firing.
Manual Inserts and retracts scaled with O(n^2.5)
LogicalInserts were O(n^3)
The documentation suggest insertion of dependent facts to simplify logic. But when dealing with tens of thousands of events triggering it did not scale sufficiently. Can anyone tell me what I may have done wrong? I have included rules and data below.
This was on 5.3 using fireAllRules. The time measured was rule execution only and excluded times due to my testing framework.
// SIMPLIFIED RULES FILE
declare DataPoint
anomaly : boolean
entityId : String
average : double
predict : double
predictDev : double
timestamp : long
end
declare AnomalyFact
entityId : String
anomalyType : AnomalyType
ruleName : String
timestamp : long
end
rule "Traffic Below 4 Deviations"
no-loop
when
$datapoint : DataPoint (
anomaly==false,
average < (predict - (predictDev * 4))
)
then
// update self only. This line was always active.
modify($datapoint) {setAnomaly(true)};
// on logicalInsert test I commented in the following line
// insertLogical( new AnomalyFact ( $datapoint.getEntityId(), AnomalyEnum.LOW, drools.getRule().getName(), $datapoint.getTimestamp()));
// on insert test I commented in the following line and added another rule to retract any anomaly with no datapoint
// insert( new AnomalyFact ( $datapoint.getEntityId(), AnomalyEnum.LOW, drools.getRule().getName(), $datapoint.getTimestamp()));
end
/* TEST RESULT CONTAINED BELOW
All data inserted triggered the rule. I was load testing for worst case behavior.
logicalInsert
n run1 run2 run3
5000 1293 1232 1767
10000 9458 8210 10079
15000 24389 31050 24311
30000 134936
manual insert/retract
n run1 run2 run3
5000 1137 1094 1169
10000 8090 5862 5616
15000 16741 16728 15874
30000 76034
update on current object only
n run1 run2 run3
5000 417 444 413
10000 600
15000 712 697 688
30000 999 970 1041 1016 1163
60000 1420
100000 1930
Thanks for any help!
*/
14 years, 2 months
(no subject)
by Alberto R. Galdo
According to the documentation:
"Rule constraints do not have direct access to variables defined inside the
> process.
> It is however possible to refer to the current process instance inside a
> rule constraint,
> by adding the process instance to the Working Memory and matching for the
> process instance in your rule constraint.
> We have added special logic to make sure that a variable processInstance
> of type WorkflowProcessInstance
> will only match to the current process instance and not to other process
> instances in the Working Memory.
> Note that you are however responsible yourself to insert the process
> instance into the session and,
> possibly, to update it, for example, using Java code or an on-entry or
> on-exit or explicit action in your process.
> The following example of a rule constraint will search for a person with
> the same name as the value stored in the variable "name" of the process:"
>
This however does not seem to be true.
The first rule is to receive a stream of objects of type MyObject.
This rule starts an associated process.
The second rule will be called by the process (from bpmn).
Within the process there a task that invokes an an asynchronous service, ie,
an implementation of WorkItemHandler that contains a Thread that will set
call
manager.completeWorkItem(workItemId,results);
when 30s have passed).
With this setup we can observe what happens when 2 MyObject instances come
into the first rule separated by a short period of time (say 5 seconds).
When the first object comes in a process is created and started (with
process id=1)
The process passes from the start node to the callMyTask node, which
invokes the slow WorkItemHandler references by MyTask
(which we have previously registered into session.getWorkItemManager() ).
Because the WorkItemHandler has a thread that waits 30s before completing
the work item, it just sits there doing nothing (so far everything is ok)
and allows the engine to process other events.
5 seconds later we insert another MyObject into the stream.
The first rule gets fired and created another process (with process id=2).
The process passes from the start node to the callMyTask node, which invoke
our slow service.
We have thus two processes running in parallel.
When the first callMyTask node completes, the next node completeTask is
invoked.
The rule "process complete" is matched as it belongs to the rule flow group
"Complete task group".
At this point we observe that the rule is matched twice spitting out:
processInstance.id 2
processInstance.id 1
.
So the rule has matched both process instances that are in working memory
and not just the one (with process id 1)
that called the "process complete" rule.
The rule file
-------------
rule "New case"
when
$myobject : MyObject(processed==false) from entry-point "myobject
stream"
then
ProcessInstance
processInstance=kcontext.getKnowledgeRuntime().createProcessInstance("com.mycompany.process.MyProcess",
parameters);
insert(processInstance);
kcontext.getKnowledgeRuntime().startProcessInstance(processInstance.getId());
modify ($myobject){
setProcessed(true)
}
end
rule "process complete"
ruleflow-group "Complete task group"
no-loop true
when
$processInstance: WorkflowProcessInstance()
then
System.out.println("processInstance.id " +
$processInstance.getId());
end
The BPMN xml
------------
<?xml version="1.0" encoding="UTF-8"?>
<definitions id="Definition"
targetNamespace="http://www.jboss.org/drools"
typeLanguage="http://www.java.com/javaTypes"
expressionLanguage="http://www.mvel.org/2.0"
xmlns="http://www.omg.org/spec/BPMN/20100524/MODEL"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.omg.org/spec/BPMN/20100524/MODELBPMN20.xsd"
xmlns:g="http://www.jboss.org/drools/flow/gpd"
xmlns:bpmndi="http://www.omg.org/spec/BPMN/20100524/DI"
xmlns:dc="http://www.omg.org/spec/DD/20100524/DC"
xmlns:di="http://www.omg.org/spec/DD/20100524/DI"
xmlns:tns="http://www.jboss.org/drools">
<process processType="Private" isExecutable="true"
id="com.mycompany.process.MyProcess" name="my process" >
<!-- process variables -->
<!-- nodes -->
<startEvent id="start" name="StartProcess" />
<task id="callMyTask" name="call my task" tns:taskName="MyTask" >
<ioSpecification>
</ioSpecification>
</task>
<businessRuleTask id="completeTask" name="Complete slow task"
g:ruleFlowGroup="Complete task group" >
</businessRuleTask>
<endEvent id="end" name="EndProcess" />
<!-- connections -->
<sequenceFlow id="start-callMyTask" sourceRef="start"
targetRef="callMyTask" />
<sequenceFlow id="callMyTask-completeTask" sourceRef="callMyTask"
targetRef="completeTask" />
<sequenceFlow id="completeTask-end" sourceRef="completeTask"
targetRef="end" />
</process>
</definitions>
What's wrong with this? Isn't WorkflowProcessInstance() supposed to be
attached to the calling ProcessInstance? Is there another way of getting
the group of rules fire only for the processinstance that called it?
Greets,
14 years, 2 months
How to get the Fact datatypes from the Guvnor
by Veera
Hi All,
Can anybody help to get the Field type(what datatype it is ).
I have declared a Facts in Guvnor 5.3 and i can able to retreive all the
package names & rules names & fact names , now i want to know what is the
datatype of each fact , i don't know which api to use for this.
Can any body help me pelase...
below code to get the all package names & rules names & fact names...:
Collection<KnowledgePackage> kpackages =
_kagent.getKnowledgeBase().getKnowledgePackages();
for(KnowledgePackage kpackage : kpackages)
{
for(org.drools.definition.rule.Rule rule1 :kpackage.getRules())
{
//printing package names & rule names.
String packname= rule1.getPackageName();
String rulename=rule1.getName();
System.out.println("This is : "+packname+" Packages and
RuleName is " +rulename);
//printing Fact names..
Field[] s=kpackage.getClass().getFields();
System.out.println(s.toString());
}
}
Thanks,
Veera
--
View this message in context: http://drools.46999.n3.nabble.com/How-to-get-the-Fact-datatypes-from-the-...
Sent from the Drools: User forum mailing list archive at Nabble.com.
14 years, 2 months
How to use database in drool
by shawn
Hi~
My project requires to use drool interact with MySQL.
Like once the rule engine received a request, the engine will ask the
database to send some fact to the engine. The engine should give a result
after matching the fact with the rule (in rule engine).
Is there any tutorial for building connection between the engine to the
database?
I know Hibernate can do it, but for SQL is there any way to do it?
Regards,
Shawn
--
View this message in context: http://drools.46999.n3.nabble.com/How-to-use-database-in-drool-tp3737584p...
Sent from the Drools: User forum mailing list archive at Nabble.com.
14 years, 2 months
time loading rules into knowledgebase giving different results when using mvel2-beta6 versus mvel2-drools4 jars
by vadlam
we have some interesting findings with loading rules PKG into knowledgebase
when using mvel2-beta6 jar versus mvel2-drools4 jar.
the PKG file was built in Guvnor 5.3.0.Final which uses mvel2-drools4 jar.
when we loaded this PKG file using Drools 5.3 jars and using mvel2-drools4
jar, we were getting OOM errors at around 1 GB .it would also take a very
long time to throw the error.
when we loaded this PKG file using Drools 5.3 jars and using mvel2-beta6
jar, we were able to load the package successfully in a lesser time and no
OOM errors at the same heap size.
Has anyone come across this issue?
is it advised/recommended to substitute mvel2-drools4 jar with mvel2-beta6
jar in our service code when loading the rules package as well as executing
the rules at runtime?
or, is there a specific reason for Guvnor 5..3 using mvel2-drools4 jar( such
as customized optimizations) instead of just mvel2 beta jars ?
--
View this message in context: http://drools.46999.n3.nabble.com/time-loading-rules-into-knowledgebase-g...
Sent from the Drools: User forum mailing list archive at Nabble.com.
14 years, 2 months
Need Camel + drools (OSGI) integration help
by Alexander Filipchik
Hello all!
Don't know if it is right list to ask, but I'm trying to make drools +
camel running on Fuse (ServiceMix container).
I couldn't even install OSGI drools artifacts for 5.3.1 version (because of
weird dependency on drools.core Snapshot),
but did it with v5.2.1. I used:
mvel2/2.1.0,
org.drools/drools-core/5.2.1.Final, org.drools/drools-compiler/5.2.1.Final,
org.drools/knowledge-api/5.2.1.Final
and org.drools/drools-camel/5.2.1.Final.
All are installed and active. Part of log:
[ 162] [Active ] [ ] [ ] [ 60] Drools :: Compiler
(5.2.1.Final)
[ 163] [Active ] [ ] [ ] [ 60] mvel2 (2.1.0.drools4)
[ 164] [Active ] [ ] [ ] [ 60] Drools :: Core
(5.2.1.Final)
[ 165] [Active ] [ ] [ ] [ 60] Knowledge API
(5.2.1.Final)
[ 172] [Active ] [ ] [ ] [ 60]
mvn:org.drools/drools-camel/5.2.1.Final
Then I tried to add my own camel-drools router (I created OSGI bundle for
it). It depends on:
Import-Package:
org.apache.activemq,org.apache.activemq.camel.component,org.apache.activemq.pool,org.apache.camel;version="[2.8,3)",org.apache.camel.builder;version="[2.8,3)",org.apache.camel.component.jms;version="[2.8,3)",org.apache.camel.model;version="[2.8,3)",org.drools.camel.component,org.osgi.service.blueprint;version="[1.0.0,2.0.0)",org.springframework.transaction;version="[3,4)"
It tried to install it to a container but all I got -
Error executing command: Unresolved constraint in bundle com.betfair.drools
[170]: Unable to resolve 170.0: missing requirement [170.0] package;
(package=org.drools.camel.component)
I even tried to put drools-camel.jar with org.drools.camel.component in a
lib folder and add package to org.osgi.framework.system.packages.extra, but
got same result.
Do you have any idea, blog posts, samples which could help me?
I'm completely stuck.
Thanks,
Alex
14 years, 2 months
Migrating repository data from Drools 5.0 to 5.3Final
by jian zhi
Hi all,
I migrated the repository from Drools 5.0 to 5.3Final. The data was stored in Microsoft SQL Server database. I exported
the repository from Drools 5.0 Guvnor and imported it back to Drools 5.3 Guvnor. After that I found
that the order of the condition columns in the web-guided decision table were
changed, and also the values for the conditions were misplaced. See the rule sources
below.
Is this a bug, or I didn’t use the right way to migrate the drools data between two versions?
Thanks in advance,
Jian
Here is the source of the decision table in Drools 5.0:
#from row number: 1
rule "Row 1 test"
dialect
"mvel"
when
consumerAction
: ConsumerWorkActionFact( checkIfExist == "true" )
accountAction
: ConsumerAccountWorkActionFact( checkIfExist == "true" )
consumerTag
: ConsumerTagAssociationFact( hasAnyConsumerTagEM == "111" )
consumerAccount
: ConsumerAccountAssociationFact( hasAnyAccountClosed == "false" )
then
consumerAction.setChangeStrategy(
"222" );
consumerAction.setHoldConsumer(
"true" );
accountAction.setAssignConsumerAccountTags(
"333" );
end
Here is the source of the decision table in Drools 5.3:
#from row number: 1
2. | rule "Row 1 test"
3. | dialect
"mvel"
4. | when
5. | consumerAction : ConsumerWorkActionFact( checkIfExist ==
"true" )
6. | consumerTag : ConsumerTagAssociationFact( hasAnyConsumerTagEM ==
"false" )
7. | consumerAccount : ConsumerAccountAssociationFact( hasAnyAccountClosed ==
true )
8. | accountAction : ConsumerAccountWorkActionFact( checkIfExist ==
"111" )
9. | then
10. | consumerAction.setChangeStrategy( "222" );
11. | consumerAction.setHoldConsumer( "true" );
12. | accountAction.setAssignConsumerAccountTags(
"333" );
13. | end
14 years, 2 months
Problem with package names and declare section
by Joaquín Díaz Vélez
Hi guys,
we are currently working with Drools Fusion in order to build a
complex event processing solution. We've modelled our events using and
object (Event) and this object is shared with everyone that needs to
publish and event.
Our scenario goes this way
1) Our shared class, located in *ar.com.fluxit.Event*
2) An example of our cep rules declared in package *rules.flux*
package *rules.flux*
import *ar.com.fluxit.Event;*
declare *Event*
@role (event)
@expires(1m)
end
rule "hello world"
when
$event:Event( eventName == "Hello World") from entry-point "*rules.flux*
"
then
System.out.println("Hello world");
end
3) When we try to insert a *ar.com.fluxit.Event *as a fact in entry point
"rules.flux" Drools Fusion throws a NullPointerException. If you read
Drools Expert documentation (
http://docs.jboss.org/drools/release/5.2.0.Final/drools-expert-docs/html/...
-
Accessing Declared Types from the Application Code) you find this paragraph:
"The first important thing to realize is that a declared fact will belong
to the package where it was declared. So, for instance, in the example
below, Person will belong to the org.drools.examples package, and so the
fully qualified name of the generated class will be
org.drools.examples.Person."
When we change package declaration to *package ar.com.fluxit *everything
works like a charm. But in our cep scenario we need that our
ar.com.fluxit.Event object to be user by all rules declared in any package.
So, there goes the question. There is a workaround for this? This is a bug?
Any help would be very appreciated
Cheers
--
Joaquín Díaz Vélez
Flux IT SA
Calle 9 N° 865 - Paseo La Panadería - Planta Alta
La Plata - Buenos Aires - Argentina
Teléfono (+54) 221 553 2980 int 306
Móvil (+54) 9221 5863322
www.fluxit.com.ar
14 years, 2 months
