8:42 p.m.
Hello,
We are trying to run a DRL file that has more than 20K rules, all of
which have the simple form:
rule "testpeoplenumber"
salience 100
dialect "mvel"
when
LineItem( inventoryItemCode == "8903210392090")
then
InventoryItem fact0 = new InventoryItem();
fact0.setCategoryCode( "ONPROMO" );
insert(fact0 );
end
However, the rule file is not getting parsed by the builder, and
either takes a very long time (well over half an hour), or gives a
Heap Overflow exception. We tried with a smaller file (about 1K rules,
and the situation is the same).
The parsing code is:
org.drools.KnowledgeBaseConfiguration kbc =
org.drools.KnowledgeBaseFactory.newKnowledgeBaseConfiguration(null,
cl);
org.drools.KnowledgeBase kbase =
org.drools.KnowledgeBaseFactory.newKnowledgeBase(kbc);
org.drools.builder.KnowledgeBuilderConfiguration
knowledgeBuilderConfig =
org.drools.builder.KnowledgeBuilderFactory.newKnowledgeBuilderConfiguration(null,
cl);
org.drools.builder.KnowledgeBuilder builder =
org.drools.builder.KnowledgeBuilderFactory.newKnowledgeBuilder(kbase,
knowledgeBuilderConfig);
org.drools.builder.impl.KnowledgeBuilderImpl builderImpl =
builder as org.drools.builder.impl.KnowledgeBuilderImpl;
builder.add(org.drools.io.ResourceFactory.newFileResource(filename),
org.drools.builder.ResourceType.DRL);
We are using Drools v5. We've tried both with the java jar as well as
in .NET (using IKVM to compile it).
Any help in understanding why this is happening and how it can be
avoided will be appreciated.
Warm Regards
Krishna.
--
http://kpowerinfinity.wordpress.com
http://www.linkedin.com/in/kpowerinfinity
10:55 p.m.
Have you tried increasing the heap size for the JVM?
"20K rules [of the same] simple form" sounds as if you are
trying to use a hammer where a screwdriver would be appropriate.
If your description is correct, these 20K rules would be nothing
but 20K if statements, in disguised form. There is nothing
in that which would gain from Drools' Rete algorithm with
its superior "many-patterns-many-object" matching capabilities.
-W
On Sat, Feb 6, 2010 at 12:42 PM, kpowerinfinity
<kpowerinfinity(a)gmail.com> wrote:
Hello,
We are trying to run a DRL file that has more than 20K rules, all of
which have the simple form:
rule "testpeoplenumber"
salience 100
dialect "mvel"
when
LineItem( inventoryItemCode == "8903210392090")
then
InventoryItem fact0 = new InventoryItem();
fact0.setCategoryCode( "ONPROMO" );
insert(fact0 );
end
However, the rule file is not getting parsed by the builder, and
either takes a very long time (well over half an hour), or gives a
Heap Overflow exception. We tried with a smaller file (about 1K rules,
and the situation is the same).
The parsing code is:
org.drools.KnowledgeBaseConfiguration kbc =
org.drools.KnowledgeBaseFactory.newKnowledgeBaseConfiguration(null,
cl);
org.drools.KnowledgeBase kbase =
org.drools.KnowledgeBaseFactory.newKnowledgeBase(kbc);
org.drools.builder.KnowledgeBuilderConfiguration
knowledgeBuilderConfig =
org.drools.builder.KnowledgeBuilderFactory.newKnowledgeBuilderConfiguration(null,
cl);
org.drools.builder.KnowledgeBuilder builder =
org.drools.builder.KnowledgeBuilderFactory.newKnowledgeBuilder(kbase,
knowledgeBuilderConfig);
org.drools.builder.impl.KnowledgeBuilderImpl builderImpl =
builder as org.drools.builder.impl.KnowledgeBuilderImpl;
builder.add(org.drools.io.ResourceFactory.newFileResource(filename),
org.drools.builder.ResourceType.DRL);
We are using Drools v5. We've tried both with the java jar as well as
in .NET (using IKVM to compile it).
Any help in understanding why this is happening and how it can be
avoided will be appreciated.
Warm Regards
Krishna.
--
http://kpowerinfinity.wordpress.com
http://www.linkedin.com/in/kpowerinfinity
_______________________________________________
rules-users mailing list
rules-users(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/rules-users
12:39 a.m.
Hi All,
"There is nothing in that which would gain from Drools' Rete algorithm with
its superior "many-patterns-many-object" matching capabilities."
I know Wolfgang knows all this, but I think we need to clarify this
statement for the general public of the list. Assuming all the rules are
indeed like he described (very simple, with a single alpha constraint), then
there are efficient and simple ways to implement it without a rules engine,
but that is not by using "if". At least he would have to use a hashing
technique, or the performance would be roughly linear on number of rules (
O(r) ) when in Drools, that does use hashing techniques for alpha
constraints, it is constant ( O(1) ). Also, it depends on the non-functional
requirements he has, like lifecycle management.
In any case, I decided to take the bate and implement a quick example
for this. I changed the HelloWorld example in Drools to generate 20k rules
in the same format you have:
rule "rule X"
dialect "mvel"
when
Message( message == "X" )
then
System.out.println( X );
end
Replacing X by the number of the rule. Then I parse, compile and build a
kbase for these rules and fire the rules for a single object. This is
**NOT** a benchmark, so I am just giving you a rough idea of the hardware...
I am running it on my laptop, with other stuff on. My laptop is a 2-years
old lenovo, dual core, 2Gb memory. I am running with -Xmx1024M.
Results I got (in ms):
1. Generating Package ...done. 163 ms
2. Dumping source ...done. 280897 ms
3. Parsing and compiling...done. 40557 ms
4. Creating kbase ...done. 79512 ms
5. Firing rules ...done. 29 ms
Please note that steps 1 and 2 are the rules generation. You don't have
that, since your source code is ready for processing, right? Step 3 is
executed by the KnowledgeBuilder and step 4 is the
kbase.addKnowledgePackages().
Again, this is a very simple case showing that it is possible to parse,
compile and load 20k rules quite easily in a drools kbase. Also remember
that kbases are designed to be shared, so you should incur that cost once
and then create sessions (that are really light to create) reusing the
kbase.
Finally, there are obviously room to improve, since the compilation and
kbase building is single thread today. We want to move to multi-thread
compilation in the future. That should improve things a bit.
The source code I used is bellow.
Cheers,
Edson
public class DroolsTest {
private static final int NUMBER_OF_RULES = 20000;
public static final void main(String[] args) {
try {
// load up the knowledge base
KnowledgeBase kbase = readKnowledgeBase();
StatefulKnowledgeSession ksession =
kbase.newStatefulKnowledgeSession();
//KnowledgeRuntimeLogger logger =
KnowledgeRuntimeLoggerFactory.newFileLogger(ksession, "test");
// go !
long ts = System.currentTimeMillis();
Message message = new Message();
message.setMessage("5");
message.setStatus(Message.HELLO);
ksession.insert(message);
ksession.fireAllRules();
System.out.print("5. Firing rules ...");
System.out.println("done.
"+(System.currentTimeMillis()-ts)+"ms");
//logger.close();
} catch (Throwable t) {
t.printStackTrace();
}
}
private static KnowledgeBase readKnowledgeBase() throws Exception {
KnowledgeBuilder kbuilder =
KnowledgeBuilderFactory.newKnowledgeBuilder();
System.out.print("1. Generating Package ...");
long ts = System.currentTimeMillis();
PackageDescr pkg = generatePackageDescr();
System.out.println("done.
"+(System.currentTimeMillis()-ts)+"ms");
System.out.print("2. Dumping source ...");
ts = System.currentTimeMillis();
String source = new DrlDumper( ).dump( pkg );
//System.out.println(source);
System.out.println("done.
"+(System.currentTimeMillis()-ts)+"ms");
System.out.print("3. Parsing and compiling...");
ts = System.currentTimeMillis();
kbuilder.add(ResourceFactory.newReaderResource( new StringReader(
source ) ), ResourceType.DRL);
KnowledgeBuilderErrors errors = kbuilder.getErrors();
if (errors.size() > 0) {
for (KnowledgeBuilderError error: errors) {
System.err.println(error);
}
throw new IllegalArgumentException("Could not parse
knowledge.");
}
System.out.println("done.
"+(System.currentTimeMillis()-ts)+"ms");
System.out.print("4. Creating kbase ...");
ts = System.currentTimeMillis();
KnowledgeBase kbase = KnowledgeBaseFactory.newKnowledgeBase();
kbase.addKnowledgePackages(kbuilder.getKnowledgePackages());
System.out.println("done.
"+(System.currentTimeMillis()-ts)+"ms");
return kbase;
}
private static PackageDescr generatePackageDescr() {
PackageDescr result = new PackageDescr("org.drools.test");
result.addImport( new ImportDescr("com.sample.DroolsTest.Message")
);
for( int i = 1; i <= NUMBER_OF_RULES; i++ ) {
RuleDescr rule = new RuleDescr("rule "+i);
AttributeDescr dialect = new AttributeDescr("dialect",
"mvel");
AndDescr lhs = new AndDescr();
PatternDescr pat = new PatternDescr("Message");
FieldConstraintDescr constr = new
FieldConstraintDescr("message");
LiteralRestrictionDescr restr = new
LiteralRestrictionDescr("==",String.valueOf( i ));
constr.addRestriction( restr );
pat.addConstraint( constr );
lhs.addDescr( pat );
rule.addAttribute( dialect );
rule.setLhs( lhs );
rule.setConsequence( "System.out.println("+i+");\n" );
result.addRule( rule );
}
return result;
}
public static class Message {
public static final int HELLO = 0;
public static final int GOODBYE = 1;
private String message;
private int status;
public String getMessage() {
return this.message;
}
public void setMessage(String message) {
this.message = message;
}
public int getStatus() {
return this.status;
}
public void setStatus(int status) {
this.status = status;
}
}
}
2010/2/6 Wolfgang Laun <wolfgang.laun(a)gmail.com>
Have you tried increasing the heap size for the JVM?
"20K rules [of the same] simple form" sounds as if you are
trying to use a hammer where a screwdriver would be appropriate.
If your description is correct, these 20K rules would be nothing
but 20K if statements, in disguised form. There is nothing
in that which would gain from Drools' Rete algorithm with
its superior "many-patterns-many-object" matching capabilities.
-W
On Sat, Feb 6, 2010 at 12:42 PM, kpowerinfinity
<kpowerinfinity(a)gmail.com> wrote:
> Hello,
>
> We are trying to run a DRL file that has more than 20K rules, all of
> which have the simple form:
>
>
>
> rule "testpeoplenumber"
> salience 100
> dialect "mvel"
> when
> LineItem( inventoryItemCode == "8903210392090")
> then
> InventoryItem fact0 = new InventoryItem();
> fact0.setCategoryCode( "ONPROMO" );
> insert(fact0 );
> end
>
> However, the rule file is not getting parsed by the builder, and
> either takes a very long time (well over half an hour), or gives a
> Heap Overflow exception. We tried with a smaller file (about 1K rules,
> and the situation is the same).
>
> The parsing code is:
> org.drools.KnowledgeBaseConfiguration kbc =
> org.drools.KnowledgeBaseFactory.newKnowledgeBaseConfiguration(null,
> cl);
> org.drools.KnowledgeBase kbase =
> org.drools.KnowledgeBaseFactory.newKnowledgeBase(kbc);
> org.drools.builder.KnowledgeBuilderConfiguration
> knowledgeBuilderConfig =
>
org.drools.builder.KnowledgeBuilderFactory.newKnowledgeBuilderConfiguration(null,
> cl);
> org.drools.builder.KnowledgeBuilder builder =
> org.drools.builder.KnowledgeBuilderFactory.newKnowledgeBuilder(kbase,
> knowledgeBuilderConfig);
> org.drools.builder.impl.KnowledgeBuilderImpl builderImpl =
> builder as org.drools.builder.impl.KnowledgeBuilderImpl;
>
>
builder.add(org.drools.io.ResourceFactory.newFileResource(filename),
> org.drools.builder.ResourceType.DRL);
>
> We are using Drools v5. We've tried both with the java jar as well as
> in .NET (using IKVM to compile it).
>
> Any help in understanding why this is happening and how it can be
> avoided will be appreciated.
>
> Warm Regards
> Krishna.
>
> --
> http://kpowerinfinity.wordpress.com
> http://www.linkedin.com/in/kpowerinfinity
> _______________________________________________
> rules-users mailing list
> rules-users(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/rules-users
>
_______________________________________________
rules-users mailing list
rules-users(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/rules-users
--
Edson Tirelli
JBoss Drools Core Development
JBoss by Red Hat @ www.jboss.com
12:04 a.m.
There are several techniques available for quickly locating an object based
on a key, as was done by the simple rule in the original posting. They vary
w.r.t. setup time, access time and memory requirement. Here are the
times (in seconds, on a 1 year old laptop) for 20000 random keys of length
13, made up from digits.
array: create 20000 elements 0.066
array: locate 20000 elements 0.024
map: create 20000 elements 0.020
map: locate 20000 elements 0.024
trie: create 20000 elements 0.028
trie: locate 20000 elements 0.025
'array' uses Arrays.sort and Arrays.binarySearch. 'map' uses
a HashMap, and 'trie' is a slapdash implementation of this
well-known data structure. Increasing the number of entries will,
eventually, show the superiority of HashMap, but its memory
requirement is certainly higher than that of an array.
I cannot claim that any of these solutions would be applicable for
the original problem where the consequence of the presented
rule inserts another fact, which might lead to more rules firing, etc.
But such techniques might be considered up front, reducing
rule-based processing to those "second generation" facts.
Using literal data in a (large) number of similar statements is always
an indication that an alternative approach should be investigated
where multiplicity is put into data instead of code.
-W
2010/2/6 Edson Tirelli <ed.tirelli(a)gmail.com>:
Hi All,
"There is nothing in that which would gain from Drools' Rete algorithm with
its superior "many-patterns-many-object" matching capabilities."
I know Wolfgang knows all this, but I think we need to clarify this
statement for the general public of the list. Assuming all the rules are
indeed like he described (very simple, with a single alpha constraint), then
there are efficient and simple ways to implement it without a rules engine,
but that is not by using "if". At least he would have to use a hashing
technique, or the performance would be roughly linear on number of rules (
O(r) ) when in Drools, that does use hashing techniques for alpha
constraints, it is constant ( O(1) ). Also, it depends on the non-functional
requirements he has, like lifecycle management.
In any case, I decided to take the bate and implement a quick example
for this. I changed the HelloWorld example in Drools to generate 20k rules
in the same format you have:
rule "rule X"
dialect "mvel"
when
Message( message == "X" )
then
System.out.println( X );
end
Replacing X by the number of the rule. Then I parse, compile and build a
kbase for these rules and fire the rules for a single object. This is
**NOT** a benchmark, so I am just giving you a rough idea of the hardware...
I am running it on my laptop, with other stuff on. My laptop is a 2-years
old lenovo, dual core, 2Gb memory. I am running with -Xmx1024M.
Results I got (in ms):
1. Generating Package ...done. 163 ms
2. Dumping source ...done. 280897 ms
3. Parsing and compiling...done. 40557 ms
4. Creating kbase ...done. 79512 ms
5. Firing rules ...done. 29 ms
Please note that steps 1 and 2 are the rules generation. You don't have
that, since your source code is ready for processing, right? Step 3 is
executed by the KnowledgeBuilder and step 4 is the
kbase.addKnowledgePackages().
Again, this is a very simple case showing that it is possible to parse,
compile and load 20k rules quite easily in a drools kbase. Also remember
that kbases are designed to be shared, so you should incur that cost once
and then create sessions (that are really light to create) reusing the
kbase.
Finally, there are obviously room to improve, since the compilation and
kbase building is single thread today. We want to move to multi-thread
compilation in the future. That should improve things a bit.
The source code I used is bellow.
Cheers,
Edson
public class DroolsTest {
private static final int NUMBER_OF_RULES = 20000;
public static final void main(String[] args) {
try {
// load up the knowledge base
KnowledgeBase kbase = readKnowledgeBase();
StatefulKnowledgeSession ksession =
kbase.newStatefulKnowledgeSession();
//KnowledgeRuntimeLogger logger =
KnowledgeRuntimeLoggerFactory.newFileLogger(ksession, "test");
// go !
long ts = System.currentTimeMillis();
Message message = new Message();
message.setMessage("5");
message.setStatus(Message.HELLO);
ksession.insert(message);
ksession.fireAllRules();
System.out.print("5. Firing rules ...");
System.out.println("done.
"+(System.currentTimeMillis()-ts)+"ms");
//logger.close();
} catch (Throwable t) {
t.printStackTrace();
}
}
private static KnowledgeBase readKnowledgeBase() throws Exception {
KnowledgeBuilder kbuilder =
KnowledgeBuilderFactory.newKnowledgeBuilder();
System.out.print("1. Generating Package ...");
long ts = System.currentTimeMillis();
PackageDescr pkg = generatePackageDescr();
System.out.println("done.
"+(System.currentTimeMillis()-ts)+"ms");
System.out.print("2. Dumping source ...");
ts = System.currentTimeMillis();
String source = new DrlDumper( ).dump( pkg );
//System.out.println(source);
System.out.println("done.
"+(System.currentTimeMillis()-ts)+"ms");
System.out.print("3. Parsing and compiling...");
ts = System.currentTimeMillis();
kbuilder.add(ResourceFactory.newReaderResource( new StringReader(
source ) ), ResourceType.DRL);
KnowledgeBuilderErrors errors = kbuilder.getErrors();
if (errors.size() > 0) {
for (KnowledgeBuilderError error: errors) {
System.err.println(error);
}
throw new IllegalArgumentException("Could not parse
knowledge.");
}
System.out.println("done.
"+(System.currentTimeMillis()-ts)+"ms");
System.out.print("4. Creating kbase ...");
ts = System.currentTimeMillis();
KnowledgeBase kbase = KnowledgeBaseFactory.newKnowledgeBase();
kbase.addKnowledgePackages(kbuilder.getKnowledgePackages());
System.out.println("done.
"+(System.currentTimeMillis()-ts)+"ms");
return kbase;
}
private static PackageDescr generatePackageDescr() {
PackageDescr result = new PackageDescr("org.drools.test");
result.addImport( new ImportDescr("com.sample.DroolsTest.Message")
);
for( int i = 1; i <= NUMBER_OF_RULES; i++ ) {
RuleDescr rule = new RuleDescr("rule "+i);
AttributeDescr dialect = new AttributeDescr("dialect",
"mvel");
AndDescr lhs = new AndDescr();
PatternDescr pat = new PatternDescr("Message");
FieldConstraintDescr constr = new
FieldConstraintDescr("message");
LiteralRestrictionDescr restr = new
LiteralRestrictionDescr("==",String.valueOf( i ));
constr.addRestriction( restr );
pat.addConstraint( constr );
lhs.addDescr( pat );
rule.addAttribute( dialect );
rule.setLhs( lhs );
rule.setConsequence( "System.out.println("+i+");\n" );
result.addRule( rule );
}
return result;
}
public static class Message {
public static final int HELLO = 0;
public static final int GOODBYE = 1;
private String message;
private int status;
public String getMessage() {
return this.message;
}
public void setMessage(String message) {
this.message = message;
}
public int getStatus() {
return this.status;
}
public void setStatus(int status) {
this.status = status;
}
}
}
2010/2/6 Wolfgang Laun <wolfgang.laun(a)gmail.com>
>
> Have you tried increasing the heap size for the JVM?
>
> "20K rules [of the same] simple form" sounds as if you are
> trying to use a hammer where a screwdriver would be appropriate.
> If your description is correct, these 20K rules would be nothing
> but 20K if statements, in disguised form. There is nothing
> in that which would gain from Drools' Rete algorithm with
> its superior "many-patterns-many-object" matching capabilities.
>
> -W
>
> On Sat, Feb 6, 2010 at 12:42 PM, kpowerinfinity
> <kpowerinfinity(a)gmail.com> wrote:
> > Hello,
> >
> > We are trying to run a DRL file that has more than 20K rules, all of
> > which have the simple form:
> >
> >
> >
> > rule "testpeoplenumber"
> > salience 100
> > dialect "mvel"
> > when
> > LineItem( inventoryItemCode == "8903210392090")
> > then
> > InventoryItem fact0 = new InventoryItem();
> > fact0.setCategoryCode( "ONPROMO" );
> > insert(fact0 );
> > end
> >
> > However, the rule file is not getting parsed by the builder, and
> > either takes a very long time (well over half an hour), or gives a
> > Heap Overflow exception. We tried with a smaller file (about 1K rules,
> > and the situation is the same).
> >
> > The parsing code is:
> > org.drools.KnowledgeBaseConfiguration kbc =
> > org.drools.KnowledgeBaseFactory.newKnowledgeBaseConfiguration(null,
> > cl);
> > org.drools.KnowledgeBase kbase =
> > org.drools.KnowledgeBaseFactory.newKnowledgeBase(kbc);
> > org.drools.builder.KnowledgeBuilderConfiguration
> > knowledgeBuilderConfig =
> >
> >
org.drools.builder.KnowledgeBuilderFactory.newKnowledgeBuilderConfiguration(null,
> > cl);
> > org.drools.builder.KnowledgeBuilder builder =
> > org.drools.builder.KnowledgeBuilderFactory.newKnowledgeBuilder(kbase,
> > knowledgeBuilderConfig);
> > org.drools.builder.impl.KnowledgeBuilderImpl builderImpl =
> > builder as org.drools.builder.impl.KnowledgeBuilderImpl;
> >
> >
> > builder.add(org.drools.io.ResourceFactory.newFileResource(filename),
> > org.drools.builder.ResourceType.DRL);
> >
> > We are using Drools v5. We've tried both with the java jar as well as
> > in .NET (using IKVM to compile it).
> >
> > Any help in understanding why this is happening and how it can be
> > avoided will be appreciated.
> >
> > Warm Regards
> > Krishna.
> >
> > --
> > http://kpowerinfinity.wordpress.com
> > http://www.linkedin.com/in/kpowerinfinity
> > _______________________________________________
> > rules-users mailing list
> > rules-users(a)lists.jboss.org
> > https://lists.jboss.org/mailman/listinfo/rules-users
> >
>
> _______________________________________________
> rules-users mailing list
> rules-users(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/rules-users
--
Edson Tirelli
JBoss Drools Core Development
JBoss by Red Hat @ www.jboss.com
_______________________________________________
rules-users mailing list
rules-users(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/rules-users
10:11 p.m.
Hi, I'v been out of this list for a while (like 2 years) but always reading,
so maybe what I'm going
to add could not be relevant... please correct me if I'm worng.
The thing is, that Krishna is doing a little job extra on the RHS, because
the action is not just
"System.out.println", it has an insert of a new fact,
InventoryItem fact0 = new InventoryItem();
fact0.setCategoryCode( "ONPROMO" );
insert(fact0 );
So that could make the difference here... maybe check this InventoryItem
constructor could help.
The other thing I would do is to put some kind of Map as a global variable
in the engine and put this
relation you are trying to make:
inventoryItemCode == "8903210392090" => InventoryItem with
categoryCode="ONPROMO"
so you could have a Map with inventoryItemCode as key and categoryCode as
value,
and finally when you finish the execution of the rule's engine you can
create new InventoryItems from
the resulting Map.
Regards.
On Sun, Feb 7, 2010 at 12:04 PM, Wolfgang Laun <wolfgang.laun(a)gmail.com>wrote:
There are several techniques available for quickly locating an object
based
on a key, as was done by the simple rule in the original posting. They vary
w.r.t. setup time, access time and memory requirement. Here are the
times (in seconds, on a 1 year old laptop) for 20000 random keys of length
13, made up from digits.
array: create 20000 elements 0.066
array: locate 20000 elements 0.024
map: create 20000 elements 0.020
map: locate 20000 elements 0.024
trie: create 20000 elements 0.028
trie: locate 20000 elements 0.025
'array' uses Arrays.sort and Arrays.binarySearch. 'map' uses
a HashMap, and 'trie' is a slapdash implementation of this
well-known data structure. Increasing the number of entries will,
eventually, show the superiority of HashMap, but its memory
requirement is certainly higher than that of an array.
I cannot claim that any of these solutions would be applicable for
the original problem where the consequence of the presented
rule inserts another fact, which might lead to more rules firing, etc.
But such techniques might be considered up front, reducing
rule-based processing to those "second generation" facts.
Using literal data in a (large) number of similar statements is always
an indication that an alternative approach should be investigated
where multiplicity is put into data instead of code.
-W
2010/2/6 Edson Tirelli <ed.tirelli(a)gmail.com>:
> Hi All,
>
> "There is nothing in that which would gain from Drools' Rete algorithm
with
> its superior "many-patterns-many-object" matching capabilities."
>
> I know Wolfgang knows all this, but I think we need to clarify this
> statement for the general public of the list. Assuming all the rules are
> indeed like he described (very simple, with a single alpha constraint),
then
> there are efficient and simple ways to implement it without a rules
engine,
> but that is not by using "if". At least he would have to use a hashing
> technique, or the performance would be roughly linear on number of rules
(
> O(r) ) when in Drools, that does use hashing techniques for alpha
> constraints, it is constant ( O(1) ). Also, it depends on the
non-functional
> requirements he has, like lifecycle management.
>
> In any case, I decided to take the bate and implement a quick
example
> for this. I changed the HelloWorld example in Drools to generate 20k
rules
> in the same format you have:
>
> rule "rule X"
> dialect "mvel"
> when
> Message( message == "X" )
> then
> System.out.println( X );
> end
>
> Replacing X by the number of the rule. Then I parse, compile and
build a
> kbase for these rules and fire the rules for a single object. This is
> **NOT** a benchmark, so I am just giving you a rough idea of the
hardware...
> I am running it on my laptop, with other stuff on. My laptop is a 2-years
> old lenovo, dual core, 2Gb memory. I am running with -Xmx1024M.
>
> Results I got (in ms):
>
> 1. Generating Package ...done. 163 ms
> 2. Dumping source ...done. 280897 ms
> 3. Parsing and compiling...done. 40557 ms
> 4. Creating kbase ...done. 79512 ms
> 5. Firing rules ...done. 29 ms
>
> Please note that steps 1 and 2 are the rules generation. You don't
have
> that, since your source code is ready for processing, right? Step 3 is
> executed by the KnowledgeBuilder and step 4 is the
> kbase.addKnowledgePackages().
>
> Again, this is a very simple case showing that it is possible to
parse,
> compile and load 20k rules quite easily in a drools kbase. Also remember
> that kbases are designed to be shared, so you should incur that cost once
> and then create sessions (that are really light to create) reusing the
> kbase.
>
> Finally, there are obviously room to improve, since the compilation
and
> kbase building is single thread today. We want to move to multi-thread
> compilation in the future. That should improve things a bit.
>
> The source code I used is bellow.
>
> Cheers,
> Edson
>
>
> public class DroolsTest {
>
> private static final int NUMBER_OF_RULES = 20000;
>
> public static final void main(String[] args) {
> try {
> // load up the knowledge base
> KnowledgeBase kbase = readKnowledgeBase();
> StatefulKnowledgeSession ksession =
> kbase.newStatefulKnowledgeSession();
> //KnowledgeRuntimeLogger logger =
> KnowledgeRuntimeLoggerFactory.newFileLogger(ksession, "test");
> // go !
> long ts = System.currentTimeMillis();
> Message message = new Message();
> message.setMessage("5");
> message.setStatus(Message.HELLO);
> ksession.insert(message);
> ksession.fireAllRules();
> System.out.print("5. Firing rules ...");
> System.out.println("done.
> "+(System.currentTimeMillis()-ts)+"ms");
> //logger.close();
> } catch (Throwable t) {
> t.printStackTrace();
> }
> }
>
> private static KnowledgeBase readKnowledgeBase() throws Exception {
> KnowledgeBuilder kbuilder =
> KnowledgeBuilderFactory.newKnowledgeBuilder();
>
> System.out.print("1. Generating Package ...");
> long ts = System.currentTimeMillis();
> PackageDescr pkg = generatePackageDescr();
> System.out.println("done.
"+(System.currentTimeMillis()-ts)+"ms");
>
> System.out.print("2. Dumping source ...");
> ts = System.currentTimeMillis();
> String source = new DrlDumper( ).dump( pkg );
> //System.out.println(source);
> System.out.println("done.
"+(System.currentTimeMillis()-ts)+"ms");
>
> System.out.print("3. Parsing and compiling...");
> ts = System.currentTimeMillis();
> kbuilder.add(ResourceFactory.newReaderResource( new StringReader(
> source ) ), ResourceType.DRL);
> KnowledgeBuilderErrors errors = kbuilder.getErrors();
> if (errors.size() > 0) {
> for (KnowledgeBuilderError error: errors) {
> System.err.println(error);
> }
> throw new IllegalArgumentException("Could not parse
> knowledge.");
> }
> System.out.println("done.
"+(System.currentTimeMillis()-ts)+"ms");
>
> System.out.print("4. Creating kbase ...");
> ts = System.currentTimeMillis();
> KnowledgeBase kbase = KnowledgeBaseFactory.newKnowledgeBase();
> kbase.addKnowledgePackages(kbuilder.getKnowledgePackages());
> System.out.println("done.
"+(System.currentTimeMillis()-ts)+"ms");
> return kbase;
> }
>
> private static PackageDescr generatePackageDescr() {
> PackageDescr result = new PackageDescr("org.drools.test");
> result.addImport( new
ImportDescr("com.sample.DroolsTest.Message")
> );
> for( int i = 1; i <= NUMBER_OF_RULES; i++ ) {
> RuleDescr rule = new RuleDescr("rule "+i);
> AttributeDescr dialect = new AttributeDescr("dialect",
"mvel");
> AndDescr lhs = new AndDescr();
> PatternDescr pat = new PatternDescr("Message");
> FieldConstraintDescr constr = new
> FieldConstraintDescr("message");
> LiteralRestrictionDescr restr = new
> LiteralRestrictionDescr("==",String.valueOf( i ));
> constr.addRestriction( restr );
> pat.addConstraint( constr );
> lhs.addDescr( pat );
> rule.addAttribute( dialect );
> rule.setLhs( lhs );
> rule.setConsequence( "System.out.println("+i+");\n"
);
> result.addRule( rule );
> }
> return result;
> }
>
> public static class Message {
>
> public static final int HELLO = 0;
> public static final int GOODBYE = 1;
>
> private String message;
>
> private int status;
>
> public String getMessage() {
> return this.message;
> }
>
> public void setMessage(String message) {
> this.message = message;
> }
>
> public int getStatus() {
> return this.status;
> }
>
> public void setStatus(int status) {
> this.status = status;
> }
>
> }
>
> }
>
>
>
> 2010/2/6 Wolfgang Laun <wolfgang.laun(a)gmail.com>
>>
>> Have you tried increasing the heap size for the JVM?
>>
>> "20K rules [of the same] simple form" sounds as if you are
>> trying to use a hammer where a screwdriver would be appropriate.
>> If your description is correct, these 20K rules would be nothing
>> but 20K if statements, in disguised form. There is nothing
>> in that which would gain from Drools' Rete algorithm with
>> its superior "many-patterns-many-object" matching capabilities.
>>
>> -W
>>
>> On Sat, Feb 6, 2010 at 12:42 PM, kpowerinfinity
>> <kpowerinfinity(a)gmail.com> wrote:
>> > Hello,
>> >
>> > We are trying to run a DRL file that has more than 20K rules, all of
>> > which have the simple form:
>> >
>> >
>> >
>> > rule "testpeoplenumber"
>> > salience 100
>> > dialect "mvel"
>> > when
>> > LineItem( inventoryItemCode == "8903210392090")
>> > then
>> > InventoryItem fact0 = new InventoryItem();
>> > fact0.setCategoryCode( "ONPROMO" );
>> > insert(fact0 );
>> > end
>> >
>> > However, the rule file is not getting parsed by the builder, and
>> > either takes a very long time (well over half an hour), or gives a
>> > Heap Overflow exception. We tried with a smaller file (about 1K rules,
>> > and the situation is the same).
>> >
>> > The parsing code is:
>> > org.drools.KnowledgeBaseConfiguration kbc =
>> > org.drools.KnowledgeBaseFactory.newKnowledgeBaseConfiguration(null,
>> > cl);
>> > org.drools.KnowledgeBase kbase =
>> > org.drools.KnowledgeBaseFactory.newKnowledgeBase(kbc);
>> > org.drools.builder.KnowledgeBuilderConfiguration
>> > knowledgeBuilderConfig =
>> >
>> >
org.drools.builder.KnowledgeBuilderFactory.newKnowledgeBuilderConfiguration(null,
>> > cl);
>> > org.drools.builder.KnowledgeBuilder builder =
>> > org.drools.builder.KnowledgeBuilderFactory.newKnowledgeBuilder(kbase,
>> > knowledgeBuilderConfig);
>> > org.drools.builder.impl.KnowledgeBuilderImpl builderImpl =
>> > builder as org.drools.builder.impl.KnowledgeBuilderImpl;
>> >
>> >
>> > builder.add(org.drools.io.ResourceFactory.newFileResource(filename),
>> > org.drools.builder.ResourceType.DRL);
>> >
>> > We are using Drools v5. We've tried both with the java jar as well as
>> > in .NET (using IKVM to compile it).
>> >
>> > Any help in understanding why this is happening and how it can be
>> > avoided will be appreciated.
>> >
>> > Warm Regards
>> > Krishna.
>> >
>> > --
>> > http://kpowerinfinity.wordpress.com
>> > http://www.linkedin.com/in/kpowerinfinity
>> > _______________________________________________
>> > rules-users mailing list
>> > rules-users(a)lists.jboss.org
>> > https://lists.jboss.org/mailman/listinfo/rules-users
>> >
>>
>> _______________________________________________
>> rules-users mailing list
>> rules-users(a)lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/rules-users
>
>
>
> --
> Edson Tirelli
> JBoss Drools Core Development
> JBoss by Red Hat @ www.jboss.com
>
> _______________________________________________
> rules-users mailing list
> rules-users(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/rules-users
>
>
_______________________________________________
rules-users mailing list
rules-users(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/rules-users
--
-----------------------------
Felipe Piccolini
felipe.piccolini(a)gmail.com
11:40 p.m.
I second Felipe -
since every time you modify working memory (eg. insert), it refires all
rules, so for 20k rules, you're looking at something like 350-400 million
firings :) Or, without no-loop, it will never end. Though perhaps salience
skips this second problem.
--
View this message in context:
http://n3.nabble.com/Reading-DRL-file-with-large-number-of-rules-tp190032...
Sent from the Drools - User mailing list archive at Nabble.com.
11:50 p.m.
But isn't the way Rete work is that it creates a decision tree to work out which rules
to evaluate. Right at the top of the tree will be a check on the fact type that is being
inserted, as the fact being inserted doesn't match against any of the 20k rules (it is
a totally different object type) the evaluation will finish immediately with no further
rule evaluations.
Thomas
-----Original Message-----
From: rules-users-bounces(a)lists.jboss.org [mailto:rules-users-
bounces(a)lists.jboss.org] On Behalf Of djb
Sent: 08 February 2010 14:41
To: rules-users(a)lists.jboss.org
Subject: Re: [rules-users] Reading DRL file with large number of rules
I second Felipe -
since every time you modify working memory (eg. insert), it refires all
rules, so for 20k rules, you're looking at something like 350-400 million
firings :) Or, without no-loop, it will never end. Though perhaps salience
skips this second problem.
--
View this message in context: http://n3.nabble.com/Reading-DRL-file-with-
large-number-of-rules-tp190032p192446.html
Sent from the Drools - User mailing list archive at Nabble.com.
_______________________________________________
rules-users mailing list
rules-users(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/rules-users
**************************************************************************************
This message is confidential and intended only for the addressee. If you have received
this message in error, please immediately notify the postmaster(a)nds.com and delete it from
your system as well as any copies. The content of e-mails as well as traffic data may be
monitored by NDS for employment and security purposes. To protect the environment please
do not print this e-mail unless necessary.
NDS Limited. Registered Office: One London Road, Staines, Middlesex, TW18 4EX, United
Kingdom. A company registered in England and Wales. Registered no. 3080780. VAT no. GB 603
8808 40-00
**************************************************************************************
This message is confidential and intended only for the addressee. If you have received
this message in error, please immediately notify the postmaster(a)nds.com and delete it from
your system as well as any copies. The content of e-mails as well as traffic data may be
monitored by NDS for employment and security purposes.
To protect the environment please do not print this e-mail unless necessary.
An NDS Group Limited company. www.nds.com
3:23 a.m.
Hi people,
We are talking about several things at the same time here, so we need to
organize our answers to not turn information into noise.
* Me and Wolfgang were suggesting that Krishna evaluates his need for a
rules engine. There are several requirements that are met by such a product,
but if all he needs is to execute a piece of code based on the value of an
attribute, there are simpler alternatives. But of course, only Krishna,
knowing all of the requirements can know what is best.
* Krishna's problem is that compilation is taking too long or not finishing
at all. The example program I sent in my e-mail, shows that it is possible
to compile 20k rules and load them in memory, so it is obvious that there is
something different in Krishna's program, but we don't have any more info on
what it could be.
* Felipe: since we are talking about compilation time only (not runtime),
and since Drools handles the consequence block as a black box, there
shouldn't be any noticeable difference in compiling a single System.out call
or an object instantiation with a few attribute setting calls. The real
overhead is in initializing the compiler to compile 20k different blocks of
code. The complexity of the block code has little impact (although obviously
there is a bit higher cost) on the compilation time compared to everything
else.
* djb, Thomas is correct. The whole point of Rete is to only check and
reactivate rules that could be impacted by each fact, minimizing the cost of
rules re-evaluation. This goes back to what me and Wolfgang were talking
about. If you would make a naive algorithm with 20k "if"s, all of them would
be executed for each new fact, causing the application to run with very poor
performance. Although, it is possible to use simple techniques and still get
good performance without using Rete ***if*** your rules are simple enough
that you can manage them. Usually, it is best to just use a rules engine.
Hope comments above help to clarify things.
Edson
2010/2/8 Swindells, Thomas <TSwindells(a)nds.com>
But isn't the way Rete work is that it creates a decision tree to
work out
which rules to evaluate. Right at the top of the tree will be a check on
the fact type that is being inserted, as the fact being inserted doesn't
match against any of the 20k rules (it is a totally different object type)
the evaluation will finish immediately with no further rule evaluations.
Thomas
> -----Original Message-----
> From: rules-users-bounces(a)lists.jboss.org [mailto:rules-users-
> bounces(a)lists.jboss.org] On Behalf Of djb
> Sent: 08 February 2010 14:41
> To: rules-users(a)lists.jboss.org
> Subject: Re: [rules-users] Reading DRL file with large number of rules
>
>
> I second Felipe -
> since every time you modify working memory (eg. insert), it refires all
> rules, so for 20k rules, you're looking at something like 350-400 million
> firings :) Or, without no-loop, it will never end. Though perhaps
salience
> skips this second problem.
>
>
>
>
> --
> View this message in context:
http://n3.nabble.com/Reading-DRL-file-with-
> large-number-of-rules-tp190032p192446.html
> Sent from the Drools - User mailing list archive at Nabble.com.
> _______________________________________________
> rules-users mailing list
> rules-users(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/rules-users
**************************************************************************************
This message is confidential and intended only for the addressee. If you
have received this message in error, please immediately notify the
postmaster(a)nds.com and delete it from your system as well as any copies.
The content of e-mails as well as traffic data may be monitored by NDS for
employment and security purposes. To protect the environment please do not
print this e-mail unless necessary.
NDS Limited. Registered Office: One London Road, Staines, Middlesex, TW18
4EX, United Kingdom. A company registered in England and Wales. Registered
no. 3080780. VAT no. GB 603 8808 40-00
**************************************************************************************
This message is confidential and intended only for the addressee. If you
have received this message in error, please immediately notify the
postmaster(a)nds.com and delete it from your system as well as any copies.
The content of e-mails as well as traffic data may be monitored by NDS for
employment and security purposes.
To protect the environment please do not print this e-mail unless
necessary.
An NDS Group Limited company. www.nds.com
_______________________________________________
rules-users mailing list
rules-users(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/rules-users
--
Edson Tirelli
JBoss Drools Core Development
JBoss by Red Hat @ www.jboss.com
1:28 p.m.
Dear all,
Thanks for the responses. We would like to use a rules engine because
we need it for a generalized set of rules. And they could be more
complicated than a map lookup. However, it does seem from the numbers
reported above that there might be some issues in reading the rule and
it should not take so much time. I will investigate and get back.
Regards
Krishna.
--
http://kpowerinfinity.wordpress.com
http://www.linkedin.com/in/kpowerinfinity
2010/2/8 Edson Tirelli <ed.tirelli(a)gmail.com>:
Hi people,
We are talking about several things at the same time here, so we need to
organize our answers to not turn information into noise.
* Me and Wolfgang were suggesting that Krishna evaluates his need for a
rules engine. There are several requirements that are met by such a product,
but if all he needs is to execute a piece of code based on the value of an
attribute, there are simpler alternatives. But of course, only Krishna,
knowing all of the requirements can know what is best.
* Krishna's problem is that compilation is taking too long or not finishing
at all. The example program I sent in my e-mail, shows that it is possible
to compile 20k rules and load them in memory, so it is obvious that there is
something different in Krishna's program, but we don't have any more info on
what it could be.
* Felipe: since we are talking about compilation time only (not runtime),
and since Drools handles the consequence block as a black box, there
shouldn't be any noticeable difference in compiling a single System.out call
or an object instantiation with a few attribute setting calls. The real
overhead is in initializing the compiler to compile 20k different blocks of
code. The complexity of the block code has little impact (although obviously
there is a bit higher cost) on the compilation time compared to everything
else.
* djb, Thomas is correct. The whole point of Rete is to only check and
reactivate rules that could be impacted by each fact, minimizing the cost of
rules re-evaluation. This goes back to what me and Wolfgang were talking
about. If you would make a naive algorithm with 20k "if"s, all of them would
be executed for each new fact, causing the application to run with very poor
performance. Although, it is possible to use simple techniques and still get
good performance without using Rete ***if*** your rules are simple enough
that you can manage them. Usually, it is best to just use a rules engine.
Hope comments above help to clarify things.
Edson
2010/2/8 Swindells, Thomas <TSwindells(a)nds.com>
>
> But isn't the way Rete work is that it creates a decision tree to work out
> which rules to evaluate. Right at the top of the tree will be a check on
> the fact type that is being inserted, as the fact being inserted doesn't
> match against any of the 20k rules (it is a totally different object type)
> the evaluation will finish immediately with no further rule evaluations.
>
> Thomas
>
> > -----Original Message-----
> > From: rules-users-bounces(a)lists.jboss.org [mailto:rules-users-
> > bounces(a)lists.jboss.org] On Behalf Of djb
> > Sent: 08 February 2010 14:41
> > To: rules-users(a)lists.jboss.org
> > Subject: Re: [rules-users] Reading DRL file with large number of rules
> >
> >
> > I second Felipe -
> > since every time you modify working memory (eg. insert), it refires all
> > rules, so for 20k rules, you're looking at something like 350-400
> > million
> > firings :) Or, without no-loop, it will never end. Though perhaps
> > salience
> > skips this second problem.
> >
> >
> >
> >
> > --
> > View this message in context:
> > http://n3.nabble.com/Reading-DRL-file-with-
> > large-number-of-rules-tp190032p192446.html
> > Sent from the Drools - User mailing list archive at Nabble.com.
> > _______________________________________________
> > rules-users mailing list
> > rules-users(a)lists.jboss.org
> > https://lists.jboss.org/mailman/listinfo/rules-users
>
>
>
>
**************************************************************************************
> This message is confidential and intended only for the addressee. If you
> have received this message in error, please immediately notify the
> postmaster(a)nds.com and delete it from your system as well as any copies. The
> content of e-mails as well as traffic data may be monitored by NDS for
> employment and security purposes. To protect the environment please do not
> print this e-mail unless necessary.
>
> NDS Limited. Registered Office: One London Road, Staines, Middlesex, TW18
> 4EX, United Kingdom. A company registered in England and Wales. Registered
> no. 3080780. VAT no. GB 603 8808 40-00
>
>
**************************************************************************************
>
> This message is confidential and intended only for the addressee. If you
> have received this message in error, please immediately notify the
> postmaster(a)nds.com and delete it from your system as well as any copies. The
> content of e-mails as well as traffic data may be monitored by NDS for
> employment and security purposes.
> To protect the environment please do not print this e-mail unless
> necessary.
>
> An NDS Group Limited company. www.nds.com
>
> _______________________________________________
> rules-users mailing list
> rules-users(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/rules-users
--
Edson Tirelli
JBoss Drools Core Development
JBoss by Red Hat @ www.jboss.com
_______________________________________________
rules-users mailing list
rules-users(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/rules-users
5201
days inactive
5205
days old
8 comments
6 participants
participants (6)
-
djb -
Edson Tirelli -
Felipe Piccolini -
kpowerinfinity -
Swindells, Thomas -
Wolfgang Laun