6:07 a.m.
Hi,
Recently I was trying to optimize the rule compilation time for my KB (~4K
rules).
I do a full KB compilation every night, as most of the rules are dynamic
generated/converted and managed by the business operators.
The compile time was not very good (around 3 hours on a 8 core Xeon CPU),
but is was acceptable.
On the other night I was alarmed that the compile time was doubled overnight
without an obvious explanation or any relevant rule count increase. After
some debuging I've traced the problem and was shocked to find that most of
the rules KB (4K rules) compile under 5 minutes, except one. This single
rule was taking all the 4h to compile!
I'm using DROOLS 5.2 Final.
The rule example below samples the problem (don't mind the nonsense Number()
conditions):
rule "CONFIG_1-3UCO07"
salience -90
when
Number(rel_4226982244: intValue)
Number(rel_7521194: intValue)
Number(rel_787633980: intValue)
Number(qty_1331544548: intValue)
Number(rel_1425187049: intValue)
Number(rel_1180441096: intValue)
Number(rel_3132221704: intValue)
Number(rel_1663554156: intValue)
Number(rel_1940612775: intValue)
Number(rel_1735126416: intValue)
Number(rel_3962361266: intValue)
Number(rel_882187: intValue)
Number(rel_1169008280: intValue)
Number(rel_3495503197: intValue)
Number(rel_70290066: intValue)
Number(rel_1333860961: intValue)
Number(rel_2793542368: intValue)
Number(rel_952404632: intValue)
Number(rel_2712335119: intValue)
Number(qty_4276673135: intValue)
Number(qty_3950051097: intValue)
Number(rel_3391032645: intValue)
Number(rel_2738029181: intValue)
Number(qty_4125201080: intValue)
Number(rel_663254919: intValue)
Number(rel_3059142355: intValue)
not((eval(qty_1331544548 == 1) or eval(rel_1425187049 == 1) or
eval(rel_1180441096 == 1) or eval(rel_787633980 >= 1) or eval(rel_7521194 >=
1) or eval(rel_4226982244 >= 1) or eval(rel_2712335119 == 1) or
eval(qty_3950051097 == 1) or eval(qty_4276673135 == 1) or
eval(rel_1940612775 == 1) or eval(rel_3132221704 == 1) or eval(rel_882187 ==
1) or eval(rel_1663554156 == 1) or eval(rel_1735126416 == 1) or
eval(rel_3962361266 == 1) or eval(rel_1169008280 == 1) or
eval(rel_2793542368 == 1) or eval(rel_3495503197 == 1) or
eval(qty_4125201080 == 1) or eval(rel_70290066 == 1) or eval(rel_3391032645
== 1) or eval(rel_1333860961 == 1) or eval(rel_2738029181 == 1) or
eval(rel_663254919 == 1) or eval(rel_952404632 == 1) or eval(rel_3059142355
== 1)))
then
vh.error(kcontext, "Error ...");
end
I've noticed that the last "not(...)" is the problematic condition. Is
there
a limit using eval() and or-clauses, or a known problem?
Regards,
Tiago Lopes
--
View this message in context:
http://drools.46999.n3.nabble.com/Slow-compilation-4h-for-a-single-rule-t...
Sent from the Drools: User forum mailing list archive at Nabble.com.
6:11 a.m.
On 24/07/2012, fx242 <drools(a)fx242.com> wrote:
Hi,
This single
rule was taking all the 4h to compile!
I'm using DROOLS 5.2 Final.
The rule example below samples the problem (don't mind the nonsense
Number()
conditions):
Well, I do mind the "nonsense Number()". There's no point in
discussing this if you don't post an exact image of your rule.
-W
6:23 a.m.
laune wrote
Well, I do mind the "nonsense Number()". There's no point in
discussing this if you don't post an exact image of your rule.
-W
There is a point, because the rule I've posted compiles and causes the same
problem, I've just removed redundant code to expose the problematic section.
TL
--
View this message in context:
http://drools.46999.n3.nabble.com/Slow-compilation-4h-for-a-single-rule-t...
Sent from the Drools: User forum mailing list archive at Nabble.com.
6:24 a.m.
As far as I remember, there were some fixes around compilation time for
edge cases in 5.4. Could you please give a try to 5.4 or even 5.5-SNAPSHOT?
Best Regards,
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
Esteban Aliverti
- Blog @ http://ilesteban.wordpress.com
On Tue, Jul 24, 2012 at 1:11 PM, Wolfgang Laun <wolfgang.laun(a)gmail.com>wrote:
On 24/07/2012, fx242 <drools(a)fx242.com> wrote:
> Hi,
>
> This single
> rule was taking all the 4h to compile!
> I'm using DROOLS 5.2 Final.
> The rule example below samples the problem (don't mind the nonsense
> Number()
> conditions):
>
Well, I do mind the "nonsense Number()". There's no point in
discussing this if you don't post an exact image of your rule.
-W
_______________________________________________
rules-users mailing list
rules-users(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/rules-users
6:57 a.m.
There are 26 patterns Number(). Now I don't know how many facts of
that type you have, but this sequence will produce all combinations of
all available facts. (This is the worst example of this anti-pattern
I've seen.)
The or-ed evals will try to create parallel nodes in the network, and
the not should guarantee that none of those exists.
I'm not sure what the rule should actually ascertain. Could you
provide a simple example, using 3 numbers, describing verbally or by a
regular boolean expression when the rule should fire?
E.g. "it should fire if there is one Number != 1, another Number < 1
and a third Number != 1"
-W
On 24/07/2012, Esteban Aliverti <esteban.aliverti(a)gmail.com> wrote:
As far as I remember, there were some fixes around compilation time
for
edge cases in 5.4. Could you please give a try to 5.4 or even 5.5-SNAPSHOT?
Best Regards,
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
Esteban Aliverti
- Blog @ http://ilesteban.wordpress.com
On Tue, Jul 24, 2012 at 1:11 PM, Wolfgang Laun
<wolfgang.laun(a)gmail.com>wrote:
> On 24/07/2012, fx242 <drools(a)fx242.com> wrote:
> > Hi,
> >
> > This single
> > rule was taking all the 4h to compile!
> > I'm using DROOLS 5.2 Final.
> > The rule example below samples the problem (don't mind the nonsense
> > Number()
> > conditions):
> >
>
> Well, I do mind the "nonsense Number()". There's no point in
> discussing this if you don't post an exact image of your rule.
>
> -W
> _______________________________________________
> rules-users mailing list
> rules-users(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/rules-users
>
8:29 a.m.
I understand that the rule looks pointless and under-optimized, but it's the
only way I could represent this kind of rule in an automated way (I didn't
write the rule by hand). Basically with these rules I try to pre-compute the
amount of some products on the WM (using count() inside a accumulate
function) resulting in a Number(), and then on the last condition I evaluate
some arbitrary arithmetic operation using the variables inside the Number()
object.
For more details for examples see my previous post where I've tried to
address this issue:
http://drools.46999.n3.nabble.com/drools-arithmetics-without-eval-td38232...
So, considerations apart of the way the rule is written, I still think there
is a bug or something similar on the RETE compiler, that causes this rule to
be processed in a very expensive way. Can someone confirm this, and try to
sugest an alternative? Maybe It's something that is already fixed on newer
versions as sugested by Esteban?
Regards,
Tiago Lopes
--
View this message in context:
http://drools.46999.n3.nabble.com/Slow-compilation-4h-for-a-single-rule-t...
Sent from the Drools: User forum mailing list archive at Nabble.com.
12:16 p.m.
You are trying to avoid the issue. As reported by others, your
conditions should be inside the patterns, not in "or'd" evals in the end,
and yes, it is possible to generate rules like that in an automated program.
Now, the reason it is taking so long is because of the "or"s. I don't
think there is any bug there. Just for you to understand, an OR is just
syntax sugar for multiple rules. So if you have a rule that says:
A() or B()
The engine is actually creating 2 rules:
rule 1 when
A()
...
rule 2 when
B()
...
I.e., it generates all possible combinations of rules, one for each of
your logical branches.
Now, I quickly counted 26 "or"s in your rule. It is quite possible that
the engine is generating all possible combinations of rules for all your
ORs, and that is roughly (there are some factors that influence the number
of rules): 2^26 == 67108864 rules. To compile 67 million rules, it can
indeed take some time.
I don't know for sure if that is the case without looking at your
application, but to solve your problem you should fix your rule generation,
not look for a problem elsewhere.
Edson
On Tue, Jul 24, 2012 at 9:29 AM, fx242 <drools(a)fx242.com> wrote:
I understand that the rule looks pointless and under-optimized, but
it's
the
only way I could represent this kind of rule in an automated way (I didn't
write the rule by hand). Basically with these rules I try to pre-compute
the
amount of some products on the WM (using count() inside a accumulate
function) resulting in a Number(), and then on the last condition I
evaluate
some arbitrary arithmetic operation using the variables inside the Number()
object.
For more details for examples see my previous post where I've tried to
address this issue:
http://drools.46999.n3.nabble.com/drools-arithmetics-without-eval-td38232...
So, considerations apart of the way the rule is written, I still think
there
is a bug or something similar on the RETE compiler, that causes this rule
to
be processed in a very expensive way. Can someone confirm this, and try to
sugest an alternative? Maybe It's something that is already fixed on newer
versions as sugested by Esteban?
Regards,
Tiago Lopes
--
View this message in context:
http://drools.46999.n3.nabble.com/Slow-compilation-4h-for-a-single-rule-t...
Sent from the Drools: User forum mailing list archive at Nabble.com.
_______________________________________________
rules-users mailing list
rules-users(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/rules-users
--
Edson Tirelli
JBoss Drools Core Development
JBoss by Red Hat @ www.jboss.com
4:49 a.m.
Thanks for your explanation Edson, it makes sense now I think!
Basically what you are saying is that there is a limit, beyond which is no
longer feasible to use more ORs on a rule.
Edson Tirelli-4 wrote
You are trying to avoid the issue. As reported by others, your
conditions should be inside the patterns, not in "or'd" evals in the end,
and yes, it is possible to generate rules like that in an automated
program.
Sadly I don't see how, as some of my rules have arbitrary arithmetic
involving more than one variable inside the Number(), for more context see
the example I've put on
http://drools.46999.n3.nabble.com/drools-arithmetics-without-eval-td38232....
For this case I think I will try to invert the logic and get rid of the ORs:
not(A or B) => not(A) and not(B).
Best regards,
TL
--
View this message in context:
http://drools.46999.n3.nabble.com/Slow-compilation-4h-for-a-single-rule-t...
Sent from the Drools: User forum mailing list archive at Nabble.com.
6:42 a.m.
A last effort from my side.
Given any number of bound variables a, b, c: to determine whether an
arbitrary boolean expression involving arithmetic and comparison
operators, a single eval() is sufficient and preferable to a CE
combination involving the CE operators "and", "or", "not"
combining
many evals.
Instead of
not( eval( a == 1 ) )
use
eval( ! (a == 1) ) // or better
eval( a != 0 )
Instead of
eval( a == 1 ) or eval( b == 1 )
use
eval( a == 1 || b == 1 )
Instead of
eval( a == 1 ) and eval( b == 1 )
use
eval( a == 1 && b == 1 )
-W
On 25/07/2012, fx242 <drools(a)fx242.com> wrote:
Thanks for your explanation Edson, it makes sense now I think!
Basically what you are saying is that there is a limit, beyond which is no
longer feasible to use more ORs on a rule.
Edson Tirelli-4 wrote
>
> You are trying to avoid the issue. As reported by others, your
> conditions should be inside the patterns, not in "or'd" evals in the
end,
> and yes, it is possible to generate rules like that in an automated
> program.
>
Sadly I don't see how, as some of my rules have arbitrary arithmetic
involving more than one variable inside the Number(), for more context see
the example I've put on
http://drools.46999.n3.nabble.com/drools-arithmetics-without-eval-td38232....
For this case I think I will try to invert the logic and get rid of the
ORs:
not(A or B) => not(A) and not(B).
Best regards,
TL
--
View this message in context:
http://drools.46999.n3.nabble.com/Slow-compilation-4h-for-a-single-rule-t...
Sent from the Drools: User forum mailing list archive at Nabble.com.
_______________________________________________
rules-users mailing list
rules-users(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/rules-users
9:29 a.m.
Listen to what others are saying in this thread... it is very good
advice, specially what wolfgang suggests bellow regarding collapsing evals.
Also, I just read the example in your link where you have multiple
accumulates in a single rule... that is *bad*....
For those that attended or saw my presentation on best practices, an
easy way to estimate the performance of accumulates is to remember that the
performance of accumulates in a single rule is polinomial: O(n^a), where
"n" is the number of matching facts and "a" is the number of
accumulates
you have in the same rule.
So, if you have 1 rule with 5 accumulates, the performance will be n^5.
If instead you break the accumulates into 5 rules with one accumulate each,
you get a performance of O(5n) that is much better.
Edson
On Wed, Jul 25, 2012 at 7:42 AM, Wolfgang Laun <wolfgang.laun(a)gmail.com>wrote:
A last effort from my side.
Given any number of bound variables a, b, c: to determine whether an
arbitrary boolean expression involving arithmetic and comparison
operators, a single eval() is sufficient and preferable to a CE
combination involving the CE operators "and", "or", "not"
combining
many evals.
Instead of
not( eval( a == 1 ) )
use
eval( ! (a == 1) ) // or better
eval( a != 0 )
Instead of
eval( a == 1 ) or eval( b == 1 )
use
eval( a == 1 || b == 1 )
Instead of
eval( a == 1 ) and eval( b == 1 )
use
eval( a == 1 && b == 1 )
-W
On 25/07/2012, fx242 <drools(a)fx242.com> wrote:
> Thanks for your explanation Edson, it makes sense now I think!
> Basically what you are saying is that there is a limit, beyond which is
no
> longer feasible to use more ORs on a rule.
>
>
> Edson Tirelli-4 wrote
>>
>> You are trying to avoid the issue. As reported by others, your
>> conditions should be inside the patterns, not in "or'd" evals in
the
end,
>> and yes, it is possible to generate rules like that in an automated
>> program.
>>
>
> Sadly I don't see how, as some of my rules have arbitrary arithmetic
> involving more than one variable inside the Number(), for more context
see
> the example I've put on
>
http://drools.46999.n3.nabble.com/drools-arithmetics-without-eval-td38232...
.
>
> For this case I think I will try to invert the logic and get rid of the
> ORs:
> not(A or B) => not(A) and not(B).
>
> Best regards,
> TL
>
>
>
>
> --
> View this message in context:
>
http://drools.46999.n3.nabble.com/Slow-compilation-4h-for-a-single-rule-t...
> Sent from the Drools: User forum mailing list archive at Nabble.com.
> _______________________________________________
> rules-users mailing list
> rules-users(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/rules-users
>
_______________________________________________
rules-users mailing list
rules-users(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/rules-users
--
Edson Tirelli
JBoss Drools Core Development
JBoss by Red Hat @ www.jboss.com
10:14 a.m.
Thank you all for your feedback, I will try to implement at least the part of
collapsing the multiple evals() into a single one.
Edson Tirelli-4 wrote
Also, I just read the example in your link where you have multiple
accumulates in a single rule... that is *bad*....
The accumulate() over-usage is a known problem for me, but so far the best
solution for my case was proposed by "Vincent Legendre" on this
http://drools.46999.n3.nabble.com/drools-arithmetics-without-eval-tp38232...
post .
As a comic note, the previous rule with the 26 ORs, had 26 accumulates on it
:)
Best regards,
TL
--
View this message in context:
http://drools.46999.n3.nabble.com/Slow-compilation-4h-for-a-single-rule-t...
Sent from the Drools: User forum mailing list archive at Nabble.com.
7:13 a.m.
Hi,
There is something that I must ask... Why do you evaluate perfectly simple
patterns in an eval, I wonder?
rule "CONFIG_1-3UCO07"
salience -90
when
Number(rel_4226982244: intValue < 1 )
Number(rel_7521194: intValue < 1 )
Number(rel_787633980: intValue < 1 )
Number(qty_1331544548: intValue != 1 )
Number(rel_1425187049: intValue != 1 )
Number(rel_1180441096: intValue != 1 )
Number(rel_3132221704: intValue != 1 )
Number(rel_1663554156: intValue != 1 )
Number(rel_1940612775: intValue != 1 )
Number(rel_1735126416: intValue != 1 )
Number(rel_3962361266: intValue != 1 )
Number(rel_882187: intValue != 1 )
Number(rel_1169008280: intValue != 1 )
Number(rel_3495503197: intValue != 1 )
Number(rel_70290066: intValue != 1 )
Number(rel_1333860961: intValue != 1 )
Number(rel_2793542368: intValue != 1 )
Number(rel_952404632: intValue != 1 )
Number(rel_2712335119: intValue != 1 )
Number(qty_4276673135: intValue != 1 )
Number(qty_3950051097: intValue != 1 )
Number(rel_3391032645: intValue != 1 )
Number(rel_2738029181: intValue != 1 )
Number(qty_4125201080: intValue != 1 )
Number(rel_663254919: intValue != 1 )
Number(rel_3059142355: intValue != 1 )
then
vh.error(kcontext, "Error ...");
end
Of course, this is still nonsense as long as you do not have a way to
constrain the number that you evaluate. The rule, as written above, will
fire from the moment you have one single number that has an intValue below
1.
Regards,
Frank
--
View this message in context:
http://drools.46999.n3.nabble.com/Slow-compilation-4h-for-a-single-rule-t...
Sent from the Drools: User forum mailing list archive at Nabble.com.
4504
days inactive
4505
days old
11 comments
5 participants
participants (5)
-
Edson Tirelli -
Esteban Aliverti -
FrankVhh -
fx242 -
Wolfgang Laun