10:44 p.m.
hi all,
I'm quite new to the drools rule engine and i've been using it for a project where
i need to filter events and execute actions accordingly.
I'm currently running a DRL file and creating a stateful session from it, and then
running 2/3 facts every time. Every X seconds I collect the available facts and call
fireAllRules() + dispose(), always within the same session. The DRL file has circa 2160
rules declared (an external program is producing them), and i'm using the
"no-loop", "lock-on-active", "activation-group" and
"agenda-group" options on almost every one.
My main problem is that the software is consuming too much memory when it loads the rules
to the knowledge base variable, it reaches +300Mb which is just not affordable in this
project. Is there any way i can work around this? I've tried setting some
configuration properties but to no avail. What techniques may i try in order to reduce
memory load on drools startup? What would be the best approach for dealing with so many
rules?
Anything would help, thanks in advance,
_ miguel
2:32 a.m.
I am not sure I understand what you are doing when you say:
Every X seconds I collect the available facts and call fireAllRules()
+
dispose(), **always within the same session**.
(emphasis is mine)
Compilation indeed uses more memory, but you can compile once and reuse
the kbase. Also, if you are calling dispose, I guess you are running
stateful sessions. Maybe you can give us a more detailed description of your
session/data/application lifecycle?
Edson
2010/3/30 Miguel Machado <mls.machado(a)gmail.com>
hi all,
I'm quite new to the drools rule engine and i've been using it for a
project where i need to filter events and execute actions accordingly.
I'm currently running a DRL file and creating a stateful session from it,
and then running 2/3 facts every time. Every X seconds I collect the
available facts and call fireAllRules() + dispose(), always within the same
session. The DRL file has circa 2160 rules declared (an external program is
producing them), and i'm using the "no-loop", "lock-on-active",
"activation-group" and "agenda-group" options on almost every one.
My main problem is that the software is consuming too much memory when it
loads the rules to the knowledge base variable, it reaches +300Mb which is
just not affordable in this project. Is there any way i can work around
this? I've tried setting some configuration properties but to no avail. What
techniques may i try in order to reduce memory load on drools startup? What
would be the best approach for dealing with so many rules?
Anything would help, thanks in advance,
_ miguel
_______________________________________________
rules-users mailing list
rules-users(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/rules-users
--
Edson Tirelli
JBoss Drools Core Development
JBoss by Red Hat @ www.jboss.com
5:53 a.m.
hi again,
On 30 Mar 2010, at 18:32, Edson Tirelli <ed.tirelli(a)gmail.com> wrote:
I am not sure I understand what you are doing when you say:
Every X seconds I collect the available facts and call fireAllRules()
+
dispose(), **always within the same session**.
(emphasis is mine)
i meant that i keep using the same object, i just create one stateful
session throughout the program execution.
Compilation indeed uses more memory, but you can compile once and reuse
the kbase.
exactly, i'm doing that, but it's a +700 rule file and its taking too much
memory. Is there anything i can do to push it down a little bit? i wouldn't
mind slowing it down to reduce memory footprint.
Also, if you are calling dispose, I guess you are running stateful sessions.
Maybe you can give us a more detailed description of your
session/data/application lifecycle?
Sure, i'll try. So i have this thread which initializes by reading the rule
base from a drl file. It then creates a stateful session from it and enters
a lifecycle in which it goes to sleep and only when some other thread wakes
it up, it collects and inserts some facts (usually 2,3 or 4..) into the
session and fire all rules. It's basically this. What i'm very concerned
about is the amount of RAM memory the whole program is consuming. I must
find a way to reduce some of it, because the rule file is being prepared by
an external program and i have no control over the number of rules i might
get. If i test it with a little over 700 rules, it reaches +300Mb which is a
lot.
Maybe the rules are too complex? Shall i try to make them simpler (how)?
what are the big memory hogs when it comes to loading a DRL file to the
knowledge base? From what i've been testing, it doesn't even matter which
rules you will actually need, it just loads the entire DRL file with no
respect to the future activations, am i right? Is there any other aspect i
should take in mind or maybe other approach?
_ miguel
2:53 p.m.
The figures (+300MB) you quote are frightening, and it would be *very*
interesting to learn what sort of rules causes this excessive amount
of heap usage. Could you post typical patterns - if they are
generated, there ought to be - and guestimates w.r,t, their relative
frequencies?
-W
On 3/30/10, miguel machado <mls.machado(a)gmail.com> wrote:
hi again,
On 30 Mar 2010, at 18:32, Edson Tirelli <ed.tirelli(a)gmail.com> wrote:
I am not sure I understand what you are doing when you say:
> Every X seconds I collect the available facts and call fireAllRules() +
dispose(), **always within the same session**.
(emphasis is mine)
i meant that i keep using the same object, i just create one stateful
session throughout the program execution.
Compilation indeed uses more memory, but you can compile once and reuse
the kbase.
exactly, i'm doing that, but it's a +700 rule file and its taking too much
memory. Is there anything i can do to push it down a little bit? i wouldn't
mind slowing it down to reduce memory footprint.
Also, if you are calling dispose, I guess you are running stateful sessions.
Maybe you can give us a more detailed description of your
session/data/application lifecycle?
Sure, i'll try. So i have this thread which initializes by reading the rule
base from a drl file. It then creates a stateful session from it and enters
a lifecycle in which it goes to sleep and only when some other thread wakes
it up, it collects and inserts some facts (usually 2,3 or 4..) into the
session and fire all rules. It's basically this. What i'm very concerned
about is the amount of RAM memory the whole program is consuming. I must
find a way to reduce some of it, because the rule file is being prepared by
an external program and i have no control over the number of rules i might
get. If i test it with a little over 700 rules, it reaches +300Mb which is a
lot.
Maybe the rules are too complex? Shall i try to make them simpler (how)?
what are the big memory hogs when it comes to loading a DRL file to the
knowledge base? From what i've been testing, it doesn't even matter which
rules you will actually need, it just loads the entire DRL file with no
respect to the future activations, am i right? Is there any other aspect i
should take in mind or maybe other approach?
_ miguel
5:44 p.m.
Hi, thanks for answering so quickly.
My ±2100 rules are grouped in 700 groups of 3 drools rules each, which are related to the
same type of event i want to generate, like the following:
rule "event1 reset" no-loop salience 9 activation-group "event1_group"
lock-on-active agenda-group "a3ce4a97-13d8-4bec-a9ab-5625fcd0f105"
when
$source : SourceBO(
ciUUID == "3938adc4-246d-43f3-b606-da4c3f09d49b",
$object : object,
$CI : ciUUID,
$sourceUUID : sourceUUID)
$value1 : SourceValueBO ( valueName == "VALUE1", numericValue < 90 )
eval (checkMayReset($source, "fb95632f-fac9-4359-9d7c-dc38d3e48e20", $CI,
$object))
then
modify ($value1) { setConditionValue(90) };
resetSource($source, "fb95632f-fac9-4359-9d7c-dc38d3e48e20", $CI, $object,
"event_version", "event_text", Arrays.asList($value1));
end
rule "event1 fire" no-loop salience 6 activation-group "event1_group"
lock-on-active agenda-group "a3ce4a97-13d8-4bec-a9ab-5625fcd0f105"
when
$source : SourceBO(
ciUUID == "3938adc4-246d-43f3-b606-da4c3f09d49b",
$object : object,
$CI : ciUUID,
$sourceUUID : sourceUUID )
$value1: SourceValueBO ( valueName == "VALUE1", numericValue > 90 )
then
modify ($value1) { setConditionValue(90) };
FireCondition $start = new FireCondition("repeat", 0, "duration",
"0s");
addToActivationQueue($source, "fb95632f-fac9-4359-9d7c-dc38d3e48e20", $CI,
$object, $start, null, "event_version", "event_text",
Arrays.asList($value1));
retract($source);
end
rule "event1 clear" no-loop salience 5 activation-group "event1_group"
lock-on-active agenda-group "a3ce4a97-13d8-4bec-a9ab-5625fcd0f105"
when
$source : SourceBO(
ciUUID == "3938adc4-246d-43f3-b606-da4c3f09d49b",
$object : object,
$CI : ciUUID,
$sourceUUID : sourceUUID)
then
clearActivationQueue($source, "fb95632f-fac9-4359-9d7c-dc38d3e48e20", $CI,
$object);
end
So, i'l try to explain briefly:
* The reset rule evaluates 2 patterns and one condition in a static method available via
"import function". As a consequence, it modifies the SourceValueBO object and
call's the reset function which is another static method.
* The fire rule is very similar to the reset rule, the difference being that the value
threshold is different (usually the opposite)
* The clear rule is like an "else" to the two above. I want it to fire in case
it didn't fall for either one of reset and fire rules.
The activation group assures that only one of these rules is fired each time i call
session.fireAllRules(). And the agenda-group allows me to stack some rules above others,
according to the type of facts i insert into memory.
Everything seems in place and it works as intended, the thing is i might be dealing with
*lots* of events (event1, event2, etc...) and lots of different UUID's which results
in a very large DRL file. For instance, if i have 700 events (event1, event ...event700),
i get 2100 drools rules (event1 reset, event1 fire, event1 clear ... event700 reset,
event700 fire, event700 clear) and over 300MB in consumed memory.
Perhaps if i took another approach? or some performance tunning, i don't know. Where
can i improve this memory hogging?
_ miguel
On 31 Mar 2010, at 06:53, Wolfgang Laun wrote:
The figures (+300MB) you quote are frightening, and it would be
*very*
interesting to learn what sort of rules causes this excessive amount
of heap usage. Could you post typical patterns - if they are
generated, there ought to be - and guestimates w.r,t, their relative
frequencies?
-W
On 3/30/10, miguel machado <mls.machado(a)gmail.com> wrote:
> hi again,
>
> On 30 Mar 2010, at 18:32, Edson Tirelli <ed.tirelli(a)gmail.com> wrote:
>
> I am not sure I understand what you are doing when you say:
>
>> Every X seconds I collect the available facts and call fireAllRules() +
> dispose(), **always within the same session**.
> (emphasis is mine)
>
> i meant that i keep using the same object, i just create one stateful
> session throughout the program execution.
>
> Compilation indeed uses more memory, but you can compile once and reuse
> the kbase.
>
>
> exactly, i'm doing that, but it's a +700 rule file and its taking too much
> memory. Is there anything i can do to push it down a little bit? i wouldn't
> mind slowing it down to reduce memory footprint.
>
> Also, if you are calling dispose, I guess you are running stateful sessions.
> Maybe you can give us a more detailed description of your
> session/data/application lifecycle?
>
> Sure, i'll try. So i have this thread which initializes by reading the rule
> base from a drl file. It then creates a stateful session from it and enters
> a lifecycle in which it goes to sleep and only when some other thread wakes
> it up, it collects and inserts some facts (usually 2,3 or 4..) into the
> session and fire all rules. It's basically this. What i'm very concerned
> about is the amount of RAM memory the whole program is consuming. I must
> find a way to reduce some of it, because the rule file is being prepared by
> an external program and i have no control over the number of rules i might
> get. If i test it with a little over 700 rules, it reaches +300Mb which is a
> lot.
>
> Maybe the rules are too complex? Shall i try to make them simpler (how)?
> what are the big memory hogs when it comes to loading a DRL file to the
> knowledge base? From what i've been testing, it doesn't even matter which
> rules you will actually need, it just loads the entire DRL file with no
> respect to the future activations, am i right? Is there any other aspect i
> should take in mind or maybe other approach?
>
> _ miguel
>
_______________________________________________
rules-users mailing list
rules-users(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/rules-users
11:03 p.m.
I can only offer a few hints:
Factor out all code that can be done in one of the called methods, e.g., in
resetSource( ..., Arrays.asList($value1));
Here, the Arrays.asList() should be done in resetSource().
Avoid multiple bindings. If you call
resetSource( $source,....)
or
checkMayReset( $source,...)
the additional parameters $object and $CI can be retrieved within
resetSource().
The funny String literals are the one thing that are strikingly different
from what you usually see in rules. String literals cost for each occurrence
unless you intern strings using javal.lang.String.intern(). (I haven't tried
whether Drools would accept that instead.) But calculating the raw cost as
(46 chars*2 bytes/char+20 bytes)*10 times * 700 rule triples only accounts
for 0.8MB, so this alone isn't the culprit.
-W
2010/3/31 Miguel Machado <mls.machado(a)gmail.com>
Hi, thanks for answering so quickly.
My ±2100 rules are grouped in 700 groups of 3 drools rules each, which are
related to the same type of event i want to generate, like the following:
rule "event1 reset" no-loop salience 9 activation-group
"event1_group"
lock-on-active agenda-group "a3ce4a97-13d8-4bec-a9ab-5625fcd0f105"
when
$source : SourceBO(
ciUUID == "3938adc4-246d-43f3-b606-da4c3f09d49b",
$object : object,
$CI : ciUUID,
$sourceUUID : sourceUUID)
$value1 : SourceValueBO ( valueName == "VALUE1", numericValue < 90 )
eval (checkMayReset($source, "fb95632f-fac9-4359-9d7c-dc38d3e48e20", $CI,
$object))
then
modify ($value1) { setConditionValue(90) };
resetSource($source, "fb95632f-fac9-4359-9d7c-dc38d3e48e20", $CI, $object,
"event_version", "event_text", Arrays.asList($value1));
end
rule "event1 fire" no-loop salience 6 activation-group
"event1_group" lock-on-active agenda-group
"a3ce4a97-13d8-4bec-a9ab-5625fcd0f105"
when
$source : SourceBO(
ciUUID == "3938adc4-246d-43f3-b606-da4c3f09d49b",
$object : object,
$CI : ciUUID,
$sourceUUID : sourceUUID )
$value1: SourceValueBO ( valueName == "VALUE1", numericValue > 90 )
then
modify ($value1) { setConditionValue(90) };
FireCondition $start = new FireCondition("repeat", 0, "duration",
"0s");
addToActivationQueue($source, "fb95632f-fac9-4359-9d7c-dc38d3e48e20", $CI,
$object, $start, null, "event_version", "event_text",
Arrays.asList($value1));
retract($source);
end
rule "event1 clear" no-loop salience 5 activation-group
"event1_group"
lock-on-active agenda-group "a3ce4a97-13d8-4bec-a9ab-5625fcd0f105"
when
$source : SourceBO(
ciUUID == "3938adc4-246d-43f3-b606-da4c3f09d49b",
$object : object,
$CI : ciUUID,
$sourceUUID : sourceUUID)
then
clearActivationQueue($source, "fb95632f-fac9-4359-9d7c-dc38d3e48e20", $CI,
$object);
end
So, i'l try to explain briefly:
* The reset rule evaluates 2 patterns and one condition in a static method
available via "import function". As a consequence, it modifies the
SourceValueBO object and call's the reset function which is another static
method.
* The fire rule is very similar to the reset rule, the difference being
that the value threshold is different (usually the opposite)
* The clear rule is like an "else" to the two above. I want it to fire in
case it didn't fall for either one of reset and fire rules.
The activation group assures that only one of these rules is fired each
time i call session.fireAllRules(). And the agenda-group allows me to stack
some rules above others, according to the type of facts i insert into
memory.
Everything seems in place and it works as intended, the thing is i might be
dealing with *lots* of events (event1, event2, etc...) and lots of different
UUID's which results in a very large DRL file. For instance, if i have 700
events (event1, event ...event700), i get 2100 drools rules (event1 reset,
event1 fire, event1 clear ... event700 reset, event700 fire, event700 clear)
and over 300MB in consumed memory.
Perhaps if i took another approach? or some performance tunning, i don't
know. Where can i improve this memory hogging?
_ miguel
On 31 Mar 2010, at 06:53, Wolfgang Laun wrote:
The figures (+300MB) you quote are frightening, and it would be *very*
interesting to learn what sort of rules causes this excessive amount
of heap usage. Could you post typical patterns - if they are
generated, there ought to be - and guestimates w.r,t, their relative
frequencies?
-W
On 3/30/10, miguel machado <mls.machado(a)gmail.com> wrote:
hi again,
On 30 Mar 2010, at 18:32, Edson Tirelli <ed.tirelli(a)gmail.com> wrote:
I am not sure I understand what you are doing when you say:
Every X seconds I collect the available facts and call fireAllRules() +
dispose(), **always within the same session**.
(emphasis is mine)
i meant that i keep using the same object, i just create one stateful
session throughout the program execution.
Compilation indeed uses more memory, but you can compile once and reuse
the kbase.
exactly, i'm doing that, but it's a +700 rule file and its taking too much
memory. Is there anything i can do to push it down a little bit? i wouldn't
mind slowing it down to reduce memory footprint.
Also, if you are calling dispose, I guess you are running stateful
sessions.
Maybe you can give us a more detailed description of your
session/data/application lifecycle?
Sure, i'll try. So i have this thread which initializes by reading the rule
base from a drl file. It then creates a stateful session from it and enters
a lifecycle in which it goes to sleep and only when some other thread wakes
it up, it collects and inserts some facts (usually 2,3 or 4..) into the
session and fire all rules. It's basically this. What i'm very concerned
about is the amount of RAM memory the whole program is consuming. I must
find a way to reduce some of it, because the rule file is being prepared by
an external program and i have no control over the number of rules i might
get. If i test it with a little over 700 rules, it reaches +300Mb which is
a
lot.
Maybe the rules are too complex? Shall i try to make them simpler (how)?
what are the big memory hogs when it comes to loading a DRL file to the
knowledge base? From what i've been testing, it doesn't even matter which
rules you will actually need, it just loads the entire DRL file with no
respect to the future activations, am i right? Is there any other aspect i
should take in mind or maybe other approach?
_ miguel
_______________________________________________
rules-users mailing list
rules-users(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/rules-users
_______________________________________________
rules-users mailing list
rules-users(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/rules-users
12:10 a.m.
Miguel,
First we need to understand where is this memory spent. Drools
externalizes the actual rule compilation to a java compiler (JDT or Janino)
and I would not be surprised if compiling 2100 rules would consume more than
300Mb **during compilation**. I don't think that after compiling, the
application will continue to use that amount of memory just because of
rules, although if you have many facts being inserted into your session, it
obviously will drive consumption up.
So, some questions:
1. Do you use one KnowledgeBuilder to compile all your rule files or do you
use one KnowledgeBuilder per file?
2. How much memory is your application using during this step, where you add
resources to your knowledgeBuilder(s)?
3. After compiling your knowledge packages, did you tried to force a GC to
check how much memory it continues to use?
4. Are you adding all rules to the same KnowledgeBase or using one kbase per
set of rules? How much memory are you consuming in this step?
5. Did you tried compiling it once, generating the kbase and serializing it
to a file, and then, instead of recompiling everything, just deserialize the
file into memory and using it? How much memory it consumes when you
deserialize from a file?
Knowing the answer to the above questions will help to determine if you
have a problem and how to work around it or fix it.
Edson
2010/3/31 Miguel Machado <mls.machado(a)gmail.com>
Hi, thanks for answering so quickly.
My ±2100 rules are grouped in 700 groups of 3 drools rules each, which are
related to the same type of event i want to generate, like the following:
rule "event1 reset" no-loop salience 9 activation-group
"event1_group"
lock-on-active agenda-group "a3ce4a97-13d8-4bec-a9ab-5625fcd0f105"
when
$source : SourceBO(
ciUUID == "3938adc4-246d-43f3-b606-da4c3f09d49b",
$object : object,
$CI : ciUUID,
$sourceUUID : sourceUUID)
$value1 : SourceValueBO ( valueName == "VALUE1", numericValue < 90 )
eval (checkMayReset($source, "fb95632f-fac9-4359-9d7c-dc38d3e48e20", $CI,
$object))
then
modify ($value1) { setConditionValue(90) };
resetSource($source, "fb95632f-fac9-4359-9d7c-dc38d3e48e20", $CI, $object,
"event_version", "event_text", Arrays.asList($value1));
end
rule "event1 fire" no-loop salience 6 activation-group
"event1_group" lock-on-active agenda-group
"a3ce4a97-13d8-4bec-a9ab-5625fcd0f105"
when
$source : SourceBO(
ciUUID == "3938adc4-246d-43f3-b606-da4c3f09d49b",
$object : object,
$CI : ciUUID,
$sourceUUID : sourceUUID )
$value1: SourceValueBO ( valueName == "VALUE1", numericValue > 90 )
then
modify ($value1) { setConditionValue(90) };
FireCondition $start = new FireCondition("repeat", 0, "duration",
"0s");
addToActivationQueue($source, "fb95632f-fac9-4359-9d7c-dc38d3e48e20", $CI,
$object, $start, null, "event_version", "event_text",
Arrays.asList($value1));
retract($source);
end
rule "event1 clear" no-loop salience 5 activation-group
"event1_group"
lock-on-active agenda-group "a3ce4a97-13d8-4bec-a9ab-5625fcd0f105"
when
$source : SourceBO(
ciUUID == "3938adc4-246d-43f3-b606-da4c3f09d49b",
$object : object,
$CI : ciUUID,
$sourceUUID : sourceUUID)
then
clearActivationQueue($source, "fb95632f-fac9-4359-9d7c-dc38d3e48e20", $CI,
$object);
end
So, i'l try to explain briefly:
* The reset rule evaluates 2 patterns and one condition in a static method
available via "import function". As a consequence, it modifies the
SourceValueBO object and call's the reset function which is another static
method.
* The fire rule is very similar to the reset rule, the difference being
that the value threshold is different (usually the opposite)
* The clear rule is like an "else" to the two above. I want it to fire in
case it didn't fall for either one of reset and fire rules.
The activation group assures that only one of these rules is fired each
time i call session.fireAllRules(). And the agenda-group allows me to stack
some rules above others, according to the type of facts i insert into
memory.
Everything seems in place and it works as intended, the thing is i might be
dealing with *lots* of events (event1, event2, etc...) and lots of different
UUID's which results in a very large DRL file. For instance, if i have 700
events (event1, event ...event700), i get 2100 drools rules (event1 reset,
event1 fire, event1 clear ... event700 reset, event700 fire, event700 clear)
and over 300MB in consumed memory.
Perhaps if i took another approach? or some performance tunning, i don't
know. Where can i improve this memory hogging?
_ miguel
On 31 Mar 2010, at 06:53, Wolfgang Laun wrote:
The figures (+300MB) you quote are frightening, and it would be *very*
interesting to learn what sort of rules causes this excessive amount
of heap usage. Could you post typical patterns - if they are
generated, there ought to be - and guestimates w.r,t, their relative
frequencies?
-W
On 3/30/10, miguel machado <mls.machado(a)gmail.com> wrote:
hi again,
On 30 Mar 2010, at 18:32, Edson Tirelli <ed.tirelli(a)gmail.com> wrote:
I am not sure I understand what you are doing when you say:
Every X seconds I collect the available facts and call fireAllRules() +
dispose(), **always within the same session**.
(emphasis is mine)
i meant that i keep using the same object, i just create one stateful
session throughout the program execution.
Compilation indeed uses more memory, but you can compile once and reuse
the kbase.
exactly, i'm doing that, but it's a +700 rule file and its taking too much
memory. Is there anything i can do to push it down a little bit? i wouldn't
mind slowing it down to reduce memory footprint.
Also, if you are calling dispose, I guess you are running stateful
sessions.
Maybe you can give us a more detailed description of your
session/data/application lifecycle?
Sure, i'll try. So i have this thread which initializes by reading the rule
base from a drl file. It then creates a stateful session from it and enters
a lifecycle in which it goes to sleep and only when some other thread wakes
it up, it collects and inserts some facts (usually 2,3 or 4..) into the
session and fire all rules. It's basically this. What i'm very concerned
about is the amount of RAM memory the whole program is consuming. I must
find a way to reduce some of it, because the rule file is being prepared by
an external program and i have no control over the number of rules i might
get. If i test it with a little over 700 rules, it reaches +300Mb which is
a
lot.
Maybe the rules are too complex? Shall i try to make them simpler (how)?
what are the big memory hogs when it comes to loading a DRL file to the
knowledge base? From what i've been testing, it doesn't even matter which
rules you will actually need, it just loads the entire DRL file with no
respect to the future activations, am i right? Is there any other aspect i
should take in mind or maybe other approach?
_ miguel
_______________________________________________
rules-users mailing list
rules-users(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/rules-users
_______________________________________________
rules-users mailing list
rules-users(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/rules-users
--
Edson Tirelli
JBoss Drools Core Development
JBoss by Red Hat @ www.jboss.com
7:38 a.m.
Hi all,
Thanks for your tips. Indeed I had some redundancy in my rule definitions,
now i've removed them and it has reduced memory usage a little bit. Any
other suggestions are welcome :)
thanks again.
_ miguel
2010/3/31 Wolfgang Laun <wolfgang.laun(a)gmail.com>
I can only offer a few hints:
Factor out all code that can be done in one of the called methods, e.g., in
resetSource( ..., Arrays.asList($value1));
Here, the Arrays.asList() should be done in resetSource().
Avoid multiple bindings. If you call
resetSource( $source,....)
or
checkMayReset( $source,...)
the additional parameters $object and $CI can be retrieved within
resetSource().
The funny String literals are the one thing that are strikingly different
from what you usually see in rules. String literals cost for each occurrence
unless you intern strings using javal.lang.String.intern(). (I haven't tried
whether Drools would accept that instead.) But calculating the raw cost as
(46 chars*2 bytes/char+20 bytes)*10 times * 700 rule triples only accounts
for 0.8MB, so this alone isn't the culprit.
-W
2010/3/31 Miguel Machado <mls.machado(a)gmail.com>
Hi, thanks for answering so quickly.
>
> My ±2100 rules are grouped in 700 groups of 3 drools rules each, which are
> related to the same type of event i want to generate, like the following:
>
>
> rule "event1 reset" no-loop salience 9 activation-group
"event1_group"
> lock-on-active agenda-group "a3ce4a97-13d8-4bec-a9ab-5625fcd0f105"
> when
> $source : SourceBO(
> ciUUID == "3938adc4-246d-43f3-b606-da4c3f09d49b",
> $object : object,
> $CI : ciUUID,
> $sourceUUID : sourceUUID)
> $value1 : SourceValueBO ( valueName == "VALUE1", numericValue < 90 )
> eval (checkMayReset($source, "fb95632f-fac9-4359-9d7c-dc38d3e48e20", $CI,
> $object))
> then
> modify ($value1) { setConditionValue(90) };
> resetSource($source, "fb95632f-fac9-4359-9d7c-dc38d3e48e20", $CI, $object,
> "event_version", "event_text", Arrays.asList($value1));
> end
>
> rule "event1 fire" no-loop salience 6 activation-group
> "event1_group" lock-on-active agenda-group
> "a3ce4a97-13d8-4bec-a9ab-5625fcd0f105"
> when
> $source : SourceBO(
> ciUUID == "3938adc4-246d-43f3-b606-da4c3f09d49b",
> $object : object,
> $CI : ciUUID,
> $sourceUUID : sourceUUID )
> $value1: SourceValueBO ( valueName == "VALUE1", numericValue > 90 )
> then
> modify ($value1) { setConditionValue(90) };
> FireCondition $start = new FireCondition("repeat", 0, "duration",
"0s");
> addToActivationQueue($source, "fb95632f-fac9-4359-9d7c-dc38d3e48e20",
> $CI, $object, $start, null, "event_version", "event_text",
> Arrays.asList($value1));
> retract($source);
> end
>
> rule "event1 clear" no-loop salience 5 activation-group
"event1_group"
> lock-on-active agenda-group "a3ce4a97-13d8-4bec-a9ab-5625fcd0f105"
> when
> $source : SourceBO(
> ciUUID == "3938adc4-246d-43f3-b606-da4c3f09d49b",
> $object : object,
> $CI : ciUUID,
> $sourceUUID : sourceUUID)
> then
> clearActivationQueue($source, "fb95632f-fac9-4359-9d7c-dc38d3e48e20", $CI,
> $object);
> end
>
> So, i'l try to explain briefly:
> * The reset rule evaluates 2 patterns and one condition in a static method
> available via "import function". As a consequence, it modifies the
> SourceValueBO object and call's the reset function which is another static
> method.
> * The fire rule is very similar to the reset rule, the difference being
> that the value threshold is different (usually the opposite)
> * The clear rule is like an "else" to the two above. I want it to fire in
> case it didn't fall for either one of reset and fire rules.
>
> The activation group assures that only one of these rules is fired each
> time i call session.fireAllRules(). And the agenda-group allows me to stack
> some rules above others, according to the type of facts i insert into
> memory.
> Everything seems in place and it works as intended, the thing is i might
> be dealing with *lots* of events (event1, event2, etc...) and lots of
> different UUID's which results in a very large DRL file. For instance, if i
> have 700 events (event1, event ...event700), i get 2100 drools rules (event1
> reset, event1 fire, event1 clear ... event700 reset, event700 fire, event700
> clear) and over 300MB in consumed memory.
>
> Perhaps if i took another approach? or some performance tunning, i don't
> know. Where can i improve this memory hogging?
> _ miguel
>
>
>
>
> On 31 Mar 2010, at 06:53, Wolfgang Laun wrote:
>
> The figures (+300MB) you quote are frightening, and it would be *very*
> interesting to learn what sort of rules causes this excessive amount
> of heap usage. Could you post typical patterns - if they are
> generated, there ought to be - and guestimates w.r,t, their relative
> frequencies?
>
> -W
>
>
> On 3/30/10, miguel machado <mls.machado(a)gmail.com> wrote:
>
> hi again,
>
>
> On 30 Mar 2010, at 18:32, Edson Tirelli <ed.tirelli(a)gmail.com> wrote:
>
>
> I am not sure I understand what you are doing when you say:
>
>
> Every X seconds I collect the available facts and call fireAllRules() +
>
> dispose(), **always within the same session**.
>
> (emphasis is mine)
>
>
> i meant that i keep using the same object, i just create one stateful
>
> session throughout the program execution.
>
>
> Compilation indeed uses more memory, but you can compile once and reuse
>
> the kbase.
>
>
>
> exactly, i'm doing that, but it's a +700 rule file and its taking too much
>
> memory. Is there anything i can do to push it down a little bit? i
> wouldn't
>
> mind slowing it down to reduce memory footprint.
>
>
> Also, if you are calling dispose, I guess you are running stateful
> sessions.
>
> Maybe you can give us a more detailed description of your
>
> session/data/application lifecycle?
>
>
> Sure, i'll try. So i have this thread which initializes by reading the
> rule
>
> base from a drl file. It then creates a stateful session from it and
> enters
>
> a lifecycle in which it goes to sleep and only when some other thread
> wakes
>
> it up, it collects and inserts some facts (usually 2,3 or 4..) into the
>
> session and fire all rules. It's basically this. What i'm very concerned
>
> about is the amount of RAM memory the whole program is consuming. I must
>
> find a way to reduce some of it, because the rule file is being prepared
> by
>
> an external program and i have no control over the number of rules i might
>
> get. If i test it with a little over 700 rules, it reaches +300Mb which is
> a
>
> lot.
>
>
> Maybe the rules are too complex? Shall i try to make them simpler (how)?
>
> what are the big memory hogs when it comes to loading a DRL file to the
>
> knowledge base? From what i've been testing, it doesn't even matter which
>
> rules you will actually need, it just loads the entire DRL file with no
>
> respect to the future activations, am i right? Is there any other aspect i
>
> should take in mind or maybe other approach?
>
>
> _ miguel
>
>
> _______________________________________________
> rules-users mailing list
> rules-users(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/rules-users
>
>
>
> _______________________________________________
> rules-users mailing list
> rules-users(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/rules-users
>
>
_______________________________________________
rules-users mailing list
rules-users(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/rules-users
--
"To understand what is recursion you must first understand recursion"
8:24 a.m.
Hi there,
First of all, thank you for your detailed and quick response. Next...
2010/3/31 Edson Tirelli <ed.tirelli(a)gmail.com>
Miguel,
First we need to understand where is this memory spent. Drools
externalizes the actual rule compilation to a java compiler (JDT or Janino)
and I would not be surprised if compiling 2100 rules would consume more than
300Mb **during compilation**. I don't think that after compiling, the
application will continue to use that amount of memory just because of
rules, although if you have many facts being inserted into your session, it
obviously will drive consumption up.
I usually load into memory 2~4 facts each time, very small objects. I then
"fire all rules" and immediately retract and dispose.
So, some questions:
1. Do you use one KnowledgeBuilder to compile all your rule files or do you
use one KnowledgeBuilder per file?
Just one, declared as a static variable.
2. How much memory is your application using during this step, where
you
add resources to your knowledgeBuilder(s)?
Its taking about ±20 MB just before it reads the DRL file, it then takes
5~10s for loading rules into memory and immediately after that line its
taking +300MB.. kinda stabilizes afterwards.
3. After compiling your knowledge packages, did you tried to force a
GC to
check how much memory it continues to use?
Just tested that, it barely makes a difference :-/ (less than 5MB saved)
4. Are you adding all rules to the same KnowledgeBase or using one
kbase
per set of rules? How much memory are you consuming in this step?
Most of the time, i've been testing with only one rule file with +2100 rules
and for that i create a single knowledge base, from which i create the
ksession. However, the whole system i'm developing has another component of
event analysis so eventually i'll be testing two separate threads doing
similar things but on different DRL files. I suppose i'll be using two
kbase's, one for each file...
This is the step where memory usage increases dramatically:
kbuilder.add( ResourceFactory.newFileSystemResource( fileName ),
ResourceType.DRL );
5. Did you tried compiling it once, generating the kbase and serializing it
to a file, and then, instead of recompiling everything, just
deserialize the
file into memory and using it? How much memory it consumes when you
deserialize from a file?
Not sure what you mean there, is it this:
http://downloads.jboss.com/drools/docs/5.0.1.26597.FINAL/drools-expert/ht...
?
if so, yes, i've tried it and noticed no difference at all, was i supposed
to?
Knowing the answer to the above questions will help to determine if you
have a problem and how to work around it or fix it.
thanks for your help so far. Any help is very much appreciated.
_ miguel
--
"To understand what is recursion you must first understand recursion"
3:27 p.m.
2010/4/1 miguel machado <mls.machado(a)gmail.com>:
2010/3/31 Edson Tirelli <ed.tirelli(a)gmail.com>
>
> 1. Do you use one KnowledgeBuilder to compile all your rule files or do
> you use one KnowledgeBuilder per file?
Just one, declared as a static variable.
That wastes, whatever the KB takes, for the remaining duration of the program
execution. After you have taken the compiled packages from the KB and
checked for errors, the KB has done his duty the KB can go :-)
>
> 5. Did you tried compiling it once, generating the kbase and serializing
> it to a file, and then, instead of recompiling everything, just deserialize
> the file into memory and using it? How much memory it consumes when you
> deserialize from a file?
Not sure what you mean there, is it
this: http://downloads.jboss.com/drools/docs/5.0.1.26597.FINAL/drools-exp...
if so, yes, i've tried it and noticed no difference at all, was i supposed
to?
For the application that just does the load part (reading the KPs from
an input stream), yes: that
should reduce memory usage.
It certainly does not, if your KB is static, and you compile and load
in the same program,
just serializing and deserializing in between.
-W
4:52 p.m.
Hi Miguel,
I assume that you've used a memory analyzer to find out conclusively that
the memory usage is by Drools, and not a memory leak?
One way to do this is to limit the virtual memory to a small amount, add
-XX:+HeapDumpOnOutOfMemoryError to your environment, and then using Memory
Analyzer to analyse the dump.
Cheers
Jevon
On Wed, Mar 31, 2010 at 2:44 AM, Miguel Machado <mls.machado(a)gmail.com>wrote:
hi all,
I'm quite new to the drools rule engine and i've been using it for a
project where i need to filter events and execute actions accordingly.
I'm currently running a DRL file and creating a stateful session from it,
and then running 2/3 facts every time. Every X seconds I collect the
available facts and call fireAllRules() + dispose(), always within the same
session. The DRL file has circa 2160 rules declared (an external program is
producing them), and i'm using the "no-loop", "lock-on-active",
"activation-group" and "agenda-group" options on almost every one.
My main problem is that the software is consuming too much memory when it
loads the rules to the knowledge base variable, it reaches +300Mb which is
just not affordable in this project. Is there any way i can work around
this? I've tried setting some configuration properties but to no avail. What
techniques may i try in order to reduce memory load on drools startup? What
would be the best approach for dealing with so many rules?
Anything would help, thanks in advance,
_ miguel
_______________________________________________
rules-users mailing list
rules-users(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/rules-users
7:31 p.m.
Hi there,
Actually that's a very good suggestion, i hadn't thought about that. Is
there any heap inspector tool you'd recommend?
In what comes to Wolfgang ideas, it makes sense what he said, but the truth
is making the knowledge builder a static variable actually made memory usage
lower (just a bit). I know it strikes as a surprise because one of the good
practices in java is to reduce the variable/object scope as much as possible
:-/ I'll try to investigate this further.
About the kbase serializing, i'll give it another try.
thanks for all support so far guys.
_ miguel
2010/4/1 Jevon Wright <jevon(a)jevon.org>
Hi Miguel,
I assume that you've used a memory analyzer to find out conclusively that
the memory usage is by Drools, and not a memory leak?
One way to do this is to limit the virtual memory to a small amount, add
-XX:+HeapDumpOnOutOfMemoryError to your environment, and then using Memory
Analyzer to analyse the dump.
Cheers
Jevon
On Wed, Mar 31, 2010 at 2:44 AM, Miguel Machado <mls.machado(a)gmail.com>wrote:
> hi all,
>
> I'm quite new to the drools rule engine and i've been using it for a
> project where i need to filter events and execute actions accordingly.
>
> I'm currently running a DRL file and creating a stateful session from it,
> and then running 2/3 facts every time. Every X seconds I collect the
> available facts and call fireAllRules() + dispose(), always within the same
> session. The DRL file has circa 2160 rules declared (an external program is
> producing them), and i'm using the "no-loop",
"lock-on-active",
> "activation-group" and "agenda-group" options on almost every
one.
>
> My main problem is that the software is consuming too much memory when it
> loads the rules to the knowledge base variable, it reaches +300Mb which is
> just not affordable in this project. Is there any way i can work around
> this? I've tried setting some configuration properties but to no avail. What
> techniques may i try in order to reduce memory load on drools startup? What
> would be the best approach for dealing with so many rules?
>
> Anything would help, thanks in advance,
>
> _ miguel
>
>
> _______________________________________________
> rules-users mailing list
> rules-users(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/rules-users
>
_______________________________________________
rules-users mailing list
rules-users(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/rules-users
--
"To understand what is recursion you must first understand recursion"
11:05 p.m.
You can use Eclipse MAT
Vijay
From: rules-users-bounces(a)lists.jboss.org [mailto:rules-users-bounces@lists.jboss.org] On
Behalf Of miguel machado
Sent: Thursday, April 01, 2010 5:31 AM
To: Rules Users List
Subject: Re: [rules-users] drools expert: memory issues
Hi there,
Actually that's a very good suggestion, i hadn't thought about that. Is there any
heap inspector tool you'd recommend?
In what comes to Wolfgang ideas, it makes sense what he said, but the truth is making the
knowledge builder a static variable actually made memory usage lower (just a bit). I know
it strikes as a surprise because one of the good practices in java is to reduce the
variable/object scope as much as possible :-/ I'll try to investigate this further.
About the kbase serializing, i'll give it another try.
thanks for all support so far guys.
_ miguel
2010/4/1 Jevon Wright <jevon@jevon.org<mailto:jevon@jevon.org>>
Hi Miguel,
I assume that you've used a memory analyzer to find out conclusively that the memory
usage is by Drools, and not a memory leak?
One way to do this is to limit the virtual memory to a small amount, add
-XX:+HeapDumpOnOutOfMemoryError to your environment, and then using Memory Analyzer to
analyse the dump.
Cheers
Jevon
On Wed, Mar 31, 2010 at 2:44 AM, Miguel Machado
<mls.machado@gmail.com<mailto:mls.machado@gmail.com>> wrote:
hi all,
I'm quite new to the drools rule engine and i've been using it for a project where
i need to filter events and execute actions accordingly.
I'm currently running a DRL file and creating a stateful session from it, and then
running 2/3 facts every time. Every X seconds I collect the available facts and call
fireAllRules() + dispose(), always within the same session. The DRL file has circa 2160
rules declared (an external program is producing them), and i'm using the
"no-loop", "lock-on-active", "activation-group" and
"agenda-group" options on almost every one.
My main problem is that the software is consuming too much memory when it loads the rules
to the knowledge base variable, it reaches +300Mb which is just not affordable in this
project. Is there any way i can work around this? I've tried setting some
configuration properties but to no avail. What techniques may i try in order to reduce
memory load on drools startup? What would be the best approach for dealing with so many
rules?
Anything would help, thanks in advance,
_ miguel
_______________________________________________
rules-users mailing list
rules-users@lists.jboss.org<mailto:rules-users@lists.jboss.org>
https://lists.jboss.org/mailman/listinfo/rules-users
_______________________________________________
rules-users mailing list
rules-users@lists.jboss.org<mailto:rules-users@lists.jboss.org>
https://lists.jboss.org/mailman/listinfo/rules-users
--
"To understand what is recursion you must first understand recursion"
11:25 p.m.
Hmm, let me complement Wolfgang's comment:
After you have taken the compiled packages from the KB and checked
for
errors, the KB has done his duty the KB ****MUST**** go :-)
The above emphasis is mine. KnowledgeBuilder is a higher level object
designed to encapsulate the parsing+compilation steps of resources. It is
**not** designed to be reused and doing so might create inconsistencies. The
memory spike you see when using the kbuilder is due to the java compiler...
Drools itself does not do anything fancy in there other than generate some
code (quite inexpensive memory wise) and compile it using either JDT or
JANINO (both quite expensive memory wise, when compared to the other steps).
So, I strongly discourage you keeping the KB around and as mentioned by
Wolfgang, doing so might keep unnecessary objects in memory.
Also, just to clarify, deserializing a compiled package means the
compiler will not be called (obviously, as everything is already compiled)
and that is why it should save you memory.
Edson
2010/4/1 Wolfgang Laun <wolfgang.laun(a)gmail.com>
2010/4/1 miguel machado <mls.machado(a)gmail.com>:
>
> 2010/3/31 Edson Tirelli <ed.tirelli(a)gmail.com>
>>
>> 1. Do you use one KnowledgeBuilder to compile all your rule files or do
>> you use one KnowledgeBuilder per file?
>
> Just one, declared as a static variable.
That wastes, whatever the KB takes, for the remaining duration of the
program
execution. After you have taken the compiled packages from the KB and
checked for errors, the KB has done his duty the KB can go :-)
>>
>> 5. Did you tried compiling it once, generating the kbase and serializing
>> it to a file, and then, instead of recompiling everything, just
deserialize
>> the file into memory and using it? How much memory it consumes when you
>> deserialize from a file?
>
> Not sure what you mean there, is it
> this:
http://downloads.jboss.com/drools/docs/5.0.1.26597.FINAL/drools-expert/ht...
?
> if so, yes, i've tried it and noticed no difference at all, was i
supposed
> to?
For the application that just does the load part (reading the KPs from
an input stream), yes: that
should reduce memory usage.
It certainly does not, if your KB is static, and you compile and load
in the same program,
just serializing and deserializing in between.
-W
_______________________________________________
rules-users mailing list
rules-users(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/rules-users
--
Edson Tirelli
JBoss Drools Core Development
JBoss by Red Hat @ www.jboss.com
2:12 a.m.
hi again,
2010/4/1 Edson Tirelli <ed.tirelli(a)gmail.com>
Hmm, let me complement Wolfgang's comment:
> After you have taken the compiled packages from the KB and checked for
errors, the KB has done his duty the KB ****MUST**** go :-)
I agree! But i don't exactly know how to do so... i mean, i call this method
which creates the KB from a DRL file and then the kbase and returns the
ksession, the KB object declaration scope is just that exact method, and
once it reaches 300MB+ in memory, it just stays there... the KB object is no
longer in use (or is it?) but memory keeps high.
I've tried setting KB to "null" and then invoking the jvm garbage
collector,
but to no avail. I wish there was a simple way i could just destroy a
particular object sigh
The above emphasis is mine. KnowledgeBuilder is a higher level object
designed to encapsulate the parsing+compilation steps of resources. It is
**not** designed to be reused and doing so might create inconsistencies. The
memory spike you see when using the kbuilder is due to the java compiler...
I understand that, and i'll have that if i must, but the thing is it's not
just a spike, it is prolonged throughout execution, it doesn't reduce
afterwards.
Drools itself does not do anything fancy in there other than generate
some
code (quite inexpensive memory wise) and compile it using either JDT or
JANINO (both quite expensive memory wise, when compared to the other steps).
So, I strongly discourage you keeping the KB around and as mentioned by
Wolfgang, doing so might keep unnecessary objects in memory.
Like i said earlier, it's not that i want to keep KB around, i just can't
remove it.
Also, just to clarify, deserializing a compiled package means the
compiler will not be called (obviously, as everything is already compiled)
and that is why it should save you memory.
Thanks for clarifying that! I've tested with a separate application to build
a compiled file and then load it in the main system and it really makes a
difference! it's gone to <100MB wow! However, this was just a
proof-of-concept, as i'm not sure how i could apply the same technique to
the main standalone application. Do i really have to create a separate
process/application just for building/compiling the rules, to make sure the
compilation-object-junk doesn't stay in memory along with the rest?
Thanks for all the feedback and input so far.
_ miguel
--
"To understand what is recursion you must first understand recursion"
9:14 p.m.
hi there,
So like i said, by pre-compiling the rules and loading them in the main
system i was able to reduce a lot memory usage and i can get better results.
However, now i've come across a different problem. I just added a fre more
dozens of rules and the java program responsible for compiling the DRL file
throws a java heap space error, something like an OutOfMemoryException.
I've read somewhere that dividing the DRL into several files and loading
e.g. 500 rules at a time might solve it. Is there another approach to this
problem?
thanks in advance,
_ miguel
On Thu, Apr 1, 2010 at 6:12 PM, miguel machado <mls.machado(a)gmail.com>wrote:
hi again,
2010/4/1 Edson Tirelli <ed.tirelli(a)gmail.com>
> Hmm, let me complement Wolfgang's comment:
>
> > After you have taken the compiled packages from the KB and checked for
> errors, the KB has done his duty the KB ****MUST**** go :-)
>
I agree! But i don't exactly know how to do so... i mean, i call this
method which creates the KB from a DRL file and then the kbase and returns
the ksession, the KB object declaration scope is just that exact method, and
once it reaches 300MB+ in memory, it just stays there... the KB object is no
longer in use (or is it?) but memory keeps high.
I've tried setting KB to "null" and then invoking the jvm garbage
collector, but to no avail. I wish there was a simple way i could just
destroy a particular object sigh
>
> The above emphasis is mine. KnowledgeBuilder is a higher level object
> designed to encapsulate the parsing+compilation steps of resources. It is
> **not** designed to be reused and doing so might create inconsistencies. The
> memory spike you see when using the kbuilder is due to the java compiler...
I understand that, and i'll have that if i must, but the thing is it's not
just a spike, it is prolonged throughout execution, it doesn't reduce
afterwards.
> Drools itself does not do anything fancy in there other than generate some
> code (quite inexpensive memory wise) and compile it using either JDT or
> JANINO (both quite expensive memory wise, when compared to the other steps).
>
>
> So, I strongly discourage you keeping the KB around and as mentioned by
> Wolfgang, doing so might keep unnecessary objects in memory.
>
Like i said earlier, it's not that i want to keep KB around, i just can't
remove it.
>
> Also, just to clarify, deserializing a compiled package means the
> compiler will not be called (obviously, as everything is already compiled)
> and that is why it should save you memory.
>
Thanks for clarifying that! I've tested with a separate application to
build a compiled file and then load it in the main system and it really
makes a difference! it's gone to <100MB wow! However, this was just a
proof-of-concept, as i'm not sure how i could apply the same technique to
the main standalone application. Do i really have to create a separate
process/application just for building/compiling the rules, to make sure the
compilation-object-junk doesn't stay in memory along with the rest?
Thanks for all the feedback and input so far.
_ miguel
--
"To understand what is recursion you must first understand recursion"
--
"To understand what is recursion you must first understand recursion"
12:11 a.m.
It should be possible to increase heap space to overcome this problem. Of
course, if a program wil start swapping, performance will go way down.
Dividing the rules file is the only way. But even a very simple script of
Perl or similar should be able to divide your (IIRC) generated rules file.
-W
2010/4/3 miguel machado <mls.machado(a)gmail.com>
hi there,
So like i said, by pre-compiling the rules and loading them in the main
system i was able to reduce a lot memory usage and i can get better results.
However, now i've come across a different problem. I just added a fre more
dozens of rules and the java program responsible for compiling the DRL file
throws a java heap space error, something like an OutOfMemoryException.
I've read somewhere that dividing the DRL into several files and loading
e.g. 500 rules at a time might solve it. Is there another approach to this
problem?
thanks in advance,
_ miguel
On Thu, Apr 1, 2010 at 6:12 PM, miguel machado <mls.machado(a)gmail.com>wrote:
> hi again,
>
> 2010/4/1 Edson Tirelli <ed.tirelli(a)gmail.com>
>
>
>> Hmm, let me complement Wolfgang's comment:
>>
>> > After you have taken the compiled packages from the KB and checked for
>> errors, the KB has done his duty the KB ****MUST**** go :-)
>>
>
> I agree! But i don't exactly know how to do so... i mean, i call this
> method which creates the KB from a DRL file and then the kbase and returns
> the ksession, the KB object declaration scope is just that exact method, and
> once it reaches 300MB+ in memory, it just stays there... the KB object is no
> longer in use (or is it?) but memory keeps high.
>
> I've tried setting KB to "null" and then invoking the jvm garbage
> collector, but to no avail. I wish there was a simple way i could just
> destroy a particular object sigh
>
>
>>
>> The above emphasis is mine. KnowledgeBuilder is a higher level object
>> designed to encapsulate the parsing+compilation steps of resources. It is
>> **not** designed to be reused and doing so might create inconsistencies. The
>> memory spike you see when using the kbuilder is due to the java compiler...
>
>
> I understand that, and i'll have that if i must, but the thing is it's not
> just a spike, it is prolonged throughout execution, it doesn't reduce
> afterwards.
>
>
>> Drools itself does not do anything fancy in there other than generate
>> some code (quite inexpensive memory wise) and compile it using either JDT or
>> JANINO (both quite expensive memory wise, when compared to the other steps).
>>
>>
>> So, I strongly discourage you keeping the KB around and as mentioned
>> by Wolfgang, doing so might keep unnecessary objects in memory.
>>
>
> Like i said earlier, it's not that i want to keep KB around, i just can't
> remove it.
>
>
>>
>> Also, just to clarify, deserializing a compiled package means the
>> compiler will not be called (obviously, as everything is already compiled)
>> and that is why it should save you memory.
>>
>
> Thanks for clarifying that! I've tested with a separate application to
> build a compiled file and then load it in the main system and it really
> makes a difference! it's gone to <100MB wow! However, this was just a
> proof-of-concept, as i'm not sure how i could apply the same technique to
> the main standalone application. Do i really have to create a separate
> process/application just for building/compiling the rules, to make sure the
> compilation-object-junk doesn't stay in memory along with the rest?
>
> Thanks for all the feedback and input so far.
> _ miguel
>
>
> --
> "To understand what is recursion you must first understand recursion"
>
--
"To understand what is recursion you must first understand recursion"
_______________________________________________
rules-users mailing list
rules-users(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/rules-users
5:07 a.m.
I'm still using good old Drools 4 and there's an ant task and a maven plugin
out there to compile your rules at build time. I'm pretty sure there's
something like that you could use as well.
Cheers,
Leonardo.
2010/4/1 miguel machado <mls.machado(a)gmail.com>
hi again,
2010/4/1 Edson Tirelli <ed.tirelli(a)gmail.com>
> Hmm, let me complement Wolfgang's comment:
>
> > After you have taken the compiled packages from the KB and checked for
> errors, the KB has done his duty the KB ****MUST**** go :-)
>
I agree! But i don't exactly know how to do so... i mean, i call this
method which creates the KB from a DRL file and then the kbase and returns
the ksession, the KB object declaration scope is just that exact method, and
once it reaches 300MB+ in memory, it just stays there... the KB object is no
longer in use (or is it?) but memory keeps high.
I've tried setting KB to "null" and then invoking the jvm garbage
collector, but to no avail. I wish there was a simple way i could just
destroy a particular object sigh
>
> The above emphasis is mine. KnowledgeBuilder is a higher level object
> designed to encapsulate the parsing+compilation steps of resources. It is
> **not** designed to be reused and doing so might create inconsistencies. The
> memory spike you see when using the kbuilder is due to the java compiler...
I understand that, and i'll have that if i must, but the thing is it's not
just a spike, it is prolonged throughout execution, it doesn't reduce
afterwards.
> Drools itself does not do anything fancy in there other than generate some
> code (quite inexpensive memory wise) and compile it using either JDT or
> JANINO (both quite expensive memory wise, when compared to the other steps).
>
>
> So, I strongly discourage you keeping the KB around and as mentioned by
> Wolfgang, doing so might keep unnecessary objects in memory.
>
Like i said earlier, it's not that i want to keep KB around, i just can't
remove it.
>
> Also, just to clarify, deserializing a compiled package means the
> compiler will not be called (obviously, as everything is already compiled)
> and that is why it should save you memory.
>
Thanks for clarifying that! I've tested with a separate application to
build a compiled file and then load it in the main system and it really
makes a difference! it's gone to <100MB wow! However, this was just a
proof-of-concept, as i'm not sure how i could apply the same technique to
the main standalone application. Do i really have to create a separate
process/application just for building/compiling the rules, to make sure the
compilation-object-junk doesn't stay in memory along with the rest?
Thanks for all the feedback and input so far.
_ miguel
--
"To understand what is recursion you must first understand recursion"
_______________________________________________
rules-users mailing list
rules-users(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/rules-users
7:50 p.m.
Hi again,
So.. i've tried doing precisely that but i am unable to fire the rules
afterwards. I am dividing the XML rule file into smaller valid XML files so
that i can compile them individually and write to disk the compiled result.
I am using the sample code shown in the docs:
// write compiled rules to disk
ObjectOutputStream out = new ObjectOutputStream(new
FileOutputStream(filename + ".temp"));
out.writeObject(kbuilder.getKnowledgePackages());
out.close();
On a later stage, i load all files holding compiled rules, one at a time, so
i can use the knowledge base and create the knowledge session. Here's what
i'm doing for each of the generated compiled files:
in = new ObjectInputStream(new FileInputStream(filename)); // the ones
*.temp
kpkgs = (Collection<KnowledgePackage>) in.readObject();
kbase.addKnowledgePackages(kpkgs);
in.close();
This all works apparently well. Drools loads the sum of all rules defined in
each smaller XML file. However, whenever i try to insert a fact into memory,
i get an exception, just like the following:
java.lang.NullPointerException
at org.drools.base.ClassFieldReader.getValue(ClassFieldReader.java:91)
at
org.drools.base.evaluators.EqualityEvaluatorsDefinition$StringEqualEvaluator.evaluate(EqualityEvaluatorsDefinition.java:1962)
at org.drools.rule.LiteralRestriction.isAllowed(LiteralRestriction.java:92)
at
org.drools.rule.OrCompositeRestriction.isAllowed(OrCompositeRestriction.java:25)
at
org.drools.rule.MultiRestrictionFieldConstraint.isAllowed(MultiRestrictionFieldConstraint.java:97)
at org.drools.reteoo.AlphaNode.assertObject(AlphaNode.java:143)
at
org.drools.reteoo.CompositeObjectSinkAdapter.doPropagateAssertObject(CompositeObjectSinkAdapter.java:360)
at
org.drools.reteoo.CompositeObjectSinkAdapter.propagateAssertObject(CompositeObjectSinkAdapter.java:344)
at org.drools.reteoo.ObjectTypeNode.assertObject(ObjectTypeNode.java:185)
at org.drools.reteoo.EntryPointNode.assertObject(EntryPointNode.java:146)
at
org.drools.common.AbstractWorkingMemory.insert(AbstractWorkingMemory.java:1046)
at
org.drools.common.AbstractWorkingMemory.insert(AbstractWorkingMemory.java:1001)
at
org.drools.common.AbstractWorkingMemory.insert(AbstractWorkingMemory.java:788)
at
org.drools.impl.StatefulKnowledgeSessionImpl.insert(StatefulKnowledgeSessionImpl.java:216)
at
pt.itsd.probe.ConfigurationSetup.parseAndCompileConfigFiles(ConfigurationSetup.java:316)
at pt.itsd.probe.ConfigurationSetup.main(ConfigurationSetup.java:131)
I have done some testing and I am unable to trace this exception to any
particular rule, or event any XML chunck. If i load the entire XML file it
takes a lot of memory compiling but it works in the end. Also, if i remove
all the <or-restriction-connective> tags, it also works no matter if i load
them to kbase.
Is there something i can do? or maybe another way of doing it? Any help is
much appreciated.
I'm using drools 5.0.0 from main download page.
_ miguel
2010/4/3 Wolfgang Laun <wolfgang.laun(a)gmail.com>
It should be possible to increase heap space to overcome this
problem. Of
course, if a program wil start swapping, performance will go way down.
Dividing the rules file is the only way. But even a very simple script of
Perl or similar should be able to divide your (IIRC) generated rules file.
-W
2010/4/3 miguel machado <mls.machado(a)gmail.com>
> hi there,
>
> So like i said, by pre-compiling the rules and loading them in the main
> system i was able to reduce a lot memory usage and i can get better results.
> However, now i've come across a different problem. I just added a fre more
> dozens of rules and the java program responsible for compiling the DRL file
> throws a java heap space error, something like an OutOfMemoryException.
>
> I've read somewhere that dividing the DRL into several files and loading
> e.g. 500 rules at a time might solve it. Is there another approach to this
> problem?
>
> thanks in advance,
> _ miguel
>
>
>
> On Thu, Apr 1, 2010 at 6:12 PM, miguel machado <mls.machado(a)gmail.com>wrote:
>
>> hi again,
>>
>> 2010/4/1 Edson Tirelli <ed.tirelli(a)gmail.com>
>>
>>
>>> Hmm, let me complement Wolfgang's comment:
>>>
>>> > After you have taken the compiled packages from the KB and checked for
>>> errors, the KB has done his duty the KB ****MUST**** go :-)
>>>
>>
>> I agree! But i don't exactly know how to do so... i mean, i call this
>> method which creates the KB from a DRL file and then the kbase and returns
>> the ksession, the KB object declaration scope is just that exact method, and
>> once it reaches 300MB+ in memory, it just stays there... the KB object is no
>> longer in use (or is it?) but memory keeps high.
>>
>> I've tried setting KB to "null" and then invoking the jvm garbage
>> collector, but to no avail. I wish there was a simple way i could just
>> destroy a particular object sigh
>>
>>
>>>
>>> The above emphasis is mine. KnowledgeBuilder is a higher level object
>>> designed to encapsulate the parsing+compilation steps of resources. It is
>>> **not** designed to be reused and doing so might create inconsistencies. The
>>> memory spike you see when using the kbuilder is due to the java compiler...
>>
>>
>> I understand that, and i'll have that if i must, but the thing is it's
>> not just a spike, it is prolonged throughout execution, it doesn't reduce
>> afterwards.
>>
>>
>>> Drools itself does not do anything fancy in there other than generate
>>> some code (quite inexpensive memory wise) and compile it using either JDT or
>>> JANINO (both quite expensive memory wise, when compared to the other steps).
>>>
>>>
>>> So, I strongly discourage you keeping the KB around and as mentioned
>>> by Wolfgang, doing so might keep unnecessary objects in memory.
>>>
>>
>> Like i said earlier, it's not that i want to keep KB around, i just
can't
>> remove it.
>>
>>
>>>
>>> Also, just to clarify, deserializing a compiled package means the
>>> compiler will not be called (obviously, as everything is already compiled)
>>> and that is why it should save you memory.
>>>
>>
>> Thanks for clarifying that! I've tested with a separate application to
>> build a compiled file and then load it in the main system and it really
>> makes a difference! it's gone to <100MB wow! However, this was just a
>> proof-of-concept, as i'm not sure how i could apply the same technique to
>> the main standalone application. Do i really have to create a separate
>> process/application just for building/compiling the rules, to make sure the
>> compilation-object-junk doesn't stay in memory along with the rest?
>>
>> Thanks for all the feedback and input so far.
>> _ miguel
>>
>>
>> --
>> "To understand what is recursion you must first understand recursion"
>>
>
>
>
> --
> "To understand what is recursion you must first understand recursion"
>
> _______________________________________________
> rules-users mailing list
> rules-users(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/rules-users
>
>
_______________________________________________
rules-users mailing list
rules-users(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/rules-users
--
"To understand what is recursion you must first understand recursion"
5125
days inactive
5153
days old
18 comments
7 participants
participants (7)
-
Edson Tirelli -
Jevon Wright -
Leonardo Gomes -
Miguel Machado -
miguel machado
-
Vijay K Pandey -
Wolfgang Laun