[rules-users] drools expert: memory issues

Wed Mar 31 11:10:26 EDT 2010

    Miguel,

    First we need to understand where is this memory spent. Drools
externalizes the actual rule compilation to a java compiler (JDT or Janino)
and I would not be surprised if compiling 2100 rules would consume more than
300Mb **during compilation**. I don't think that after compiling, the
application will continue to use that amount of memory just because of
rules, although if you have many facts being inserted into your session, it
obviously will drive consumption up.

    So, some questions:

1. Do you use one KnowledgeBuilder to compile all your rule files or do you
use one KnowledgeBuilder per file?
2. How much memory is your application using during this step, where you add
resources to your knowledgeBuilder(s)?
3. After compiling your knowledge packages, did you tried to force a GC to
check how much memory it continues to use?
4. Are you adding all rules to the same KnowledgeBase or using one kbase per
set of rules? How much memory are you consuming in this step?
5. Did you tried compiling it once, generating the kbase and serializing it
to a file, and then, instead of recompiling everything, just deserialize the
file into memory and using it? How much memory it consumes when you
deserialize from a file?

   Knowing the answer to the above questions will help to determine if you
have a problem and how to work around it or fix it.

   Edson

2010/3/31 Miguel Machado <mls.machado at gmail.com>

> Hi, thanks for answering so quickly.
>
> My ±2100 rules are grouped in 700 groups of 3 drools rules each, which are
> related to the same type of event i want to generate, like the following:
>
>
> rule "event1 reset" no-loop salience 9 activation-group "event1_group"
> lock-on-active agenda-group "a3ce4a97-13d8-4bec-a9ab-5625fcd0f105"
> when
> $source : SourceBO(
> ciUUID == "3938adc4-246d-43f3-b606-da4c3f09d49b",
> $object : object,
> $CI : ciUUID,
> $sourceUUID : sourceUUID)
> $value1 : SourceValueBO ( valueName == "VALUE1", numericValue < 90 )
> eval (checkMayReset($source, "fb95632f-fac9-4359-9d7c-dc38d3e48e20", $CI,
> $object))
> then
> modify ($value1) { setConditionValue(90) };
> resetSource($source, "fb95632f-fac9-4359-9d7c-dc38d3e48e20", $CI, $object,
> "event_version", "event_text", Arrays.asList($value1));
> end
>
> rule "event1 fire" no-loop salience 6 activation-group
> "event1_group" lock-on-active agenda-group
> "a3ce4a97-13d8-4bec-a9ab-5625fcd0f105"
> when
> $source : SourceBO(
> ciUUID == "3938adc4-246d-43f3-b606-da4c3f09d49b",
> $object : object,
> $CI : ciUUID,
> $sourceUUID : sourceUUID )
> $value1: SourceValueBO ( valueName == "VALUE1", numericValue > 90 )
> then
> modify ($value1) { setConditionValue(90) };
> FireCondition $start = new FireCondition("repeat", 0, "duration", "0s");
> addToActivationQueue($source, "fb95632f-fac9-4359-9d7c-dc38d3e48e20", $CI,
> $object, $start, null, "event_version", "event_text",
> Arrays.asList($value1));
> retract($source);
> end
>
> rule "event1 clear" no-loop salience 5 activation-group "event1_group"
> lock-on-active agenda-group "a3ce4a97-13d8-4bec-a9ab-5625fcd0f105"
> when
> $source : SourceBO(
> ciUUID == "3938adc4-246d-43f3-b606-da4c3f09d49b",
> $object : object,
> $CI : ciUUID,
> $sourceUUID : sourceUUID)
> then
> clearActivationQueue($source, "fb95632f-fac9-4359-9d7c-dc38d3e48e20", $CI,
> $object);
> end
>
> So, i'l try to explain briefly:
> * The reset rule evaluates 2 patterns and one condition in a static method
> available via "import function". As a consequence, it modifies the
> SourceValueBO object and call's the reset function which is another static
> method.
> * The fire rule is very similar to the reset rule, the difference being
> that the value threshold is different (usually the opposite)
> * The clear rule is like an "else" to the two above. I want it to fire in
> case it didn't fall for either one of reset and fire rules.
>
> The activation group assures that only one of these rules is fired each
> time i call session.fireAllRules(). And the agenda-group allows me to stack
> some rules above others, according to the type of facts i insert into
> memory.
> Everything seems in place and it works as intended, the thing is i might be
> dealing with *lots* of events (event1, event2, etc...) and lots of different
> UUID's which results in a very large DRL file. For instance, if i have 700
> events (event1, event ...event700), i get 2100 drools rules (event1 reset,
> event1 fire, event1 clear ... event700 reset, event700 fire, event700 clear)
> and over 300MB in consumed memory.
>
> Perhaps if i took another approach? or some performance tunning, i don't
> know. Where can i improve this memory hogging?
> _ miguel
>
>
>
>
> On 31 Mar 2010, at 06:53, Wolfgang Laun wrote:
>
> The figures (+300MB) you quote are frightening, and it would be *very*
> interesting to learn what sort of rules causes this excessive amount
> of heap usage. Could you post typical patterns - if they are
> generated, there ought to be - and guestimates w.r,t, their relative
> frequencies?
>
> -W
>
>
> On 3/30/10, miguel machado <mls.machado at gmail.com> wrote:
>
> hi again,
>
>
> On 30 Mar 2010, at 18:32, Edson Tirelli <ed.tirelli at gmail.com> wrote:
>
>
>   I am not sure I understand what you are doing when you say:
>
>
> Every X seconds I collect the available facts and call fireAllRules() +
>
> dispose(), **always within the same session**.
>
> (emphasis is mine)
>
>
> i meant that i keep using the same object, i just create one stateful
>
> session throughout the program execution.
>
>
>   Compilation indeed uses more memory, but you can compile once and reuse
>
> the kbase.
>
>
>
> exactly, i'm doing that, but it's a +700 rule file and its taking too much
>
> memory. Is there anything i can do to push it down a little bit? i wouldn't
>
> mind slowing it down to reduce memory footprint.
>
>
> Also, if you are calling dispose, I guess you are running stateful
> sessions.
>
> Maybe you can give us a more detailed description of your
>
> session/data/application lifecycle?
>
>
> Sure, i'll try. So i have this thread which initializes by reading the rule
>
> base from a drl file. It then creates a stateful session from it and enters
>
> a lifecycle in which it goes to sleep and only when some other thread wakes
>
> it up, it collects and inserts some facts (usually 2,3 or 4..) into the
>
> session and fire all rules. It's basically this. What i'm very concerned
>
> about is the amount of RAM memory the whole program is consuming. I must
>
> find a way to reduce some of it, because the rule file is being prepared by
>
> an external program and i have no control over the number of rules i might
>
> get. If i test it with a little over 700 rules, it reaches +300Mb which is
> a
>
> lot.
>
>
> Maybe the rules are too complex? Shall i try to make them simpler (how)?
>
> what are the big memory hogs when it comes to loading a DRL file to the
>
> knowledge base? From what i've been testing, it doesn't even matter which
>
> rules you will actually need, it just loads the entire DRL file with no
>
> respect to the future activations, am i right? Is there any other aspect i
>
> should take in mind or maybe other approach?
>
>
> _ miguel
>
>
> _______________________________________________
> rules-users mailing list
> rules-users at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/rules-users
>
>
>
> _______________________________________________
> rules-users mailing list
> rules-users at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/rules-users
>
>

-- 
 Edson Tirelli
 JBoss Drools Core Development
 JBoss by Red Hat @ www.jboss.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/rules-users/attachments/20100331/1159ae76/attachment.html 