[rules-users] drools expert: memory issues

Tue Apr 27 06:50:51 EDT 2010

Hi again,

So.. i've tried doing precisely that but i am unable to fire the rules
afterwards. I am dividing the XML rule file into smaller valid XML files so
that i can compile them individually and write to disk the compiled result.
I am using the sample code shown in the docs:

// write compiled rules to disk
ObjectOutputStream out = new ObjectOutputStream(new
FileOutputStream(filename + ".temp"));
out.writeObject(kbuilder.getKnowledgePackages());
out.close();

On a later stage, i load all files holding compiled rules, one at a time, so
i can use the knowledge base and create the knowledge session. Here's what
i'm doing for each of the generated compiled files:

in = new ObjectInputStream(new FileInputStream(filename)); // the ones
*.temp
kpkgs = (Collection<KnowledgePackage>) in.readObject();
kbase.addKnowledgePackages(kpkgs);
in.close();

This all works apparently well. Drools loads the sum of all rules defined in
each smaller XML file. However, whenever i try to insert a fact into memory,
i get an exception, just like the following:

java.lang.NullPointerException
at org.drools.base.ClassFieldReader.getValue(ClassFieldReader.java:91)
at
org.drools.base.evaluators.EqualityEvaluatorsDefinition$StringEqualEvaluator.evaluate(EqualityEvaluatorsDefinition.java:1962)
at org.drools.rule.LiteralRestriction.isAllowed(LiteralRestriction.java:92)
at
org.drools.rule.OrCompositeRestriction.isAllowed(OrCompositeRestriction.java:25)
at
org.drools.rule.MultiRestrictionFieldConstraint.isAllowed(MultiRestrictionFieldConstraint.java:97)
at org.drools.reteoo.AlphaNode.assertObject(AlphaNode.java:143)
at
org.drools.reteoo.CompositeObjectSinkAdapter.doPropagateAssertObject(CompositeObjectSinkAdapter.java:360)
at
org.drools.reteoo.CompositeObjectSinkAdapter.propagateAssertObject(CompositeObjectSinkAdapter.java:344)
at org.drools.reteoo.ObjectTypeNode.assertObject(ObjectTypeNode.java:185)
at org.drools.reteoo.EntryPointNode.assertObject(EntryPointNode.java:146)
at
org.drools.common.AbstractWorkingMemory.insert(AbstractWorkingMemory.java:1046)
at
org.drools.common.AbstractWorkingMemory.insert(AbstractWorkingMemory.java:1001)
at
org.drools.common.AbstractWorkingMemory.insert(AbstractWorkingMemory.java:788)
at
org.drools.impl.StatefulKnowledgeSessionImpl.insert(StatefulKnowledgeSessionImpl.java:216)
at
pt.itsd.probe.ConfigurationSetup.parseAndCompileConfigFiles(ConfigurationSetup.java:316)
at pt.itsd.probe.ConfigurationSetup.main(ConfigurationSetup.java:131)

I have done some testing and I am unable to trace this exception to any
particular rule, or event any XML chunck. If i load the entire XML file it
takes a lot of memory compiling but it works in the end. Also, if i remove
all the <or-restriction-connective> tags, it also works no matter if i load
them to kbase.

Is there something i can do? or maybe another way of doing it? Any help is
much appreciated.

I'm using drools 5.0.0 from main download page.
_ miguel

2010/4/3 Wolfgang Laun <wolfgang.laun at gmail.com>

> It should be possible to increase heap space to overcome this problem. Of
> course, if a program wil start swapping, performance will go way down.
>
> Dividing the rules file is the only way. But even a very simple script of
> Perl or similar should be able to divide your (IIRC) generated rules file.
>
> -W
>
> 2010/4/3 miguel machado <mls.machado at gmail.com>
>
>> hi there,
>>
>> So like i said, by pre-compiling the rules and loading them in the main
>> system i was able to reduce a lot memory usage and i can get better results.
>> However, now i've come across a different problem. I just added a fre more
>> dozens of rules and the java program responsible for compiling the DRL file
>> throws a java heap space error, something like an OutOfMemoryException.
>>
>> I've read somewhere that dividing the DRL into several files and loading
>> e.g. 500 rules at a time might solve it. Is there another approach to this
>> problem?
>>
>> thanks in advance,
>> _ miguel
>>
>>
>>
>> On Thu, Apr 1, 2010 at 6:12 PM, miguel machado <mls.machado at gmail.com>wrote:
>>
>>> hi again,
>>>
>>>  2010/4/1 Edson Tirelli <ed.tirelli at gmail.com>
>>>
>>>
>>>>    Hmm, let me complement Wolfgang's comment:
>>>>
>>>> > After you have taken the compiled packages from the KB and checked for
>>>> errors, the KB has done his duty the KB ****MUST**** go :-)
>>>>
>>>
>>> I agree! But i don't exactly know how to do so... i mean, i call this
>>> method which creates the KB from a DRL file and then the kbase and returns
>>> the ksession, the KB object declaration scope is just that exact method, and
>>> once it reaches 300MB+ in memory, it just stays there... the KB object is no
>>> longer in use (or is it?) but memory keeps high.
>>>
>>> I've tried setting KB to "null" and then invoking the jvm garbage
>>> collector, but to no avail. I wish there was a simple way i could just
>>> destroy a particular object sigh
>>>
>>>
>>>>
>>>>    The above emphasis is mine. KnowledgeBuilder is a higher level object
>>>> designed to encapsulate the parsing+compilation steps of resources. It is
>>>> **not** designed to be reused and doing so might create inconsistencies. The
>>>> memory spike you see when using the kbuilder is due to the java compiler...
>>>
>>>
>>> I understand that, and i'll have that if i must, but the thing is it's
>>> not just a spike, it is prolonged throughout execution, it doesn't reduce
>>> afterwards.
>>>
>>>
>>>> Drools itself does not do anything fancy in there other than generate
>>>> some code (quite inexpensive memory wise) and compile it using either JDT or
>>>> JANINO (both quite expensive memory wise, when compared to the other steps).
>>>>
>>>>
>>>>    So, I strongly discourage you keeping the KB around and as mentioned
>>>> by Wolfgang, doing so might keep unnecessary objects in memory.
>>>>
>>>
>>> Like i said earlier, it's not that i want to keep KB around, i just can't
>>> remove it.
>>>
>>>
>>>>
>>>>    Also, just to clarify, deserializing a compiled package means the
>>>> compiler will not be called (obviously, as everything is already compiled)
>>>> and that is why it should save you memory.
>>>>
>>>
>>> Thanks for clarifying that! I've tested with a separate application to
>>> build a compiled file and then load it in the main system and it really
>>> makes a difference! it's gone to <100MB wow! However, this was just a
>>> proof-of-concept, as i'm not sure how i could apply the same technique to
>>> the main standalone application. Do i really have to create a separate
>>> process/application just for building/compiling the rules, to make sure the
>>> compilation-object-junk doesn't stay in memory along with the rest?
>>>
>>> Thanks for all the feedback and input so far.
>>> _ miguel
>>>
>>>
>>> --
>>> "To understand what is recursion you must first understand recursion"
>>>
>>
>>
>>
>> --
>> "To understand what is recursion you must first understand recursion"
>>
>> _______________________________________________
>> rules-users mailing list
>> rules-users at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/rules-users
>>
>>
>
> _______________________________________________
> rules-users mailing list
> rules-users at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/rules-users
>
>

-- 
"To understand what is recursion you must first understand recursion"
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/rules-users/attachments/20100427/f5883466/attachment.html 