[rules-dev] [rules-users] Memory Leak in 5.4.0 and 5.5.0 [was: Garbage collection and sliding windows (Drools 5.5.0 Final)]
Mark Proctor
mproctor at codehaus.org
Tue Feb 19 03:45:02 EST 2013
On 17 Feb 2013, at 17:29, Wolfgang Laun <wolfgang.laun at gmail.com> wrote:
> Something that may or may not be related to that other problem -
>
> org.drools.reteoo.ReteooWorkingMemory.WorkingMemoryReteExpireAction
> method execute(InternalWorkingMemory workingMemory)
>
> Look at lines 517 - 523:
>
> context.evaluateActionQueue( workingMemory ); // <x>
> // if no activations for this expired event
> if ( ((EventFactHandle)
> factHandle).getActivationsCount() == 0 ) {
> // remove it from the object store and clean up resources
> ((EventFactHandle)
> factHandle).getEntryPoint().retract( factHandle );
> }
> context.evaluateActionQueue( workingMemory ); // <x>
>
> The repetition of <x> looks somewhat strange to me. Surely the
> getActivationsCount() doesn't have a side effect so that
> context/workingMemory have changed? If the retract(factHandle) causes
> a change, then the second call should be inside the if statement, too?
yes, but it wouldn't be the source of a memory leak - it's just potentially a wasted queue check.
>
> -W
>
>
> On 17/02/2013, Wolfgang Laun <wolfgang.laun at gmail.com> wrote:
>> Looking at org.drools.reteoo.WindowNode and assertObject(...), where
>> final WindowMemory memory =
>> (WindowMemory)workingMemory.getNodeMemory(this);
>> you can quickly convince yourself that
>> memory.events.size()
>> keeps increasing but is never decreased - apparently method
>> retractObject never gets called.
>>
>> -W
>>
>>
>> On 17/02/2013, Mark Proctor <mproctor at codehaus.org> wrote:
>>> If we can get a unit test for this, we are just about to put out 5.6
>>> (which
>>> has a lot of important fixes in it already), we'll try and have it fixed
>>> for
>>> then.
>>>
>>> Mark
>>> On 17 Feb 2013, at 08:55, Wolfgang Laun <wolfgang.laun at gmail.com> wrote:
>>>
>>>> With Drools 5.4.0.Final and 5.5.0.Final, automatic event retraction in
>>>> combination with a sliding window is broken, i.e., it suffers from a
>>>> memory leak. Full code for a demo has been posted by tai-atari; so I
>>>> just add some diagnostics that clearly indicate what goes wrong.
>>>>
>>>> In the snapshot taken with jmap -histo:live after running the program
>>>> for several seconds and the insertion of 15800 AnEvent facts you can
>>>> see:
>>>> * several classes with 2000 instances each - AnEvents and the ones
>>>> required for bookkeeping the future expiry
>>>> * EventFactHandle, WindowTupleList, ObjectHashMap$ObjectEntry all
>>>> growing without restraint, indicating a memory leak.
>>>>
>>>>
>>>> num #instances #bytes class name
>>>> ----------------------------------------------
>>>> 1: 8747 8696096 [S
>>>> 2: 26985 4110776 <constMethodKlass>
>>>> 3: 26985 2161888 <methodKlass>
>>>> 4: 47816 2046304 <symbolKlass>
>>>> 5: 2414 1482672 <constantPoolKlass>
>>>> 6: 6142 1434280 [I
>>>> 7: 13800 1324800 org.drools.common.EventFactHandle
>>>> 8: 2414 1056424 <instanceKlassKlass>
>>>> 9: 1891 854664 <constantPoolCacheKlass>
>>>> 10: 8348 644704 [C
>>>> 11: 13800 441600 org.drools.reteoo.WindowTupleList
>>>> 12: 2571 426976 [B
>>>> 13: 15800 379200
>>>> org.drools.core.util.ObjectHashMap$ObjectEntry
>>>> 14: 2672 256512 java.lang.Class
>>>> 15: 545 246776 <methodDataKlass>
>>>> 16: 7852 188448 java.lang.String
>>>> 17: 4084 185496 [[I
>>>> 18: 2000 176000
>>>> org.drools.common.PropagationContextImpl
>>>> 19: 8 164288 [Lorg.drools.core.util.Entry;
>>>> 20: 2000 160000 org.drools.common.AgendaItem
>>>> 21: 2005 128320 org.drools.reteoo.WindowTuple
>>>> 22: 2000 128000
>>>> org.drools.reteoo.RuleTerminalNodeLeftTuple
>>>> 23: 2001 96048 java.util.concurrent.FutureTask$Sync
>>>> 24: 2001 80040
>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask
>>>> 25: 1165 79152 [J
>>>> 26: 229 75112 <objArrayKlassKlass>
>>>> 27: 2001 64032
>>>> org.drools.time.impl.JDKTimerService$JDKJobHandle
>>>> 28: 2001 64032
>>>> org.drools.time.impl.DefaultTimerJobInstance
>>>> 29: 423 54728 [Ljava.lang.Object;
>>>> 30: 2150 51600 java.util.LinkedList$Entry
>>>> 31: 2052 49248 java.util.LinkedList
>>>> 32: 2002 48048
>>>> org.drools.core.util.ObjectHashSet$ObjectEntry
>>>> 33: 2001 48024 java.util.Date
>>>> 34: 2000 48000
>>>> org.drools.reteoo.ObjectTypeNode$ExpireJobContext
>>>> 35: 1958 46992 java.util.HashMap$Entry
>>>> 36: 2261 36176 java.lang.Integer
>>>> 37: 2006 32096
>>>> java.util.concurrent.atomic.AtomicBoolean
>>>> 38: 2001 32016
>>>> org.drools.time.impl.PointInTimeTrigger
>>>> 39: 2000 32000
>>>> org.drools.reteoo.ReteooWorkingMemory$WorkingMemoryReteExpireAction
>>>> 40: 2000 32000 memcons.AnEvent
>>>>
>>>>
>>>>
>>>>
>>>> On 17/02/2013, tai-atari <p00temkin at gmail.com> wrote:
>>>>> Hi laune,
>>>>>
>>>>> Really appreciate you trying to reproduce the issue. Since you are not
>>>>> experiencing the same behavior I'm obviously missing something. I've
>>>>> attached a very simple example below which reproduces the scenario for
>>>>> the
>>>>> Drools versions I have tried so far: 5.2, 5.4 and 5.5.
>>>>>
>>>>> Basically this example runs an infinite loop which inserts 200 events
>>>>> about
>>>>> every second, and keeps a sliding window of 10 seconds. Every loop will
>>>>> print the current wm event count, which will increase to 2000 and stay
>>>>> put.
>>>>> This is where I expected the expired events to be available for garbage
>>>>> collection (since only 2000 are concurrently active in wm). But
>>>>> VisualVM
>>>>> shows that org.drools.common.EventFactHandle uses an increasing amount
>>>>> of
>>>>> memory over time.
>>>>>
>>>>> Start.java:
>>>>> ==================================
>>>>> package org.drools.example;
>>>>>
>>>>> import java.util.Random;
>>>>>
>>>>> import org.drools.KnowledgeBase;
>>>>> import org.drools.KnowledgeBaseConfiguration;
>>>>> import org.drools.KnowledgeBaseFactory;
>>>>> import org.drools.builder.KnowledgeBuilder;
>>>>> import org.drools.builder.KnowledgeBuilderError;
>>>>> import org.drools.builder.KnowledgeBuilderErrors;
>>>>> import org.drools.builder.KnowledgeBuilderFactory;
>>>>> import org.drools.builder.ResourceType;
>>>>> import org.drools.conf.EventProcessingOption;
>>>>> import org.drools.io.ResourceFactory;
>>>>> import org.drools.runtime.StatefulKnowledgeSession;
>>>>> import org.drools.runtime.rule.WorkingMemoryEntryPoint;
>>>>>
>>>>> public class Start {
>>>>>
>>>>> public static final void main(String[] args) {
>>>>> try {
>>>>>
>>>>> System.out.println("Init");
>>>>> KnowledgeBase kbase = readKnowledgeBase();
>>>>> StatefulKnowledgeSession ksession =
>>>>> kbase.newStatefulKnowledgeSession();
>>>>> WorkingMemoryEntryPoint eventStream =
>>>>> ksession.getWorkingMemoryEntryPoint("TheEventStream");
>>>>> Random randGen = new Random();
>>>>>
>>>>> while( true ) {
>>>>>
>>>>> // Insert 200
>>>>> int x = 0;
>>>>> while( x < 200 ) {
>>>>> AnEvent anEvent = new AnEvent();
>>>>> anEvent.setSource(randGen.nextInt());
>>>>> eventStream.insert(anEvent);
>>>>> ksession.fireAllRules();
>>>>> x++;
>>>>> }
>>>>>
>>>>> System.out.println("current event count in wm" + ": " +
>>>>> eventStream.getFactCount());
>>>>> Thread.sleep(1000);
>>>>> }
>>>>>
>>>>> //ksession.dispose();
>>>>>
>>>>> } catch (Throwable t) {
>>>>> t.printStackTrace();
>>>>> }
>>>>> }
>>>>>
>>>>> private static KnowledgeBase readKnowledgeBase() throws Exception {
>>>>>
>>>>> KnowledgeBuilder kbuilder =
>>>>> KnowledgeBuilderFactory.newKnowledgeBuilder();
>>>>> kbuilder.add(ResourceFactory.newClassPathResource("Sample.drl"),
>>>>> ResourceType.DRL);
>>>>> KnowledgeBuilderErrors errors = kbuilder.getErrors();
>>>>> if (errors.size() > 0) {
>>>>> for (KnowledgeBuilderError error: errors) {
>>>>> System.err.println(error);
>>>>> }
>>>>> throw new IllegalArgumentException("Could not parse
>>>>> knowledge.");
>>>>> }
>>>>>
>>>>> final KnowledgeBaseConfiguration kbConfig =
>>>>> KnowledgeBaseFactory.newKnowledgeBaseConfiguration();
>>>>> kbConfig.setOption(EventProcessingOption.STREAM);
>>>>> KnowledgeBase kbase =
>>>>> KnowledgeBaseFactory.newKnowledgeBase(kbConfig);
>>>>>
>>>>> kbase.addKnowledgePackages(kbuilder.getKnowledgePackages());
>>>>> return kbase;
>>>>> }
>>>>>
>>>>> }
>>>>> ==================================
>>>>>
>>>>> AnEvent.java:
>>>>> ==================================
>>>>> package org.drools.example;
>>>>>
>>>>> public class AnEvent {
>>>>>
>>>>> private Integer source;
>>>>>
>>>>> public AnEvent () {
>>>>> }
>>>>>
>>>>> public Integer getSource() {
>>>>> return source;
>>>>> }
>>>>>
>>>>> public void setSource(Integer source) {
>>>>> this.source = source;
>>>>> }
>>>>>
>>>>> }
>>>>> ==================================
>>>>>
>>>>> Sample.drl
>>>>> ==================================
>>>>> package org.drools.example
>>>>> import org.drools.example.AnEvent;
>>>>>
>>>>> declare AnEvent
>>>>> @role( event )
>>>>> end
>>>>>
>>>>> rule "Event print"
>>>>> when
>>>>> AnEvent ( $src: source ) over window:time(10s) from entry-point
>>>>> TheEventStream
>>>>> then
>>>>> System.out.println("---------> Received AnEvent from " + $src);
>>>>> end
>>>>> ==================================
>>>>>
>>>>> Also tried using an immutable pojo as you suggested cusmaimatteo, with
>>>>>
>>>>> AnEvent.java:
>>>>> ==================================
>>>>> package org.drools.example;
>>>>>
>>>>> public class AnEvent {
>>>>>
>>>>> private final Integer source;
>>>>>
>>>>> public AnEvent (Integer i) {
>>>>> this.source = i;
>>>>> }
>>>>>
>>>>> public Integer getSource() {
>>>>> return source;
>>>>> }
>>>>> }
>>>>> ==================================
>>>>>
>>>>> and using instead using
>>>>>
>>>>> eventStream.insert(new AnEvent(randGen.nextInt()));
>>>>>
>>>>> but with the same memory issues noted.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> View this message in context:
>>>>> http://drools.46999.n3.nabble.com/Garbage-collection-and-sliding-windows-Drools-5-5-0-Final-tp4022183p4022350.html
>>>>> Sent from the Drools: User forum mailing list archive at Nabble.com.
>>>>> _______________________________________________
>>>>> rules-users mailing list
>>>>> rules-users at lists.jboss.org
>>>>> https://lists.jboss.org/mailman/listinfo/rules-users
>>>>>
>>>> _______________________________________________
>>>> rules-users mailing list
>>>> rules-users at lists.jboss.org
>>>> https://lists.jboss.org/mailman/listinfo/rules-users
>>>
>>>
>>> _______________________________________________
>>> rules-users mailing list
>>> rules-users at lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/rules-users
>>>
>>
More information about the rules-dev
mailing list