If we can get a unit test for this, we are just about to put out 5.6 (which has a lot of important fixes in it already), we'll try and have it fixed for then. Mark On 17 Feb 2013, at 08:55, Wolfgang Laun <wolfgang.laun(a)gmail.com&gt; wrote: > With Drools 5.4.0.Final and 5.5.0.Final, automatic event retraction in > combination with a sliding window is broken, i.e., it suffers from a > memory leak. Full code for a demo has been posted by tai-atari; so I > just add some diagnostics that clearly indicate what goes wrong. > > In the snapshot taken with jmap -histo:live after running the program > for several seconds and the insertion of 15800 AnEvent facts you can > see: > * several classes with 2000 instances each - AnEvents and the ones > required for bookkeeping the future expiry > * EventFactHandle, WindowTupleList, ObjectHashMap$ObjectEntry all > growing without restraint, indicating a memory leak. > > > num #instances #bytes class name > ---------------------------------------------- > 1: 8747 8696096 [S > 2: 26985 4110776 <constMethodKlass> > 3: 26985 2161888 <methodKlass> > 4: 47816 2046304 <symbolKlass> > 5: 2414 1482672 <constantPoolKlass> > 6: 6142 1434280 [I > 7: 13800 1324800 org.drools.common.EventFactHandle > 8: 2414 1056424 <instanceKlassKlass> > 9: 1891 854664 <constantPoolCacheKlass> > 10: 8348 644704 [C > 11: 13800 441600 org.drools.reteoo.WindowTupleList > 12: 2571 426976 [B > 13: 15800 379200 > org.drools.core.util.ObjectHashMap$ObjectEntry > 14: 2672 256512 java.lang.Class > 15: 545 246776 <methodDataKlass> > 16: 7852 188448 java.lang.String > 17: 4084 185496 [[I > 18: 2000 176000 > org.drools.common.PropagationContextImpl > 19: 8 164288 [Lorg.drools.core.util.Entry; > 20: 2000 160000 org.drools.common.AgendaItem > 21: 2005 128320 org.drools.reteoo.WindowTuple > 22: 2000 128000 > org.drools.reteoo.RuleTerminalNodeLeftTuple > 23: 2001 96048 java.util.concurrent.FutureTask$Sync > 24: 2001 80040 > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask > 25: 1165 79152 [J > 26: 229 75112 <objArrayKlassKlass> > 27: 2001 64032 > org.drools.time.impl.JDKTimerService$JDKJobHandle > 28: 2001 64032 > org.drools.time.impl.DefaultTimerJobInstance > 29: 423 54728 [Ljava.lang.Object; > 30: 2150 51600 java.util.LinkedList$Entry > 31: 2052 49248 java.util.LinkedList > 32: 2002 48048 > org.drools.core.util.ObjectHashSet$ObjectEntry > 33: 2001 48024 java.util.Date > 34: 2000 48000 > org.drools.reteoo.ObjectTypeNode$ExpireJobContext > 35: 1958 46992 java.util.HashMap$Entry > 36: 2261 36176 java.lang.Integer > 37: 2006 32096 > java.util.concurrent.atomic.AtomicBoolean > 38: 2001 32016 > org.drools.time.impl.PointInTimeTrigger > 39: 2000 32000 > org.drools.reteoo.ReteooWorkingMemory$WorkingMemoryReteExpireAction > 40: 2000 32000 memcons.AnEvent > > > > > On 17/02/2013, tai-atari <p00temkin(a)gmail.com&gt; wrote: >> Hi laune, >> >> Really appreciate you trying to reproduce the issue. Since you are not >> experiencing the same behavior I'm obviously missing something. I've >> attached a very simple example below which reproduces the scenario for >> the >> Drools versions I have tried so far: 5.2, 5.4 and 5.5. >> >> Basically this example runs an infinite loop which inserts 200 events >> about >> every second, and keeps a sliding window of 10 seconds. Every loop will >> print the current wm event count, which will increase to 2000 and stay >> put. >> This is where I expected the expired events to be available for garbage >> collection (since only 2000 are concurrently active in wm). But VisualVM >> shows that org.drools.common.EventFactHandle uses an increasing amount >> of >> memory over time. >> >> Start.java: >> ================================== >> package org.drools.example; >> >> import java.util.Random; >> >> import org.drools.KnowledgeBase; >> import org.drools.KnowledgeBaseConfiguration; >> import org.drools.KnowledgeBaseFactory; >> import org.drools.builder.KnowledgeBuilder; >> import org.drools.builder.KnowledgeBuilderError; >> import org.drools.builder.KnowledgeBuilderErrors; >> import org.drools.builder.KnowledgeBuilderFactory; >> import org.drools.builder.ResourceType; >> import org.drools.conf.EventProcessingOption; >> import org.drools.io.ResourceFactory; >> import org.drools.runtime.StatefulKnowledgeSession; >> import org.drools.runtime.rule.WorkingMemoryEntryPoint; >> >> public class Start { >> >> public static final void main(String[] args) { >> try { >> >> System.out.println("Init"); >> KnowledgeBase kbase = readKnowledgeBase(); >> StatefulKnowledgeSession ksession = >> kbase.newStatefulKnowledgeSession(); >> WorkingMemoryEntryPoint eventStream = >> ksession.getWorkingMemoryEntryPoint("TheEventStream"); >> Random randGen = new Random(); >> >> while( true ) { >> >> // Insert 200 >> int x = 0; >> while( x < 200 ) { >> AnEvent anEvent = new AnEvent(); >> anEvent.setSource(randGen.nextInt()); >> eventStream.insert(anEvent); >> ksession.fireAllRules(); >> x++; >> } >> >> System.out.println("current event count in wm" + ": " + >> eventStream.getFactCount()); >> Thread.sleep(1000); >> } >> >> //ksession.dispose(); >> >> } catch (Throwable t) { >> t.printStackTrace(); >> } >> } >> >> private static KnowledgeBase readKnowledgeBase() throws Exception { >> >> KnowledgeBuilder kbuilder = >> KnowledgeBuilderFactory.newKnowledgeBuilder(); >> kbuilder.add(ResourceFactory.newClassPathResource("Sample.drl"), >> ResourceType.DRL); >> KnowledgeBuilderErrors errors = kbuilder.getErrors(); >> if (errors.size() > 0) { >> for (KnowledgeBuilderError error: errors) { >> System.err.println(error); >> } >> throw new IllegalArgumentException("Could not parse >> knowledge."); >> } >> >> final KnowledgeBaseConfiguration kbConfig = >> KnowledgeBaseFactory.newKnowledgeBaseConfiguration(); >> kbConfig.setOption(EventProcessingOption.STREAM); >> KnowledgeBase kbase = >> KnowledgeBaseFactory.newKnowledgeBase(kbConfig); >> >> kbase.addKnowledgePackages(kbuilder.getKnowledgePackages()); >> return kbase; >> } >> >> } >> ================================== >> >> AnEvent.java: >> ================================== >> package org.drools.example; >> >> public class AnEvent { >> >> private Integer source; >> >> public AnEvent () { >> } >> >> public Integer getSource() { >> return source; >> } >> >> public void setSource(Integer source) { >> this.source = source; >> } >> >> } >> ================================== >> >> Sample.drl >> ================================== >> package org.drools.example >> import org.drools.example.AnEvent; >> >> declare AnEvent >> @role( event ) >> end >> >> rule "Event print" >> when >> AnEvent ( $src: source ) over window:time(10s) from entry-point >> TheEventStream >> then >> System.out.println("---------> Received AnEvent from " + $src); >> end >> ================================== >> >> Also tried using an immutable pojo as you suggested cusmaimatteo, with >> >> AnEvent.java: >> ================================== >> package org.drools.example; >> >> public class AnEvent { >> >> private final Integer source; >> >> public AnEvent (Integer i) { >> this.source = i; >> } >> >> public Integer getSource() { >> return source; >> } >> } >> ================================== >> >> and using instead using >> >> eventStream.insert(new AnEvent(randGen.nextInt())); >> >> but with the same memory issues noted. >> >> >> >> >> >> -- >> View this message in context: >> http://drools.46999.n3.nabble.com/Garbage-collection-and-sliding-windows-... >> Sent from the Drools: User forum mailing list archive at Nabble.com. >> _______________________________________________ >> rules-users mailing list >> rules-users(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/rules-users >> > _______________________________________________ > rules-users mailing list > rules-users(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/rules-users _______________________________________________ rules-users mailing list rules-users(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/rules-users

Something that may or may not be related to that other problem - org.drools.reteoo.ReteooWorkingMemory.WorkingMemoryReteExpireAction method execute(InternalWorkingMemory workingMemory) Look at lines 517 - 523: context.evaluateActionQueue( workingMemory ); // <x> // if no activations for this expired event if ( ((EventFactHandle) factHandle).getActivationsCount() == 0 ) { // remove it from the object store and clean up resources ((EventFactHandle) factHandle).getEntryPoint().retract( factHandle ); } context.evaluateActionQueue( workingMemory ); // <x> The repetition of <x> looks somewhat strange to me. Surely the getActivationsCount() doesn't have a side effect so that context/workingMemory have changed? If the retract(factHandle) causes a change, then the second call should be inside the if statement, too?

-W On 17/02/2013, Wolfgang Laun <wolfgang.laun(a)gmail.com&gt; wrote: > Looking at org.drools.reteoo.WindowNode and assertObject(...), where > final WindowMemory memory = > (WindowMemory)workingMemory.getNodeMemory(this); > you can quickly convince yourself that > memory.events.size() > keeps increasing but is never decreased - apparently method > retractObject never gets called. > > -W > > > On 17/02/2013, Mark Proctor <mproctor(a)codehaus.org&gt; wrote: >> If we can get a unit test for this, we are just about to put out 5.6 >> (which >> has a lot of important fixes in it already), we'll try and have it fixed >> for >> then. >> >> Mark >> On 17 Feb 2013, at 08:55, Wolfgang Laun <wolfgang.laun(a)gmail.com&gt; wrote: >> >>> With Drools 5.4.0.Final and 5.5.0.Final, automatic event retraction in >>> combination with a sliding window is broken, i.e., it suffers from a >>> memory leak. Full code for a demo has been posted by tai-atari; so I >>> just add some diagnostics that clearly indicate what goes wrong. >>> >>> In the snapshot taken with jmap -histo:live after running the program >>> for several seconds and the insertion of 15800 AnEvent facts you can >>> see: >>> * several classes with 2000 instances each - AnEvents and the ones >>> required for bookkeeping the future expiry >>> * EventFactHandle, WindowTupleList, ObjectHashMap$ObjectEntry all >>> growing without restraint, indicating a memory leak. >>> >>> >>> num #instances #bytes class name >>> ---------------------------------------------- >>> 1: 8747 8696096 [S >>> 2: 26985 4110776 <constMethodKlass> >>> 3: 26985 2161888 <methodKlass> >>> 4: 47816 2046304 <symbolKlass> >>> 5: 2414 1482672 <constantPoolKlass> >>> 6: 6142 1434280 [I >>> 7: 13800 1324800 org.drools.common.EventFactHandle >>> 8: 2414 1056424 <instanceKlassKlass> >>> 9: 1891 854664 <constantPoolCacheKlass> >>> 10: 8348 644704 [C >>> 11: 13800 441600 org.drools.reteoo.WindowTupleList >>> 12: 2571 426976 [B >>> 13: 15800 379200 >>> org.drools.core.util.ObjectHashMap$ObjectEntry >>> 14: 2672 256512 java.lang.Class >>> 15: 545 246776 <methodDataKlass> >>> 16: 7852 188448 java.lang.String >>> 17: 4084 185496 [[I >>> 18: 2000 176000 >>> org.drools.common.PropagationContextImpl >>> 19: 8 164288 [Lorg.drools.core.util.Entry; >>> 20: 2000 160000 org.drools.common.AgendaItem >>> 21: 2005 128320 org.drools.reteoo.WindowTuple >>> 22: 2000 128000 >>> org.drools.reteoo.RuleTerminalNodeLeftTuple >>> 23: 2001 96048 java.util.concurrent.FutureTask$Sync >>> 24: 2001 80040 >>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask >>> 25: 1165 79152 [J >>> 26: 229 75112 <objArrayKlassKlass> >>> 27: 2001 64032 >>> org.drools.time.impl.JDKTimerService$JDKJobHandle >>> 28: 2001 64032 >>> org.drools.time.impl.DefaultTimerJobInstance >>> 29: 423 54728 [Ljava.lang.Object; >>> 30: 2150 51600 java.util.LinkedList$Entry >>> 31: 2052 49248 java.util.LinkedList >>> 32: 2002 48048 >>> org.drools.core.util.ObjectHashSet$ObjectEntry >>> 33: 2001 48024 java.util.Date >>> 34: 2000 48000 >>> org.drools.reteoo.ObjectTypeNode$ExpireJobContext >>> 35: 1958 46992 java.util.HashMap$Entry >>> 36: 2261 36176 java.lang.Integer >>> 37: 2006 32096 >>> java.util.concurrent.atomic.AtomicBoolean >>> 38: 2001 32016 >>> org.drools.time.impl.PointInTimeTrigger >>> 39: 2000 32000 >>> org.drools.reteoo.ReteooWorkingMemory$WorkingMemoryReteExpireAction >>> 40: 2000 32000 memcons.AnEvent >>> >>> >>> >>> >>> On 17/02/2013, tai-atari <p00temkin(a)gmail.com&gt; wrote: >>>> Hi laune, >>>> >>>> Really appreciate you trying to reproduce the issue. Since you are not >>>> experiencing the same behavior I'm obviously missing something. I've >>>> attached a very simple example below which reproduces the scenario for >>>> the >>>> Drools versions I have tried so far: 5.2, 5.4 and 5.5. >>>> >>>> Basically this example runs an infinite loop which inserts 200 events >>>> about >>>> every second, and keeps a sliding window of 10 seconds. Every loop will >>>> print the current wm event count, which will increase to 2000 and stay >>>> put. >>>> This is where I expected the expired events to be available for garbage >>>> collection (since only 2000 are concurrently active in wm). But >>>> VisualVM >>>> shows that org.drools.common.EventFactHandle uses an increasing amount >>>> of >>>> memory over time. >>>> >>>> Start.java: >>>> ================================== >>>> package org.drools.example; >>>> >>>> import java.util.Random; >>>> >>>> import org.drools.KnowledgeBase; >>>> import org.drools.KnowledgeBaseConfiguration; >>>> import org.drools.KnowledgeBaseFactory; >>>> import org.drools.builder.KnowledgeBuilder; >>>> import org.drools.builder.KnowledgeBuilderError; >>>> import org.drools.builder.KnowledgeBuilderErrors; >>>> import org.drools.builder.KnowledgeBuilderFactory; >>>> import org.drools.builder.ResourceType; >>>> import org.drools.conf.EventProcessingOption; >>>> import org.drools.io.ResourceFactory; >>>> import org.drools.runtime.StatefulKnowledgeSession; >>>> import org.drools.runtime.rule.WorkingMemoryEntryPoint; >>>> >>>> public class Start { >>>> >>>> public static final void main(String[] args) { >>>> try { >>>> >>>> System.out.println("Init"); >>>> KnowledgeBase kbase = readKnowledgeBase(); >>>> StatefulKnowledgeSession ksession = >>>> kbase.newStatefulKnowledgeSession(); >>>> WorkingMemoryEntryPoint eventStream = >>>> ksession.getWorkingMemoryEntryPoint("TheEventStream"); >>>> Random randGen = new Random(); >>>> >>>> while( true ) { >>>> >>>> // Insert 200 >>>> int x = 0; >>>> while( x < 200 ) { >>>> AnEvent anEvent = new AnEvent(); >>>> anEvent.setSource(randGen.nextInt()); >>>> eventStream.insert(anEvent); >>>> ksession.fireAllRules(); >>>> x++; >>>> } >>>> >>>> System.out.println("current event count in wm" + ": " + >>>> eventStream.getFactCount()); >>>> Thread.sleep(1000); >>>> } >>>> >>>> //ksession.dispose(); >>>> >>>> } catch (Throwable t) { >>>> t.printStackTrace(); >>>> } >>>> } >>>> >>>> private static KnowledgeBase readKnowledgeBase() throws Exception { >>>> >>>> KnowledgeBuilder kbuilder = >>>> KnowledgeBuilderFactory.newKnowledgeBuilder(); >>>> kbuilder.add(ResourceFactory.newClassPathResource("Sample.drl"), >>>> ResourceType.DRL); >>>> KnowledgeBuilderErrors errors = kbuilder.getErrors(); >>>> if (errors.size() > 0) { >>>> for (KnowledgeBuilderError error: errors) { >>>> System.err.println(error); >>>> } >>>> throw new IllegalArgumentException("Could not parse >>>> knowledge."); >>>> } >>>> >>>> final KnowledgeBaseConfiguration kbConfig = >>>> KnowledgeBaseFactory.newKnowledgeBaseConfiguration(); >>>> kbConfig.setOption(EventProcessingOption.STREAM); >>>> KnowledgeBase kbase = >>>> KnowledgeBaseFactory.newKnowledgeBase(kbConfig); >>>> >>>> kbase.addKnowledgePackages(kbuilder.getKnowledgePackages()); >>>> return kbase; >>>> } >>>> >>>> } >>>> ================================== >>>> >>>> AnEvent.java: >>>> ================================== >>>> package org.drools.example; >>>> >>>> public class AnEvent { >>>> >>>> private Integer source; >>>> >>>> public AnEvent () { >>>> } >>>> >>>> public Integer getSource() { >>>> return source; >>>> } >>>> >>>> public void setSource(Integer source) { >>>> this.source = source; >>>> } >>>> >>>> } >>>> ================================== >>>> >>>> Sample.drl >>>> ================================== >>>> package org.drools.example >>>> import org.drools.example.AnEvent; >>>> >>>> declare AnEvent >>>> @role( event ) >>>> end >>>> >>>> rule "Event print" >>>> when >>>> AnEvent ( $src: source ) over window:time(10s) from entry-point >>>> TheEventStream >>>> then >>>> System.out.println("---------> Received AnEvent from " + $src); >>>> end >>>> ================================== >>>> >>>> Also tried using an immutable pojo as you suggested cusmaimatteo, with >>>> >>>> AnEvent.java: >>>> ================================== >>>> package org.drools.example; >>>> >>>> public class AnEvent { >>>> >>>> private final Integer source; >>>> >>>> public AnEvent (Integer i) { >>>> this.source = i; >>>> } >>>> >>>> public Integer getSource() { >>>> return source; >>>> } >>>> } >>>> ================================== >>>> >>>> and using instead using >>>> >>>> eventStream.insert(new AnEvent(randGen.nextInt())); >>>> >>>> but with the same memory issues noted. >>>> >>>> >>>> >>>> >>>> >>>> -- >>>> View this message in context: >>>> http://drools.46999.n3.nabble.com/Garbage-collection-and-sliding-windows-... >>>> Sent from the Drools: User forum mailing list archive at Nabble.com. >>>> _______________________________________________ >>>> rules-users mailing list >>>> rules-users(a)lists.jboss.org >>>> https://lists.jboss.org/mailman/listinfo/rules-users >>>> >>> _______________________________________________ >>> rules-users mailing list >>> rules-users(a)lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/rules-users >> >> >> _______________________________________________ >> rules-users mailing list >> rules-users(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/rules-users >> >