With Drools 5.4.0.Final and 5.5.0.Final, automatic event retraction in combination with a sliding window is broken, i.e., it suffers from a memory leak. Full code for a demo has been posted by tai-atari; so I just add some diagnostics that clearly indicate what goes wrong. In the snapshot taken with jmap -histo:live after running the program for several seconds and the insertion of 15800 AnEvent facts you can see: * several classes with 2000 instances each - AnEvents and the ones required for bookkeeping the future expiry * EventFactHandle, WindowTupleList, ObjectHashMap$ObjectEntry all growing without restraint, indicating a memory leak. num #instances #bytes class name ---------------------------------------------- 1: 8747 8696096 [S 2: 26985 4110776 <constMethodKlass> 3: 26985 2161888 <methodKlass> 4: 47816 2046304 <symbolKlass> 5: 2414 1482672 <constantPoolKlass> 6: 6142 1434280 [I 7: 13800 1324800 org.drools.common.EventFactHandle 8: 2414 1056424 <instanceKlassKlass> 9: 1891 854664 <constantPoolCacheKlass> 10: 8348 644704 [C 11: 13800 441600 org.drools.reteoo.WindowTupleList 12: 2571 426976 [B 13: 15800 379200 org.drools.core.util.ObjectHashMap$ObjectEntry 14: 2672 256512 java.lang.Class 15: 545 246776 <methodDataKlass> 16: 7852 188448 java.lang.String 17: 4084 185496 [[I 18: 2000 176000 org.drools.common.PropagationContextImpl 19: 8 164288 [Lorg.drools.core.util.Entry; 20: 2000 160000 org.drools.common.AgendaItem 21: 2005 128320 org.drools.reteoo.WindowTuple 22: 2000 128000 org.drools.reteoo.RuleTerminalNodeLeftTuple 23: 2001 96048 java.util.concurrent.FutureTask$Sync 24: 2001 80040 java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask 25: 1165 79152 [J 26: 229 75112 <objArrayKlassKlass> 27: 2001 64032 org.drools.time.impl.JDKTimerService$JDKJobHandle 28: 2001 64032 org.drools.time.impl.DefaultTimerJobInstance 29: 423 54728 [Ljava.lang.Object; 30: 2150 51600 java.util.LinkedList$Entry 31: 2052 49248 java.util.LinkedList 32: 2002 48048 org.drools.core.util.ObjectHashSet$ObjectEntry 33: 2001 48024 java.util.Date 34: 2000 48000 org.drools.reteoo.ObjectTypeNode$ExpireJobContext 35: 1958 46992 java.util.HashMap$Entry 36: 2261 36176 java.lang.Integer 37: 2006 32096 java.util.concurrent.atomic.AtomicBoolean 38: 2001 32016 org.drools.time.impl.PointInTimeTrigger 39: 2000 32000 org.drools.reteoo.ReteooWorkingMemory$WorkingMemoryReteExpireAction 40: 2000 32000 memcons.AnEvent On 17/02/2013, tai-atari <p00temkin(a)gmail.com&gt; wrote: > Hi laune, > > Really appreciate you trying to reproduce the issue. Since you are not > experiencing the same behavior I'm obviously missing something. I've > attached a very simple example below which reproduces the scenario for the > Drools versions I have tried so far: 5.2, 5.4 and 5.5. > > Basically this example runs an infinite loop which inserts 200 events about > every second, and keeps a sliding window of 10 seconds. Every loop will > print the current wm event count, which will increase to 2000 and stay put. > This is where I expected the expired events to be available for garbage > collection (since only 2000 are concurrently active in wm). But VisualVM > shows that org.drools.common.EventFactHandle uses an increasing amount of > memory over time. > > Start.java: > ================================== > package org.drools.example; > > import java.util.Random; > > import org.drools.KnowledgeBase; > import org.drools.KnowledgeBaseConfiguration; > import org.drools.KnowledgeBaseFactory; > import org.drools.builder.KnowledgeBuilder; > import org.drools.builder.KnowledgeBuilderError; > import org.drools.builder.KnowledgeBuilderErrors; > import org.drools.builder.KnowledgeBuilderFactory; > import org.drools.builder.ResourceType; > import org.drools.conf.EventProcessingOption; > import org.drools.io.ResourceFactory; > import org.drools.runtime.StatefulKnowledgeSession; > import org.drools.runtime.rule.WorkingMemoryEntryPoint; > > public class Start { > > public static final void main(String[] args) { > try { > > System.out.println("Init"); > KnowledgeBase kbase = readKnowledgeBase(); > StatefulKnowledgeSession ksession = > kbase.newStatefulKnowledgeSession(); > WorkingMemoryEntryPoint eventStream = > ksession.getWorkingMemoryEntryPoint("TheEventStream"); > Random randGen = new Random(); > > while( true ) { > > // Insert 200 > int x = 0; > while( x < 200 ) { > AnEvent anEvent = new AnEvent(); > anEvent.setSource(randGen.nextInt()); > eventStream.insert(anEvent); > ksession.fireAllRules(); > x++; > } > > System.out.println("current event count in wm" + ": " + > eventStream.getFactCount()); > Thread.sleep(1000); > } > > //ksession.dispose(); > > } catch (Throwable t) { > t.printStackTrace(); > } > } > > private static KnowledgeBase readKnowledgeBase() throws Exception { > > KnowledgeBuilder kbuilder = > KnowledgeBuilderFactory.newKnowledgeBuilder(); > kbuilder.add(ResourceFactory.newClassPathResource("Sample.drl"), > ResourceType.DRL); > KnowledgeBuilderErrors errors = kbuilder.getErrors(); > if (errors.size() > 0) { > for (KnowledgeBuilderError error: errors) { > System.err.println(error); > } > throw new IllegalArgumentException("Could not parse > knowledge."); > } > > final KnowledgeBaseConfiguration kbConfig = > KnowledgeBaseFactory.newKnowledgeBaseConfiguration(); > kbConfig.setOption(EventProcessingOption.STREAM); > KnowledgeBase kbase = KnowledgeBaseFactory.newKnowledgeBase(kbConfig); > > kbase.addKnowledgePackages(kbuilder.getKnowledgePackages()); > return kbase; > } > > } > ================================== > > AnEvent.java: > ================================== > package org.drools.example; > > public class AnEvent { > > private Integer source; > > public AnEvent () { > } > > public Integer getSource() { > return source; > } > > public void setSource(Integer source) { > this.source = source; > } > > } > ================================== > > Sample.drl > ================================== > package org.drools.example > import org.drools.example.AnEvent; > > declare AnEvent > @role( event ) > end > > rule "Event print" > when > AnEvent ( $src: source ) over window:time(10s) from entry-point > TheEventStream > then > System.out.println("---------> Received AnEvent from " + $src); > end > ================================== > > Also tried using an immutable pojo as you suggested cusmaimatteo, with > > AnEvent.java: > ================================== > package org.drools.example; > > public class AnEvent { > > private final Integer source; > > public AnEvent (Integer i) { > this.source = i; > } > > public Integer getSource() { > return source; > } > } > ================================== > > and using instead using > > eventStream.insert(new AnEvent(randGen.nextInt())); > > but with the same memory issues noted. > > > > > > -- > View this message in context: > http://drools.46999.n3.nabble.com/Garbage-collection-and-sliding-windows-... > Sent from the Drools: User forum mailing list archive at Nabble.com. > _______________________________________________ > rules-users mailing list > rules-users(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/rules-users > _______________________________________________ rules-users mailing list rules-users(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/rules-users

Give me a JUnit framework where you see the increasing memory consumption...? On 17/02/2013, Mark Proctor <mproctor(a)codehaus.org&gt; wrote: > If we can get a unit test for this, we are just about to put out 5.6 > (which > has a lot of important fixes in it already), we'll try and have it fixed > for > then. > > Mark > On 17 Feb 2013, at 08:55, Wolfgang Laun <wolfgang.laun(a)gmail.com&gt; wrote: > >> With Drools 5.4.0.Final and 5.5.0.Final, automatic event retraction in >> combination with a sliding window is broken, i.e., it suffers from a >> memory leak. Full code for a demo has been posted by tai-atari; so I >> just add some diagnostics that clearly indicate what goes wrong. >> >> In the snapshot taken with jmap -histo:live after running the program >> for several seconds and the insertion of 15800 AnEvent facts you can >> see: >> * several classes with 2000 instances each - AnEvents and the ones >> required for bookkeeping the future expiry >> * EventFactHandle, WindowTupleList, ObjectHashMap$ObjectEntry all >> growing without restraint, indicating a memory leak. >> >> >> num #instances #bytes class name >> ---------------------------------------------- >> 1: 8747 8696096 [S >> 2: 26985 4110776 <constMethodKlass> >> 3: 26985 2161888 <methodKlass> >> 4: 47816 2046304 <symbolKlass> >> 5: 2414 1482672 <constantPoolKlass> >> 6: 6142 1434280 [I >> 7: 13800 1324800 org.drools.common.EventFactHandle >> 8: 2414 1056424 <instanceKlassKlass> >> 9: 1891 854664 <constantPoolCacheKlass> >> 10: 8348 644704 [C >> 11: 13800 441600 org.drools.reteoo.WindowTupleList >> 12: 2571 426976 [B >> 13: 15800 379200 >> org.drools.core.util.ObjectHashMap$ObjectEntry >> 14: 2672 256512 java.lang.Class >> 15: 545 246776 <methodDataKlass> >> 16: 7852 188448 java.lang.String >> 17: 4084 185496 [[I >> 18: 2000 176000 >> org.drools.common.PropagationContextImpl >> 19: 8 164288 [Lorg.drools.core.util.Entry; >> 20: 2000 160000 org.drools.common.AgendaItem >> 21: 2005 128320 org.drools.reteoo.WindowTuple >> 22: 2000 128000 >> org.drools.reteoo.RuleTerminalNodeLeftTuple >> 23: 2001 96048 java.util.concurrent.FutureTask$Sync >> 24: 2001 80040 >> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask >> 25: 1165 79152 [J >> 26: 229 75112 <objArrayKlassKlass> >> 27: 2001 64032 >> org.drools.time.impl.JDKTimerService$JDKJobHandle >> 28: 2001 64032 >> org.drools.time.impl.DefaultTimerJobInstance >> 29: 423 54728 [Ljava.lang.Object; >> 30: 2150 51600 java.util.LinkedList$Entry >> 31: 2052 49248 java.util.LinkedList >> 32: 2002 48048 >> org.drools.core.util.ObjectHashSet$ObjectEntry >> 33: 2001 48024 java.util.Date >> 34: 2000 48000 >> org.drools.reteoo.ObjectTypeNode$ExpireJobContext >> 35: 1958 46992 java.util.HashMap$Entry >> 36: 2261 36176 java.lang.Integer >> 37: 2006 32096 >> java.util.concurrent.atomic.AtomicBoolean >> 38: 2001 32016 >> org.drools.time.impl.PointInTimeTrigger >> 39: 2000 32000 >> org.drools.reteoo.ReteooWorkingMemory$WorkingMemoryReteExpireAction >> 40: 2000 32000 memcons.AnEvent >> >> >> >> >> On 17/02/2013, tai-atari <p00temkin(a)gmail.com&gt; wrote: >>> Hi laune, >>> >>> Really appreciate you trying to reproduce the issue. Since you are not >>> experiencing the same behavior I'm obviously missing something. I've >>> attached a very simple example below which reproduces the scenario for >>> the >>> Drools versions I have tried so far: 5.2, 5.4 and 5.5. >>> >>> Basically this example runs an infinite loop which inserts 200 events >>> about >>> every second, and keeps a sliding window of 10 seconds. Every loop will >>> print the current wm event count, which will increase to 2000 and stay >>> put. >>> This is where I expected the expired events to be available for garbage >>> collection (since only 2000 are concurrently active in wm). But >>> VisualVM >>> shows that org.drools.common.EventFactHandle uses an increasing amount >>> of >>> memory over time. >>> >>> Start.java: >>> ================================== >>> package org.drools.example; >>> >>> import java.util.Random; >>> >>> import org.drools.KnowledgeBase; >>> import org.drools.KnowledgeBaseConfiguration; >>> import org.drools.KnowledgeBaseFactory; >>> import org.drools.builder.KnowledgeBuilder; >>> import org.drools.builder.KnowledgeBuilderError; >>> import org.drools.builder.KnowledgeBuilderErrors; >>> import org.drools.builder.KnowledgeBuilderFactory; >>> import org.drools.builder.ResourceType; >>> import org.drools.conf.EventProcessingOption; >>> import org.drools.io.ResourceFactory; >>> import org.drools.runtime.StatefulKnowledgeSession; >>> import org.drools.runtime.rule.WorkingMemoryEntryPoint; >>> >>> public class Start { >>> >>> public static final void main(String[] args) { >>> try { >>> >>> System.out.println("Init"); >>> KnowledgeBase kbase = readKnowledgeBase(); >>> StatefulKnowledgeSession ksession = >>> kbase.newStatefulKnowledgeSession(); >>> WorkingMemoryEntryPoint eventStream = >>> ksession.getWorkingMemoryEntryPoint("TheEventStream"); >>> Random randGen = new Random(); >>> >>> while( true ) { >>> >>> // Insert 200 >>> int x = 0; >>> while( x < 200 ) { >>> AnEvent anEvent = new AnEvent(); >>> anEvent.setSource(randGen.nextInt()); >>> eventStream.insert(anEvent); >>> ksession.fireAllRules(); >>> x++; >>> } >>> >>> System.out.println("current event count in wm" + ": " + >>> eventStream.getFactCount()); >>> Thread.sleep(1000); >>> } >>> >>> //ksession.dispose(); >>> >>> } catch (Throwable t) { >>> t.printStackTrace(); >>> } >>> } >>> >>> private static KnowledgeBase readKnowledgeBase() throws Exception { >>> >>> KnowledgeBuilder kbuilder = >>> KnowledgeBuilderFactory.newKnowledgeBuilder(); >>> kbuilder.add(ResourceFactory.newClassPathResource("Sample.drl"), >>> ResourceType.DRL); >>> KnowledgeBuilderErrors errors = kbuilder.getErrors(); >>> if (errors.size() > 0) { >>> for (KnowledgeBuilderError error: errors) { >>> System.err.println(error); >>> } >>> throw new IllegalArgumentException("Could not parse >>> knowledge."); >>> } >>> >>> final KnowledgeBaseConfiguration kbConfig = >>> KnowledgeBaseFactory.newKnowledgeBaseConfiguration(); >>> kbConfig.setOption(EventProcessingOption.STREAM); >>> KnowledgeBase kbase = >>> KnowledgeBaseFactory.newKnowledgeBase(kbConfig); >>> >>> kbase.addKnowledgePackages(kbuilder.getKnowledgePackages()); >>> return kbase; >>> } >>> >>> } >>> ================================== >>> >>> AnEvent.java: >>> ================================== >>> package org.drools.example; >>> >>> public class AnEvent { >>> >>> private Integer source; >>> >>> public AnEvent () { >>> } >>> >>> public Integer getSource() { >>> return source; >>> } >>> >>> public void setSource(Integer source) { >>> this.source = source; >>> } >>> >>> } >>> ================================== >>> >>> Sample.drl >>> ================================== >>> package org.drools.example >>> import org.drools.example.AnEvent; >>> >>> declare AnEvent >>> @role( event ) >>> end >>> >>> rule "Event print" >>> when >>> AnEvent ( $src: source ) over window:time(10s) from entry-point >>> TheEventStream >>> then >>> System.out.println("---------> Received AnEvent from " + $src); >>> end >>> ================================== >>> >>> Also tried using an immutable pojo as you suggested cusmaimatteo, with >>> >>> AnEvent.java: >>> ================================== >>> package org.drools.example; >>> >>> public class AnEvent { >>> >>> private final Integer source; >>> >>> public AnEvent (Integer i) { >>> this.source = i; >>> } >>> >>> public Integer getSource() { >>> return source; >>> } >>> } >>> ================================== >>> >>> and using instead using >>> >>> eventStream.insert(new AnEvent(randGen.nextInt())); >>> >>> but with the same memory issues noted. >>> >>> >>> >>> >>> >>> -- >>> View this message in context: >>> http://drools.46999.n3.nabble.com/Garbage-collection-and-sliding-windows-... >>> Sent from the Drools: User forum mailing list archive at Nabble.com. >>> _______________________________________________ >>> rules-users mailing list >>> rules-users(a)lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/rules-users >>> >> _______________________________________________ >> rules-users mailing list >> rules-users(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/rules-users > > > _______________________________________________ > rules-users mailing list > rules-users(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/rules-users >