2:25 p.m.
Hi everyone,
I'm currently doing some performance tests with Drools Fusion (5.5.0) and
have run into some issues with EventFactHandles which are not being garbage
collected. VisualVM tells me that for every 200k inserted events, about 15k
EventFactHandle objects are still left as live objects.
Current setup:
1) Drools 5.5.0 Final
2) Oracle JVM 1.7.0_06 on Windows 7
3) STREAM mode
4) 1 ksession at startup
5) 1 drl file with
- a single sliding window "over window:time(10s)"
- 1 entry-point
- declared event class as '@role( event )'
- only action being taken as events shows up is a println()
Events are being expired (and removed) properly from the 10 second sliding
window, which is verified by checking the
getFactCount() from the WorkingMemoryEntryPoint. Iterating through all
EventFactHandles also show that isExpired is false on all of them (and the
count is correct). But even though the events are removed from the sliding
window, they do not seem to be available to the GC.
If dispose() is called on the ksession, garbage collection is performed
properly. I get this same behavior with over "window:length" as well. And
even without using sliding windows at all unless I specifically add a
"@expires ( 1s )" to the event class declaration.
Has anyone else seen this behavior? After a couple of million events the
memory consumption becomes a pain and the only recovery action which seems
to be available is to dispose the ksession and start over. This means state
is lost and it requires the application to save and reinsert all facts to
the ksession, then start adding events again.
Any help on how to troubleshoot this further would be really appreciated.
--
View this message in context:
http://drools.46999.n3.nabble.com/Garbage-collection-and-sliding-windows-...
Sent from the Drools: User forum mailing list archive at Nabble.com.
6:13 p.m.
Are you sure that you don't have any rule based on time and on these
events? From my experience the @expire annotation doesn't work if the event
could be used from some rules.
On 9 Feb 2013 14:26, "tai-atari" <p00temkin(a)gmail.com> wrote:
Hi everyone,
I'm currently doing some performance tests with Drools Fusion (5.5.0) and
have run into some issues with EventFactHandles which are not being garbage
collected. VisualVM tells me that for every 200k inserted events, about 15k
EventFactHandle objects are still left as live objects.
Current setup:
1) Drools 5.5.0 Final
2) Oracle JVM 1.7.0_06 on Windows 7
3) STREAM mode
4) 1 ksession at startup
5) 1 drl file with
- a single sliding window "over window:time(10s)"
- 1 entry-point
- declared event class as '@role( event )'
- only action being taken as events shows up is a println()
Events are being expired (and removed) properly from the 10 second sliding
window, which is verified by checking the
getFactCount() from the WorkingMemoryEntryPoint. Iterating through all
EventFactHandles also show that isExpired is false on all of them (and the
count is correct). But even though the events are removed from the sliding
window, they do not seem to be available to the GC.
If dispose() is called on the ksession, garbage collection is performed
properly. I get this same behavior with over "window:length" as well. And
even without using sliding windows at all unless I specifically add a
"@expires ( 1s )" to the event class declaration.
Has anyone else seen this behavior? After a couple of million events the
memory consumption becomes a pain and the only recovery action which seems
to be available is to dispose the ksession and start over. This means state
is lost and it requires the application to save and reinsert all facts to
the ksession, then start adding events again.
Any help on how to troubleshoot this further would be really appreciated.
--
View this message in context:
http://drools.46999.n3.nabble.com/Garbage-collection-and-sliding-windows-...
Sent from the Drools: User forum mailing list archive at Nabble.com.
_______________________________________________
rules-users mailing list
rules-users(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/rules-users
9:29 p.m.
Hi cusmaimatteo and thanks for your quick reply.
The drl file loaded is really simple (see below), so there should not be any
additional rule dependencies.
<code>
package com.droolstest
import com.droolstest.AnEvent;
dialect "java"
declare AnEvent
@role( event )
end
rule "Event print"
when
AnEvent( $src : source) over window:time(10s) from entry-point
"TheEventStream"
then
System.out.println("---------> Received AnEvent from " + $src);
end
</code>
I even tried to marshall the ksession to disk as described in the Drools
Developers Cookbook. This also showed that the expired events are still
being written to the ksession.info file which just keeps growing over time.
Unmarshalling just puts me back into the same state.
For the above entry-point ("TheEventStream"), I verify the number of
remaining items using
<code>
WorkingMemoryEntryPoint eventStream = (WorkingMemoryEntryPoint)
ksession.getWorkingMemoryEntryPoint( "TheEventStream");
System.out.println("current event count " + ": " +
eventStream.getFactCount());
</code>
which shows that this event count is increasing/decreasing properly over
time. During bursts its value shows thousands of events, and after silence
for 10 seconds it goes down to 0. So everything seems fine unless I profile
the application and run it live with high throughput.
Am I missing some configuration option available for ksession to make
expired events available to the GC?
--
View this message in context:
http://drools.46999.n3.nabble.com/Garbage-collection-and-sliding-windows-...
Sent from the Drools: User forum mailing list archive at Nabble.com.
6:32 p.m.
We had the same problem - unless you call dispose (We tested with Drools
5.2.0), there was a big memory leak that we could not identify.
--
View this message in context:
http://drools.46999.n3.nabble.com/Garbage-collection-and-sliding-windows-...
Sent from the Drools: User forum mailing list archive at Nabble.com.
9:10 a.m.
Thanks for the info karlunho, since you experienced the same problem it may
be a generic Drools Fusion bug. Will make an update on any findings.
--
View this message in context:
http://drools.46999.n3.nabble.com/Garbage-collection-and-sliding-windows-...
Sent from the Drools: User forum mailing list archive at Nabble.com.
9:43 a.m.
I am using drools 5.4 now in a production system, and I haven't noticed
this issue.
Are you sure that the event a simple bean and it is immutable?
On 12 Feb 2013 09:10, "tai-atari" <p00temkin(a)gmail.com> wrote:
Thanks for the info karlunho, since you experienced the same problem
it may
be a generic Drools Fusion bug. Will make an update on any findings.
--
View this message in context:
http://drools.46999.n3.nabble.com/Garbage-collection-and-sliding-windows-...
Sent from the Drools: User forum mailing list archive at Nabble.com.
_______________________________________________
rules-users mailing list
rules-users(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/rules-users
3:50 p.m.
The original post mentioned "performance tests with Drools Fusion" in
combination with using the feature of automatic event retraction. I
don't think that OutOfMemory problems in this kind of scenario are due
to some bug in Drools (Fusion) - at least not until I see a complete
test app that reproduces the claimed problem.
It's important to note that automatic event retraction incurs a
considerable overhead in addition to the work required for getting the
event fact off the Rete
network. Essentially, a "Job" (i.e., the retraction) is scheduled for
punctual execution using a TimerService. One must therefore take care
that the system can cope with the additional work of disbanding all of
this contraband - otherwise it'll just clog, and croak.
Running fireUntilHalt() in one thread while inserting events from
another one and expecting automatic event retraction to happen after
10s (due to a window:time(10s)) is one way of risking that the system
runs into overload. With time slices being allotted equably to
threads, a backlog will accumulate if the expire/retract takes longer
than the insert/start-timer.
I've run the aforementioned scenario with System.sleep(nms) after
every so many insertions, and this keeps the system in balance.
Reducing nms gradually will - eventually - stress the system beyond
its breaking point, and OutOfMemory occurs. (To fill up memory more
quickly, the Event class was stuffed with some "dead data" -
char[2000].)
-W
1:27 a.m.
Hi laune,
Really appreciate you trying to reproduce the issue. Since you are not
experiencing the same behavior I'm obviously missing something. I've
attached a very simple example below which reproduces the scenario for the
Drools versions I have tried so far: 5.2, 5.4 and 5.5.
Basically this example runs an infinite loop which inserts 200 events about
every second, and keeps a sliding window of 10 seconds. Every loop will
print the current wm event count, which will increase to 2000 and stay put.
This is where I expected the expired events to be available for garbage
collection (since only 2000 are concurrently active in wm). But VisualVM
shows that org.drools.common.EventFactHandle uses an increasing amount of
memory over time.
Start.java:
==================================
package org.drools.example;
import java.util.Random;
import org.drools.KnowledgeBase;
import org.drools.KnowledgeBaseConfiguration;
import org.drools.KnowledgeBaseFactory;
import org.drools.builder.KnowledgeBuilder;
import org.drools.builder.KnowledgeBuilderError;
import org.drools.builder.KnowledgeBuilderErrors;
import org.drools.builder.KnowledgeBuilderFactory;
import org.drools.builder.ResourceType;
import org.drools.conf.EventProcessingOption;
import org.drools.io.ResourceFactory;
import org.drools.runtime.StatefulKnowledgeSession;
import org.drools.runtime.rule.WorkingMemoryEntryPoint;
public class Start {
public static final void main(String[] args) {
try {
System.out.println("Init");
KnowledgeBase kbase = readKnowledgeBase();
StatefulKnowledgeSession ksession =
kbase.newStatefulKnowledgeSession();
WorkingMemoryEntryPoint eventStream =
ksession.getWorkingMemoryEntryPoint("TheEventStream");
Random randGen = new Random();
while( true ) {
// Insert 200
int x = 0;
while( x < 200 ) {
AnEvent anEvent = new AnEvent();
anEvent.setSource(randGen.nextInt());
eventStream.insert(anEvent);
ksession.fireAllRules();
x++;
}
System.out.println("current event count in wm" + ": "
+
eventStream.getFactCount());
Thread.sleep(1000);
}
//ksession.dispose();
} catch (Throwable t) {
t.printStackTrace();
}
}
private static KnowledgeBase readKnowledgeBase() throws Exception {
KnowledgeBuilder kbuilder =
KnowledgeBuilderFactory.newKnowledgeBuilder();
kbuilder.add(ResourceFactory.newClassPathResource("Sample.drl"),
ResourceType.DRL);
KnowledgeBuilderErrors errors = kbuilder.getErrors();
if (errors.size() > 0) {
for (KnowledgeBuilderError error: errors) {
System.err.println(error);
}
throw new IllegalArgumentException("Could not parse
knowledge.");
}
final KnowledgeBaseConfiguration kbConfig =
KnowledgeBaseFactory.newKnowledgeBaseConfiguration();
kbConfig.setOption(EventProcessingOption.STREAM);
KnowledgeBase kbase = KnowledgeBaseFactory.newKnowledgeBase(kbConfig);
kbase.addKnowledgePackages(kbuilder.getKnowledgePackages());
return kbase;
}
}
==================================
AnEvent.java:
==================================
package org.drools.example;
public class AnEvent {
private Integer source;
public AnEvent () {
}
public Integer getSource() {
return source;
}
public void setSource(Integer source) {
this.source = source;
}
}
==================================
Sample.drl
==================================
package org.drools.example
import org.drools.example.AnEvent;
declare AnEvent
@role( event )
end
rule "Event print"
when
AnEvent ( $src: source ) over window:time(10s) from entry-point
TheEventStream
then
System.out.println("---------> Received AnEvent from " + $src);
end
==================================
Also tried using an immutable pojo as you suggested cusmaimatteo, with
AnEvent.java:
==================================
package org.drools.example;
public class AnEvent {
private final Integer source;
public AnEvent (Integer i) {
this.source = i;
}
public Integer getSource() {
return source;
}
}
==================================
and using instead using
eventStream.insert(new AnEvent(randGen.nextInt()));
but with the same memory issues noted.
--
View this message in context:
http://drools.46999.n3.nabble.com/Garbage-collection-and-sliding-windows-...
Sent from the Drools: User forum mailing list archive at Nabble.com.
8:19 a.m.
Could this be related to the issue at
https://issues.jboss.org/browse/JBRULES-3721? It started with performance
issues but eventually led to this case where a NPE is thrown by the
expire/retract action. They seem quite similar and perhaps the issue
attached project could help solve this issue as well.
laune wrote
The original post mentioned "performance tests with Drools
Fusion" in
combination with using the feature of automatic event retraction. I
don't think that OutOfMemory problems in this kind of scenario are due
to some bug in Drools (Fusion) - at least not until I see a complete
test app that reproduces the claimed problem.
It's important to note that automatic event retraction incurs a
considerable overhead in addition to the work required for getting the
event fact off the Rete
network. Essentially, a "Job" (i.e., the retraction) is scheduled for
punctual execution using a TimerService. One must therefore take care
that the system can cope with the additional work of disbanding all of
this contraband - otherwise it'll just clog, and croak.
Running fireUntilHalt() in one thread while inserting events from
another one and expecting automatic event retraction to happen after
10s (due to a window:time(10s)) is one way of risking that the system
runs into overload. With time slices being allotted equably to
threads, a backlog will accumulate if the expire/retract takes longer
than the insert/start-timer.
I've run the aforementioned scenario with System.sleep(nms) after
every so many insertions, and this keeps the system in balance.
Reducing nms gradually will - eventually - stress the system beyond
its breaking point, and OutOfMemory occurs. (To fill up memory more
quickly, the Event class was stuffed with some "dead data" -
char[2000].)
-W
_______________________________________________
rules-users mailing list
rules-users@.jboss
--
View this message in context:
http://drools.46999.n3.nabble.com/Garbage-collection-and-sliding-windows-...
Sent from the Drools: User forum mailing list archive at Nabble.com.
4075
days inactive
4087
days old
8 comments
5 participants
participants (5)
-
karlunho -
marianbuenosayres -
Matteo Cusmai -
tai-atari -
Wolfgang Laun