11:57 p.m.
This is great. It sounds like I have to go back to the drawing board. It
may take a while to work through this list. I'll circle back with outcomes.
Thank you.
5:11 a.m.
New subject: GC Overhead Limit Exceeded and 1B JoinLeftNode Objects
Ok, thanks. Here's what I have so far:
1) I am not retracting facts or setting expiration. This is by design
since I am doing the next item. Do I need to retract globals?
2) I have disposed the session when I ran with a StatefulSession. I
understood this means I do not need to retract facts. I do not attempt to
reuse the session. I am now trying to use a StatelessSession.
3) Unfortunately, the rule base is very large and this will take a long
time. I am hoping to at least get to a point where this runs end-to-end.
If it takes several hours, I am ok with that.
4) I would expect everything except "eval" statements to take advantage of
indexing in my rules. Are you talking about BetaNode indexing? In all
cases, I use the property access over getters.
5) Since I am using a stateless session, I would expect no recursion.
6) Got it. Thanks.
7) Does this only apply with a shared KnowledgeBase? What if I spawn
multiple sessions in separate threads?
All in all, I reduced the allocated heap size and ran jmap as recommended
by Wolfgang with a 6GB heap, 4 threads running sessions via a ForkJoinPool
and less than 100K facts. Here is a snapshot of the top hits. I ran this
multiple times, and just like JProfiler, the JoinNodeLeftTuple continue to
grow and grow. I would expect this to fluctuate up and down in count and
size since I am not re-using a session. Not being familiar with the
internals of Drools, I am hoping someone could provide a sense of whether
or not the below points to one of the issues Mark mentioned above. Also,
during GC events only Eden space gets freed up so these objects appear to
be living in Tenured space. This further concerns me that something is not
being cleaned up.
I fear it is all not so simple and will continue looking into Mark's list
for opportunities in my code base as well as work towards simple test case.
I appreciate all the time taken to read my rather lengthy emails. I am
hoping that this detail will help others as it has me. Thank you.
num #instances #bytes class name
----------------------------------------------
1: 61754567 4940365360 org.drools.reteoo.JoinNodeLeftTuple
2: 1954644 109460064 org.drools.reteoo.RightTuple
3: 302247 24179760 org.drools.common.AgendaItem
4: 447814 21495072 com.mycompany.loader.FactsLoader <- A
bunch of callable objects that get executed to load data to send to Drools
when the processor(s) is (are) idle.
5: 302247 19343808
org.drools.reteoo.RuleTerminalNodeLeftTuple
6: 897091 14353456 java.lang.Integer
7: 447814 14330048 java.util.RandomAccessSubList
8: 300383 11743152 [C
9: 447815 10747560
java.util.Collections$SynchronizedRandomAccessList
On Sat, Feb 23, 2013 at 5:57 PM, Julian Klein <julianklein(a)gmail.com> wrote:
This is great. It sounds like I have to go back to the drawing
board. It
may take a while to work through this list. I'll circle back with outcomes.
Thank you.
5:37 a.m.
New subject: GC Overhead Limit Exceeded and 1B JoinLeftNode Objects
On 24 Feb 2013, at 04:11, Julian Klein <julianklein(a)gmail.com> wrote:
Ok, thanks. Here's what I have so far:
1) I am not retracting facts or setting expiration. This is by design since I am doing
the next item. Do I need to retract globals?
2) I have disposed the session when I ran with a StatefulSession. I understood this
means I do not need to retract facts. I do not attempt to reuse the session. I am now
trying to use a StatelessSession.
it makes no difference. Stateless is just a
convenience wrapper for stateful, that avoids having to call "dispose" at the
end.
3) Unfortunately, the rule base is very large and this will take a
long time. I am hoping to at least get to a point where this runs end-to-end. If it
takes several hours, I am ok with that.
Don't let an application get this big,
before you start testing.
4) I would expect everything except "eval" statements to
take advantage of indexing in my rules. Are you talking about BetaNode indexing? In all
cases, I use the property access over getters.
no not everything is indexed. Only
'==' operators are indexed, and you must put the field name to the left of the
operator. Until recently anything with nested accessors would not be indexed either, but
that should have improved in 5.5
5) Since I am using a stateless session, I would expect no recursion.
Statless session is just a wrapper for stateful. If the consequences modify or
insert data, you get recursions.
6) Got it. Thanks.
7) Does this only apply with a shared KnowledgeBase? What if I spawn multiple sessions
in separate threads?
The thread lock is per session.
All in all, I reduced the allocated heap size and ran jmap as recommended by Wolfgang
with a 6GB heap, 4 threads running sessions via a ForkJoinPool and less than 100K facts.
Here is a snapshot of the top hits. I ran this multiple times, and just like JProfiler,
the JoinNodeLeftTuple continue to grow and grow. I would expect this to fluctuate up and
down in count and size since I am not re-using a session. Not being familiar with the
internals of Drools, I am hoping someone could provide a sense of whether or not the below
points to one of the issues Mark mentioned above. Also, during GC events only Eden space
gets freed up so these objects appear to be living in Tenured space. This further
concerns me that something is not being cleaned up.
I fear it is all not so simple and will continue looking into Mark's list for
opportunities in my code base as well as work towards simple test case. I appreciate all
the time taken to read my rather lengthy emails. I am hoping that this detail will help
others as it has me. Thank you.
num #instances #bytes class name
----------------------------------------------
1: 61754567 4940365360 org.drools.reteoo.JoinNodeLeftTuple
2: 1954644 109460064 org.drools.reteoo.RightTuple
3: 302247 24179760 org.drools.common.AgendaItem
4: 447814 21495072 com.mycompany.loader.FactsLoader <- A bunch of
callable objects that get executed to load data to send to Drools when the processor(s) is
(are) idle.
5: 302247 19343808 org.drools.reteoo.RuleTerminalNodeLeftTuple
6: 897091 14353456 java.lang.Integer
7: 447814 14330048 java.util.RandomAccessSubList
8: 300383 11743152 [C
9: 447815 10747560 java.util.Collections$SynchronizedRandomAccessList
On Sat, Feb 23, 2013 at 5:57 PM, Julian Klein <julianklein(a)gmail.com> wrote:
This is great. It sounds like I have to go back to the drawing board. It may take a
while to work through this list. I'll circle back with outcomes.
Thank you.
_______________________________________________
rules-users mailing list
rules-users(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/rules-users
7:12 a.m.
New subject: GC Overhead Limit Exceeded and 1B JoinLeftNode Objects
On 24/02/2013, Julian Klein <julianklein(a)gmail.com> wrote:
Ok, thanks. Here's what I have so far:
1) I am not retracting facts or setting expiration. This is by design
since I am doing the next item. Do I need to retract globals?
Be careful with globals when you run alike sessions in parallel - the
object(s) is (are) shared.
2) I have disposed the session when I ran with a StatefulSession. I
understood this means I do not need to retract facts. I do not attempt to
reuse the session. I am now trying to use a StatelessSession.
Don't change things when you try to narrow down one effect. A stateful
session lets you inspect things after fireAllRules has returned.
3) Unfortunately, the rule base is very large and this will take a
long
time. I am hoping to at least get to a point where this runs end-to-end.
If it takes several hours, I am ok with that.
4) I would expect everything except "eval" statements to take advantage of
indexing in my rules. Are you talking about BetaNode indexing? In all
cases, I use the property access over getters.
5) Since I am using a stateless session, I would expect no recursion.
6) Got it. Thanks.
7) Does this only apply with a shared KnowledgeBase? What if I spawn
multiple sessions in separate threads?
Don't do this when trying to find a memory leak.
Originally I got the impression you were running sessions one after
the other until running out of memory. Running jmap between sessions
will give you a clear indication of what is left over after dispose().
A jmap is useful to observe trends, and I don't think only
JoinNodeLeftTuple grows with the number of past sessions - don't just
look at the highrunner. Compare its results after 10, 100, 1000,...
single-threaded executions of identical sessions.
-W
All in all, I reduced the allocated heap size and ran jmap as recommended
by Wolfgang with a 6GB heap, 4 threads running sessions via a ForkJoinPool
and less than 100K facts. Here is a snapshot of the top hits. I ran this
multiple times, and just like JProfiler, the JoinNodeLeftTuple continue to
grow and grow. I would expect this to fluctuate up and down in count and
size since I am not re-using a session. Not being familiar with the
internals of Drools, I am hoping someone could provide a sense of whether
or not the below points to one of the issues Mark mentioned above. Also,
during GC events only Eden space gets freed up so these objects appear to
be living in Tenured space. This further concerns me that something is not
being cleaned up.
I fear it is all not so simple and will continue looking into Mark's list
for opportunities in my code base as well as work towards simple test case.
I appreciate all the time taken to read my rather lengthy emails. I am
hoping that this detail will help others as it has me. Thank you.
num #instances #bytes class name
----------------------------------------------
1: 61754567 4940365360 org.drools.reteoo.JoinNodeLeftTuple
2: 1954644 109460064 org.drools.reteoo.RightTuple
3: 302247 24179760 org.drools.common.AgendaItem
4: 447814 21495072 com.mycompany.loader.FactsLoader <- A
bunch of callable objects that get executed to load data to send to Drools
when the processor(s) is (are) idle.
5: 302247 19343808
org.drools.reteoo.RuleTerminalNodeLeftTuple
6: 897091 14353456 java.lang.Integer
7: 447814 14330048 java.util.RandomAccessSubList
8: 300383 11743152 [C
9: 447815 10747560
java.util.Collections$SynchronizedRandomAccessList
On Sat, Feb 23, 2013 at 5:57 PM, Julian Klein <julianklein(a)gmail.com>
wrote:
> This is great. It sounds like I have to go back to the drawing board.
> It
> may take a while to work through this list. I'll circle back with
> outcomes.
>
> Thank you.
>
7:25 a.m.
New subject: GC Overhead Limit Exceeded and 1B JoinLeftNode Objects
You also need to give us some indication of what your rules look like. Are you using just
patterns, what conditional elements are you using, are you using any temporal operators,
scheduling, tms?
Mark
On 24 Feb 2013, at 06:12, Wolfgang Laun <wolfgang.laun(a)gmail.com> wrote:
On 24/02/2013, Julian Klein <julianklein(a)gmail.com> wrote:
> Ok, thanks. Here's what I have so far:
>
> 1) I am not retracting facts or setting expiration. This is by design
> since I am doing the next item. Do I need to retract globals?
Be careful with globals when you run alike sessions in parallel - the
object(s) is (are) shared.
> 2) I have disposed the session when I ran with a StatefulSession. I
> understood this means I do not need to retract facts. I do not attempt to
> reuse the session. I am now trying to use a StatelessSession.
Don't change things when you try to narrow down one effect. A stateful
session lets you inspect things after fireAllRules has returned.
> 3) Unfortunately, the rule base is very large and this will take a long
> time. I am hoping to at least get to a point where this runs end-to-end.
> If it takes several hours, I am ok with that.
> 4) I would expect everything except "eval" statements to take advantage of
> indexing in my rules. Are you talking about BetaNode indexing? In all
> cases, I use the property access over getters.
> 5) Since I am using a stateless session, I would expect no recursion.
> 6) Got it. Thanks.
> 7) Does this only apply with a shared KnowledgeBase? What if I spawn
> multiple sessions in separate threads?
Don't do this when trying to find a memory leak.
Originally I got the impression you were running sessions one after
the other until running out of memory. Running jmap between sessions
will give you a clear indication of what is left over after dispose().
A jmap is useful to observe trends, and I don't think only
JoinNodeLeftTuple grows with the number of past sessions - don't just
look at the highrunner. Compare its results after 10, 100, 1000,...
single-threaded executions of identical sessions.
-W
>
> All in all, I reduced the allocated heap size and ran jmap as recommended
> by Wolfgang with a 6GB heap, 4 threads running sessions via a ForkJoinPool
> and less than 100K facts. Here is a snapshot of the top hits. I ran this
> multiple times, and just like JProfiler, the JoinNodeLeftTuple continue to
> grow and grow. I would expect this to fluctuate up and down in count and
> size since I am not re-using a session. Not being familiar with the
> internals of Drools, I am hoping someone could provide a sense of whether
> or not the below points to one of the issues Mark mentioned above. Also,
> during GC events only Eden space gets freed up so these objects appear to
> be living in Tenured space. This further concerns me that something is not
> being cleaned up.
>
> I fear it is all not so simple and will continue looking into Mark's list
> for opportunities in my code base as well as work towards simple test case.
> I appreciate all the time taken to read my rather lengthy emails. I am
> hoping that this detail will help others as it has me. Thank you.
>
> num #instances #bytes class name
> ----------------------------------------------
> 1: 61754567 4940365360 org.drools.reteoo.JoinNodeLeftTuple
> 2: 1954644 109460064 org.drools.reteoo.RightTuple
> 3: 302247 24179760 org.drools.common.AgendaItem
> 4: 447814 21495072 com.mycompany.loader.FactsLoader <- A
> bunch of callable objects that get executed to load data to send to Drools
> when the processor(s) is (are) idle.
> 5: 302247 19343808
> org.drools.reteoo.RuleTerminalNodeLeftTuple
> 6: 897091 14353456 java.lang.Integer
> 7: 447814 14330048 java.util.RandomAccessSubList
> 8: 300383 11743152 [C
> 9: 447815 10747560
> java.util.Collections$SynchronizedRandomAccessList
>
>
>
> On Sat, Feb 23, 2013 at 5:57 PM, Julian Klein <julianklein(a)gmail.com>
> wrote:
>
>> This is great. It sounds like I have to go back to the drawing board.
>> It
>> may take a while to work through this list. I'll circle back with
>> outcomes.
>>
>> Thank you.
>>
>
_______________________________________________
rules-users mailing list
rules-users(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/rules-users
7:07 p.m.
New subject: GC Overhead Limit Exceeded and 1B JoinLeftNode Objects
Thanks for all the advice once again. I understand their is a distinct
possibility that my rules are highly inefficient and therefore leading to a
large memory requirement. I will keep pairing this down to one thread and
one use case.
So here are two example rules I have. This demonstrates the approach I am
using (pattern matching, temporal reasoning, regex matches, etc). You'll
notice that rule 001n only fires when 001d exists. It is operating on
a separate agenda as well. I am not sure that makes sense, but it will
come down to whether a Stateless with Sequential mode is better for my
scenarios than a Stateful session with one agenda (or two with the agenda
being switched after the first call to fireAllRules). For
reference, durationCycleYear and utils are globals. One is a I can see at
the least that I need to move some variables to the right hand side of the
== statements. Most of the rules are similar in their approach.
rule "001 [d]"
@id("001d")
agenda-group 'd-rules-agenda'
when
$sv1 : SiteVisit( yearRecorded == durationCycleYear, !annualVisit )
FaultCode( $sv1.ID==svID, code matches "366.\\d+|743.3\\d?" )
$inspector: Insepector (ID == $sv1.insepectortID)
$sv2 : SitVisit( yearRecorded == durationCycleYear, !annualVisit )
#make sure we are dealing with the same inspector
Inspector (ID == $sv2.inspectorID, EUID == $inspector.EUID)
( FaultCode( $sv2.ID == svID, code matches "45.61" ) or
ServiceCode($sv2.ID == svID, code matches
"66(8[4-9][0-9]|9([0-3][0-9]|40))|66982|66984|66983" ) )
then
insert(utils.saveAndReturnEvent(kcontext, $sv2));
end
rule "001 [n]"
@id("001n")
agenda-group 'n-rules-agenda'
when
$event : Event( externalID == "001d")
$encD : SiteVisit($event.svID == ID)
$inspector : Inspector (ID == $svD.inspectorID)
$svN : SiteVisit( datetimeRecorded after[0ms, 90d]
$svD.datetimeRecorded, !annualVisit )
#make sure we are dealing with the same inspector
Inspector (ID == $svN.inspectorID, EUID == $inspector.EUID)
FaultCode($svN.ID == svID, code matches
"361.\\d+|362.4[23]|371.\\d+|360.0\\d?|360.1|362.53|998.82|998.9" )
then
insert(utils.saveAndReturnEvent(kcontext, $svN));
end
On Sun, Feb 24, 2013 at 1:25 AM, Mark Proctor <mproctor(a)codehaus.org> wrote:
You also need to give us some indication of what your rules look
like. Are
you using just patterns, what conditional elements are you using, are you
using any temporal operators, scheduling, tms?
Mark
On 24 Feb 2013, at 06:12, Wolfgang Laun <wolfgang.laun(a)gmail.com> wrote:
> On 24/02/2013, Julian Klein <julianklein(a)gmail.com> wrote:
>> Ok, thanks. Here's what I have so far:
>>
>> 1) I am not retracting facts or setting expiration. This is by design
>> since I am doing the next item. Do I need to retract globals?
>
> Be careful with globals when you run alike sessions in parallel - the
> object(s) is (are) shared.
>
>
>> 2) I have disposed the session when I ran with a StatefulSession. I
>> understood this means I do not need to retract facts. I do not attempt
to
>> reuse the session. I am now trying to use a StatelessSession.
>
> Don't change things when you try to narrow down one effect. A stateful
> session lets you inspect things after fireAllRules has returned.
>
>
>> 3) Unfortunately, the rule base is very large and this will take a long
>> time. I am hoping to at least get to a point where this runs
end-to-end.
>> If it takes several hours, I am ok with that.
>> 4) I would expect everything except "eval" statements to take
advantage
of
>> indexing in my rules. Are you talking about BetaNode indexing? In all
>> cases, I use the property access over getters.
>> 5) Since I am using a stateless session, I would expect no recursion.
>> 6) Got it. Thanks.
>> 7) Does this only apply with a shared KnowledgeBase? What if I spawn
>> multiple sessions in separate threads?
>
> Don't do this when trying to find a memory leak.
>
> Originally I got the impression you were running sessions one after
> the other until running out of memory. Running jmap between sessions
> will give you a clear indication of what is left over after dispose().
> A jmap is useful to observe trends, and I don't think only
> JoinNodeLeftTuple grows with the number of past sessions - don't just
> look at the highrunner. Compare its results after 10, 100, 1000,...
> single-threaded executions of identical sessions.
>
> -W
>
>>
>> All in all, I reduced the allocated heap size and ran jmap as
recommended
>> by Wolfgang with a 6GB heap, 4 threads running sessions via a
ForkJoinPool
>> and less than 100K facts. Here is a snapshot of the top hits. I ran
this
>> multiple times, and just like JProfiler, the JoinNodeLeftTuple continue
to
>> grow and grow. I would expect this to fluctuate up and down in count
and
>> size since I am not re-using a session. Not being familiar with the
>> internals of Drools, I am hoping someone could provide a sense of
whether
>> or not the below points to one of the issues Mark mentioned above.
Also,
>> during GC events only Eden space gets freed up so these objects appear
to
>> be living in Tenured space. This further concerns me that something is
not
>> being cleaned up.
>>
>> I fear it is all not so simple and will continue looking into Mark's
list
>> for opportunities in my code base as well as work towards simple test
case.
>> I appreciate all the time taken to read my rather lengthy emails. I am
>> hoping that this detail will help others as it has me. Thank you.
>>
>> num #instances #bytes class name
>> ----------------------------------------------
>> 1: 61754567 4940365360 org.drools.reteoo.JoinNodeLeftTuple
>> 2: 1954644 109460064 org.drools.reteoo.RightTuple
>> 3: 302247 24179760 org.drools.common.AgendaItem
>> 4: 447814 21495072 com.mycompany.loader.FactsLoader <-
A
>> bunch of callable objects that get executed to load data to send to
Drools
>> when the processor(s) is (are) idle.
>> 5: 302247 19343808
>> org.drools.reteoo.RuleTerminalNodeLeftTuple
>> 6: 897091 14353456 java.lang.Integer
>> 7: 447814 14330048 java.util.RandomAccessSubList
>> 8: 300383 11743152 [C
>> 9: 447815 10747560
>> java.util.Collections$SynchronizedRandomAccessList
>>
>>
>>
>> On Sat, Feb 23, 2013 at 5:57 PM, Julian Klein <julianklein(a)gmail.com>
>> wrote:
>>
>>> This is great. It sounds like I have to go back to the drawing board.
>>> It
>>> may take a while to work through this list. I'll circle back with
>>> outcomes.
>>>
>>> Thank you.
>>>
>>
> _______________________________________________
> rules-users mailing list
> rules-users(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/rules-users
_______________________________________________
rules-users mailing list
rules-users(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/rules-users
7:50 p.m.
New subject: GC Overhead Limit Exceeded and 1B JoinLeftNode Objects
Below are some comments on rules "001 [d]" and "001[n]" . I have
silently ignored all the obvious spelling errors.
On 24/02/2013, Julian Klein <julianklein(a)gmail.com> wrote:
Thanks for all the advice once again. I understand their is a
distinct
possibility that my rules are highly inefficient and therefore leading to a
large memory requirement. I will keep pairing this down to one thread and
one use case.
So here are two example rules I have. This demonstrates the approach I am
using (pattern matching, temporal reasoning, regex matches, etc). You'll
notice that rule 001n only fires when 001d exists. It is operating on
a separate agenda as well. I am not sure that makes sense, but it will
come down to whether a Stateless with Sequential mode is better for my
scenarios than a Stateful session with one agenda (or two with the agenda
being switched after the first call to fireAllRules). For
reference, durationCycleYear and utils are globals. One is a I can see at
the least that I need to move some variables to the right hand side of the
== statements. Most of the rules are similar in their approach.
rule "001 [d]"
@id("001d")
agenda-group 'd-rules-agenda'
when
$sv1 : SiteVisit( yearRecorded == durationCycleYear, !annualVisit )
Hopefully durationCycleYear is a global and constant?
FaultCode( $sv1.ID==svID, code matches
"366.\\d+|743.3\\d?" )
In these patterns, is '.' supposed to match any character? A literal
'.' would have to be escaped.
$inspector: Insepector (ID == $sv1.insepectortID)
$sv2 : SitVisit( yearRecorded == durationCycleYear, !annualVisit )
Bad. This matches any other SiteVisit according to these constraints,
but it also matches the
same one as already matched $sv1. Moreover, if there is a pair of two
matching SiteVisits
(SV1, SV2) there will be 4 activations with SV1+SV2, SV2+SV1, SV1+SV1, SV2+SV2.
#make sure we are dealing with the same inspector
Inspector (ID == $sv2.inspectorID, EUID == $inspector.EUID)
This constraint should be added in the second SiteVisit pattern.
There's really no need to re-bind the same Inspector - we already have
it bound to $inspector.
( FaultCode( $sv2.ID == svID, code matches "45.61" ) or
ServiceCode($sv2.ID == svID, code matches
"66(8[4-9][0-9]|9([0-3][0-9]|40))|66982|66984|66983" ) )
then
insert(utils.saveAndReturnEvent(kcontext, $sv2));
(Let's hope that the returned Event is a "slim" Pojo and doesn't try
to conserve tons of data from kcontext.)
rule "001 [n]"
@id("001n")
agenda-group 'n-rules-agenda'
when
$event : Event( externalID == "001d")
$encD : SiteVisit($event.svID == ID)
$inspector : Inspector (ID == $svD.inspectorID)
$svN : SiteVisit( datetimeRecorded after[0ms, 90d]
$svD.datetimeRecorded, !annualVisit )
Note that A after[0ms,...] B does not enforce that A and B are
different. See the previous comment starting with "Bad.".
#make sure we are dealing with the same inspector
Inspector (ID == $svN.inspectorID, EUID == $inspector.EUID)
See the corresponding comment re the 1st rule.
FaultCode($svN.ID == svID, code matches
"361.\\d+|362.4[23]|371.\\d+|360.0\\d?|360.1|362.53|998.82|998.9" )
Again: '.'?
then
insert(utils.saveAndReturnEvent(kcontext, $svN));
end
Doesn't look too good, IMHO. A thorough revision of your rule base
appears indicated.
-W
On Sun, Feb 24, 2013 at 1:25 AM, Mark Proctor <mproctor(a)codehaus.org>
wrote:
> You also need to give us some indication of what your rules look like.
> Are
> you using just patterns, what conditional elements are you using, are you
> using any temporal operators, scheduling, tms?
>
> Mark
> On 24 Feb 2013, at 06:12, Wolfgang Laun <wolfgang.laun(a)gmail.com> wrote:
>
> > On 24/02/2013, Julian Klein <julianklein(a)gmail.com> wrote:
> >> Ok, thanks. Here's what I have so far:
> >>
> >> 1) I am not retracting facts or setting expiration. This is by design
> >> since I am doing the next item. Do I need to retract globals?
> >
> > Be careful with globals when you run alike sessions in parallel - the
> > object(s) is (are) shared.
> >
> >
> >> 2) I have disposed the session when I ran with a StatefulSession. I
> >> understood this means I do not need to retract facts. I do not
> >> attempt
> to
> >> reuse the session. I am now trying to use a StatelessSession.
> >
> > Don't change things when you try to narrow down one effect. A stateful
> > session lets you inspect things after fireAllRules has returned.
> >
> >
> >> 3) Unfortunately, the rule base is very large and this will take a
> >> long
> >> time. I am hoping to at least get to a point where this runs
> end-to-end.
> >> If it takes several hours, I am ok with that.
> >> 4) I would expect everything except "eval" statements to take
> >> advantage
> of
> >> indexing in my rules. Are you talking about BetaNode indexing? In
> >> all
> >> cases, I use the property access over getters.
> >> 5) Since I am using a stateless session, I would expect no recursion.
> >> 6) Got it. Thanks.
> >> 7) Does this only apply with a shared KnowledgeBase? What if I spawn
> >> multiple sessions in separate threads?
> >
> > Don't do this when trying to find a memory leak.
> >
> > Originally I got the impression you were running sessions one after
> > the other until running out of memory. Running jmap between sessions
> > will give you a clear indication of what is left over after dispose().
> > A jmap is useful to observe trends, and I don't think only
> > JoinNodeLeftTuple grows with the number of past sessions - don't just
> > look at the highrunner. Compare its results after 10, 100, 1000,...
> > single-threaded executions of identical sessions.
> >
> > -W
> >
> >>
> >> All in all, I reduced the allocated heap size and ran jmap as
> recommended
> >> by Wolfgang with a 6GB heap, 4 threads running sessions via a
> ForkJoinPool
> >> and less than 100K facts. Here is a snapshot of the top hits. I ran
> this
> >> multiple times, and just like JProfiler, the JoinNodeLeftTuple
> >> continue
> to
> >> grow and grow. I would expect this to fluctuate up and down in count
> and
> >> size since I am not re-using a session. Not being familiar with the
> >> internals of Drools, I am hoping someone could provide a sense of
> whether
> >> or not the below points to one of the issues Mark mentioned above.
> Also,
> >> during GC events only Eden space gets freed up so these objects appear
> to
> >> be living in Tenured space. This further concerns me that something
> >> is
> not
> >> being cleaned up.
> >>
> >> I fear it is all not so simple and will continue looking into Mark's
> list
> >> for opportunities in my code base as well as work towards simple test
> case.
> >> I appreciate all the time taken to read my rather lengthy emails. I
> >> am
> >> hoping that this detail will help others as it has me. Thank you.
> >>
> >> num #instances #bytes class name
> >> ----------------------------------------------
> >> 1: 61754567 4940365360 org.drools.reteoo.JoinNodeLeftTuple
> >> 2: 1954644 109460064 org.drools.reteoo.RightTuple
> >> 3: 302247 24179760 org.drools.common.AgendaItem
> >> 4: 447814 21495072 com.mycompany.loader.FactsLoader
> >> <-
> A
> >> bunch of callable objects that get executed to load data to send to
> Drools
> >> when the processor(s) is (are) idle.
> >> 5: 302247 19343808
> >> org.drools.reteoo.RuleTerminalNodeLeftTuple
> >> 6: 897091 14353456 java.lang.Integer
> >> 7: 447814 14330048 java.util.RandomAccessSubList
> >> 8: 300383 11743152 [C
> >> 9: 447815 10747560
> >> java.util.Collections$SynchronizedRandomAccessList
> >>
> >>
> >>
> >> On Sat, Feb 23, 2013 at 5:57 PM, Julian Klein <julianklein(a)gmail.com>
> >> wrote:
> >>
> >>> This is great. It sounds like I have to go back to the drawing
> >>> board.
> >>> It
> >>> may take a while to work through this list. I'll circle back with
> >>> outcomes.
> >>>
> >>> Thank you.
> >>>
> >>
> > _______________________________________________
> > rules-users mailing list
> > rules-users(a)lists.jboss.org
> > https://lists.jboss.org/mailman/listinfo/rules-users
>
>
> _______________________________________________
> rules-users mailing list
> rules-users(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/rules-users
>
8:55 p.m.
New subject: GC Overhead Limit Exceeded and 1B JoinLeftNode Objects
Sorry for the typos. I rewrote the domain to save me some trouble.
1) durationCycleYear is a constant globals value
2) I see what you mean regarding the literal "." in the regular
expressions. It should be a literal ("366\.\\d+|743\.3\\d?) . I'll see
about reducing activations and unneeded optionals in these patterns.
3) Regarding the four activations with SiteVisits. I believe this is
desired as the second site visit may or may not be the same visit as the
first. We want to capture both scenarios. This is also potentially a
complex data cleansing issue. I don't have many options here AFAICT.
4) Finding the same inspector, good point. This a data cleansing issue.
I'll think about options upstream.
5) The return event that inspects the kcontext, is a slim one and does not
have a reference to kcontext in it. It only has about five, non-composite
fields.
Thank you so very much!
7:58 a.m.
New subject: GC Overhead Limit Exceeded and 1B JoinLeftNode Objects
On 24/02/2013, Julian Klein <julianklein(a)gmail.com> wrote:
2) I see what you mean regarding the literal "." in the
regular
expressions. It should be a literal ("366\.\\d+|743\.3\\d?) .
"366\\.\\d+|743\\.\\d?"
3) Regarding the four activations with SiteVisits. I believe this
is
desired as the second site visit may or may not be the same visit as the
first.
[Hmm, words like "first" and "second" imply two distinct events, so
they can't
be "the same". You may have a mess in your data so that there could be two
reports (!) of the *same* visit.]
We want to capture both scenarios. This is also potentially a
complex data cleansing issue. I don't have many options here AFAICT.
I'd say it's a "get-your-specs-cleansed" issue. Consider the rule
(after eliminating the useless binding of Inspector and some
simplifications):
when
$sv1 : SiteVisit( yearRecorded == durationCycleYear, !annualVisit )
FaultCode( svID == $sv1.ID, code matches "366.12" )
$sv2 : SiteVisit( yearRecorded == durationCycleYear, !annualVisit,
inspectorID == $sv1.inspectorID ) // ensure
same Inspector
FaultCode( svID == $sv2.ID, code matches "45.61" )
then
Now, not knowing how SiteVisit and FaultCode can be related, I can
only guess that possible data could be:
(1)
SiteVisit( 2013, "13/42", false ) // year, ID, annualVisit
FaultCode( 366.12, "13/42" )
SiteVisit( 2013, "13/55", false )
FaultCode( 45.61, "13/55" )
Two distinct visits, fires once.
(2)
SiteVisit( 2013, "13/42", false )
FaultCode( 366.12, "13/42" )
FaultCode( 45.61, "13/42" )
There is only one visit, but fires once nonetheless.
(3)
SiteVisit( 2013, "13/42", false )
FaultCode( 366.12, "13/42" )
FaultCode( 45.61, "13/42" )
SiteVisit( 2013, "13/55", false )
FaultCode( 366.12, "13/55" )
FaultCode( 45.61, "13/55" )
Two visits, fires twice.
It's possible that multiple FaultCodes per SiteVisit aren't possible,
but the data model (AFAIK) suggests otherwise, and if this is so, you
should be prepared for every possible situation. I can't imagine
handling all three situations adequately with this single rule.
4) Finding the same inspector, good point. This a data cleansing
issue.
I'll think about options upstream.
No, it is definitely not "data cleansing" - it is an issue of writing
rules well. (Sorry for the harsh wording.)
-W
9:50 p.m.
New subject: GC Overhead Limit Exceeded and 1B JoinLeftNode Objects
On 24 Feb 2013, at 18:07, Julian Klein <julianklein(a)gmail.com> wrote:
Thanks for all the advice once again. I understand their is a
distinct possibility that my rules are highly inefficient and therefore leading to a large
memory requirement. I will keep pairing this down to one thread and one use case.
So here are two example rules I have. This demonstrates the approach I am using (pattern
matching, temporal reasoning, regex matches, etc). You'll notice that rule 001n only
fires when 001d exists. It is operating on a separate agenda as well. I am not sure that
makes sense, but it will come down to whether a Stateless with Sequential mode is better
for my scenarios than a Stateful session with one agenda (or two with the agenda being
switched after the first call to fireAllRules). For reference, durationCycleYear and
utils are globals. One is a I can see at the least that I need to move some variables to
the right hand side of the == statements. Most of the rules are similar in their
approach.
rule "001 [d]"
@id("001d")
agenda-group 'd-rules-agenda'
when
$sv1 : SiteVisit( yearRecorded == durationCycleYear, !annualVisit )
FaultCode( $sv1.ID==svID, code matches "366.\\d+|743.3\\d?" )
This
is wrong if you want indexing. It'll work, it just doesn't get picked up by the
indexer:
FaultCode( svID == $sv1.ID)
$inspector: Insepector (ID == $sv1.insepectortID)
$sv2 : SitVisit( yearRecorded == durationCycleYear, !annualVisit )
#make sure we are dealing with the same inspector
Inspector (ID == $sv2.inspectorID, EUID == $inspector.EUID)
( FaultCode( $sv2.ID == svID, code matches "45.61" ) or
ServiceCode($sv2.ID == svID, code matches
"66(8[4-9][0-9]|9([0-3][0-9]|40))|66982|66984|66983" ) )
same here
Are you sure you want 'or' and not ||. The 'or' CE can result in two rules
firing, if they are not mutually exclusive logical conditions.
then
insert(utils.saveAndReturnEvent(kcontext, $sv2));
end
rule "001 [n]"
@id("001n")
agenda-group 'n-rules-agenda'
when
$event : Event( externalID == "001d")
$encD : SiteVisit($event.svID == ID)
$inspector : Inspector (ID == $svD.inspectorID)
$svN : SiteVisit( datetimeRecorded after[0ms, 90d] $svD.datetimeRecorded,
!annualVisit )
#make sure we are dealing with the same inspector
Inspector (ID == $svN.inspectorID, EUID == $inspector.EUID)
FaultCode($svN.ID == svID, code matches
"361.\\d+|362.4[23]|371.\\d+|360.0\\d?|360.1|362.53|998.82|998.9" )
then
insert(utils.saveAndReturnEvent(kcontext, $svN));
end
On Sun, Feb 24, 2013 at 1:25 AM, Mark Proctor <mproctor(a)codehaus.org> wrote:
You also need to give us some indication of what your rules look like. Are you using just
patterns, what conditional elements are you using, are you using any temporal operators,
scheduling, tms?
Mark
On 24 Feb 2013, at 06:12, Wolfgang Laun <wolfgang.laun(a)gmail.com> wrote:
> On 24/02/2013, Julian Klein <julianklein(a)gmail.com> wrote:
>> Ok, thanks. Here's what I have so far:
>>
>> 1) I am not retracting facts or setting expiration. This is by design
>> since I am doing the next item. Do I need to retract globals?
>
> Be careful with globals when you run alike sessions in parallel - the
> object(s) is (are) shared.
>
>
>> 2) I have disposed the session when I ran with a StatefulSession. I
>> understood this means I do not need to retract facts. I do not attempt to
>> reuse the session. I am now trying to use a StatelessSession.
>
> Don't change things when you try to narrow down one effect. A stateful
> session lets you inspect things after fireAllRules has returned.
>
>
>> 3) Unfortunately, the rule base is very large and this will take a long
>> time. I am hoping to at least get to a point where this runs end-to-end.
>> If it takes several hours, I am ok with that.
>> 4) I would expect everything except "eval" statements to take
advantage of
>> indexing in my rules. Are you talking about BetaNode indexing? In all
>> cases, I use the property access over getters.
>> 5) Since I am using a stateless session, I would expect no recursion.
>> 6) Got it. Thanks.
>> 7) Does this only apply with a shared KnowledgeBase? What if I spawn
>> multiple sessions in separate threads?
>
> Don't do this when trying to find a memory leak.
>
> Originally I got the impression you were running sessions one after
> the other until running out of memory. Running jmap between sessions
> will give you a clear indication of what is left over after dispose().
> A jmap is useful to observe trends, and I don't think only
> JoinNodeLeftTuple grows with the number of past sessions - don't just
> look at the highrunner. Compare its results after 10, 100, 1000,...
> single-threaded executions of identical sessions.
>
> -W
>
>>
>> All in all, I reduced the allocated heap size and ran jmap as recommended
>> by Wolfgang with a 6GB heap, 4 threads running sessions via a ForkJoinPool
>> and less than 100K facts. Here is a snapshot of the top hits. I ran this
>> multiple times, and just like JProfiler, the JoinNodeLeftTuple continue to
>> grow and grow. I would expect this to fluctuate up and down in count and
>> size since I am not re-using a session. Not being familiar with the
>> internals of Drools, I am hoping someone could provide a sense of whether
>> or not the below points to one of the issues Mark mentioned above. Also,
>> during GC events only Eden space gets freed up so these objects appear to
>> be living in Tenured space. This further concerns me that something is not
>> being cleaned up.
>>
>> I fear it is all not so simple and will continue looking into Mark's list
>> for opportunities in my code base as well as work towards simple test case.
>> I appreciate all the time taken to read my rather lengthy emails. I am
>> hoping that this detail will help others as it has me. Thank you.
>>
>> num #instances #bytes class name
>> ----------------------------------------------
>> 1: 61754567 4940365360 org.drools.reteoo.JoinNodeLeftTuple
>> 2: 1954644 109460064 org.drools.reteoo.RightTuple
>> 3: 302247 24179760 org.drools.common.AgendaItem
>> 4: 447814 21495072 com.mycompany.loader.FactsLoader <- A
>> bunch of callable objects that get executed to load data to send to Drools
>> when the processor(s) is (are) idle.
>> 5: 302247 19343808
>> org.drools.reteoo.RuleTerminalNodeLeftTuple
>> 6: 897091 14353456 java.lang.Integer
>> 7: 447814 14330048 java.util.RandomAccessSubList
>> 8: 300383 11743152 [C
>> 9: 447815 10747560
>> java.util.Collections$SynchronizedRandomAccessList
>>
>>
>>
>> On Sat, Feb 23, 2013 at 5:57 PM, Julian Klein <julianklein(a)gmail.com>
>> wrote:
>>
>>> This is great. It sounds like I have to go back to the drawing board.
>>> It
>>> may take a while to work through this list. I'll circle back with
>>> outcomes.
>>>
>>> Thank you.
>>>
>>
> _______________________________________________
> rules-users mailing list
> rules-users(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/rules-users
_______________________________________________
rules-users mailing list
rules-users(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/rules-users
_______________________________________________
rules-users mailing list
rules-users(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/rules-users
10:57 p.m.
New subject: GC Overhead Limit Exceeded and 1B JoinLeftNode Objects
As far as $sv.ID == svID, I understand and have changed that throughout my
rules. I've actually applied all your advice so far, thank you. I am
still reviewing the rule base as I continue to get OoM Exceptions. I also
have changed the structure of my model to reduce redundancy in the
Inspector facts. This has shortened my rules a bit.
In regards to "or" versus "||", the 5.5 docs state that "||"
is deprecated
for CEs. Therefore I chose ||. Also, if I match on either ServiceCode or
FaultCode, I should get a result. I am not sure how || or "or" makes a
difference here since the docs state they are interchangeable. Would it
result in a shortcut preventing both generated sub-rules from firing?
On Sun, Feb 24, 2013 at 3:50 PM, Mark Proctor <mproctor(a)codehaus.org> wrote:
On 24 Feb 2013, at 18:07, Julian Klein <julianklein(a)gmail.com> wrote:
Thanks for all the advice once again. I understand their is a distinct
possibility that my rules are highly inefficient and therefore leading to a
large memory requirement. I will keep pairing this down to one thread and
one use case.
So here are two example rules I have. This demonstrates the approach I am
using (pattern matching, temporal reasoning, regex matches, etc). You'll
notice that rule 001n only fires when 001d exists. It is operating on
a separate agenda as well. I am not sure that makes sense, but it will
come down to whether a Stateless with Sequential mode is better for my
scenarios than a Stateful session with one agenda (or two with the agenda
being switched after the first call to fireAllRules). For
reference, durationCycleYear and utils are globals. One is a I can see at
the least that I need to move some variables to the right hand side of the
== statements. Most of the rules are similar in their approach.
rule "001 [d]"
@id("001d")
agenda-group 'd-rules-agenda'
when
$sv1 : SiteVisit( yearRecorded == durationCycleYear, !annualVisit )
FaultCode( $sv1.ID==svID, code matches "366.\\d+|743.3\\d?" )
This is wrong if you want indexing. It'll work, it just doesn't get picked
up by the indexer:
FaultCode( svID == $sv1.ID)
$inspector: Insepector (ID == $sv1.insepectortID)
$sv2 : SitVisit( yearRecorded == durationCycleYear, !annualVisit )
#make sure we are dealing with the same inspector
Inspector (ID == $sv2.inspectorID, EUID == $inspector.EUID)
( FaultCode( $sv2.ID == svID, code matches "45.61" ) or
ServiceCode($sv2.ID == svID, code matches
"66(8[4-9][0-9]|9([0-3][0-9]|40))|66982|66984|66983" ) )
same here
Are you sure you want 'or' and not ||. The 'or' CE can result in two
rules
firing, if they are not mutually exclusive logical conditions.
then
insert(utils.saveAndReturnEvent(kcontext, $sv2));
end
rule "001 [n]"
@id("001n")
agenda-group 'n-rules-agenda'
when
$event : Event( externalID == "001d")
$encD : SiteVisit($event.svID == ID)
$inspector : Inspector (ID == $svD.inspectorID)
$svN : SiteVisit( datetimeRecorded after[0ms, 90d]
$svD.datetimeRecorded, !annualVisit )
#make sure we are dealing with the same inspector
Inspector (ID == $svN.inspectorID, EUID == $inspector.EUID)
FaultCode($svN.ID == svID, code matches
"361.\\d+|362.4[23]|371.\\d+|360.0\\d?|360.1|362.53|998.82|998.9" )
then
insert(utils.saveAndReturnEvent(kcontext, $svN));
end
On Sun, Feb 24, 2013 at 1:25 AM, Mark Proctor <mproctor(a)codehaus.org>wrote:
> You also need to give us some indication of what your rules look like.
> Are you using just patterns, what conditional elements are you using, are
> you using any temporal operators, scheduling, tms?
>
> Mark
> On 24 Feb 2013, at 06:12, Wolfgang Laun <wolfgang.laun(a)gmail.com> wrote:
>
> > On 24/02/2013, Julian Klein <julianklein(a)gmail.com> wrote:
> >> Ok, thanks. Here's what I have so far:
> >>
> >> 1) I am not retracting facts or setting expiration. This is by design
> >> since I am doing the next item. Do I need to retract globals?
> >
> > Be careful with globals when you run alike sessions in parallel - the
> > object(s) is (are) shared.
> >
> >
> >> 2) I have disposed the session when I ran with a StatefulSession. I
> >> understood this means I do not need to retract facts. I do not
> attempt to
> >> reuse the session. I am now trying to use a StatelessSession.
> >
> > Don't change things when you try to narrow down one effect. A stateful
> > session lets you inspect things after fireAllRules has returned.
> >
> >
> >> 3) Unfortunately, the rule base is very large and this will take a long
> >> time. I am hoping to at least get to a point where this runs
> end-to-end.
> >> If it takes several hours, I am ok with that.
> >> 4) I would expect everything except "eval" statements to take
> advantage of
> >> indexing in my rules. Are you talking about BetaNode indexing? In
> all
> >> cases, I use the property access over getters.
> >> 5) Since I am using a stateless session, I would expect no recursion.
> >> 6) Got it. Thanks.
> >> 7) Does this only apply with a shared KnowledgeBase? What if I spawn
> >> multiple sessions in separate threads?
> >
> > Don't do this when trying to find a memory leak.
> >
> > Originally I got the impression you were running sessions one after
> > the other until running out of memory. Running jmap between sessions
> > will give you a clear indication of what is left over after dispose().
> > A jmap is useful to observe trends, and I don't think only
> > JoinNodeLeftTuple grows with the number of past sessions - don't just
> > look at the highrunner. Compare its results after 10, 100, 1000,...
> > single-threaded executions of identical sessions.
> >
> > -W
> >
> >>
> >> All in all, I reduced the allocated heap size and ran jmap as
> recommended
> >> by Wolfgang with a 6GB heap, 4 threads running sessions via a
> ForkJoinPool
> >> and less than 100K facts. Here is a snapshot of the top hits. I ran
> this
> >> multiple times, and just like JProfiler, the JoinNodeLeftTuple
> continue to
> >> grow and grow. I would expect this to fluctuate up and down in count
> and
> >> size since I am not re-using a session. Not being familiar with the
> >> internals of Drools, I am hoping someone could provide a sense of
> whether
> >> or not the below points to one of the issues Mark mentioned above.
> Also,
> >> during GC events only Eden space gets freed up so these objects appear
> to
> >> be living in Tenured space. This further concerns me that something
> is not
> >> being cleaned up.
> >>
> >> I fear it is all not so simple and will continue looking into Mark's
> list
> >> for opportunities in my code base as well as work towards simple test
> case.
> >> I appreciate all the time taken to read my rather lengthy emails. I am
> >> hoping that this detail will help others as it has me. Thank you.
> >>
> >> num #instances #bytes class name
> >> ----------------------------------------------
> >> 1: 61754567 4940365360 org.drools.reteoo.JoinNodeLeftTuple
> >> 2: 1954644 109460064 org.drools.reteoo.RightTuple
> >> 3: 302247 24179760 org.drools.common.AgendaItem
> >> 4: 447814 21495072 com.mycompany.loader.FactsLoader
> <- A
> >> bunch of callable objects that get executed to load data to send to
> Drools
> >> when the processor(s) is (are) idle.
> >> 5: 302247 19343808
> >> org.drools.reteoo.RuleTerminalNodeLeftTuple
> >> 6: 897091 14353456 java.lang.Integer
> >> 7: 447814 14330048 java.util.RandomAccessSubList
> >> 8: 300383 11743152 [C
> >> 9: 447815 10747560
> >> java.util.Collections$SynchronizedRandomAccessList
> >>
> >>
> >>
> >> On Sat, Feb 23, 2013 at 5:57 PM, Julian Klein <julianklein(a)gmail.com>
> >> wrote:
> >>
> >>> This is great. It sounds like I have to go back to the drawing board.
> >>> It
> >>> may take a while to work through this list. I'll circle back with
> >>> outcomes.
> >>>
> >>> Thank you.
> >>>
> >>
> > _______________________________________________
> > rules-users mailing list
> > rules-users(a)lists.jboss.org
> > https://lists.jboss.org/mailman/listinfo/rules-users
>
>
> _______________________________________________
> rules-users mailing list
> rules-users(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/rules-users
>
_______________________________________________
rules-users mailing list
rules-users(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/rules-users
_______________________________________________
rules-users mailing list
rules-users(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/rules-users
1:48 a.m.
New subject: GC Overhead Limit Exceeded and 1B JoinLeftNode Objects
On 24 Feb 2013, at 21:57, Julian Klein <julianklein(a)gmail.com> wrote:
As far as $sv.ID == svID, I understand and have changed that
throughout my rules. I've actually applied all your advice so far, thank you. I am
still reviewing the rule base as I continue to get OoM Exceptions. I also have changed
the structure of my model to reduce redundancy in the Inspector facts. This has shortened
my rules a bit.
In regards to "or" versus "||", the 5.5 docs state that
"||" is deprecated for CEs. Therefore I chose ||.
'||' is
deprecated between CE's, so that it can be used for inside of expressions. Early
versions of drools did not have compound expressions, it only supported conditional
element or. Later we added compound expression. The behaviour of a CE 'or' and a
compound expression are different. the 'or' CE is removed and logical rewrite
rules applied to create rules for each possible outcome. The result is you can have
multiple rules firing.
Also, if I match on either ServiceCode or FaultCode, I should get a
result. I am not sure how || or "or" makes a difference here since the docs
state they are interchangeable. Would it result in a shortcut preventing both generated
sub-rules from firing?
The docs differentiate between 'or' CE and ||
compound expression.
MArk
On Sun, Feb 24, 2013 at 3:50 PM, Mark Proctor <mproctor(a)codehaus.org> wrote:
On 24 Feb 2013, at 18:07, Julian Klein <julianklein(a)gmail.com> wrote:
> Thanks for all the advice once again. I understand their is a distinct possibility
that my rules are highly inefficient and therefore leading to a large memory requirement.
I will keep pairing this down to one thread and one use case.
>
> So here are two example rules I have. This demonstrates the approach I am using
(pattern matching, temporal reasoning, regex matches, etc). You'll notice that rule
001n only fires when 001d exists. It is operating on a separate agenda as well. I am not
sure that makes sense, but it will come down to whether a Stateless with Sequential mode
is better for my scenarios than a Stateful session with one agenda (or two with the agenda
being switched after the first call to fireAllRules). For reference, durationCycleYear
and utils are globals. One is a I can see at the least that I need to move some variables
to the right hand side of the == statements. Most of the rules are similar in their
approach.
>
> rule "001 [d]"
> @id("001d")
> agenda-group 'd-rules-agenda'
>
> when
> $sv1 : SiteVisit( yearRecorded == durationCycleYear, !annualVisit )
> FaultCode( $sv1.ID==svID, code matches "366.\\d+|743.3\\d?" )
This is wrong if you want indexing. It'll work, it just doesn't get picked up by
the indexer:
FaultCode( svID == $sv1.ID)
> $inspector: Insepector (ID == $sv1.insepectortID)
>
> $sv2 : SitVisit( yearRecorded == durationCycleYear, !annualVisit )
> #make sure we are dealing with the same inspector
> Inspector (ID == $sv2.inspectorID, EUID == $inspector.EUID)
>
> ( FaultCode( $sv2.ID == svID, code matches "45.61" ) or
> ServiceCode($sv2.ID == svID, code matches
"66(8[4-9][0-9]|9([0-3][0-9]|40))|66982|66984|66983" ) )
>
same here
Are you sure you want 'or' and not ||. The 'or' CE can result in two
rules firing, if they are not mutually exclusive logical conditions.
>
> then
> insert(utils.saveAndReturnEvent(kcontext, $sv2));
> end
>
> rule "001 [n]"
> @id("001n")
> agenda-group 'n-rules-agenda'
> when
> $event : Event( externalID == "001d")
> $encD : SiteVisit($event.svID == ID)
> $inspector : Inspector (ID == $svD.inspectorID)
>
> $svN : SiteVisit( datetimeRecorded after[0ms, 90d] $svD.datetimeRecorded,
!annualVisit )
> #make sure we are dealing with the same inspector
> Inspector (ID == $svN.inspectorID, EUID == $inspector.EUID)
>
> FaultCode($svN.ID == svID, code matches
"361.\\d+|362.4[23]|371.\\d+|360.0\\d?|360.1|362.53|998.82|998.9" )
>
> then
> insert(utils.saveAndReturnEvent(kcontext, $svN));
> end
>
>
> On Sun, Feb 24, 2013 at 1:25 AM, Mark Proctor <mproctor(a)codehaus.org> wrote:
> You also need to give us some indication of what your rules look like. Are you using
just patterns, what conditional elements are you using, are you using any temporal
operators, scheduling, tms?
>
> Mark
> On 24 Feb 2013, at 06:12, Wolfgang Laun <wolfgang.laun(a)gmail.com> wrote:
>
> > On 24/02/2013, Julian Klein <julianklein(a)gmail.com> wrote:
> >> Ok, thanks. Here's what I have so far:
> >>
> >> 1) I am not retracting facts or setting expiration. This is by design
> >> since I am doing the next item. Do I need to retract globals?
> >
> > Be careful with globals when you run alike sessions in parallel - the
> > object(s) is (are) shared.
> >
> >
> >> 2) I have disposed the session when I ran with a StatefulSession. I
> >> understood this means I do not need to retract facts. I do not attempt to
> >> reuse the session. I am now trying to use a StatelessSession.
> >
> > Don't change things when you try to narrow down one effect. A stateful
> > session lets you inspect things after fireAllRules has returned.
> >
> >
> >> 3) Unfortunately, the rule base is very large and this will take a long
> >> time. I am hoping to at least get to a point where this runs end-to-end.
> >> If it takes several hours, I am ok with that.
> >> 4) I would expect everything except "eval" statements to take
advantage of
> >> indexing in my rules. Are you talking about BetaNode indexing? In all
> >> cases, I use the property access over getters.
> >> 5) Since I am using a stateless session, I would expect no recursion.
> >> 6) Got it. Thanks.
> >> 7) Does this only apply with a shared KnowledgeBase? What if I spawn
> >> multiple sessions in separate threads?
> >
> > Don't do this when trying to find a memory leak.
> >
> > Originally I got the impression you were running sessions one after
> > the other until running out of memory. Running jmap between sessions
> > will give you a clear indication of what is left over after dispose().
> > A jmap is useful to observe trends, and I don't think only
> > JoinNodeLeftTuple grows with the number of past sessions - don't just
> > look at the highrunner. Compare its results after 10, 100, 1000,...
> > single-threaded executions of identical sessions.
> >
> > -W
> >
> >>
> >> All in all, I reduced the allocated heap size and ran jmap as recommended
> >> by Wolfgang with a 6GB heap, 4 threads running sessions via a ForkJoinPool
> >> and less than 100K facts. Here is a snapshot of the top hits. I ran this
> >> multiple times, and just like JProfiler, the JoinNodeLeftTuple continue to
> >> grow and grow. I would expect this to fluctuate up and down in count and
> >> size since I am not re-using a session. Not being familiar with the
> >> internals of Drools, I am hoping someone could provide a sense of whether
> >> or not the below points to one of the issues Mark mentioned above. Also,
> >> during GC events only Eden space gets freed up so these objects appear to
> >> be living in Tenured space. This further concerns me that something is not
> >> being cleaned up.
> >>
> >> I fear it is all not so simple and will continue looking into Mark's
list
> >> for opportunities in my code base as well as work towards simple test case.
> >> I appreciate all the time taken to read my rather lengthy emails. I am
> >> hoping that this detail will help others as it has me. Thank you.
> >>
> >> num #instances #bytes class name
> >> ----------------------------------------------
> >> 1: 61754567 4940365360 org.drools.reteoo.JoinNodeLeftTuple
> >> 2: 1954644 109460064 org.drools.reteoo.RightTuple
> >> 3: 302247 24179760 org.drools.common.AgendaItem
> >> 4: 447814 21495072 com.mycompany.loader.FactsLoader <-
A
> >> bunch of callable objects that get executed to load data to send to Drools
> >> when the processor(s) is (are) idle.
> >> 5: 302247 19343808
> >> org.drools.reteoo.RuleTerminalNodeLeftTuple
> >> 6: 897091 14353456 java.lang.Integer
> >> 7: 447814 14330048 java.util.RandomAccessSubList
> >> 8: 300383 11743152 [C
> >> 9: 447815 10747560
> >> java.util.Collections$SynchronizedRandomAccessList
> >>
> >>
> >>
> >> On Sat, Feb 23, 2013 at 5:57 PM, Julian Klein <julianklein(a)gmail.com>
> >> wrote:
> >>
> >>> This is great. It sounds like I have to go back to the drawing board.
> >>> It
> >>> may take a while to work through this list. I'll circle back with
> >>> outcomes.
> >>>
> >>> Thank you.
> >>>
> >>
> > _______________________________________________
> > rules-users mailing list
> > rules-users(a)lists.jboss.org
> > https://lists.jboss.org/mailman/listinfo/rules-users
>
>
> _______________________________________________
> rules-users mailing list
> rules-users(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/rules-users
>
> _______________________________________________
> rules-users mailing list
> rules-users(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/rules-users
_______________________________________________
rules-users mailing list
rules-users(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/rules-users
_______________________________________________
rules-users mailing list
rules-users(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/rules-users
7:46 a.m.
New subject: GC Overhead Limit Exceeded and 1B JoinLeftNode Objects
On 24/02/2013, Mark Proctor <mproctor(a)codehaus.org> wrote:
> when
> $sv1 : SiteVisit( yearRecorded == durationCycleYear, !annualVisit )
> FaultCode( $sv1.ID==svID, code matches "366.\\d+|743.3\\d?" )
> $inspector: Insepector (ID == $sv1.insepectortID)
>
> $sv2 : SitVisit( yearRecorded == durationCycleYear, !annualVisit )
> #make sure we are dealing with the same inspector
> Inspector (ID == $sv2.inspectorID, EUID == $inspector.EUID)
>
> ( FaultCode( $sv2.ID == svID, code matches "45.61" ) or
> ServiceCode($sv2.ID == svID, code matches
> "66(8[4-9][0-9]|9([0-3][0-9]|40))|66982|66984|66983" ) )
>
same here
Are you sure you want 'or' and not ||.
Mark, how would you suggest to avoid 'or' and use '||' (and I don't
mean the deprecated alternative to 'or')? You are beginning to confuse
(even) me!
-W
The 'or' CE can result in two rules
firing, if they are not mutually exclusive logical conditions.
9:13 a.m.
New subject: GC Overhead Limit Exceeded and 1B JoinLeftNode Objects
On 25 Feb 2013, at 06:46, Wolfgang Laun <wolfgang.laun(a)gmail.com> wrote:
On 24/02/2013, Mark Proctor <mproctor(a)codehaus.org> wrote:
>> when
>> $sv1 : SiteVisit( yearRecorded == durationCycleYear, !annualVisit )
>> FaultCode( $sv1.ID==svID, code matches "366.\\d+|743.3\\d?" )
>> $inspector: Insepector (ID == $sv1.insepectortID)
>>
>> $sv2 : SitVisit( yearRecorded == durationCycleYear, !annualVisit )
>> #make sure we are dealing with the same inspector
>> Inspector (ID == $sv2.inspectorID, EUID == $inspector.EUID)
>>
>> ( FaultCode( $sv2.ID == svID, code matches "45.61" ) or
>> ServiceCode($sv2.ID == svID, code matches
>> "66(8[4-9][0-9]|9([0-3][0-9]|40))|66982|66984|66983" ) )
>>
> same here
>
> Are you sure you want 'or' and not ||.
Mark, how would you suggest to avoid 'or' and use '||' (and I don't
mean the deprecated alternative to 'or')? You are beginning to confuse
(even) me!
Many people do not want CE 'or' they want infix '||'
used in compound expressions. I later clarified the difference in the follow up email.
Mark
-W
> The 'or' CE can result in two rules
> firing, if they are not mutually exclusive logical conditions.
_______________________________________________
rules-users mailing list
rules-users(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/rules-users
11:56 p.m.
New subject: GC Overhead Limit Exceeded and 1B JoinLeftNode Objects
Hi guys...
We have hundreds of rules and We have this problem as well. We get either
OutOfMemoryError or GC Overhead Limit Exceeded, no matter I cache
kknowledgebase or create a new one. The only difference is that with caching
knowledge-base it runs faster.
I even increased PermGen to 768 but no lock!
--
View this message in context:
http://drools.46999.n3.nabble.com/rules-users-GC-Overhead-Limit-Exceeded-...
Sent from the Drools: User forum mailing list archive at Nabble.com.
5:35 a.m.
New subject: GC Overhead Limit Exceeded and 1B JoinLeftNode Objects
Not sure about this but I found this code in dispose() method in stateful
session:
this.session.dispose();
this.session = DisposedReteooWorkingMemory.INSTANCE;
That means that the created session will never become eligible to be GCed,
instead it remains in memory pointing to the same instance (an instance of
DisposedReteooWorkingMemory)
I could be wrong... please correct me if I'm wrong!
--
View this message in context:
http://drools.46999.n3.nabble.com/rules-users-GC-Overhead-Limit-Exceeded-...
Sent from the Drools: User forum mailing list archive at Nabble.com.
6:09 a.m.
New subject: GC Overhead Limit Exceeded and 1B JoinLeftNode Objects
The GC uses **incoming** references to determine the liveness of an
object and thus its eligibility for GC.
The code here makes sure that the internal ReteooStatefulSession is no
longer referenced by
the StatefulKnowldegeSessionImpl wrapper, making it eligible for GC
unless it's referenced somewhere else.
The StatefulKnowledgeSessionImpl itself now may or may not be reachable
anymore, depending on who
was holding references to it. In any case, it is now pointing to the
"Disposed" singleton, a mock, shared empty
object.
This said, you will need to provide more details about your
configuration and usage. Just knowing that
you have "hundreds of rules" does not help much in diagnosing the
problem... :)
Did your system work, say, in 5.4 and then started generating these
OOMEs after moving to 5.5?
Could the problem be reproduced systematically? What are your policies
for retracting facts OR expiring sessions?
See the previous messages in this thread for a list of other questions
you may want to answer...
Only then we may be able to provide some advice :)
Best
Davide
On 02/25/2013 09:35 PM, ismaximum wrote:
Not sure about this but I found this code in dispose() method in
stateful
session:
this.session.dispose();
this.session = DisposedReteooWorkingMemory.INSTANCE;
That means that the created session will never become eligible to be GCed,
instead it remains in memory pointing to the same instance (an instance of
DisposedReteooWorkingMemory)
I could be wrong... please correct me if I'm wrong!
--
View this message in context:
http://drools.46999.n3.nabble.com/rules-users-GC-Overhead-Limit-Exceeded-...
Sent from the Drools: User forum mailing list archive at Nabble.com.
_______________________________________________
rules-users mailing list
rules-users(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/rules-users
7:17 a.m.
New subject: GC Overhead Limit Exceeded and 1B JoinLeftNode Objects
Well, actually we are migrating from 4.x to 5.5, a big jump :)
So far we had a lot of issue with this migration and now this is occurring.
We had no issue with v4 and since we are still working on this migration I
can't say whether the system works properly or not because we need to fix
rules otherwise they don't work at all.
We are using Stateless sessions and we don't have any retracting policy.
This retracting policy you said, made me a bit concerned. Now the question
is where is the right place to do this? I put below code in the event
listener, is that the right place and right way to do it?
@Override
public void afterRuleFlowGroupDeactivated(
org.drools.event.rule.RuleFlowGroupDeactivatedEvent e) {
Collection<FactHandle> handlers =
e.getKnowledgeRuntime().getFactHandles();
if(handlers != null) {
for(FactHandle fh : handlers) {
e.getKnowledgeRuntime().retract(fh);
}
}
}
Thanks
--
View this message in context:
http://drools.46999.n3.nabble.com/rules-users-GC-Overhead-Limit-Exceeded-...
Sent from the Drools: User forum mailing list archive at Nabble.com.
8:11 a.m.
New subject: GC Overhead Limit Exceeded and 1B JoinLeftNode Objects
The combination of a stateLESS session and this particular handling of the
RuleFlowGroupDeactivatedEvent looks strange to me. Consider that after
this code executes, all facts are gone, but a stateLESS session can't
be populated with fresh facts, since after the return from the
execute(.) call, the session itself is a goner, having been
dispose()-d automatically.
-W
On 26/02/2013, ismaximum <mnrz57(a)gmail.com> wrote:
Well, actually we are migrating from 4.x to 5.5, a big jump :)
So far we had a lot of issue with this migration and now this is occurring.
We had no issue with v4 and since we are still working on this migration I
can't say whether the system works properly or not because we need to fix
rules otherwise they don't work at all.
We are using Stateless sessions and we don't have any retracting policy.
This retracting policy you said, made me a bit concerned. Now the question
is where is the right place to do this? I put below code in the event
listener, is that the right place and right way to do it?
@Override
public void afterRuleFlowGroupDeactivated(
org.drools.event.rule.RuleFlowGroupDeactivatedEvent e) {
Collection<FactHandle> handlers =
e.getKnowledgeRuntime().getFactHandles();
if(handlers != null) {
for(FactHandle fh : handlers) {
e.getKnowledgeRuntime().retract(fh);
}
}
}
Thanks
--
View this message in context:
http://drools.46999.n3.nabble.com/rules-users-GC-Overhead-Limit-Exceeded-...
Sent from the Drools: User forum mailing list archive at Nabble.com.
_______________________________________________
rules-users mailing list
rules-users(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/rules-users
1:59 a.m.
New subject: GC Overhead Limit Exceeded and 1B JoinLeftNode Objects
Well.. I don't know.. I am just asking whether that's correct or not.
But it seems it's not correct... I don't know if I really need to retract
facts when we are using stateless Kn. Because the session will be expired as
soon as the it exists the execute() method...
But I don't know why we are getting OutOfMemoryError!
laune wrote
The combination of a stateLESS session and this particular handling
of the
RuleFlowGroupDeactivatedEvent looks strange to me. Consider that after
this code executes, all facts are gone, but a stateLESS session can't
be populated with fresh facts, since after the return from the
execute(.) call, the session itself is a goner, having been
dispose()-d automatically.
-W
--
View this message in context:
http://drools.46999.n3.nabble.com/rules-users-GC-Overhead-Limit-Exceeded-...
Sent from the Drools: User forum mailing list archive at Nabble.com.
4:55 a.m.
New subject: GC Overhead Limit Exceeded and 1B JoinLeftNode Objects
Hi Guys... I attached the result from profiler after it failed with
OutOfMemoryError...
I thought it might be helpful.
I can't see our objects are the cause of this problem.
This image shows instances of Drools :
<http://drools.46999.n3.nabble.com/file/n4022604/drools.png>
This image shows all instances:
<http://drools.46999.n3.nabble.com/file/n4022604/drools-all-object.png>
Our instances in memory:
<http://drools.46999.n3.nabble.com/file/n4022604/drools-our-objectst.png>
--
View this message in context:
http://drools.46999.n3.nabble.com/rules-users-GC-Overhead-Limit-Exceeded-...
Sent from the Drools: User forum mailing list archive at Nabble.com.
6:38 a.m.
New subject: GC Overhead Limit Exceeded and 1B JoinLeftNode Objects
What are your JVM settings?
If you got an OOME/GCE when ~32Mb of live char[] were occupying ~25% of
your memory,
I'd suspect that you are running your application with a maximum heap
size of 128M...
On 02/26/2013 08:55 PM, ismaximum wrote:
Hi Guys... I attached the result from profiler after it failed with
OutOfMemoryError...
I thought it might be helpful.
I can't see our objects are the cause of this problem.
This image shows instances of Drools :
<http://drools.46999.n3.nabble.com/file/n4022604/drools.png>
This image shows all instances:
<http://drools.46999.n3.nabble.com/file/n4022604/drools-all-object.png>
Our instances in memory:
<http://drools.46999.n3.nabble.com/file/n4022604/drools-our-objectst.png>
--
View this message in context:
http://drools.46999.n3.nabble.com/rules-users-GC-Overhead-Limit-Exceeded-...
Sent from the Drools: User forum mailing list archive at Nabble.com.
_______________________________________________
rules-users mailing list
rules-users(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/rules-users
3 p.m.
New subject: GC Overhead Limit Exceeded and 1B JoinLeftNode Objects
I wanted to circle back before this thread gets too far off the original
topic and folks searching in the future don't get confused. I am still
confused between the 'or' and '||' as explained here and in the
documentation. I will attempt to review this in more detail. In the
meantime, I have simplified my rules per the above comments and am able to
run them with about 10M facts in under 3 hours. The system required more
than 40GB of RAM. I am chunking the data and running multiple threads. I
will look to reduce the overhead where possible as I am certain their are
further improvements that can be made. Once I complete out my refactoring
and reviews, I'll circle back again. Needless to say, things have moved in
a very positive direction. I am not sure what memory expectations I should
have so I'll take a complete run in this timeframe without errors as a win.
Thank you, Wolgang and Mark. You guys have been fantastic!
On Mon, Feb 25, 2013 at 3:13 AM, Mark Proctor <mproctor(a)codehaus.org> wrote:
On 25 Feb 2013, at 06:46, Wolfgang Laun <wolfgang.laun(a)gmail.com> wrote:
> On 24/02/2013, Mark Proctor <mproctor(a)codehaus.org> wrote:
>
>>> when
>>> $sv1 : SiteVisit( yearRecorded == durationCycleYear, !annualVisit
)
>>> FaultCode( $sv1.ID==svID, code matches "366.\\d+|743.3\\d?"
)
>>> $inspector: Insepector (ID == $sv1.insepectortID)
>>>
>>> $sv2 : SitVisit( yearRecorded == durationCycleYear, !annualVisit )
>>> #make sure we are dealing with the same inspector
>>> Inspector (ID == $sv2.inspectorID, EUID == $inspector.EUID)
>>>
>>> ( FaultCode( $sv2.ID == svID, code matches "45.61" ) or
>>> ServiceCode($sv2.ID == svID, code matches
>>> "66(8[4-9][0-9]|9([0-3][0-9]|40))|66982|66984|66983" ) )
>>>
>> same here
>>
>> Are you sure you want 'or' and not ||.
>
> Mark, how would you suggest to avoid 'or' and use '||' (and I
don't
> mean the deprecated alternative to 'or')? You are beginning to confuse
> (even) me!
Many people do not want CE 'or' they want infix '||' used in compound
expressions. I later clarified the difference in the follow up email.
Mark
>
> -W
>
>> The 'or' CE can result in two rules
>> firing, if they are not mutually exclusive logical conditions.
> _______________________________________________
> rules-users mailing list
> rules-users(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/rules-users
_______________________________________________
rules-users mailing list
rules-users(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/rules-users
4076
days inactive
4080
days old
22 comments
5 participants
participants (5)
-
Davide Sottara -
ismaximum -
Julian Klein -
Mark Proctor -
Wolfgang Laun