[rules-users] GC Overhead Limit Exceeded and 1B JoinLeftNode Objects
Julian Klein
julianklein at gmail.com
Sun Feb 24 13:07:57 EST 2013
Thanks for all the advice once again. I understand their is a distinct
possibility that my rules are highly inefficient and therefore leading to a
large memory requirement. I will keep pairing this down to one thread and
one use case.
So here are two example rules I have. This demonstrates the approach I am
using (pattern matching, temporal reasoning, regex matches, etc). You'll
notice that rule 001n only fires when 001d exists. It is operating on
a separate agenda as well. I am not sure that makes sense, but it will
come down to whether a Stateless with Sequential mode is better for my
scenarios than a Stateful session with one agenda (or two with the agenda
being switched after the first call to fireAllRules). For
reference, durationCycleYear and utils are globals. One is a I can see at
the least that I need to move some variables to the right hand side of the
== statements. Most of the rules are similar in their approach.
rule "001 [d]"
@id("001d")
agenda-group 'd-rules-agenda'
when
$sv1 : SiteVisit( yearRecorded == durationCycleYear, !annualVisit )
FaultCode( $sv1.ID==svID, code matches "366.\\d+|743.3\\d?" )
$inspector: Insepector (ID == $sv1.insepectortID)
$sv2 : SitVisit( yearRecorded == durationCycleYear, !annualVisit )
#make sure we are dealing with the same inspector
Inspector (ID == $sv2.inspectorID, EUID == $inspector.EUID)
( FaultCode( $sv2.ID == svID, code matches "45.61" ) or
ServiceCode($sv2.ID == svID, code matches
"66(8[4-9][0-9]|9([0-3][0-9]|40))|66982|66984|66983" ) )
then
insert(utils.saveAndReturnEvent(kcontext, $sv2));
end
rule "001 [n]"
@id("001n")
agenda-group 'n-rules-agenda'
when
$event : Event( externalID == "001d")
$encD : SiteVisit($event.svID == ID)
$inspector : Inspector (ID == $svD.inspectorID)
$svN : SiteVisit( datetimeRecorded after[0ms, 90d]
$svD.datetimeRecorded, !annualVisit )
#make sure we are dealing with the same inspector
Inspector (ID == $svN.inspectorID, EUID == $inspector.EUID)
FaultCode($svN.ID == svID, code matches
"361.\\d+|362.4[23]|371.\\d+|360.0\\d?|360.1|362.53|998.82|998.9" )
then
insert(utils.saveAndReturnEvent(kcontext, $svN));
end
On Sun, Feb 24, 2013 at 1:25 AM, Mark Proctor <mproctor at codehaus.org> wrote:
> You also need to give us some indication of what your rules look like. Are
> you using just patterns, what conditional elements are you using, are you
> using any temporal operators, scheduling, tms?
>
> Mark
> On 24 Feb 2013, at 06:12, Wolfgang Laun <wolfgang.laun at gmail.com> wrote:
>
> > On 24/02/2013, Julian Klein <julianklein at gmail.com> wrote:
> >> Ok, thanks. Here's what I have so far:
> >>
> >> 1) I am not retracting facts or setting expiration. This is by design
> >> since I am doing the next item. Do I need to retract globals?
> >
> > Be careful with globals when you run alike sessions in parallel - the
> > object(s) is (are) shared.
> >
> >
> >> 2) I have disposed the session when I ran with a StatefulSession. I
> >> understood this means I do not need to retract facts. I do not attempt
> to
> >> reuse the session. I am now trying to use a StatelessSession.
> >
> > Don't change things when you try to narrow down one effect. A stateful
> > session lets you inspect things after fireAllRules has returned.
> >
> >
> >> 3) Unfortunately, the rule base is very large and this will take a long
> >> time. I am hoping to at least get to a point where this runs
> end-to-end.
> >> If it takes several hours, I am ok with that.
> >> 4) I would expect everything except "eval" statements to take advantage
> of
> >> indexing in my rules. Are you talking about BetaNode indexing? In all
> >> cases, I use the property access over getters.
> >> 5) Since I am using a stateless session, I would expect no recursion.
> >> 6) Got it. Thanks.
> >> 7) Does this only apply with a shared KnowledgeBase? What if I spawn
> >> multiple sessions in separate threads?
> >
> > Don't do this when trying to find a memory leak.
> >
> > Originally I got the impression you were running sessions one after
> > the other until running out of memory. Running jmap between sessions
> > will give you a clear indication of what is left over after dispose().
> > A jmap is useful to observe trends, and I don't think only
> > JoinNodeLeftTuple grows with the number of past sessions - don't just
> > look at the highrunner. Compare its results after 10, 100, 1000,...
> > single-threaded executions of identical sessions.
> >
> > -W
> >
> >>
> >> All in all, I reduced the allocated heap size and ran jmap as
> recommended
> >> by Wolfgang with a 6GB heap, 4 threads running sessions via a
> ForkJoinPool
> >> and less than 100K facts. Here is a snapshot of the top hits. I ran
> this
> >> multiple times, and just like JProfiler, the JoinNodeLeftTuple continue
> to
> >> grow and grow. I would expect this to fluctuate up and down in count
> and
> >> size since I am not re-using a session. Not being familiar with the
> >> internals of Drools, I am hoping someone could provide a sense of
> whether
> >> or not the below points to one of the issues Mark mentioned above.
> Also,
> >> during GC events only Eden space gets freed up so these objects appear
> to
> >> be living in Tenured space. This further concerns me that something is
> not
> >> being cleaned up.
> >>
> >> I fear it is all not so simple and will continue looking into Mark's
> list
> >> for opportunities in my code base as well as work towards simple test
> case.
> >> I appreciate all the time taken to read my rather lengthy emails. I am
> >> hoping that this detail will help others as it has me. Thank you.
> >>
> >> num #instances #bytes class name
> >> ----------------------------------------------
> >> 1: 61754567 4940365360 org.drools.reteoo.JoinNodeLeftTuple
> >> 2: 1954644 109460064 org.drools.reteoo.RightTuple
> >> 3: 302247 24179760 org.drools.common.AgendaItem
> >> 4: 447814 21495072 com.mycompany.loader.FactsLoader <-
> A
> >> bunch of callable objects that get executed to load data to send to
> Drools
> >> when the processor(s) is (are) idle.
> >> 5: 302247 19343808
> >> org.drools.reteoo.RuleTerminalNodeLeftTuple
> >> 6: 897091 14353456 java.lang.Integer
> >> 7: 447814 14330048 java.util.RandomAccessSubList
> >> 8: 300383 11743152 [C
> >> 9: 447815 10747560
> >> java.util.Collections$SynchronizedRandomAccessList
> >>
> >>
> >>
> >> On Sat, Feb 23, 2013 at 5:57 PM, Julian Klein <julianklein at gmail.com>
> >> wrote:
> >>
> >>> This is great. It sounds like I have to go back to the drawing board.
> >>> It
> >>> may take a while to work through this list. I'll circle back with
> >>> outcomes.
> >>>
> >>> Thank you.
> >>>
> >>
> > _______________________________________________
> > rules-users mailing list
> > rules-users at lists.jboss.org
> > https://lists.jboss.org/mailman/listinfo/rules-users
>
>
> _______________________________________________
> rules-users mailing list
> rules-users at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/rules-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/rules-users/attachments/20130224/22f19bc0/attachment.html
More information about the rules-users
mailing list