[rules-users] GC Overhead Limit Exceeded and 1B JoinLeftNode Objects
Mark Proctor
mproctor at codehaus.org
Sun Feb 24 15:50:47 EST 2013
On 24 Feb 2013, at 18:07, Julian Klein <julianklein at gmail.com> wrote:
> Thanks for all the advice once again. I understand their is a distinct possibility that my rules are highly inefficient and therefore leading to a large memory requirement. I will keep pairing this down to one thread and one use case.
>
> So here are two example rules I have. This demonstrates the approach I am using (pattern matching, temporal reasoning, regex matches, etc). You'll notice that rule 001n only fires when 001d exists. It is operating on a separate agenda as well. I am not sure that makes sense, but it will come down to whether a Stateless with Sequential mode is better for my scenarios than a Stateful session with one agenda (or two with the agenda being switched after the first call to fireAllRules). For reference, durationCycleYear and utils are globals. One is a I can see at the least that I need to move some variables to the right hand side of the == statements. Most of the rules are similar in their approach.
>
> rule "001 [d]"
> @id("001d")
> agenda-group 'd-rules-agenda'
>
> when
> $sv1 : SiteVisit( yearRecorded == durationCycleYear, !annualVisit )
> FaultCode( $sv1.ID==svID, code matches "366.\\d+|743.3\\d?" )
This is wrong if you want indexing. It'll work, it just doesn't get picked up by the indexer:
FaultCode( svID == $sv1.ID)
> $inspector: Insepector (ID == $sv1.insepectortID)
>
> $sv2 : SitVisit( yearRecorded == durationCycleYear, !annualVisit )
> #make sure we are dealing with the same inspector
> Inspector (ID == $sv2.inspectorID, EUID == $inspector.EUID)
>
> ( FaultCode( $sv2.ID == svID, code matches "45.61" ) or
> ServiceCode($sv2.ID == svID, code matches "66(8[4-9][0-9]|9([0-3][0-9]|40))|66982|66984|66983" ) )
>
same here
Are you sure you want 'or' and not ||. The 'or' CE can result in two rules firing, if they are not mutually exclusive logical conditions.
>
> then
> insert(utils.saveAndReturnEvent(kcontext, $sv2));
> end
>
> rule "001 [n]"
> @id("001n")
> agenda-group 'n-rules-agenda'
> when
> $event : Event( externalID == "001d")
> $encD : SiteVisit($event.svID == ID)
> $inspector : Inspector (ID == $svD.inspectorID)
>
> $svN : SiteVisit( datetimeRecorded after[0ms, 90d] $svD.datetimeRecorded, !annualVisit )
> #make sure we are dealing with the same inspector
> Inspector (ID == $svN.inspectorID, EUID == $inspector.EUID)
>
> FaultCode($svN.ID == svID, code matches "361.\\d+|362.4[23]|371.\\d+|360.0\\d?|360.1|362.53|998.82|998.9" )
>
> then
> insert(utils.saveAndReturnEvent(kcontext, $svN));
> end
>
>
> On Sun, Feb 24, 2013 at 1:25 AM, Mark Proctor <mproctor at codehaus.org> wrote:
> You also need to give us some indication of what your rules look like. Are you using just patterns, what conditional elements are you using, are you using any temporal operators, scheduling, tms?
>
> Mark
> On 24 Feb 2013, at 06:12, Wolfgang Laun <wolfgang.laun at gmail.com> wrote:
>
> > On 24/02/2013, Julian Klein <julianklein at gmail.com> wrote:
> >> Ok, thanks. Here's what I have so far:
> >>
> >> 1) I am not retracting facts or setting expiration. This is by design
> >> since I am doing the next item. Do I need to retract globals?
> >
> > Be careful with globals when you run alike sessions in parallel - the
> > object(s) is (are) shared.
> >
> >
> >> 2) I have disposed the session when I ran with a StatefulSession. I
> >> understood this means I do not need to retract facts. I do not attempt to
> >> reuse the session. I am now trying to use a StatelessSession.
> >
> > Don't change things when you try to narrow down one effect. A stateful
> > session lets you inspect things after fireAllRules has returned.
> >
> >
> >> 3) Unfortunately, the rule base is very large and this will take a long
> >> time. I am hoping to at least get to a point where this runs end-to-end.
> >> If it takes several hours, I am ok with that.
> >> 4) I would expect everything except "eval" statements to take advantage of
> >> indexing in my rules. Are you talking about BetaNode indexing? In all
> >> cases, I use the property access over getters.
> >> 5) Since I am using a stateless session, I would expect no recursion.
> >> 6) Got it. Thanks.
> >> 7) Does this only apply with a shared KnowledgeBase? What if I spawn
> >> multiple sessions in separate threads?
> >
> > Don't do this when trying to find a memory leak.
> >
> > Originally I got the impression you were running sessions one after
> > the other until running out of memory. Running jmap between sessions
> > will give you a clear indication of what is left over after dispose().
> > A jmap is useful to observe trends, and I don't think only
> > JoinNodeLeftTuple grows with the number of past sessions - don't just
> > look at the highrunner. Compare its results after 10, 100, 1000,...
> > single-threaded executions of identical sessions.
> >
> > -W
> >
> >>
> >> All in all, I reduced the allocated heap size and ran jmap as recommended
> >> by Wolfgang with a 6GB heap, 4 threads running sessions via a ForkJoinPool
> >> and less than 100K facts. Here is a snapshot of the top hits. I ran this
> >> multiple times, and just like JProfiler, the JoinNodeLeftTuple continue to
> >> grow and grow. I would expect this to fluctuate up and down in count and
> >> size since I am not re-using a session. Not being familiar with the
> >> internals of Drools, I am hoping someone could provide a sense of whether
> >> or not the below points to one of the issues Mark mentioned above. Also,
> >> during GC events only Eden space gets freed up so these objects appear to
> >> be living in Tenured space. This further concerns me that something is not
> >> being cleaned up.
> >>
> >> I fear it is all not so simple and will continue looking into Mark's list
> >> for opportunities in my code base as well as work towards simple test case.
> >> I appreciate all the time taken to read my rather lengthy emails. I am
> >> hoping that this detail will help others as it has me. Thank you.
> >>
> >> num #instances #bytes class name
> >> ----------------------------------------------
> >> 1: 61754567 4940365360 org.drools.reteoo.JoinNodeLeftTuple
> >> 2: 1954644 109460064 org.drools.reteoo.RightTuple
> >> 3: 302247 24179760 org.drools.common.AgendaItem
> >> 4: 447814 21495072 com.mycompany.loader.FactsLoader <- A
> >> bunch of callable objects that get executed to load data to send to Drools
> >> when the processor(s) is (are) idle.
> >> 5: 302247 19343808
> >> org.drools.reteoo.RuleTerminalNodeLeftTuple
> >> 6: 897091 14353456 java.lang.Integer
> >> 7: 447814 14330048 java.util.RandomAccessSubList
> >> 8: 300383 11743152 [C
> >> 9: 447815 10747560
> >> java.util.Collections$SynchronizedRandomAccessList
> >>
> >>
> >>
> >> On Sat, Feb 23, 2013 at 5:57 PM, Julian Klein <julianklein at gmail.com>
> >> wrote:
> >>
> >>> This is great. It sounds like I have to go back to the drawing board.
> >>> It
> >>> may take a while to work through this list. I'll circle back with
> >>> outcomes.
> >>>
> >>> Thank you.
> >>>
> >>
> > _______________________________________________
> > rules-users mailing list
> > rules-users at lists.jboss.org
> > https://lists.jboss.org/mailman/listinfo/rules-users
>
>
> _______________________________________________
> rules-users mailing list
> rules-users at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/rules-users
>
> _______________________________________________
> rules-users mailing list
> rules-users at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/rules-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/rules-users/attachments/20130224/768e943e/attachment-0001.html
More information about the rules-users
mailing list