[rules-users] GC Overhead Limit Exceeded and 1B JoinLeftNode Objects
Wolfgang Laun
wolfgang.laun at gmail.com
Sun Feb 24 13:50:37 EST 2013
Below are some comments on rules "001 [d]" and "001[n]" . I have
silently ignored all the obvious spelling errors.
On 24/02/2013, Julian Klein <julianklein at gmail.com> wrote:
> Thanks for all the advice once again. I understand their is a distinct
> possibility that my rules are highly inefficient and therefore leading to a
> large memory requirement. I will keep pairing this down to one thread and
> one use case.
>
> So here are two example rules I have. This demonstrates the approach I am
> using (pattern matching, temporal reasoning, regex matches, etc). You'll
> notice that rule 001n only fires when 001d exists. It is operating on
> a separate agenda as well. I am not sure that makes sense, but it will
> come down to whether a Stateless with Sequential mode is better for my
> scenarios than a Stateful session with one agenda (or two with the agenda
> being switched after the first call to fireAllRules). For
> reference, durationCycleYear and utils are globals. One is a I can see at
> the least that I need to move some variables to the right hand side of the
> == statements. Most of the rules are similar in their approach.
>
> rule "001 [d]"
> @id("001d")
> agenda-group 'd-rules-agenda'
>
> when
> $sv1 : SiteVisit( yearRecorded == durationCycleYear, !annualVisit )
Hopefully durationCycleYear is a global and constant?
> FaultCode( $sv1.ID==svID, code matches "366.\\d+|743.3\\d?" )
In these patterns, is '.' supposed to match any character? A literal
'.' would have to be escaped.
> $inspector: Insepector (ID == $sv1.insepectortID)
>
> $sv2 : SitVisit( yearRecorded == durationCycleYear, !annualVisit )
Bad. This matches any other SiteVisit according to these constraints,
but it also matches the
same one as already matched $sv1. Moreover, if there is a pair of two
matching SiteVisits
(SV1, SV2) there will be 4 activations with SV1+SV2, SV2+SV1, SV1+SV1, SV2+SV2.
> #make sure we are dealing with the same inspector
> Inspector (ID == $sv2.inspectorID, EUID == $inspector.EUID)
This constraint should be added in the second SiteVisit pattern.
There's really no need to re-bind the same Inspector - we already have
it bound to $inspector.
>
> ( FaultCode( $sv2.ID == svID, code matches "45.61" ) or
> ServiceCode($sv2.ID == svID, code matches "66(8[4-9][0-9]|9([0-3][0-9]|40))|66982|66984|66983" ) )
>
>
> then
> insert(utils.saveAndReturnEvent(kcontext, $sv2));
(Let's hope that the returned Event is a "slim" Pojo and doesn't try
to conserve tons of data from kcontext.)
>
> rule "001 [n]"
> @id("001n")
> agenda-group 'n-rules-agenda'
> when
> $event : Event( externalID == "001d")
> $encD : SiteVisit($event.svID == ID)
> $inspector : Inspector (ID == $svD.inspectorID)
>
> $svN : SiteVisit( datetimeRecorded after[0ms, 90d]
> $svD.datetimeRecorded, !annualVisit )
Note that A after[0ms,...] B does not enforce that A and B are
different. See the previous comment starting with "Bad.".
> #make sure we are dealing with the same inspector
> Inspector (ID == $svN.inspectorID, EUID == $inspector.EUID)
See the corresponding comment re the 1st rule.
>
> FaultCode($svN.ID == svID, code matches
> "361.\\d+|362.4[23]|371.\\d+|360.0\\d?|360.1|362.53|998.82|998.9" )
Again: '.'?
>
> then
> insert(utils.saveAndReturnEvent(kcontext, $svN));
> end
>
Doesn't look too good, IMHO. A thorough revision of your rule base
appears indicated.
-W
>
> On Sun, Feb 24, 2013 at 1:25 AM, Mark Proctor <mproctor at codehaus.org>
> wrote:
>
>> You also need to give us some indication of what your rules look like.
>> Are
>> you using just patterns, what conditional elements are you using, are you
>> using any temporal operators, scheduling, tms?
>>
>> Mark
>> On 24 Feb 2013, at 06:12, Wolfgang Laun <wolfgang.laun at gmail.com> wrote:
>>
>> > On 24/02/2013, Julian Klein <julianklein at gmail.com> wrote:
>> >> Ok, thanks. Here's what I have so far:
>> >>
>> >> 1) I am not retracting facts or setting expiration. This is by design
>> >> since I am doing the next item. Do I need to retract globals?
>> >
>> > Be careful with globals when you run alike sessions in parallel - the
>> > object(s) is (are) shared.
>> >
>> >
>> >> 2) I have disposed the session when I ran with a StatefulSession. I
>> >> understood this means I do not need to retract facts. I do not
>> >> attempt
>> to
>> >> reuse the session. I am now trying to use a StatelessSession.
>> >
>> > Don't change things when you try to narrow down one effect. A stateful
>> > session lets you inspect things after fireAllRules has returned.
>> >
>> >
>> >> 3) Unfortunately, the rule base is very large and this will take a
>> >> long
>> >> time. I am hoping to at least get to a point where this runs
>> end-to-end.
>> >> If it takes several hours, I am ok with that.
>> >> 4) I would expect everything except "eval" statements to take
>> >> advantage
>> of
>> >> indexing in my rules. Are you talking about BetaNode indexing? In
>> >> all
>> >> cases, I use the property access over getters.
>> >> 5) Since I am using a stateless session, I would expect no recursion.
>> >> 6) Got it. Thanks.
>> >> 7) Does this only apply with a shared KnowledgeBase? What if I spawn
>> >> multiple sessions in separate threads?
>> >
>> > Don't do this when trying to find a memory leak.
>> >
>> > Originally I got the impression you were running sessions one after
>> > the other until running out of memory. Running jmap between sessions
>> > will give you a clear indication of what is left over after dispose().
>> > A jmap is useful to observe trends, and I don't think only
>> > JoinNodeLeftTuple grows with the number of past sessions - don't just
>> > look at the highrunner. Compare its results after 10, 100, 1000,...
>> > single-threaded executions of identical sessions.
>> >
>> > -W
>> >
>> >>
>> >> All in all, I reduced the allocated heap size and ran jmap as
>> recommended
>> >> by Wolfgang with a 6GB heap, 4 threads running sessions via a
>> ForkJoinPool
>> >> and less than 100K facts. Here is a snapshot of the top hits. I ran
>> this
>> >> multiple times, and just like JProfiler, the JoinNodeLeftTuple
>> >> continue
>> to
>> >> grow and grow. I would expect this to fluctuate up and down in count
>> and
>> >> size since I am not re-using a session. Not being familiar with the
>> >> internals of Drools, I am hoping someone could provide a sense of
>> whether
>> >> or not the below points to one of the issues Mark mentioned above.
>> Also,
>> >> during GC events only Eden space gets freed up so these objects appear
>> to
>> >> be living in Tenured space. This further concerns me that something
>> >> is
>> not
>> >> being cleaned up.
>> >>
>> >> I fear it is all not so simple and will continue looking into Mark's
>> list
>> >> for opportunities in my code base as well as work towards simple test
>> case.
>> >> I appreciate all the time taken to read my rather lengthy emails. I
>> >> am
>> >> hoping that this detail will help others as it has me. Thank you.
>> >>
>> >> num #instances #bytes class name
>> >> ----------------------------------------------
>> >> 1: 61754567 4940365360 org.drools.reteoo.JoinNodeLeftTuple
>> >> 2: 1954644 109460064 org.drools.reteoo.RightTuple
>> >> 3: 302247 24179760 org.drools.common.AgendaItem
>> >> 4: 447814 21495072 com.mycompany.loader.FactsLoader
>> >> <-
>> A
>> >> bunch of callable objects that get executed to load data to send to
>> Drools
>> >> when the processor(s) is (are) idle.
>> >> 5: 302247 19343808
>> >> org.drools.reteoo.RuleTerminalNodeLeftTuple
>> >> 6: 897091 14353456 java.lang.Integer
>> >> 7: 447814 14330048 java.util.RandomAccessSubList
>> >> 8: 300383 11743152 [C
>> >> 9: 447815 10747560
>> >> java.util.Collections$SynchronizedRandomAccessList
>> >>
>> >>
>> >>
>> >> On Sat, Feb 23, 2013 at 5:57 PM, Julian Klein <julianklein at gmail.com>
>> >> wrote:
>> >>
>> >>> This is great. It sounds like I have to go back to the drawing
>> >>> board.
>> >>> It
>> >>> may take a while to work through this list. I'll circle back with
>> >>> outcomes.
>> >>>
>> >>> Thank you.
>> >>>
>> >>
>> > _______________________________________________
>> > rules-users mailing list
>> > rules-users at lists.jboss.org
>> > https://lists.jboss.org/mailman/listinfo/rules-users
>>
>>
>> _______________________________________________
>> rules-users mailing list
>> rules-users at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/rules-users
>>
>
More information about the rules-users
mailing list