[rules-users] GC Overhead Limit Exceeded and 1B JoinLeftNode Objects
Julian Klein
julianklein at gmail.com
Sun Feb 24 16:57:19 EST 2013
As far as $sv.ID == svID, I understand and have changed that throughout my
rules. I've actually applied all your advice so far, thank you. I am
still reviewing the rule base as I continue to get OoM Exceptions. I also
have changed the structure of my model to reduce redundancy in the
Inspector facts. This has shortened my rules a bit.
In regards to "or" versus "||", the 5.5 docs state that "||" is deprecated
for CEs. Therefore I chose ||. Also, if I match on either ServiceCode or
FaultCode, I should get a result. I am not sure how || or "or" makes a
difference here since the docs state they are interchangeable. Would it
result in a shortcut preventing both generated sub-rules from firing?
On Sun, Feb 24, 2013 at 3:50 PM, Mark Proctor <mproctor at codehaus.org> wrote:
>
> On 24 Feb 2013, at 18:07, Julian Klein <julianklein at gmail.com> wrote:
>
> Thanks for all the advice once again. I understand their is a distinct
> possibility that my rules are highly inefficient and therefore leading to a
> large memory requirement. I will keep pairing this down to one thread and
> one use case.
>
> So here are two example rules I have. This demonstrates the approach I am
> using (pattern matching, temporal reasoning, regex matches, etc). You'll
> notice that rule 001n only fires when 001d exists. It is operating on
> a separate agenda as well. I am not sure that makes sense, but it will
> come down to whether a Stateless with Sequential mode is better for my
> scenarios than a Stateful session with one agenda (or two with the agenda
> being switched after the first call to fireAllRules). For
> reference, durationCycleYear and utils are globals. One is a I can see at
> the least that I need to move some variables to the right hand side of the
> == statements. Most of the rules are similar in their approach.
>
> rule "001 [d]"
> @id("001d")
> agenda-group 'd-rules-agenda'
>
> when
> $sv1 : SiteVisit( yearRecorded == durationCycleYear, !annualVisit )
> FaultCode( $sv1.ID==svID, code matches "366.\\d+|743.3\\d?" )
>
> This is wrong if you want indexing. It'll work, it just doesn't get picked
> up by the indexer:
> FaultCode( svID == $sv1.ID)
>
> $inspector: Insepector (ID == $sv1.insepectortID)
>
> $sv2 : SitVisit( yearRecorded == durationCycleYear, !annualVisit )
> #make sure we are dealing with the same inspector
> Inspector (ID == $sv2.inspectorID, EUID == $inspector.EUID)
>
> ( FaultCode( $sv2.ID == svID, code matches "45.61" ) or
> ServiceCode($sv2.ID == svID, code matches
> "66(8[4-9][0-9]|9([0-3][0-9]|40))|66982|66984|66983" ) )
>
> same here
>
> Are you sure you want 'or' and not ||. The 'or' CE can result in two rules
> firing, if they are not mutually exclusive logical conditions.
>
>
> then
> insert(utils.saveAndReturnEvent(kcontext, $sv2));
> end
>
> rule "001 [n]"
> @id("001n")
> agenda-group 'n-rules-agenda'
> when
> $event : Event( externalID == "001d")
> $encD : SiteVisit($event.svID == ID)
> $inspector : Inspector (ID == $svD.inspectorID)
>
> $svN : SiteVisit( datetimeRecorded after[0ms, 90d]
> $svD.datetimeRecorded, !annualVisit )
> #make sure we are dealing with the same inspector
> Inspector (ID == $svN.inspectorID, EUID == $inspector.EUID)
>
> FaultCode($svN.ID == svID, code matches
> "361.\\d+|362.4[23]|371.\\d+|360.0\\d?|360.1|362.53|998.82|998.9" )
>
> then
> insert(utils.saveAndReturnEvent(kcontext, $svN));
> end
>
>
> On Sun, Feb 24, 2013 at 1:25 AM, Mark Proctor <mproctor at codehaus.org>wrote:
>
>> You also need to give us some indication of what your rules look like.
>> Are you using just patterns, what conditional elements are you using, are
>> you using any temporal operators, scheduling, tms?
>>
>> Mark
>> On 24 Feb 2013, at 06:12, Wolfgang Laun <wolfgang.laun at gmail.com> wrote:
>>
>> > On 24/02/2013, Julian Klein <julianklein at gmail.com> wrote:
>> >> Ok, thanks. Here's what I have so far:
>> >>
>> >> 1) I am not retracting facts or setting expiration. This is by design
>> >> since I am doing the next item. Do I need to retract globals?
>> >
>> > Be careful with globals when you run alike sessions in parallel - the
>> > object(s) is (are) shared.
>> >
>> >
>> >> 2) I have disposed the session when I ran with a StatefulSession. I
>> >> understood this means I do not need to retract facts. I do not
>> attempt to
>> >> reuse the session. I am now trying to use a StatelessSession.
>> >
>> > Don't change things when you try to narrow down one effect. A stateful
>> > session lets you inspect things after fireAllRules has returned.
>> >
>> >
>> >> 3) Unfortunately, the rule base is very large and this will take a long
>> >> time. I am hoping to at least get to a point where this runs
>> end-to-end.
>> >> If it takes several hours, I am ok with that.
>> >> 4) I would expect everything except "eval" statements to take
>> advantage of
>> >> indexing in my rules. Are you talking about BetaNode indexing? In
>> all
>> >> cases, I use the property access over getters.
>> >> 5) Since I am using a stateless session, I would expect no recursion.
>> >> 6) Got it. Thanks.
>> >> 7) Does this only apply with a shared KnowledgeBase? What if I spawn
>> >> multiple sessions in separate threads?
>> >
>> > Don't do this when trying to find a memory leak.
>> >
>> > Originally I got the impression you were running sessions one after
>> > the other until running out of memory. Running jmap between sessions
>> > will give you a clear indication of what is left over after dispose().
>> > A jmap is useful to observe trends, and I don't think only
>> > JoinNodeLeftTuple grows with the number of past sessions - don't just
>> > look at the highrunner. Compare its results after 10, 100, 1000,...
>> > single-threaded executions of identical sessions.
>> >
>> > -W
>> >
>> >>
>> >> All in all, I reduced the allocated heap size and ran jmap as
>> recommended
>> >> by Wolfgang with a 6GB heap, 4 threads running sessions via a
>> ForkJoinPool
>> >> and less than 100K facts. Here is a snapshot of the top hits. I ran
>> this
>> >> multiple times, and just like JProfiler, the JoinNodeLeftTuple
>> continue to
>> >> grow and grow. I would expect this to fluctuate up and down in count
>> and
>> >> size since I am not re-using a session. Not being familiar with the
>> >> internals of Drools, I am hoping someone could provide a sense of
>> whether
>> >> or not the below points to one of the issues Mark mentioned above.
>> Also,
>> >> during GC events only Eden space gets freed up so these objects appear
>> to
>> >> be living in Tenured space. This further concerns me that something
>> is not
>> >> being cleaned up.
>> >>
>> >> I fear it is all not so simple and will continue looking into Mark's
>> list
>> >> for opportunities in my code base as well as work towards simple test
>> case.
>> >> I appreciate all the time taken to read my rather lengthy emails. I am
>> >> hoping that this detail will help others as it has me. Thank you.
>> >>
>> >> num #instances #bytes class name
>> >> ----------------------------------------------
>> >> 1: 61754567 4940365360 org.drools.reteoo.JoinNodeLeftTuple
>> >> 2: 1954644 109460064 org.drools.reteoo.RightTuple
>> >> 3: 302247 24179760 org.drools.common.AgendaItem
>> >> 4: 447814 21495072 com.mycompany.loader.FactsLoader
>> <- A
>> >> bunch of callable objects that get executed to load data to send to
>> Drools
>> >> when the processor(s) is (are) idle.
>> >> 5: 302247 19343808
>> >> org.drools.reteoo.RuleTerminalNodeLeftTuple
>> >> 6: 897091 14353456 java.lang.Integer
>> >> 7: 447814 14330048 java.util.RandomAccessSubList
>> >> 8: 300383 11743152 [C
>> >> 9: 447815 10747560
>> >> java.util.Collections$SynchronizedRandomAccessList
>> >>
>> >>
>> >>
>> >> On Sat, Feb 23, 2013 at 5:57 PM, Julian Klein <julianklein at gmail.com>
>> >> wrote:
>> >>
>> >>> This is great. It sounds like I have to go back to the drawing board.
>> >>> It
>> >>> may take a while to work through this list. I'll circle back with
>> >>> outcomes.
>> >>>
>> >>> Thank you.
>> >>>
>> >>
>> > _______________________________________________
>> > rules-users mailing list
>> > rules-users at lists.jboss.org
>> > https://lists.jboss.org/mailman/listinfo/rules-users
>>
>>
>> _______________________________________________
>> rules-users mailing list
>> rules-users at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/rules-users
>>
>
> _______________________________________________
> rules-users mailing list
> rules-users at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/rules-users
>
>
>
> _______________________________________________
> rules-users mailing list
> rules-users at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/rules-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/rules-users/attachments/20130224/b2322f8c/attachment.html
More information about the rules-users
mailing list