[rules-users] Help with MR3
Mark Proctor
mproctor at codehaus.org
Wed Jul 4 12:42:25 EDT 2007
I think it's 3. I notice that while we lock existing WMs during package
addition, we don't block new WMs being created. I've added syncronised
blocks on the package map in the relevant places, I hope this fixes the
issue.
http://jira.jboss.org/jira/browse/JBRULES-971
Could you try trunk and let me know?
Mark
Mark Proctor wrote:
> actually I thought i tracked it down, was wrong. Anyway once the
> RuleBase is built the "ObjectHashMap objectTypeNodes" in Rete should
> not change....Only I things I can think of are:
> 1) Some how a thread is seeing a stale version of the map
> 2) the rulebase is getting updated while propagation is happening,
> maybe combine with an issue from 1. In reality the rulebase addition
> should obey the standard locking mechanism and stop this from happening.
> 3) The initial RuleBase hasn't finished building yet, and threads are
> already being spun off to assert data.
>
> I have a feeling its 3...
>
> Mark
> Mark Proctor wrote:
>> ok, that might be it. We generate code in a singleton classloader, I
>> suspect that each thread is generating its' own getters. I suspect
>> that created singleton classloader is not getting GCd and releasing
>> the perm gen. I'm not sure how to fix this. You can check this
>> yourself by attaching a jprofiler instance (available for free trial)
>> and looking at the object counts.
>>
>> I've tracked down the concurrency issue. The Rete node has a HashMap
>> of ObjectTypeNodes that is built on the fly, that was global to the
>> rulebase. I'll have to make it local to the working Memory. I'll fix
>> that today.
>>
>> Mark
>> s erel wrote:
>>> Our server creates hundreds of stateful rule sessions concurrently.
>>> Each created rule seesion is specific to a thread. We did not encounter
>>> any memory problems with the previous version. As I've said,
>>> it's difficult (actually, impossible) for me to provide you with a
>>> self contained example
>>> since the drl is complicated and contains dozens of complex objects.
>>> However, I am ready to provide you with any information that can help
>>> us in that manner.
>>> I've also mentioned in the earlier threads the concurrency problem
>>> I've encountered with AbstractHashTable.
>>> Can this has something to do with it?
>>>
>>>
>>> On 7/4/07, *Mark Proctor* <mproctor at codehaus.org
>>> <mailto:mproctor at codehaus.org>> wrote:
>>>
>>> I don't beleive there is anything in 4.0 that is going to cause
>>> such quick loss of permgen. Can you create a self contained
>>> example that illustrates this behaviour? So we can reproduce this?
>>>
>>> Mark
>>>
>>> s erel wrote:
>>>> We've tried to increase the permGen to 256mb. It did not help
>>>> and the space run out really fast.
>>>> Regarding MVEL, is turning code generation off something that
>>>> can (or will) be done with a configuration parameter/factory
>>>> method or do I need to track down all the places in the code?
>>>>
>>>> We did not experience such memory behavior with the previous
>>>> version we used (3.06) when running the same tests.
>>>> Bugs in 3.06 (no longer present in 4M3) are forcing us to upgrade.
>>>>
>>>> Is there another reason for such behaviour?
>>>> Should we wait for release candidate?
>>>>
>>>>
>>>> On 7/4/07, *Mark Proctor* <mproctor at codehaus.org
>>>> <mailto:mproctor at codehaus.org>> wrote:
>>>>
>>>> increase your perm gen space,or use the MVEL dialect with
>>>> code generation off.
>>>>
>>>>
>>>> Mark
>>>> s erel wrote:
>>>>> Hello,
>>>>>
>>>>> During capacity tests we've received permGen OOM
>>>>> exception. The occupied space in the permGen area
>>>>> increases rapidly. Any opinions?
>>>>>
>>>>>
>>>>> On 7/3/07, *s erel* <erelsagi at gmail.com
>>>>> <mailto:erelsagi at gmail.com>> wrote:
>>>>>
>>>>> In our project we are creating a
>>>>> StatefulRuleSession and saving it in a per-thread
>>>>> context (i.e. Each thread has it's own
>>>>> StatefulRuleSession):
>>>>>
>>>>> ruleServiceProvider.getRuleRuntime().createRuleSession(contextName,
>>>>> properties, RuleRuntime.STATEFUL_SESSION_TYPE);
>>>>>
>>>>> When a thread session ends, we are calling release on
>>>>> the previously created StatefulRuleSession.
>>>>>
>>>>>
>>>>> Changing the following lines:
>>>>>
>>>>>
>>>>> public abstract class AbstractHashTable
>>>>>
>>>>>
>>>>> ...
>>>>>
>>>>> public Iterator iterator() {
>>>>> // if ( this.iterator == null ) {
>>>>> // this.iterator = new HashTableIterator(
>>>>> this );
>>>>> // }
>>>>> //
>>>>> // this.iterator.reset();
>>>>> // return this.iterator;
>>>>>
>>>>> HashTableIterator iterator = new
>>>>> HashTableIterator(this);
>>>>> iterator.reset();
>>>>>
>>>>> return iterator;
>>>>> }
>>>>>
>>>>> Seems to solve the problem I've encountered. What's
>>>>> your opinion?
>>>>>
>>>>>
>>>>> On 7/2/07, *Mark Proctor* <mproctor at codehaus.org
>>>>> <mailto:mproctor at codehaus.org>> wrote:
>>>>>
>>>>> a working memory should be single threaded, so not
>>>>> sure how this could be a race condition?
>>>>>
>>>>> Mark
>>>>>
>>>>> s erel wrote:
>>>>>> I've done a little debugging. The code fails
>>>>>> in the following segment:
>>>>>>
>>>>>> public static class HashTableIterator
>>>>>> ...
>>>>>> while ( this.entry == null ) {
>>>>>> this.row++;
>>>>>> if ( this.row == this.length ) {
>>>>>> return null;
>>>>>> }
>>>>>> this.entry =
>>>>>> this.table[this.row]; *// ---> index out of
>>>>>> bounds exception*
>>>>>> }
>>>>>> }
>>>>>>
>>>>>> this.row has the same value as this.length
>>>>>> despite the condition above it. Probably a race
>>>>>> condition issue.
>>>>>>
>>>>>>
>>>>>> On 7/2/07, *Mark Proctor* <mproctor at codehaus.org
>>>>>> <mailto:mproctor at codehaus.org>> wrote:
>>>>>>
>>>>>> Not really :(
>>>>>>
>>>>>> In your situation I tend to keep removing
>>>>>> rules and data while still making sure the
>>>>>> error happens, to get it down to a minimum.
>>>>>> Please do try, as this isn't an error that
>>>>>> should happen. Or alterntaively you can open
>>>>>> drools-core and drools-compiler in eclipse
>>>>>> and execuse and debug this yourself - in your
>>>>>> situation this might best. you can put in a
>>>>>> breakpoint to listen for that particular
>>>>>> exception.
>>>>>>
>>>>>> Mark
>>>>>>
>>>>>> s erel wrote:
>>>>>>> It's hard for me to provide a self contained
>>>>>>> project. The drl is long and uses several
>>>>>>> business objects. It's the same drl as we've
>>>>>>> been using for 306 minus the keyword changes.
>>>>>>> Is there anything else i can check or
>>>>>>> provide you in order to solve this matter.
>>>>>>>
>>>>>>> Thanks
>>>>>>>
>>>>>>> On 7/1/07, *Mark Proctor*
>>>>>>> <mproctor at codehaus.org
>>>>>>> <mailto:mproctor at codehaus.org>> wrote:
>>>>>>>
>>>>>>> Can you provide us a self contained
>>>>>>> project which creates this error? Unless
>>>>>>> we can recreate it, it will be very hard
>>>>>>> to track it down. Please attach the
>>>>>>> project to a jira and we'll make it a
>>>>>>> priority.
>>>>>>>
>>>>>>> Mark
>>>>>>> s erel wrote:
>>>>>>>> Hello,
>>>>>>>>
>>>>>>>> I've just started integrating MR3 into
>>>>>>>> my project (I've previously
>>>>>>>> used 3.06). The drl compiles and
>>>>>>>> everything seems fine, but during
>>>>>>>> tests the following exception is thrown
>>>>>>>> for time to time:
>>>>>>>>
>>>>>>>> java.lang.ArrayIndexOutOfBoundsException:
>>>>>>>> 17
>>>>>>>> at
>>>>>>>> org.drools.util.AbstractHashTable$HashTableIterator.next(AbstractHashTable.java:250)
>>>>>>>> at
>>>>>>>> org.drools.reteoo.Rete$ObjectTypeConf.buildCache(Rete.java:434)
>>>>>>>> at
>>>>>>>> org.drools.reteoo.Rete$ObjectTypeConf.getObjectTypeNodes(Rete.java:425)
>>>>>>>> at
>>>>>>>> org.drools.reteoo.Rete.assertObject(Rete.java:172)
>>>>>>>> at
>>>>>>>> org.drools.reteoo.ReteooRuleBase.assertObject(ReteooRuleBase.java:190)
>>>>>>>> at
>>>>>>>> org.drools.reteoo.ReteooWorkingMemory$WorkingMemoryReteAssertAction.execute
>>>>>>>> (ReteooWorkingMemory.java:163)
>>>>>>>> at
>>>>>>>> org.drools.common.AbstractWorkingMemory.executeQueuedActions(AbstractWorkingMemory.java:1135)
>>>>>>>> at
>>>>>>>> org.drools.common.AbstractWorkingMemory.insert(AbstractWorkingMemory.java:781)
>>>>>>>> at
>>>>>>>> org.drools.common.AbstractWorkingMemory.insert(AbstractWorkingMemory.java:584)
>>>>>>>> at
>>>>>>>> org.drools.jsr94.rules.StatefulRuleSessionImpl.addObject(StatefulRuleSessionImpl.java:162)
>>>>>>>>
>>>>>>>> This only happens during high load tests.
>>>>>>>> Can anyone help me?
>>>>>>>>
>>>>>>>> Thanks
>>>>>>>>
>>>>>>>> ------------------------------------------------------------------------
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> rules-users mailing list
>>>>>>>> rules-users at lists.jboss.org <mailto:rules-users at lists.jboss.org>
>>>>>>>> https://lists.jboss.org/mailman/listinfo/rules-users
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> rules-users mailing list
>>>>>>> rules-users at lists.jboss.org
>>>>>>> <mailto:rules-users at lists.jboss.org>
>>>>>>> https://lists.jboss.org/mailman/listinfo/rules-users
>>>>>>> <https://lists.jboss.org/mailman/listinfo/rules-users>
>>>>>>>
>>>>>>>
>>>>>>> ------------------------------------------------------------------------
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> rules-users mailing list
>>>>>>> rules-users at lists.jboss.org <mailto:rules-users at lists.jboss.org>
>>>>>>> https://lists.jboss.org/mailman/listinfo/rules-users
>>>>>>>
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> rules-users mailing list
>>>>>> rules-users at lists.jboss.org
>>>>>> <mailto:rules-users at lists.jboss.org>
>>>>>> https://lists.jboss.org/mailman/listinfo/rules-users
>>>>>> <https://lists.jboss.org/mailman/listinfo/rules-users>
>>>>>>
>>>>>>
>>>>>> ------------------------------------------------------------------------
>>>>>>
>>>>>> _______________________________________________
>>>>>> rules-users mailing list
>>>>>> rules-users at lists.jboss.org <mailto:rules-users at lists.jboss.org>
>>>>>> https://lists.jboss.org/mailman/listinfo/rules-users
>>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> rules-users mailing list
>>>>> rules-users at lists.jboss.org
>>>>> <mailto:rules-users at lists.jboss.org>
>>>>> https://lists.jboss.org/mailman/listinfo/rules-users
>>>>> <https://lists.jboss.org/mailman/listinfo/rules-users>
>>>>>
>>>>>
>>>>>
>>>>> ------------------------------------------------------------------------
>>>>>
>>>>> _______________________________________________
>>>>> rules-users mailing list
>>>>> rules-users at lists.jboss.org <mailto:rules-users at lists.jboss.org>
>>>>> https://lists.jboss.org/mailman/listinfo/rules-users
>>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> rules-users mailing list
>>>> rules-users at lists.jboss.org
>>>> <mailto:rules-users at lists.jboss.org>
>>>> https://lists.jboss.org/mailman/listinfo/rules-users
>>>> <https://lists.jboss.org/mailman/listinfo/rules-users>
>>>>
>>>>
>>>> ------------------------------------------------------------------------
>>>>
>>>> _______________________________________________
>>>> rules-users mailing list
>>>> rules-users at lists.jboss.org <mailto:rules-users at lists.jboss.org>
>>>> https://lists.jboss.org/mailman/listinfo/rules-users
>>>>
>>>
>>>
>>> _______________________________________________
>>> rules-users mailing list
>>> rules-users at lists.jboss.org <mailto:rules-users at lists.jboss.org>
>>> https://lists.jboss.org/mailman/listinfo/rules-users
>>> <https://lists.jboss.org/mailman/listinfo/rules-users>
>>>
>>>
>>> ------------------------------------------------------------------------
>>>
>>> _______________________________________________
>>> rules-users mailing list
>>> rules-users at lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/rules-users
>>>
>>
>> ------------------------------------------------------------------------
>>
>> _______________________________________________
>> rules-users mailing list
>> rules-users at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/rules-users
>>
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> rules-users mailing list
> rules-users at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/rules-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/rules-users/attachments/20070704/cca7854e/attachment.html
More information about the rules-users
mailing list