[rules-users] Help with MR3
Mark Proctor
mproctor at codehaus.org
Thu Jul 5 09:33:11 EDT 2007
mic, has been working on this all day and cannot reproduce it. spinning
off a hundred threads and sesssions; you'll have to provide some code
that recreates the situation for us.
Mark
s erel wrote:
> Please let me know if the permGen issue can be fixed somehow.
> Thanks
>
>
> On 7/5/07, *Mark Proctor* <mproctor at codehaus.org
> <mailto:mproctor at codehaus.org>> wrote:
>
> actually, yes you are right - that was indeed the problem. That
> iterator is normally only used by the working memory, which is
> single threaded, but in this case its on the rulebase, which is
> not suppose to hold state. I'll fix this now, well found :)
>
> Mark
>
> s erel wrote:
>> Do you know if the fix I mentioned (returning a new iterator each
>> time) solves the problem or only mask it?
>> We create the initial RuleBase on server start-up.
>>
>> On 7/4/07, *Mark Proctor* <mproctor at codehaus.org
>> <mailto:mproctor at codehaus.org>> wrote:
>>
>> actually I thought i tracked it down, was wrong. Anyway once
>> the RuleBase is built the "ObjectHashMap objectTypeNodes" in
>> Rete should not change....Only I things I can think of are:
>> 1) Some how a thread is seeing a stale version of the map
>> 2) the rulebase is getting updated while propagation is
>> happening, maybe combine with an issue from 1. In reality the
>> rulebase addition should obey the standard locking mechanism
>> and stop this from happening.
>> 3) The initial RuleBase hasn't finished building yet, and
>> threads are already being spun off to assert data.
>>
>> I have a feeling its 3...
>>
>> Mark
>>
>> Mark Proctor wrote:
>>> ok, that might be it. We generate code in a singleton
>>> classloader, I suspect that each thread is generating its'
>>> own getters. I suspect that created singleton classloader is
>>> not getting GCd and releasing the perm gen. I'm not sure how
>>> to fix this. You can check this yourself by attaching a
>>> jprofiler instance (available for free trial) and looking at
>>> the object counts.
>>>
>>> I've tracked down the concurrency issue. The Rete node has a
>>> HashMap of ObjectTypeNodes that is built on the fly, that
>>> was global to the rulebase. I'll have to make it local to
>>> the working Memory. I'll fix that today.
>>>
>>> Mark
>>> s erel wrote:
>>>> Our server creates hundreds of stateful rule sessions
>>>> concurrently. Each created rule seesion is specific to a
>>>> thread. We did not encounter
>>>> any memory problems with the previous version. As I've
>>>> said, it's difficult (actually, impossible) for me to
>>>> provide you with a self contained example
>>>> since the drl is complicated and contains dozens of complex
>>>> objects. However, I am ready to provide you with any
>>>> information that can help
>>>> us in that manner.
>>>> I've also mentioned in the earlier threads the concurrency
>>>> problem I've encountered with AbstractHashTable.
>>>> Can this has something to do with it?
>>>>
>>>>
>>>> On 7/4/07, *Mark Proctor* <mproctor at codehaus.org
>>>> <mailto:mproctor at codehaus.org>> wrote:
>>>>
>>>> I don't beleive there is anything in 4.0 that is going
>>>> to cause such quick loss of permgen. Can you create a
>>>> self contained example that illustrates this behaviour?
>>>> So we can reproduce this?
>>>>
>>>> Mark
>>>>
>>>> s erel wrote:
>>>>> We've tried to increase the permGen to 256mb. It did
>>>>> not help and the space run out really fast.
>>>>> Regarding MVEL, is turning code generation
>>>>> off something that can (or will) be done with a
>>>>> configuration parameter/factory method or do I need to
>>>>> track down all the places in the code?
>>>>>
>>>>> We did not experience such memory behavior with the
>>>>> previous version we used (3.06) when running the same
>>>>> tests.
>>>>> Bugs in 3.06 (no longer present in 4M3) are forcing us
>>>>> to upgrade.
>>>>>
>>>>> Is there another reason for such behaviour?
>>>>> Should we wait for release candidate?
>>>>>
>>>>>
>>>>> On 7/4/07, *Mark Proctor* <mproctor at codehaus.org
>>>>> <mailto:mproctor at codehaus.org>> wrote:
>>>>>
>>>>> increase your perm gen space,or use the MVEL
>>>>> dialect with code generation off.
>>>>>
>>>>>
>>>>> Mark
>>>>> s erel wrote:
>>>>>> Hello,
>>>>>>
>>>>>> During capacity tests we've received permGen OOM
>>>>>> exception. The occupied space in the permGen area
>>>>>> increases rapidly. Any opinions?
>>>>>>
>>>>>>
>>>>>> On 7/3/07, *s erel* <erelsagi at gmail.com
>>>>>> <mailto:erelsagi at gmail.com>> wrote:
>>>>>>
>>>>>> In our project we are creating a
>>>>>> StatefulRuleSession and saving it in a
>>>>>> per-thread context (i.e. Each thread has it's
>>>>>> own StatefulRuleSession):
>>>>>>
>>>>>> ruleServiceProvider.getRuleRuntime().createRuleSession(contextName,
>>>>>> properties, RuleRuntime.STATEFUL_SESSION_TYPE);
>>>>>>
>>>>>> When a thread session ends, we are calling
>>>>>> release on the previously created
>>>>>> StatefulRuleSession.
>>>>>>
>>>>>>
>>>>>> Changing the following lines:
>>>>>>
>>>>>>
>>>>>> public abstract class AbstractHashTable
>>>>>>
>>>>>>
>>>>>> ...
>>>>>>
>>>>>> public Iterator iterator() {
>>>>>> // if ( this.iterator == null ) {
>>>>>> // this.iterator = new
>>>>>> HashTableIterator( this );
>>>>>> // }
>>>>>> //
>>>>>> // this.iterator.reset();
>>>>>> // return this.iterator;
>>>>>>
>>>>>> HashTableIterator iterator = new
>>>>>> HashTableIterator(this);
>>>>>> iterator.reset();
>>>>>>
>>>>>> return iterator;
>>>>>> }
>>>>>>
>>>>>> Seems to solve the problem I've encountered.
>>>>>> What's your opinion?
>>>>>>
>>>>>>
>>>>>> On 7/2/07, *Mark Proctor*
>>>>>> <mproctor at codehaus.org
>>>>>> <mailto:mproctor at codehaus.org>> wrote:
>>>>>>
>>>>>> a working memory should be single
>>>>>> threaded, so not sure how this could be a
>>>>>> race condition?
>>>>>>
>>>>>> Mark
>>>>>>
>>>>>> s erel wrote:
>>>>>>> I've done a little debugging. The code
>>>>>>> fails in the following segment:
>>>>>>>
>>>>>>> public static class HashTableIterator
>>>>>>> ...
>>>>>>> while ( this.entry == null ) {
>>>>>>> this.row++;
>>>>>>> if ( this.row ==
>>>>>>> this.length ) {
>>>>>>> return null;
>>>>>>> }
>>>>>>> this.entry =
>>>>>>> this.table[this.row]; *// ---> index out
>>>>>>> of bounds exception*
>>>>>>> }
>>>>>>> }
>>>>>>>
>>>>>>> this.row has the same value as
>>>>>>> this.length despite the condition above
>>>>>>> it. Probably a race condition issue.
>>>>>>>
>>>>>>>
>>>>>>> On 7/2/07, *Mark Proctor*
>>>>>>> <mproctor at codehaus.org
>>>>>>> <mailto:mproctor at codehaus.org>> wrote:
>>>>>>>
>>>>>>> Not really :(
>>>>>>>
>>>>>>> In your situation I tend to keep
>>>>>>> removing rules and data while still
>>>>>>> making sure the error happens, to
>>>>>>> get it down to a minimum. Please do
>>>>>>> try, as this isn't an error that
>>>>>>> should happen. Or alterntaively you
>>>>>>> can open drools-core and
>>>>>>> drools-compiler in eclipse and
>>>>>>> execuse and debug this yourself - in
>>>>>>> your situation this might best. you
>>>>>>> can put in a breakpoint to listen
>>>>>>> for that particular exception.
>>>>>>>
>>>>>>> Mark
>>>>>>>
>>>>>>> s erel wrote:
>>>>>>>> It's hard for me to provide a self
>>>>>>>> contained project. The drl is long
>>>>>>>> and uses several business objects.
>>>>>>>> It's the same drl as we've been
>>>>>>>> using for 306 minus the keyword
>>>>>>>> changes.
>>>>>>>> Is there anything else i can check
>>>>>>>> or provide you in order to solve
>>>>>>>> this matter.
>>>>>>>>
>>>>>>>> Thanks
>>>>>>>>
>>>>>>>> On 7/1/07, *Mark Proctor*
>>>>>>>> <mproctor at codehaus.org
>>>>>>>> <mailto:mproctor at codehaus.org>> wrote:
>>>>>>>>
>>>>>>>> Can you provide us a self
>>>>>>>> contained project which creates
>>>>>>>> this error? Unless we can
>>>>>>>> recreate it, it will be very
>>>>>>>> hard to track it down. Please
>>>>>>>> attach the project to a jira
>>>>>>>> and we'll make it a priority.
>>>>>>>>
>>>>>>>> Mark
>>>>>>>> s erel wrote:
>>>>>>>>> Hello,
>>>>>>>>>
>>>>>>>>> I've just started
>>>>>>>>> integrating MR3 into my
>>>>>>>>> project (I've previously
>>>>>>>>> used 3.06). The drl compiles
>>>>>>>>> and everything seems fine, but
>>>>>>>>> during
>>>>>>>>> tests the following exception
>>>>>>>>> is thrown for time to time:
>>>>>>>>>
>>>>>>>>> java.lang.ArrayIndexOutOfBoundsException:
>>>>>>>>> 17
>>>>>>>>> at
>>>>>>>>> org.drools.util.AbstractHashTable$HashTableIterator.next(AbstractHashTable.java:250)
>>>>>>>>> at
>>>>>>>>> org.drools.reteoo.Rete$ObjectTypeConf.buildCache(Rete.java:434)
>>>>>>>>> at
>>>>>>>>> org.drools.reteoo.Rete$ObjectTypeConf.getObjectTypeNodes(Rete.java:425)
>>>>>>>>> at
>>>>>>>>> org.drools.reteoo.Rete.assertObject(Rete.java:172)
>>>>>>>>> at
>>>>>>>>> org.drools.reteoo.ReteooRuleBase.assertObject(ReteooRuleBase.java:190)
>>>>>>>>> at
>>>>>>>>> org.drools.reteoo.ReteooWorkingMemory$WorkingMemoryReteAssertAction.execute
>>>>>>>>> (ReteooWorkingMemory.java:163)
>>>>>>>>> at
>>>>>>>>> org.drools.common.AbstractWorkingMemory.executeQueuedActions(AbstractWorkingMemory.java:1135)
>>>>>>>>> at
>>>>>>>>> org.drools.common.AbstractWorkingMemory.insert(AbstractWorkingMemory.java:781)
>>>>>>>>> at
>>>>>>>>> org.drools.common.AbstractWorkingMemory.insert(AbstractWorkingMemory.java:584)
>>>>>>>>> at
>>>>>>>>> org.drools.jsr94.rules.StatefulRuleSessionImpl.addObject(StatefulRuleSessionImpl.java:162)
>>>>>>>>>
>>>>>>>>> This only happens during high
>>>>>>>>> load tests.
>>>>>>>>> Can anyone help me?
>>>>>>>>>
>>>>>>>>> Thanks
>>>>>>>>>
>>>>>>>>> ------------------------------------------------------------------------
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> rules-users mailing list
>>>>>>>>> rules-users at lists.jboss.org <mailto:rules-users at lists.jboss.org>
>>>>>>>>> https://lists.jboss.org/mailman/listinfo/rules-users
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> rules-users mailing list
>>>>>>>> rules-users at lists.jboss.org
>>>>>>>> <mailto:rules-users at lists.jboss.org>
>>>>>>>> https://lists.jboss.org/mailman/listinfo/rules-users
>>>>>>>> <https://lists.jboss.org/mailman/listinfo/rules-users>
>>>>>>>>
>>>>>>>>
>>>>>>>> ------------------------------------------------------------------------
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> rules-users mailing list
>>>>>>>> rules-users at lists.jboss.org <mailto:rules-users at lists.jboss.org>
>>>>>>>> https://lists.jboss.org/mailman/listinfo/rules-users
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> rules-users mailing list
>>>>>>> rules-users at lists.jboss.org
>>>>>>> <mailto:rules-users at lists.jboss.org>
>>>>>>> https://lists.jboss.org/mailman/listinfo/rules-users
>>>>>>> <https://lists.jboss.org/mailman/listinfo/rules-users>
>>>>>>>
>>>>>>>
>>>>>>> ------------------------------------------------------------------------
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> rules-users mailing list
>>>>>>> rules-users at lists.jboss.org <mailto:rules-users at lists.jboss.org>
>>>>>>> https://lists.jboss.org/mailman/listinfo/rules-users
>>>>>>>
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> rules-users mailing list
>>>>>> rules-users at lists.jboss.org
>>>>>> <mailto:rules-users at lists.jboss.org>
>>>>>> https://lists.jboss.org/mailman/listinfo/rules-users
>>>>>> <https://lists.jboss.org/mailman/listinfo/rules-users>
>>>>>>
>>>>>>
>>>>>>
>>>>>> ------------------------------------------------------------------------
>>>>>>
>>>>>> _______________________________________________
>>>>>> rules-users mailing list
>>>>>> rules-users at lists.jboss.org <mailto:rules-users at lists.jboss.org>
>>>>>> https://lists.jboss.org/mailman/listinfo/rules-users
>>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> rules-users mailing list
>>>>> rules-users at lists.jboss.org
>>>>> <mailto:rules-users at lists.jboss.org>
>>>>> https://lists.jboss.org/mailman/listinfo/rules-users
>>>>> <https://lists.jboss.org/mailman/listinfo/rules-users>
>>>>>
>>>>>
>>>>> ------------------------------------------------------------------------
>>>>>
>>>>> _______________________________________________
>>>>> rules-users mailing list
>>>>> rules-users at lists.jboss.org <mailto:rules-users at lists.jboss.org>
>>>>> https://lists.jboss.org/mailman/listinfo/rules-users
>>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> rules-users mailing list
>>>> rules-users at lists.jboss.org
>>>> <mailto:rules-users at lists.jboss.org>
>>>> https://lists.jboss.org/mailman/listinfo/rules-users
>>>> <https://lists.jboss.org/mailman/listinfo/rules-users>
>>>>
>>>>
>>>> ------------------------------------------------------------------------
>>>>
>>>> _______________________________________________
>>>> rules-users mailing list
>>>> rules-users at lists.jboss.org <mailto:rules-users at lists.jboss.org>
>>>> https://lists.jboss.org/mailman/listinfo/rules-users
>>>>
>>>
>>> ------------------------------------------------------------------------
>>>
>>> _______________________________________________
>>> rules-users mailing list
>>> rules-users at lists.jboss.org <mailto:rules-users at lists.jboss.org>
>>> https://lists.jboss.org/mailman/listinfo/rules-users
>>>
>>
>>
>> _______________________________________________
>> rules-users mailing list
>> rules-users at lists.jboss.org <mailto:rules-users at lists.jboss.org>
>> https://lists.jboss.org/mailman/listinfo/rules-users
>> <https://lists.jboss.org/mailman/listinfo/rules-users>
>>
>>
>> ------------------------------------------------------------------------
>>
>> _______________________________________________
>> rules-users mailing list
>> rules-users at lists.jboss.org <mailto:rules-users at lists.jboss.org>
>> https://lists.jboss.org/mailman/listinfo/rules-users
>>
>
>
> _______________________________________________
> rules-users mailing list
> rules-users at lists.jboss.org <mailto:rules-users at lists.jboss.org>
> https://lists.jboss.org/mailman/listinfo/rules-users
> <https://lists.jboss.org/mailman/listinfo/rules-users>
>
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> rules-users mailing list
> rules-users at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/rules-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/rules-users/attachments/20070705/adcb6f57/attachment.html
More information about the rules-users
mailing list