Hello, I've just started integrating MR3 into my project (I've previously used 3.06). The drl compiles and everything seems fine, but during tests the following exception is thrown for time to time: java.lang.ArrayIndexOutOfBoundsException: 17 at org.drools.util.AbstractHashTable$HashTableIterator.next(AbstractHashTable.java:250) at org.drools.reteoo.Rete$ObjectTypeConf.buildCache(Rete.java:434) at org.drools.reteoo.Rete$ObjectTypeConf.getObjectTypeNodes(Rete.java:425) at org.drools.reteoo.Rete.assertObject(Rete.java:172) at org.drools.reteoo.ReteooRuleBase.assertObject(ReteooRuleBase.java:190) at org.drools.reteoo.ReteooWorkingMemory$WorkingMemoryReteAssertAction.execute (ReteooWorkingMemory.java:163) at org.drools.common.AbstractWorkingMemory.executeQueuedActions(AbstractWorkingMemory.java:1135) at org.drools.common.AbstractWorkingMemory.insert(AbstractWorkingMemory.java:781) at org.drools.common.AbstractWorkingMemory.insert(AbstractWorkingMemory.java:584) at org.drools.jsr94.rules.StatefulRuleSessionImpl.addObject(StatefulRuleSessionImpl.java:162) This only happens during high load tests. Can anyone help me? Thanks ------------------------------------------------------------------------ _______________________________________________ rules-users mailing list rules-users(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/rules-users

It's hard for me to provide a self contained project. The drl is long and uses several business objects. It's the same drl as we've been using for 306 minus the keyword changes. Is there anything else i can check or provide you in order to solve this matter. Thanks On 7/1/07, *Mark Proctor* <mproctor(a)codehaus.org <mailto:mproctor@codehaus.org>> wrote: Can you provide us a self contained project which creates this error? Unless we can recreate it, it will be very hard to track it down. Please attach the project to a jira and we'll make it a priority. Mark s erel wrote: > Hello, > > I've just started integrating MR3 into my project (I've > previously used 3.06). The drl compiles and everything seems > fine, but during > tests the following exception is thrown for time to time: > > java.lang.ArrayIndexOutOfBoundsException: 17 > at > org.drools.util.AbstractHashTable$HashTableIterator.next(AbstractHashTable.java:250) > at org.drools.reteoo.Rete$ObjectTypeConf.buildCache(Rete.java:434) > at > org.drools.reteoo.Rete$ObjectTypeConf.getObjectTypeNodes(Rete.java:425) > at org.drools.reteoo.Rete.assertObject(Rete.java:172) > at > org.drools.reteoo.ReteooRuleBase.assertObject(ReteooRuleBase.java:190) > at > org.drools.reteoo.ReteooWorkingMemory$WorkingMemoryReteAssertAction.execute > (ReteooWorkingMemory.java:163) > at > org.drools.common.AbstractWorkingMemory.executeQueuedActions(AbstractWorkingMemory.java:1135) > at > org.drools.common.AbstractWorkingMemory.insert(AbstractWorkingMemory.java:781) > at > org.drools.common.AbstractWorkingMemory.insert(AbstractWorkingMemory.java:584) > at > org.drools.jsr94.rules.StatefulRuleSessionImpl.addObject(StatefulRuleSessionImpl.java:162) > > This only happens during high load tests. > Can anyone help me? > > Thanks > > ------------------------------------------------------------------------ > > _______________________________________________ > rules-users mailing list > rules-users(a)lists.jboss.org <mailto:rules-users@lists.jboss.org> > https://lists.jboss.org/mailman/listinfo/rules-users > _______________________________________________ rules-users mailing list rules-users(a)lists.jboss.org <mailto:rules-users@lists.jboss.org> https://lists.jboss.org/mailman/listinfo/rules-users ------------------------------------------------------------------------ _______________________________________________ rules-users mailing list rules-users(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/rules-users

I've done a little debugging. The code fails in the following segment: public static class HashTableIterator ... while ( this.entry == null ) { this.row++; if ( this.row == this.length ) { return null; } this.entry = this.table[this.row]; *// ---> index out of bounds exception* } } this.row has the same value as this.length despite the condition above it. Probably a race condition issue. On 7/2/07, *Mark Proctor* <mproctor(a)codehaus.org <mailto:mproctor@codehaus.org>> wrote: Not really :( In your situation I tend to keep removing rules and data while still making sure the error happens, to get it down to a minimum. Please do try, as this isn't an error that should happen. Or alterntaively you can open drools-core and drools-compiler in eclipse and execuse and debug this yourself - in your situation this might best. you can put in a breakpoint to listen for that particular exception. Mark s erel wrote: > It's hard for me to provide a self contained project. The drl > is long and uses several business objects. It's the same drl as > we've been using for 306 minus the keyword changes. > Is there anything else i can check or provide you in order to > solve this matter. > > Thanks > > On 7/1/07, *Mark Proctor* <mproctor(a)codehaus.org > <mailto:mproctor@codehaus.org>> wrote: > > Can you provide us a self contained project which creates > this error? Unless we can recreate it, it will be very hard > to track it down. Please attach the project to a jira and > we'll make it a priority. > > Mark > s erel wrote: >> Hello, >> >> I've just started integrating MR3 into my project (I've >> previously used 3.06). The drl compiles and everything seems >> fine, but during >> tests the following exception is thrown for time to time: >> >> java.lang.ArrayIndexOutOfBoundsException: 17 >> at >> org.drools.util.AbstractHashTable$HashTableIterator.next(AbstractHashTable.java:250) >> at >> org.drools.reteoo.Rete$ObjectTypeConf.buildCache(Rete.java:434) >> at >> org.drools.reteoo.Rete$ObjectTypeConf.getObjectTypeNodes(Rete.java:425) >> at org.drools.reteoo.Rete.assertObject(Rete.java:172) >> at >> org.drools.reteoo.ReteooRuleBase.assertObject(ReteooRuleBase.java:190) >> at >> org.drools.reteoo.ReteooWorkingMemory$WorkingMemoryReteAssertAction.execute >> (ReteooWorkingMemory.java:163) >> at >> org.drools.common.AbstractWorkingMemory.executeQueuedActions(AbstractWorkingMemory.java:1135) >> at >> org.drools.common.AbstractWorkingMemory.insert(AbstractWorkingMemory.java:781) >> at >> org.drools.common.AbstractWorkingMemory.insert(AbstractWorkingMemory.java:584) >> at >> org.drools.jsr94.rules.StatefulRuleSessionImpl.addObject(StatefulRuleSessionImpl.java:162) >> >> This only happens during high load tests. >> Can anyone help me? >> >> Thanks >> >> ------------------------------------------------------------------------ >> >> _______________________________________________ >> rules-users mailing list >> rules-users(a)lists.jboss.org <mailto:rules-users@lists.jboss.org> >> https://lists.jboss.org/mailman/listinfo/rules-users >> > > > > _______________________________________________ > rules-users mailing list > rules-users(a)lists.jboss.org <mailto:rules-users@lists.jboss.org> > https://lists.jboss.org/mailman/listinfo/rules-users > <https://lists.jboss.org/mailman/listinfo/rules-users> > > > ------------------------------------------------------------------------ > > _______________________________________________ > rules-users mailing list > rules-users(a)lists.jboss.org <mailto:rules-users@lists.jboss.org> > https://lists.jboss.org/mailman/listinfo/rules-users > _______________________________________________ rules-users mailing list rules-users(a)lists.jboss.org <mailto:rules-users@lists.jboss.org> https://lists.jboss.org/mailman/listinfo/rules-users <https://lists.jboss.org/mailman/listinfo/rules-users> ------------------------------------------------------------------------ _______________________________________________ rules-users mailing list rules-users(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/rules-users

In our project we are creating a StatefulRuleSession and saving it in a per-thread context (i.e. Each thread has it's own StatefulRuleSession): ruleServiceProvider.getRuleRuntime().createRuleSession(contextName, properties, RuleRuntime.STATEFUL_SESSION_TYPE); When a thread session ends, we are calling release on the previously created StatefulRuleSession. Changing the following lines: public abstract class AbstractHashTable ... public Iterator iterator() { // if ( this.iterator == null ) { // this.iterator = new HashTableIterator( this ); // } // // this.iterator.reset(); // return this.iterator; HashTableIterator iterator = new HashTableIterator(this); iterator.reset(); return iterator; } Seems to solve the problem I've encountered. What's your opinion? On 7/2/07, Mark Proctor <mproctor(a)codehaus.org > wrote: > > a working memory should be single threaded, so not sure how this could > be a race condition? > > Mark > s erel wrote: > > I've done a little debugging. The code fails in the following segment: > > public static class HashTableIterator > ... > while ( this.entry == null ) { > this.row++; > if ( this.row == this.length ) { > return null; > } > this.entry = this.table[this.row]; *// ---> index > out of bounds exception* > } > } > > this.row has the same value as this.length despite the condition above > it. Probably a race condition issue. > > > On 7/2/07, Mark Proctor <mproctor(a)codehaus.org > wrote: > > > > Not really :( > > > > In your situation I tend to keep removing rules and data while still > > making sure the error happens, to get it down to a minimum. Please do try, > > as this isn't an error that should happen. Or alterntaively you can open > > drools-core and drools-compiler in eclipse and execuse and debug this > > yourself - in your situation this might best. you can put in a breakpoint to > > listen for that particular exception. > > > > Mark > > s erel wrote: > > > > It's hard for me to provide a self contained project. The drl is long > > and uses several business objects. It's the same drl as we've been using for > > 306 minus the keyword changes. > > Is there anything else i can check or provide you in order to solve > > this matter. > > > > Thanks > > > > On 7/1/07, Mark Proctor <mproctor(a)codehaus.org > wrote: > > > > > > Can you provide us a self contained project which creates this > > > error? Unless we can recreate it, it will be very hard to track it down. > > > Please attach the project to a jira and we'll make it a priority. > > > > > > Mark > > > s erel wrote: > > > > > > Hello, > > > > > > I've just started integrating MR3 into my project (I've previously > > > used 3.06). The drl compiles and everything seems fine, but during > > > tests the following exception is thrown for time to time: > > > > > > java.lang.ArrayIndexOutOfBoundsException: 17 > > > at org.drools.util.AbstractHashTable$HashTableIterator.next( > > > AbstractHashTable.java:250) > > > at org.drools.reteoo.Rete$ObjectTypeConf.buildCache(Rete.java:434) > > > at org.drools.reteoo.Rete$ObjectTypeConf.getObjectTypeNodes( > > > Rete.java:425) > > > at org.drools.reteoo.Rete.assertObject(Rete.java:172) > > > at org.drools.reteoo.ReteooRuleBase.assertObject( > > > ReteooRuleBase.java:190) > > > at > > > org.drools.reteoo.ReteooWorkingMemory$WorkingMemoryReteAssertAction.execute( > > > ReteooWorkingMemory.java:163) > > > at org.drools.common.AbstractWorkingMemory.executeQueuedActions( > > > AbstractWorkingMemory.java:1135) > > > at org.drools.common.AbstractWorkingMemory.insert( > > > AbstractWorkingMemory.java:781) > > > at org.drools.common.AbstractWorkingMemory.insert( > > > AbstractWorkingMemory.java:584) > > > at org.drools.jsr94.rules.StatefulRuleSessionImpl.addObject( > > > StatefulRuleSessionImpl.java:162) > > > > > > This only happens during high load tests. > > > Can anyone help me? > > > > > > Thanks > > > > > > > > > ------------------------------ > > > > > > _______________________________________________ > > > rules-users mailing listrules-users@lists.jboss.orghttps://lists.jboss.org/mailman/listinfo/rules-users > > > > > > > > > > > > > > > _______________________________________________ > > > rules-users mailing list > > > rules-users(a)lists.jboss.org > > > https://lists.jboss.org/mailman/listinfo/rules-users > > > > > > > > ------------------------------ > > > > _______________________________________________ > > rules-users mailing listrules-users@lists.jboss.orghttps://lists.jboss.org/mailman/listinfo/rules-users > > > > > > > > _______________________________________________ > > rules-users mailing list > > rules-users(a)lists.jboss.org > > https://lists.jboss.org/mailman/listinfo/rules-users > > > > > ------------------------------ > > _______________________________________________ > rules-users mailing listrules-users@lists.jboss.orghttps://lists.jboss.org/mailman/listinfo/rules-users > > > > _______________________________________________ > rules-users mailing list > rules-users(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/rules-users > >

increase your perm gen space,or use the MVEL dialect with code generation off. Mark s erel wrote: Hello, During capacity tests we've received permGen OOM exception. The occupied space in the permGen area increases rapidly. Any opinions? On 7/3/07, s erel <erelsagi(a)gmail.com&gt; wrote: > > In our project we are creating a StatefulRuleSession and saving it in a > per-thread context (i.e. Each thread has it's own StatefulRuleSession): > > ruleServiceProvider.getRuleRuntime().createRuleSession(contextName, > properties, RuleRuntime.STATEFUL_SESSION_TYPE); > > When a thread session ends, we are calling release on the previously > created StatefulRuleSession. > > > Changing the following lines: > > > public abstract class AbstractHashTable > > > ... > > public Iterator iterator() { > // if ( this.iterator == null ) { > // this.iterator = new HashTableIterator( this ); > // } > // > // this.iterator.reset(); > // return this.iterator; > > HashTableIterator iterator = new HashTableIterator(this); > iterator.reset(); > > return iterator; > } > Seems to solve the problem I've encountered. What's your opinion? > > > On 7/2/07, Mark Proctor <mproctor(a)codehaus.org > wrote: > > > > a working memory should be single threaded, so not sure how this could > > be a race condition? > > > > Mark > > s erel wrote: > > > > I've done a little debugging. The code fails in the following segment: > > > > public static class HashTableIterator > > ... > > while ( this.entry == null ) { > > this.row++; > > if ( this.row == this.length ) { > > return null; > > } > > this.entry = this.table[this.row]; *// ---> index > > out of bounds exception* > > } > > } > > > > this.row has the same value as this.length despite the condition above > > it. Probably a race condition issue. > > > > > > On 7/2/07, Mark Proctor <mproctor(a)codehaus.org > wrote: > > > > > > Not really :( > > > > > > In your situation I tend to keep removing rules and data while still > > > making sure the error happens, to get it down to a minimum. Please do try, > > > as this isn't an error that should happen. Or alterntaively you can open > > > drools-core and drools-compiler in eclipse and execuse and debug this > > > yourself - in your situation this might best. you can put in a breakpoint to > > > listen for that particular exception. > > > > > > Mark > > > s erel wrote: > > > > > > It's hard for me to provide a self contained project. The drl > > > is long and uses several business objects. It's the same drl as we've been > > > using for 306 minus the keyword changes. > > > Is there anything else i can check or provide you in order to solve > > > this matter. > > > > > > Thanks > > > > > > On 7/1/07, Mark Proctor <mproctor(a)codehaus.org > wrote: > > > > > > > > Can you provide us a self contained project which creates this > > > > error? Unless we can recreate it, it will be very hard to track it down. > > > > Please attach the project to a jira and we'll make it a priority. > > > > > > > > Mark > > > > s erel wrote: > > > > > > > > Hello, > > > > > > > > I've just started integrating MR3 into my project (I've previously > > > > used 3.06). The drl compiles and everything seems fine, but during > > > > > > > > tests the following exception is thrown for time to time: > > > > > > > > java.lang.ArrayIndexOutOfBoundsException: 17 > > > > at org.drools.util.AbstractHashTable$HashTableIterator.next( > > > > AbstractHashTable.java:250) > > > > at org.drools.reteoo.Rete$ObjectTypeConf.buildCache(Rete.java > > > > :434) > > > > at org.drools.reteoo.Rete$ObjectTypeConf.getObjectTypeNodes( > > > > Rete.java:425) > > > > at org.drools.reteoo.Rete.assertObject(Rete.java:172) > > > > at org.drools.reteoo.ReteooRuleBase.assertObject( > > > > ReteooRuleBase.java:190) > > > > at > > > > org.drools.reteoo.ReteooWorkingMemory$WorkingMemoryReteAssertAction.execute( > > > > ReteooWorkingMemory.java:163) > > > > at org.drools.common.AbstractWorkingMemory.executeQueuedActions( > > > > AbstractWorkingMemory.java:1135) > > > > at org.drools.common.AbstractWorkingMemory.insert( > > > > AbstractWorkingMemory.java:781) > > > > at org.drools.common.AbstractWorkingMemory.insert( > > > > AbstractWorkingMemory.java:584) > > > > at org.drools.jsr94.rules.StatefulRuleSessionImpl.addObject( > > > > StatefulRuleSessionImpl.java:162) > > > > > > > > This only happens during high load tests. > > > > Can anyone help me? > > > > > > > > Thanks > > > > > > > > > > > > ------------------------------ > > > > > > > > _______________________________________________ > > > > rules-users mailing listrules-users@lists.jboss.orghttps://lists.jboss.org/mailman/listinfo/rules-users > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > rules-users mailing list > > > > rules-users(a)lists.jboss.org > > > > https://lists.jboss.org/mailman/listinfo/rules-users > > > > > > > > > > > ------------------------------ > > > > > > _______________________________________________ > > > rules-users mailing listrules-users@lists.jboss.orghttps://lists.jboss.org/mailman/listinfo/rules-users > > > > > > > > > > > > _______________________________________________ > > > rules-users mailing list > > > rules-users(a)lists.jboss.org > > > https://lists.jboss.org/mailman/listinfo/rules-users > > > > > > > > ------------------------------ > > > > _______________________________________________ > > rules-users mailing listrules-users@lists.jboss.orghttps://lists.jboss.org/mailman/listinfo/rules-users > > > > > > > > _______________________________________________ > > rules-users mailing list > > rules-users(a)lists.jboss.org > > https://lists.jboss.org/mailman/listinfo/rules-users > > > > > ------------------------------ _______________________________________________ rules-users mailing listrules-users@lists.jboss.orghttps://lists.jboss.org/mailman/listinfo/rules-users _______________________________________________ rules-users mailing list rules-users(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/rules-users

I don't beleive there is anything in 4.0 that is going to cause such quick loss of permgen. Can you create a self contained example that illustrates this behaviour? So we can reproduce this? Mark s erel wrote: We've tried to increase the permGen to 256mb. It did not help and the space run out really fast. Regarding MVEL, is turning code generation off something that can (or will) be done with a configuration parameter/factory method or do I need to track down all the places in the code? We did not experience such memory behavior with the previous version we used (3.06) when running the same tests. Bugs in 3.06 (no longer present in 4M3) are forcing us to upgrade. Is there another reason for such behaviour? Should we wait for release candidate? On 7/4/07, Mark Proctor <mproctor(a)codehaus.org&gt; wrote: > > increase your perm gen space,or use the MVEL dialect with code > generation off. > > Mark > s erel wrote: > > Hello, > > During capacity tests we've received permGen OOM exception. The occupied > space in the permGen area increases rapidly. Any opinions? > > > On 7/3/07, s erel <erelsagi(a)gmail.com&gt; wrote: > > > > In our project we are creating a StatefulRuleSession and saving it in > > a per-thread context (i.e. Each thread has it's own > > StatefulRuleSession): > > > > ruleServiceProvider.getRuleRuntime().createRuleSession(contextName, > > properties, RuleRuntime.STATEFUL_SESSION_TYPE); > > > > When a thread session ends, we are calling release on the previously > > created StatefulRuleSession. > > > > > > Changing the following lines: > > > > > > public abstract class AbstractHashTable > > > > > > ... > > > > public Iterator iterator() { > > // if ( this.iterator == null ) { > > // this.iterator = new HashTableIterator( this ); > > // } > > // > > // this.iterator.reset(); > > // return this.iterator; > > > > HashTableIterator iterator = new HashTableIterator(this); > > iterator.reset(); > > > > return iterator; > > } > > Seems to solve the problem I've encountered. What's your opinion? > > > > > > On 7/2/07, Mark Proctor <mproctor(a)codehaus.org > wrote: > > > > > > a working memory should be single threaded, so not sure how this > > > could be a race condition? > > > > > > Mark > > > s erel wrote: > > > > > > I've done a little debugging. The code fails in the following > > > segment: > > > > > > public static class HashTableIterator > > > ... > > > while ( this.entry == null ) { > > > this.row++; > > > if ( this.row == this.length ) { > > > return null; > > > } > > > this.entry = this.table[this.row]; *// ---> > > > index out of bounds exception* > > > } > > > } > > > > > > this.row has the same value as this.length despite the condition > > > above it. Probably a race condition issue. > > > > > > > > > On 7/2/07, Mark Proctor <mproctor(a)codehaus.org > wrote: > > > > > > > > Not really :( > > > > > > > > In your situation I tend to keep removing rules and data while > > > > still making sure the error happens, to get it down to a minimum. Please do > > > > try, as this isn't an error that should happen. Or alterntaively you can > > > > open drools-core and drools-compiler in eclipse and execuse and debug this > > > > yourself - in your situation this might best. you can put in a breakpoint to > > > > listen for that particular exception. > > > > > > > > Mark > > > > s erel wrote: > > > > > > > > It's hard for me to provide a self contained project. The drl > > > > is long and uses several business objects. It's the same drl as we've been > > > > using for 306 minus the keyword changes. > > > > Is there anything else i can check or provide you in order to > > > > solve this matter. > > > > > > > > Thanks > > > > > > > > On 7/1/07, Mark Proctor <mproctor(a)codehaus.org > wrote: > > > > > > > > > > Can you provide us a self contained project which creates this > > > > > error? Unless we can recreate it, it will be very hard to track it down. > > > > > Please attach the project to a jira and we'll make it a priority. > > > > > > > > > > Mark > > > > > s erel wrote: > > > > > > > > > > Hello, > > > > > > > > > > I've just started integrating MR3 into my project (I've > > > > > previously used 3.06). The drl compiles and everything seems > > > > > fine, but during > > > > > tests the following exception is thrown for time to time: > > > > > > > > > > java.lang.ArrayIndexOutOfBoundsException: 17 > > > > > at org.drools.util.AbstractHashTable$HashTableIterator.next( > > > > > AbstractHashTable.java:250) > > > > > at org.drools.reteoo.Rete$ObjectTypeConf.buildCache(Rete.java > > > > > :434) > > > > > at org.drools.reteoo.Rete$ObjectTypeConf.getObjectTypeNodes( > > > > > Rete.java:425) > > > > > at org.drools.reteoo.Rete.assertObject(Rete.java:172) > > > > > at org.drools.reteoo.ReteooRuleBase.assertObject( > > > > > ReteooRuleBase.java:190) > > > > > at > > > > > org.drools.reteoo.ReteooWorkingMemory$WorkingMemoryReteAssertAction.execute( > > > > > ReteooWorkingMemory.java:163) > > > > > at org.drools.common.AbstractWorkingMemory.executeQueuedActions > > > > > (AbstractWorkingMemory.java:1135) > > > > > at org.drools.common.AbstractWorkingMemory.insert( > > > > > AbstractWorkingMemory.java:781) > > > > > at org.drools.common.AbstractWorkingMemory.insert( > > > > > AbstractWorkingMemory.java:584) > > > > > at org.drools.jsr94.rules.StatefulRuleSessionImpl.addObject( > > > > > StatefulRuleSessionImpl.java:162) > > > > > > > > > > This only happens during high load tests. > > > > > Can anyone help me? > > > > > > > > > > Thanks > > > > > > > > > > > > > > > ------------------------------ > > > > > > > > > > _______________________________________________ > > > > > rules-users mailing listrules-users@lists.jboss.orghttps://lists.jboss.org/mailman/listinfo/rules-users > > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > rules-users mailing list > > > > > rules-users(a)lists.jboss.org > > > > > https://lists.jboss.org/mailman/listinfo/rules-users > > > > > > > > > > > > > > ------------------------------ > > > > > > > > _______________________________________________ > > > > rules-users mailing listrules-users@lists.jboss.orghttps://lists.jboss.org/mailman/listinfo/rules-users > > > > > > > > > > > > > > > > _______________________________________________ > > > > rules-users mailing list > > > > rules-users(a)lists.jboss.org > > > > https://lists.jboss.org/mailman/listinfo/rules-users > > > > > > > > > > > ------------------------------ > > > > > > _______________________________________________ > > > rules-users mailing listrules-users@lists.jboss.orghttps://lists.jboss.org/mailman/listinfo/rules-users > > > > > > > > > > > > _______________________________________________ > > > rules-users mailing list > > > rules-users(a)lists.jboss.org > > > https://lists.jboss.org/mailman/listinfo/rules-users > > > > > > > > > ------------------------------ > > _______________________________________________ > rules-users mailing listrules-users@lists.jboss.orghttps://lists.jboss.org/mailman/listinfo/rules-users > > > > _______________________________________________ > rules-users mailing list > rules-users(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/rules-users > > ------------------------------ _______________________________________________ rules-users mailing listrules-users@lists.jboss.orghttps://lists.jboss.org/mailman/listinfo/rules-users _______________________________________________ rules-users mailing list rules-users(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/rules-users

ok, that might be it. We generate code in a singleton classloader, I suspect that each thread is generating its' own getters. I suspect that created singleton classloader is not getting GCd and releasing the perm gen. I'm not sure how to fix this. You can check this yourself by attaching a jprofiler instance (available for free trial) and looking at the object counts. I've tracked down the concurrency issue. The Rete node has a HashMap of ObjectTypeNodes that is built on the fly, that was global to the rulebase. I'll have to make it local to the working Memory. I'll fix that today. Mark s erel wrote: > Our server creates hundreds of stateful rule sessions concurrently. > Each created rule seesion is specific to a thread. We did not encounter > any memory problems with the previous version. As I've said, > it's difficult (actually, impossible) for me to provide you with a > self contained example > since the drl is complicated and contains dozens of complex objects. > However, I am ready to provide you with any information that can help > us in that manner. > I've also mentioned in the earlier threads the concurrency problem > I've encountered with AbstractHashTable. > Can this has something to do with it? > > > On 7/4/07, *Mark Proctor* <mproctor(a)codehaus.org > <mailto:mproctor@codehaus.org>> wrote: > > I don't beleive there is anything in 4.0 that is going to cause > such quick loss of permgen. Can you create a self contained > example that illustrates this behaviour? So we can reproduce this? > > Mark > > s erel wrote: >> We've tried to increase the permGen to 256mb. It did not help >> and the space run out really fast. >> Regarding MVEL, is turning code generation off something that >> can (or will) be done with a configuration parameter/factory >> method or do I need to track down all the places in the code? >> >> We did not experience such memory behavior with the previous >> version we used (3.06) when running the same tests. >> Bugs in 3.06 (no longer present in 4M3) are forcing us to upgrade. >> >> Is there another reason for such behaviour? >> Should we wait for release candidate? >> >> >> On 7/4/07, *Mark Proctor* <mproctor(a)codehaus.org >> <mailto:mproctor@codehaus.org>> wrote: >> >> increase your perm gen space,or use the MVEL dialect with >> code generation off. >> >> >> Mark >> s erel wrote: >>> Hello, >>> >>> During capacity tests we've received permGen OOM >>> exception. The occupied space in the permGen area increases >>> rapidly. Any opinions? >>> >>> >>> On 7/3/07, *s erel* <erelsagi(a)gmail.com >>> <mailto:erelsagi@gmail.com>> wrote: >>> >>> In our project we are creating a >>> StatefulRuleSession and saving it in a per-thread >>> context (i.e. Each thread has it's own >>> StatefulRuleSession): >>> >>> ruleServiceProvider.getRuleRuntime().createRuleSession(contextName, >>> properties, RuleRuntime.STATEFUL_SESSION_TYPE); >>> >>> When a thread session ends, we are calling release on >>> the previously created StatefulRuleSession. >>> >>> >>> Changing the following lines: >>> >>> >>> public abstract class AbstractHashTable >>> >>> >>> ... >>> >>> public Iterator iterator() { >>> // if ( this.iterator == null ) { >>> // this.iterator = new HashTableIterator( >>> this ); >>> // } >>> // >>> // this.iterator.reset(); >>> // return this.iterator; >>> >>> HashTableIterator iterator = new >>> HashTableIterator(this); >>> iterator.reset(); >>> >>> return iterator; >>> } >>> >>> Seems to solve the problem I've encountered. What's >>> your opinion? >>> >>> >>> On 7/2/07, *Mark Proctor* <mproctor(a)codehaus.org >>> <mailto:mproctor@codehaus.org>> wrote: >>> >>> a working memory should be single threaded, so not >>> sure how this could be a race condition? >>> >>> Mark >>> >>> s erel wrote: >>>> I've done a little debugging. The code fails >>>> in the following segment: >>>> >>>> public static class HashTableIterator >>>> ... >>>> while ( this.entry == null ) { >>>> this.row++; >>>> if ( this.row == this.length ) { >>>> return null; >>>> } >>>> this.entry = >>>> this.table[this.row]; *// ---> index out of bounds >>>> exception* >>>> } >>>> } >>>> >>>> this.row has the same value as this.length despite >>>> the condition above it. Probably a race condition >>>> issue. >>>> >>>> >>>> On 7/2/07, *Mark Proctor* <mproctor(a)codehaus.org >>>> <mailto:mproctor@codehaus.org>> wrote: >>>> >>>> Not really :( >>>> >>>> In your situation I tend to keep removing >>>> rules and data while still making sure the >>>> error happens, to get it down to a minimum. >>>> Please do try, as this isn't an error that >>>> should happen. Or alterntaively you can open >>>> drools-core and drools-compiler in eclipse and >>>> execuse and debug this yourself - in your >>>> situation this might best. you can put in a >>>> breakpoint to listen for that particular >>>> exception. >>>> >>>> Mark >>>> >>>> s erel wrote: >>>>> It's hard for me to provide a self contained >>>>> project. The drl is long and uses several >>>>> business objects. It's the same drl as we've >>>>> been using for 306 minus the keyword changes. >>>>> Is there anything else i can check or provide >>>>> you in order to solve this matter. >>>>> >>>>> Thanks >>>>> >>>>> On 7/1/07, *Mark Proctor* >>>>> <mproctor(a)codehaus.org >>>>> <mailto:mproctor@codehaus.org>> wrote: >>>>> >>>>> Can you provide us a self contained >>>>> project which creates this error? Unless >>>>> we can recreate it, it will be very hard >>>>> to track it down. Please attach the >>>>> project to a jira and we'll make it a >>>>> priority. >>>>> >>>>> Mark >>>>> s erel wrote: >>>>>> Hello, >>>>>> >>>>>> I've just started integrating MR3 into >>>>>> my project (I've previously >>>>>> used 3.06). The drl compiles and >>>>>> everything seems fine, but during >>>>>> tests the following exception is thrown >>>>>> for time to time: >>>>>> >>>>>> java.lang.ArrayIndexOutOfBoundsException: 17 >>>>>> at >>>>>> org.drools.util.AbstractHashTable$HashTableIterator.next(AbstractHashTable.java:250) >>>>>> at >>>>>> org.drools.reteoo.Rete$ObjectTypeConf.buildCache(Rete.java:434) >>>>>> at >>>>>> org.drools.reteoo.Rete$ObjectTypeConf.getObjectTypeNodes(Rete.java:425) >>>>>> at >>>>>> org.drools.reteoo.Rete.assertObject(Rete.java:172) >>>>>> at >>>>>> org.drools.reteoo.ReteooRuleBase.assertObject(ReteooRuleBase.java:190) >>>>>> at >>>>>> org.drools.reteoo.ReteooWorkingMemory$WorkingMemoryReteAssertAction.execute >>>>>> (ReteooWorkingMemory.java:163) >>>>>> at >>>>>> org.drools.common.AbstractWorkingMemory.executeQueuedActions(AbstractWorkingMemory.java:1135) >>>>>> at >>>>>> org.drools.common.AbstractWorkingMemory.insert(AbstractWorkingMemory.java:781) >>>>>> at >>>>>> org.drools.common.AbstractWorkingMemory.insert(AbstractWorkingMemory.java:584) >>>>>> at >>>>>> org.drools.jsr94.rules.StatefulRuleSessionImpl.addObject(StatefulRuleSessionImpl.java:162) >>>>>> >>>>>> This only happens during high load tests. >>>>>> Can anyone help me? >>>>>> >>>>>> Thanks >>>>>> >>>>>> ------------------------------------------------------------------------ >>>>>> >>>>>> _______________________________________________ >>>>>> rules-users mailing list >>>>>> rules-users(a)lists.jboss.org <mailto:rules-users@lists.jboss.org> >>>>>> https://lists.jboss.org/mailman/listinfo/rules-users >>>>>> >>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> rules-users mailing list >>>>> rules-users(a)lists.jboss.org >>>>> <mailto:rules-users@lists.jboss.org> >>>>> https://lists.jboss.org/mailman/listinfo/rules-users >>>>> <https://lists.jboss.org/mailman/listinfo/rules-users> >>>>> >>>>> >>>>> ------------------------------------------------------------------------ >>>>> >>>>> _______________________________________________ >>>>> rules-users mailing list >>>>> rules-users(a)lists.jboss.org <mailto:rules-users@lists.jboss.org> >>>>> https://lists.jboss.org/mailman/listinfo/rules-users >>>>> >>>> >>>> >>>> _______________________________________________ >>>> rules-users mailing list >>>> rules-users(a)lists.jboss.org >>>> <mailto:rules-users@lists.jboss.org> >>>> https://lists.jboss.org/mailman/listinfo/rules-users >>>> <https://lists.jboss.org/mailman/listinfo/rules-users> >>>> >>>> >>>> ------------------------------------------------------------------------ >>>> >>>> _______________________________________________ >>>> rules-users mailing list >>>> rules-users(a)lists.jboss.org <mailto:rules-users@lists.jboss.org> >>>> https://lists.jboss.org/mailman/listinfo/rules-users >>>> >>> >>> >>> _______________________________________________ >>> rules-users mailing list >>> rules-users(a)lists.jboss.org >>> <mailto:rules-users@lists.jboss.org> >>> https://lists.jboss.org/mailman/listinfo/rules-users >>> <https://lists.jboss.org/mailman/listinfo/rules-users> >>> >>> >>> >>> ------------------------------------------------------------------------ >>> >>> _______________________________________________ >>> rules-users mailing list >>> rules-users(a)lists.jboss.org <mailto:rules-users@lists.jboss.org> >>> https://lists.jboss.org/mailman/listinfo/rules-users >>> >> >> >> _______________________________________________ >> rules-users mailing list >> rules-users(a)lists.jboss.org <mailto:rules-users@lists.jboss.org> >> https://lists.jboss.org/mailman/listinfo/rules-users >> <https://lists.jboss.org/mailman/listinfo/rules-users> >> >> >> ------------------------------------------------------------------------ >> >> _______________________________________________ >> rules-users mailing list >> rules-users(a)lists.jboss.org <mailto:rules-users@lists.jboss.org> >> https://lists.jboss.org/mailman/listinfo/rules-users >> > > > _______________________________________________ > rules-users mailing list > rules-users(a)lists.jboss.org <mailto:rules-users@lists.jboss.org> > https://lists.jboss.org/mailman/listinfo/rules-users > <https://lists.jboss.org/mailman/listinfo/rules-users> > > > ------------------------------------------------------------------------ > > _______________________________________________ > rules-users mailing list > rules-users(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/rules-users > ------------------------------------------------------------------------ _______________________________________________ rules-users mailing list rules-users(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/rules-users

actually I thought i tracked it down, was wrong. Anyway once the RuleBase is built the "ObjectHashMap objectTypeNodes" in Rete should not change....Only I things I can think of are: 1) Some how a thread is seeing a stale version of the map 2) the rulebase is getting updated while propagation is happening, maybe combine with an issue from 1. In reality the rulebase addition should obey the standard locking mechanism and stop this from happening. 3) The initial RuleBase hasn't finished building yet, and threads are already being spun off to assert data. I have a feeling its 3... Mark Mark Proctor wrote: > ok, that might be it. We generate code in a singleton classloader, I > suspect that each thread is generating its' own getters. I suspect > that created singleton classloader is not getting GCd and releasing > the perm gen. I'm not sure how to fix this. You can check this > yourself by attaching a jprofiler instance (available for free trial) > and looking at the object counts. > > I've tracked down the concurrency issue. The Rete node has a HashMap > of ObjectTypeNodes that is built on the fly, that was global to the > rulebase. I'll have to make it local to the working Memory. I'll fix > that today. > > Mark > s erel wrote: >> Our server creates hundreds of stateful rule sessions concurrently. >> Each created rule seesion is specific to a thread. We did not encounter >> any memory problems with the previous version. As I've said, >> it's difficult (actually, impossible) for me to provide you with a >> self contained example >> since the drl is complicated and contains dozens of complex objects. >> However, I am ready to provide you with any information that can help >> us in that manner. >> I've also mentioned in the earlier threads the concurrency problem >> I've encountered with AbstractHashTable. >> Can this has something to do with it? >> >> >> On 7/4/07, *Mark Proctor* <mproctor(a)codehaus.org >> <mailto:mproctor@codehaus.org>> wrote: >> >> I don't beleive there is anything in 4.0 that is going to cause >> such quick loss of permgen. Can you create a self contained >> example that illustrates this behaviour? So we can reproduce this? >> >> Mark >> >> s erel wrote: >>> We've tried to increase the permGen to 256mb. It did not help >>> and the space run out really fast. >>> Regarding MVEL, is turning code generation off something that >>> can (or will) be done with a configuration parameter/factory >>> method or do I need to track down all the places in the code? >>> >>> We did not experience such memory behavior with the previous >>> version we used (3.06) when running the same tests. >>> Bugs in 3.06 (no longer present in 4M3) are forcing us to upgrade. >>> >>> Is there another reason for such behaviour? >>> Should we wait for release candidate? >>> >>> >>> On 7/4/07, *Mark Proctor* <mproctor(a)codehaus.org >>> <mailto:mproctor@codehaus.org>> wrote: >>> >>> increase your perm gen space,or use the MVEL dialect with >>> code generation off. >>> >>> >>> Mark >>> s erel wrote: >>>> Hello, >>>> >>>> During capacity tests we've received permGen OOM >>>> exception. The occupied space in the permGen area >>>> increases rapidly. Any opinions? >>>> >>>> >>>> On 7/3/07, *s erel* <erelsagi(a)gmail.com >>>> <mailto:erelsagi@gmail.com>> wrote: >>>> >>>> In our project we are creating a >>>> StatefulRuleSession and saving it in a per-thread >>>> context (i.e. Each thread has it's own >>>> StatefulRuleSession): >>>> >>>> ruleServiceProvider.getRuleRuntime().createRuleSession(contextName, >>>> properties, RuleRuntime.STATEFUL_SESSION_TYPE); >>>> >>>> When a thread session ends, we are calling release on >>>> the previously created StatefulRuleSession. >>>> >>>> >>>> Changing the following lines: >>>> >>>> >>>> public abstract class AbstractHashTable >>>> >>>> >>>> ... >>>> >>>> public Iterator iterator() { >>>> // if ( this.iterator == null ) { >>>> // this.iterator = new HashTableIterator( >>>> this ); >>>> // } >>>> // >>>> // this.iterator.reset(); >>>> // return this.iterator; >>>> >>>> HashTableIterator iterator = new >>>> HashTableIterator(this); >>>> iterator.reset(); >>>> >>>> return iterator; >>>> } >>>> >>>> Seems to solve the problem I've encountered. What's >>>> your opinion? >>>> >>>> >>>> On 7/2/07, *Mark Proctor* <mproctor(a)codehaus.org >>>> <mailto:mproctor@codehaus.org>> wrote: >>>> >>>> a working memory should be single threaded, so not >>>> sure how this could be a race condition? >>>> >>>> Mark >>>> >>>> s erel wrote: >>>>> I've done a little debugging. The code fails >>>>> in the following segment: >>>>> >>>>> public static class HashTableIterator >>>>> ... >>>>> while ( this.entry == null ) { >>>>> this.row++; >>>>> if ( this.row == this.length ) { >>>>> return null; >>>>> } >>>>> this.entry = >>>>> this.table[this.row]; *// ---> index out of >>>>> bounds exception* >>>>> } >>>>> } >>>>> >>>>> this.row has the same value as this.length >>>>> despite the condition above it. Probably a race >>>>> condition issue. >>>>> >>>>> >>>>> On 7/2/07, *Mark Proctor* <mproctor(a)codehaus.org >>>>> <mailto:mproctor@codehaus.org>> wrote: >>>>> >>>>> Not really :( >>>>> >>>>> In your situation I tend to keep removing >>>>> rules and data while still making sure the >>>>> error happens, to get it down to a minimum. >>>>> Please do try, as this isn't an error that >>>>> should happen. Or alterntaively you can open >>>>> drools-core and drools-compiler in eclipse >>>>> and execuse and debug this yourself - in your >>>>> situation this might best. you can put in a >>>>> breakpoint to listen for that particular >>>>> exception. >>>>> >>>>> Mark >>>>> >>>>> s erel wrote: >>>>>> It's hard for me to provide a self contained >>>>>> project. The drl is long and uses several >>>>>> business objects. It's the same drl as we've >>>>>> been using for 306 minus the keyword changes. >>>>>> Is there anything else i can check or >>>>>> provide you in order to solve this matter. >>>>>> >>>>>> Thanks >>>>>> >>>>>> On 7/1/07, *Mark Proctor* >>>>>> <mproctor(a)codehaus.org >>>>>> <mailto:mproctor@codehaus.org>> wrote: >>>>>> >>>>>> Can you provide us a self contained >>>>>> project which creates this error? Unless >>>>>> we can recreate it, it will be very hard >>>>>> to track it down. Please attach the >>>>>> project to a jira and we'll make it a >>>>>> priority. >>>>>> >>>>>> Mark >>>>>> s erel wrote: >>>>>>> Hello, >>>>>>> >>>>>>> I've just started integrating MR3 into >>>>>>> my project (I've previously >>>>>>> used 3.06). The drl compiles and >>>>>>> everything seems fine, but during >>>>>>> tests the following exception is thrown >>>>>>> for time to time: >>>>>>> >>>>>>> java.lang.ArrayIndexOutOfBoundsException: >>>>>>> 17 >>>>>>> at >>>>>>> org.drools.util.AbstractHashTable$HashTableIterator.next(AbstractHashTable.java:250) >>>>>>> at >>>>>>> org.drools.reteoo.Rete$ObjectTypeConf.buildCache(Rete.java:434) >>>>>>> at >>>>>>> org.drools.reteoo.Rete$ObjectTypeConf.getObjectTypeNodes(Rete.java:425) >>>>>>> at >>>>>>> org.drools.reteoo.Rete.assertObject(Rete.java:172) >>>>>>> at >>>>>>> org.drools.reteoo.ReteooRuleBase.assertObject(ReteooRuleBase.java:190) >>>>>>> at >>>>>>> org.drools.reteoo.ReteooWorkingMemory$WorkingMemoryReteAssertAction.execute >>>>>>> (ReteooWorkingMemory.java:163) >>>>>>> at >>>>>>> org.drools.common.AbstractWorkingMemory.executeQueuedActions(AbstractWorkingMemory.java:1135) >>>>>>> at >>>>>>> org.drools.common.AbstractWorkingMemory.insert(AbstractWorkingMemory.java:781) >>>>>>> at >>>>>>> org.drools.common.AbstractWorkingMemory.insert(AbstractWorkingMemory.java:584) >>>>>>> at >>>>>>> org.drools.jsr94.rules.StatefulRuleSessionImpl.addObject(StatefulRuleSessionImpl.java:162) >>>>>>> >>>>>>> This only happens during high load tests. >>>>>>> Can anyone help me? >>>>>>> >>>>>>> Thanks >>>>>>> >>>>>>> ------------------------------------------------------------------------ >>>>>>> >>>>>>> _______________________________________________ >>>>>>> rules-users mailing list >>>>>>> rules-users(a)lists.jboss.org <mailto:rules-users@lists.jboss.org> >>>>>>> https://lists.jboss.org/mailman/listinfo/rules-users >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> rules-users mailing list >>>>>> rules-users(a)lists.jboss.org >>>>>> <mailto:rules-users@lists.jboss.org> >>>>>> https://lists.jboss.org/mailman/listinfo/rules-users >>>>>> <https://lists.jboss.org/mailman/listinfo/rules-users> >>>>>> >>>>>> >>>>>> ------------------------------------------------------------------------ >>>>>> >>>>>> _______________________________________________ >>>>>> rules-users mailing list >>>>>> rules-users(a)lists.jboss.org <mailto:rules-users@lists.jboss.org> >>>>>> https://lists.jboss.org/mailman/listinfo/rules-users >>>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> rules-users mailing list >>>>> rules-users(a)lists.jboss.org >>>>> <mailto:rules-users@lists.jboss.org> >>>>> https://lists.jboss.org/mailman/listinfo/rules-users >>>>> <https://lists.jboss.org/mailman/listinfo/rules-users> >>>>> >>>>> >>>>> ------------------------------------------------------------------------ >>>>> >>>>> _______________________________________________ >>>>> rules-users mailing list >>>>> rules-users(a)lists.jboss.org <mailto:rules-users@lists.jboss.org> >>>>> https://lists.jboss.org/mailman/listinfo/rules-users >>>>> >>>> >>>> >>>> _______________________________________________ >>>> rules-users mailing list >>>> rules-users(a)lists.jboss.org >>>> <mailto:rules-users@lists.jboss.org> >>>> https://lists.jboss.org/mailman/listinfo/rules-users >>>> <https://lists.jboss.org/mailman/listinfo/rules-users> >>>> >>>> >>>> >>>> ------------------------------------------------------------------------ >>>> >>>> _______________________________________________ >>>> rules-users mailing list >>>> rules-users(a)lists.jboss.org <mailto:rules-users@lists.jboss.org> >>>> https://lists.jboss.org/mailman/listinfo/rules-users >>>> >>> >>> >>> _______________________________________________ >>> rules-users mailing list >>> rules-users(a)lists.jboss.org >>> <mailto:rules-users@lists.jboss.org> >>> https://lists.jboss.org/mailman/listinfo/rules-users >>> <https://lists.jboss.org/mailman/listinfo/rules-users> >>> >>> >>> ------------------------------------------------------------------------ >>> >>> _______________________________________________ >>> rules-users mailing list >>> rules-users(a)lists.jboss.org <mailto:rules-users@lists.jboss.org> >>> https://lists.jboss.org/mailman/listinfo/rules-users >>> >> >> >> _______________________________________________ >> rules-users mailing list >> rules-users(a)lists.jboss.org <mailto:rules-users@lists.jboss.org> >> https://lists.jboss.org/mailman/listinfo/rules-users >> <https://lists.jboss.org/mailman/listinfo/rules-users> >> >> >> ------------------------------------------------------------------------ >> >> _______________________________________________ >> rules-users mailing list >> rules-users(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/rules-users >> > > ------------------------------------------------------------------------ > > _______________________________________________ > rules-users mailing list > rules-users(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/rules-users > ------------------------------------------------------------------------ _______________________________________________ rules-users mailing list rules-users(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/rules-users

Do you know if the fix I mentioned (returning a new iterator each time) solves the problem or only mask it? We create the initial RuleBase on server start-up. On 7/4/07, *Mark Proctor* <mproctor(a)codehaus.org <mailto:mproctor@codehaus.org>> wrote: actually I thought i tracked it down, was wrong. Anyway once the RuleBase is built the "ObjectHashMap objectTypeNodes" in Rete should not change....Only I things I can think of are: 1) Some how a thread is seeing a stale version of the map 2) the rulebase is getting updated while propagation is happening, maybe combine with an issue from 1. In reality the rulebase addition should obey the standard locking mechanism and stop this from happening. 3) The initial RuleBase hasn't finished building yet, and threads are already being spun off to assert data. I have a feeling its 3... Mark Mark Proctor wrote: > ok, that might be it. We generate code in a singleton > classloader, I suspect that each thread is generating its' own > getters. I suspect that created singleton classloader is not > getting GCd and releasing the perm gen. I'm not sure how to fix > this. You can check this yourself by attaching a jprofiler > instance (available for free trial) and looking at the object > counts. > > I've tracked down the concurrency issue. The Rete node has a > HashMap of ObjectTypeNodes that is built on the fly, that was > global to the rulebase. I'll have to make it local to the working > Memory. I'll fix that today. > > Mark > s erel wrote: >> Our server creates hundreds of stateful rule sessions >> concurrently. Each created rule seesion is specific to a thread. >> We did not encounter >> any memory problems with the previous version. As I've said, >> it's difficult (actually, impossible) for me to provide you with >> a self contained example >> since the drl is complicated and contains dozens of complex >> objects. However, I am ready to provide you with any information >> that can help >> us in that manner. >> I've also mentioned in the earlier threads the concurrency >> problem I've encountered with AbstractHashTable. >> Can this has something to do with it? >> >> >> On 7/4/07, *Mark Proctor* <mproctor(a)codehaus.org >> <mailto:mproctor@codehaus.org>> wrote: >> >> I don't beleive there is anything in 4.0 that is going to >> cause such quick loss of permgen. Can you create a self >> contained example that illustrates this behaviour? So we can >> reproduce this? >> >> Mark >> >> s erel wrote: >>> We've tried to increase the permGen to 256mb. It did not >>> help and the space run out really fast. >>> Regarding MVEL, is turning code generation off something >>> that can (or will) be done with a configuration >>> parameter/factory method or do I need to track down all the >>> places in the code? >>> >>> We did not experience such memory behavior with the >>> previous version we used (3.06) when running the same tests. >>> Bugs in 3.06 (no longer present in 4M3) are forcing us to >>> upgrade. >>> >>> Is there another reason for such behaviour? >>> Should we wait for release candidate? >>> >>> >>> On 7/4/07, *Mark Proctor* <mproctor(a)codehaus.org >>> <mailto:mproctor@codehaus.org>> wrote: >>> >>> increase your perm gen space,or use the MVEL dialect >>> with code generation off. >>> >>> >>> Mark >>> s erel wrote: >>>> Hello, >>>> >>>> During capacity tests we've received permGen OOM >>>> exception. The occupied space in the permGen area >>>> increases rapidly. Any opinions? >>>> >>>> >>>> On 7/3/07, *s erel* <erelsagi(a)gmail.com >>>> <mailto:erelsagi@gmail.com>> wrote: >>>> >>>> In our project we are creating a >>>> StatefulRuleSession and saving it in a per-thread >>>> context (i.e. Each thread has it's own >>>> StatefulRuleSession): >>>> >>>> ruleServiceProvider.getRuleRuntime().createRuleSession(contextName, >>>> properties, RuleRuntime.STATEFUL_SESSION_TYPE); >>>> >>>> When a thread session ends, we are calling release >>>> on the previously created StatefulRuleSession. >>>> >>>> >>>> Changing the following lines: >>>> >>>> >>>> public abstract class AbstractHashTable >>>> >>>> >>>> ... >>>> >>>> public Iterator iterator() { >>>> // if ( this.iterator == null ) { >>>> // this.iterator = new >>>> HashTableIterator( this ); >>>> // } >>>> // >>>> // this.iterator.reset(); >>>> // return this.iterator; >>>> >>>> HashTableIterator iterator = new >>>> HashTableIterator(this); >>>> iterator.reset(); >>>> >>>> return iterator; >>>> } >>>> >>>> Seems to solve the problem I've encountered. >>>> What's your opinion? >>>> >>>> >>>> On 7/2/07, *Mark Proctor* <mproctor(a)codehaus.org >>>> <mailto:mproctor@codehaus.org>> wrote: >>>> >>>> a working memory should be single threaded, so >>>> not sure how this could be a race condition? >>>> >>>> Mark >>>> >>>> s erel wrote: >>>>> I've done a little debugging. The code fails >>>>> in the following segment: >>>>> >>>>> public static class HashTableIterator >>>>> ... >>>>> while ( this.entry == null ) { >>>>> this.row++; >>>>> if ( this.row == >>>>> this.length ) { >>>>> return null; >>>>> } >>>>> this.entry = >>>>> this.table[this.row]; *// ---> index out of >>>>> bounds exception* >>>>> } >>>>> } >>>>> >>>>> this.row has the same value as this.length >>>>> despite the condition above it. Probably a >>>>> race condition issue. >>>>> >>>>> >>>>> On 7/2/07, *Mark Proctor* >>>>> <mproctor(a)codehaus.org >>>>> <mailto:mproctor@codehaus.org>> wrote: >>>>> >>>>> Not really :( >>>>> >>>>> In your situation I tend to keep removing >>>>> rules and data while still making sure >>>>> the error happens, to get it down to a >>>>> minimum. Please do try, as this isn't an >>>>> error that should happen. Or >>>>> alterntaively you can open drools-core >>>>> and drools-compiler in eclipse and >>>>> execuse and debug this yourself - in your >>>>> situation this might best. you can put in >>>>> a breakpoint to listen for that >>>>> particular exception. >>>>> >>>>> Mark >>>>> >>>>> s erel wrote: >>>>>> It's hard for me to provide a self >>>>>> contained project. The drl is long >>>>>> and uses several business objects. It's >>>>>> the same drl as we've been using for 306 >>>>>> minus the keyword changes. >>>>>> Is there anything else i can check or >>>>>> provide you in order to solve this matter. >>>>>> >>>>>> Thanks >>>>>> >>>>>> On 7/1/07, *Mark Proctor* >>>>>> <mproctor(a)codehaus.org >>>>>> <mailto:mproctor@codehaus.org>> wrote: >>>>>> >>>>>> Can you provide us a self contained >>>>>> project which creates this error? >>>>>> Unless we can recreate it, it will >>>>>> be very hard to track it down. >>>>>> Please attach the project to a jira >>>>>> and we'll make it a priority. >>>>>> >>>>>> Mark >>>>>> s erel wrote: >>>>>>> Hello, >>>>>>> >>>>>>> I've just started >>>>>>> integrating MR3 into my project >>>>>>> (I've previously used 3.06). The >>>>>>> drl compiles and everything seems >>>>>>> fine, but during >>>>>>> tests the following exception is >>>>>>> thrown for time to time: >>>>>>> >>>>>>> java.lang.ArrayIndexOutOfBoundsException: >>>>>>> 17 >>>>>>> at >>>>>>> org.drools.util.AbstractHashTable$HashTableIterator.next(AbstractHashTable.java:250) >>>>>>> at >>>>>>> org.drools.reteoo.Rete$ObjectTypeConf.buildCache(Rete.java:434) >>>>>>> at >>>>>>> org.drools.reteoo.Rete$ObjectTypeConf.getObjectTypeNodes(Rete.java:425) >>>>>>> at >>>>>>> org.drools.reteoo.Rete.assertObject(Rete.java:172) >>>>>>> at >>>>>>> org.drools.reteoo.ReteooRuleBase.assertObject(ReteooRuleBase.java:190) >>>>>>> at >>>>>>> org.drools.reteoo.ReteooWorkingMemory$WorkingMemoryReteAssertAction.execute >>>>>>> (ReteooWorkingMemory.java:163) >>>>>>> at >>>>>>> org.drools.common.AbstractWorkingMemory.executeQueuedActions(AbstractWorkingMemory.java:1135) >>>>>>> at >>>>>>> org.drools.common.AbstractWorkingMemory.insert(AbstractWorkingMemory.java:781) >>>>>>> at >>>>>>> org.drools.common.AbstractWorkingMemory.insert(AbstractWorkingMemory.java:584) >>>>>>> at >>>>>>> org.drools.jsr94.rules.StatefulRuleSessionImpl.addObject(StatefulRuleSessionImpl.java:162) >>>>>>> >>>>>>> This only happens during high load >>>>>>> tests. >>>>>>> Can anyone help me? >>>>>>> >>>>>>> Thanks >>>>>>> >>>>>>> ------------------------------------------------------------------------ >>>>>>> >>>>>>> _______________________________________________ >>>>>>> rules-users mailing list >>>>>>> rules-users(a)lists.jboss.org <mailto:rules-users@lists.jboss.org> >>>>>>> https://lists.jboss.org/mailman/listinfo/rules-users >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> rules-users mailing list >>>>>> rules-users(a)lists.jboss.org >>>>>> <mailto:rules-users@lists.jboss.org> >>>>>> https://lists.jboss.org/mailman/listinfo/rules-users >>>>>> <https://lists.jboss.org/mailman/listinfo/rules-users> >>>>>> >>>>>> >>>>>> ------------------------------------------------------------------------ >>>>>> >>>>>> _______________________________________________ >>>>>> rules-users mailing list >>>>>> rules-users(a)lists.jboss.org <mailto:rules-users@lists.jboss.org> >>>>>> https://lists.jboss.org/mailman/listinfo/rules-users >>>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> rules-users mailing list >>>>> rules-users(a)lists.jboss.org >>>>> <mailto:rules-users@lists.jboss.org> >>>>> https://lists.jboss.org/mailman/listinfo/rules-users >>>>> <https://lists.jboss.org/mailman/listinfo/rules-users> >>>>> >>>>> >>>>> ------------------------------------------------------------------------ >>>>> >>>>> _______________________________________________ >>>>> rules-users mailing list >>>>> rules-users(a)lists.jboss.org <mailto:rules-users@lists.jboss.org> >>>>> https://lists.jboss.org/mailman/listinfo/rules-users >>>>> >>>> >>>> >>>> _______________________________________________ >>>> rules-users mailing list >>>> rules-users(a)lists.jboss.org >>>> <mailto:rules-users@lists.jboss.org> >>>> https://lists.jboss.org/mailman/listinfo/rules-users >>>> <https://lists.jboss.org/mailman/listinfo/rules-users> >>>> >>>> >>>> >>>> ------------------------------------------------------------------------ >>>> >>>> _______________________________________________ >>>> rules-users mailing list >>>> rules-users(a)lists.jboss.org <mailto:rules-users@lists.jboss.org> >>>> https://lists.jboss.org/mailman/listinfo/rules-users >>>> >>> >>> >>> _______________________________________________ >>> rules-users mailing list >>> rules-users(a)lists.jboss.org >>> <mailto:rules-users@lists.jboss.org> >>> https://lists.jboss.org/mailman/listinfo/rules-users >>> <https://lists.jboss.org/mailman/listinfo/rules-users> >>> >>> >>> ------------------------------------------------------------------------ >>> >>> _______________________________________________ >>> rules-users mailing list >>> rules-users(a)lists.jboss.org <mailto:rules-users@lists.jboss.org> >>> https://lists.jboss.org/mailman/listinfo/rules-users >>> >> >> >> _______________________________________________ >> rules-users mailing list >> rules-users(a)lists.jboss.org <mailto:rules-users@lists.jboss.org> >> https://lists.jboss.org/mailman/listinfo/rules-users >> <https://lists.jboss.org/mailman/listinfo/rules-users> >> >> >> ------------------------------------------------------------------------ >> >> _______________________________________________ >> rules-users mailing list >> rules-users(a)lists.jboss.org <mailto:rules-users@lists.jboss.org> >> https://lists.jboss.org/mailman/listinfo/rules-users >> > > ------------------------------------------------------------------------ > > _______________________________________________ > rules-users mailing list > rules-users(a)lists.jboss.org <mailto:rules-users@lists.jboss.org> > https://lists.jboss.org/mailman/listinfo/rules-users > _______________________________________________ rules-users mailing list rules-users(a)lists.jboss.org <mailto:rules-users@lists.jboss.org> https://lists.jboss.org/mailman/listinfo/rules-users <https://lists.jboss.org/mailman/listinfo/rules-users> ------------------------------------------------------------------------ _______________________________________________ rules-users mailing list rules-users(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/rules-users

Please let me know if the permGen issue can be fixed somehow. Thanks On 7/5/07, *Mark Proctor* <mproctor(a)codehaus.org <mailto:mproctor@codehaus.org>> wrote: actually, yes you are right - that was indeed the problem. That iterator is normally only used by the working memory, which is single threaded, but in this case its on the rulebase, which is not suppose to hold state. I'll fix this now, well found :) Mark s erel wrote: > Do you know if the fix I mentioned (returning a new iterator each > time) solves the problem or only mask it? > We create the initial RuleBase on server start-up. > > On 7/4/07, *Mark Proctor* <mproctor(a)codehaus.org > <mailto:mproctor@codehaus.org>> wrote: > > actually I thought i tracked it down, was wrong. Anyway once > the RuleBase is built the "ObjectHashMap objectTypeNodes" in > Rete should not change....Only I things I can think of are: > 1) Some how a thread is seeing a stale version of the map > 2) the rulebase is getting updated while propagation is > happening, maybe combine with an issue from 1. In reality the > rulebase addition should obey the standard locking mechanism > and stop this from happening. > 3) The initial RuleBase hasn't finished building yet, and > threads are already being spun off to assert data. > > I have a feeling its 3... > > Mark > > Mark Proctor wrote: >> ok, that might be it. We generate code in a singleton >> classloader, I suspect that each thread is generating its' >> own getters. I suspect that created singleton classloader is >> not getting GCd and releasing the perm gen. I'm not sure how >> to fix this. You can check this yourself by attaching a >> jprofiler instance (available for free trial) and looking at >> the object counts. >> >> I've tracked down the concurrency issue. The Rete node has a >> HashMap of ObjectTypeNodes that is built on the fly, that >> was global to the rulebase. I'll have to make it local to >> the working Memory. I'll fix that today. >> >> Mark >> s erel wrote: >>> Our server creates hundreds of stateful rule sessions >>> concurrently. Each created rule seesion is specific to a >>> thread. We did not encounter >>> any memory problems with the previous version. As I've >>> said, it's difficult (actually, impossible) for me to >>> provide you with a self contained example >>> since the drl is complicated and contains dozens of complex >>> objects. However, I am ready to provide you with any >>> information that can help >>> us in that manner. >>> I've also mentioned in the earlier threads the concurrency >>> problem I've encountered with AbstractHashTable. >>> Can this has something to do with it? >>> >>> >>> On 7/4/07, *Mark Proctor* <mproctor(a)codehaus.org >>> <mailto:mproctor@codehaus.org>> wrote: >>> >>> I don't beleive there is anything in 4.0 that is going >>> to cause such quick loss of permgen. Can you create a >>> self contained example that illustrates this behaviour? >>> So we can reproduce this? >>> >>> Mark >>> >>> s erel wrote: >>>> We've tried to increase the permGen to 256mb. It did >>>> not help and the space run out really fast. >>>> Regarding MVEL, is turning code generation >>>> off something that can (or will) be done with a >>>> configuration parameter/factory method or do I need to >>>> track down all the places in the code? >>>> >>>> We did not experience such memory behavior with the >>>> previous version we used (3.06) when running the same >>>> tests. >>>> Bugs in 3.06 (no longer present in 4M3) are forcing us >>>> to upgrade. >>>> >>>> Is there another reason for such behaviour? >>>> Should we wait for release candidate? >>>> >>>> >>>> On 7/4/07, *Mark Proctor* <mproctor(a)codehaus.org >>>> <mailto:mproctor@codehaus.org>> wrote: >>>> >>>> increase your perm gen space,or use the MVEL >>>> dialect with code generation off. >>>> >>>> >>>> Mark >>>> s erel wrote: >>>>> Hello, >>>>> >>>>> During capacity tests we've received permGen OOM >>>>> exception. The occupied space in the permGen area >>>>> increases rapidly. Any opinions? >>>>> >>>>> >>>>> On 7/3/07, *s erel* <erelsagi(a)gmail.com >>>>> <mailto:erelsagi@gmail.com>> wrote: >>>>> >>>>> In our project we are creating a >>>>> StatefulRuleSession and saving it in a >>>>> per-thread context (i.e. Each thread has it's >>>>> own StatefulRuleSession): >>>>> >>>>> ruleServiceProvider.getRuleRuntime().createRuleSession(contextName, >>>>> properties, RuleRuntime.STATEFUL_SESSION_TYPE); >>>>> >>>>> When a thread session ends, we are calling >>>>> release on the previously created >>>>> StatefulRuleSession. >>>>> >>>>> >>>>> Changing the following lines: >>>>> >>>>> >>>>> public abstract class AbstractHashTable >>>>> >>>>> >>>>> ... >>>>> >>>>> public Iterator iterator() { >>>>> // if ( this.iterator == null ) { >>>>> // this.iterator = new >>>>> HashTableIterator( this ); >>>>> // } >>>>> // >>>>> // this.iterator.reset(); >>>>> // return this.iterator; >>>>> >>>>> HashTableIterator iterator = new >>>>> HashTableIterator(this); >>>>> iterator.reset(); >>>>> >>>>> return iterator; >>>>> } >>>>> >>>>> Seems to solve the problem I've encountered. >>>>> What's your opinion? >>>>> >>>>> >>>>> On 7/2/07, *Mark Proctor* >>>>> <mproctor(a)codehaus.org >>>>> <mailto:mproctor@codehaus.org>> wrote: >>>>> >>>>> a working memory should be single >>>>> threaded, so not sure how this could be a >>>>> race condition? >>>>> >>>>> Mark >>>>> >>>>> s erel wrote: >>>>>> I've done a little debugging. The code >>>>>> fails in the following segment: >>>>>> >>>>>> public static class HashTableIterator >>>>>> ... >>>>>> while ( this.entry == null ) { >>>>>> this.row++; >>>>>> if ( this.row == >>>>>> this.length ) { >>>>>> return null; >>>>>> } >>>>>> this.entry = >>>>>> this.table[this.row]; *// ---> index out >>>>>> of bounds exception* >>>>>> } >>>>>> } >>>>>> >>>>>> this.row has the same value as >>>>>> this.length despite the condition above >>>>>> it. Probably a race condition issue. >>>>>> >>>>>> >>>>>> On 7/2/07, *Mark Proctor* >>>>>> <mproctor(a)codehaus.org >>>>>> <mailto:mproctor@codehaus.org>> wrote: >>>>>> >>>>>> Not really :( >>>>>> >>>>>> In your situation I tend to keep >>>>>> removing rules and data while still >>>>>> making sure the error happens, to >>>>>> get it down to a minimum. Please do >>>>>> try, as this isn't an error that >>>>>> should happen. Or alterntaively you >>>>>> can open drools-core and >>>>>> drools-compiler in eclipse and >>>>>> execuse and debug this yourself - in >>>>>> your situation this might best. you >>>>>> can put in a breakpoint to listen >>>>>> for that particular exception. >>>>>> >>>>>> Mark >>>>>> >>>>>> s erel wrote: >>>>>>> It's hard for me to provide a self >>>>>>> contained project. The drl is long >>>>>>> and uses several business objects. >>>>>>> It's the same drl as we've been >>>>>>> using for 306 minus the keyword >>>>>>> changes. >>>>>>> Is there anything else i can check >>>>>>> or provide you in order to solve >>>>>>> this matter. >>>>>>> >>>>>>> Thanks >>>>>>> >>>>>>> On 7/1/07, *Mark Proctor* >>>>>>> <mproctor(a)codehaus.org >>>>>>> <mailto:mproctor@codehaus.org>> wrote: >>>>>>> >>>>>>> Can you provide us a self >>>>>>> contained project which creates >>>>>>> this error? Unless we can >>>>>>> recreate it, it will be very >>>>>>> hard to track it down. Please >>>>>>> attach the project to a jira >>>>>>> and we'll make it a priority. >>>>>>> >>>>>>> Mark >>>>>>> s erel wrote: >>>>>>>> Hello, >>>>>>>> >>>>>>>> I've just started >>>>>>>> integrating MR3 into my >>>>>>>> project (I've previously >>>>>>>> used 3.06). The drl compiles >>>>>>>> and everything seems fine, but >>>>>>>> during >>>>>>>> tests the following exception >>>>>>>> is thrown for time to time: >>>>>>>> >>>>>>>> java.lang.ArrayIndexOutOfBoundsException: >>>>>>>> 17 >>>>>>>> at >>>>>>>> org.drools.util.AbstractHashTable$HashTableIterator.next(AbstractHashTable.java:250) >>>>>>>> at >>>>>>>> org.drools.reteoo.Rete$ObjectTypeConf.buildCache(Rete.java:434) >>>>>>>> at >>>>>>>> org.drools.reteoo.Rete$ObjectTypeConf.getObjectTypeNodes(Rete.java:425) >>>>>>>> at >>>>>>>> org.drools.reteoo.Rete.assertObject(Rete.java:172) >>>>>>>> at >>>>>>>> org.drools.reteoo.ReteooRuleBase.assertObject(ReteooRuleBase.java:190) >>>>>>>> at >>>>>>>> org.drools.reteoo.ReteooWorkingMemory$WorkingMemoryReteAssertAction.execute >>>>>>>> (ReteooWorkingMemory.java:163) >>>>>>>> at >>>>>>>> org.drools.common.AbstractWorkingMemory.executeQueuedActions(AbstractWorkingMemory.java:1135) >>>>>>>> at >>>>>>>> org.drools.common.AbstractWorkingMemory.insert(AbstractWorkingMemory.java:781) >>>>>>>> at >>>>>>>> org.drools.common.AbstractWorkingMemory.insert(AbstractWorkingMemory.java:584) >>>>>>>> at >>>>>>>> org.drools.jsr94.rules.StatefulRuleSessionImpl.addObject(StatefulRuleSessionImpl.java:162) >>>>>>>> >>>>>>>> This only happens during high >>>>>>>> load tests. >>>>>>>> Can anyone help me? >>>>>>>> >>>>>>>> Thanks >>>>>>>> >>>>>>>> ------------------------------------------------------------------------ >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> rules-users mailing list >>>>>>>> rules-users(a)lists.jboss.org <mailto:rules-users@lists.jboss.org> >>>>>>>> https://lists.jboss.org/mailman/listinfo/rules-users >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> rules-users mailing list >>>>>>> rules-users(a)lists.jboss.org >>>>>>> <mailto:rules-users@lists.jboss.org> >>>>>>> https://lists.jboss.org/mailman/listinfo/rules-users >>>>>>> <https://lists.jboss.org/mailman/listinfo/rules-users> >>>>>>> >>>>>>> >>>>>>> ------------------------------------------------------------------------ >>>>>>> >>>>>>> _______________________________________________ >>>>>>> rules-users mailing list >>>>>>> rules-users(a)lists.jboss.org <mailto:rules-users@lists.jboss.org> >>>>>>> https://lists.jboss.org/mailman/listinfo/rules-users >>>>>>> >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> rules-users mailing list >>>>>> rules-users(a)lists.jboss.org >>>>>> <mailto:rules-users@lists.jboss.org> >>>>>> https://lists.jboss.org/mailman/listinfo/rules-users >>>>>> <https://lists.jboss.org/mailman/listinfo/rules-users> >>>>>> >>>>>> >>>>>> ------------------------------------------------------------------------ >>>>>> >>>>>> _______________________________________________ >>>>>> rules-users mailing list >>>>>> rules-users(a)lists.jboss.org <mailto:rules-users@lists.jboss.org> >>>>>> https://lists.jboss.org/mailman/listinfo/rules-users >>>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> rules-users mailing list >>>>> rules-users(a)lists.jboss.org >>>>> <mailto:rules-users@lists.jboss.org> >>>>> https://lists.jboss.org/mailman/listinfo/rules-users >>>>> <https://lists.jboss.org/mailman/listinfo/rules-users> >>>>> >>>>> >>>>> >>>>> ------------------------------------------------------------------------ >>>>> >>>>> _______________________________________________ >>>>> rules-users mailing list >>>>> rules-users(a)lists.jboss.org <mailto:rules-users@lists.jboss.org> >>>>> https://lists.jboss.org/mailman/listinfo/rules-users >>>>> >>>> >>>> >>>> _______________________________________________ >>>> rules-users mailing list >>>> rules-users(a)lists.jboss.org >>>> <mailto:rules-users@lists.jboss.org> >>>> https://lists.jboss.org/mailman/listinfo/rules-users >>>> <https://lists.jboss.org/mailman/listinfo/rules-users> >>>> >>>> >>>> ------------------------------------------------------------------------ >>>> >>>> _______________________________________________ >>>> rules-users mailing list >>>> rules-users(a)lists.jboss.org <mailto:rules-users@lists.jboss.org> >>>> https://lists.jboss.org/mailman/listinfo/rules-users >>>> >>> >>> >>> _______________________________________________ >>> rules-users mailing list >>> rules-users(a)lists.jboss.org >>> <mailto:rules-users@lists.jboss.org> >>> https://lists.jboss.org/mailman/listinfo/rules-users >>> <https://lists.jboss.org/mailman/listinfo/rules-users> >>> >>> >>> ------------------------------------------------------------------------ >>> >>> _______________________________________________ >>> rules-users mailing list >>> rules-users(a)lists.jboss.org <mailto:rules-users@lists.jboss.org> >>> https://lists.jboss.org/mailman/listinfo/rules-users >>> >> >> ------------------------------------------------------------------------ >> >> _______________________________________________ >> rules-users mailing list >> rules-users(a)lists.jboss.org <mailto:rules-users@lists.jboss.org> >> https://lists.jboss.org/mailman/listinfo/rules-users >> > > > _______________________________________________ > rules-users mailing list > rules-users(a)lists.jboss.org <mailto:rules-users@lists.jboss.org> > https://lists.jboss.org/mailman/listinfo/rules-users > <https://lists.jboss.org/mailman/listinfo/rules-users> > > > ------------------------------------------------------------------------ > > _______________________________________________ > rules-users mailing list > rules-users(a)lists.jboss.org <mailto:rules-users@lists.jboss.org> > https://lists.jboss.org/mailman/listinfo/rules-users > _______________________________________________ rules-users mailing list rules-users(a)lists.jboss.org <mailto:rules-users@lists.jboss.org> https://lists.jboss.org/mailman/listinfo/rules-users <https://lists.jboss.org/mailman/listinfo/rules-users> ------------------------------------------------------------------------ _______________________________________________ rules-users mailing list rules-users(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/rules-users