Hi All, We have an application that uses drools stateless session over a set of 35000 objects (taking about 40 MB heap) of different types. During profiling, I saw that for rules that perform joins between two huge object types data sets, lots of org.drools.reteoo.LeftTuple objects causing out of memory errors. We are using 5G heap, and have set property drools.sequential=true too. Is there any way we can decrease the memory usage while applying these rules? Drools documentation mentions  for sequential mode, we should do the following, some of them I am not sure how to set: Turn off all node memories, except the right-input Object memory. Disconnect the LeftInputAdapterNode propagation, and have the Object plus the Node referenced in a Command object, which is added to a list on the WorkingMemory for later execution. Assert all objects, when all assertions are finished and thus right-input node memories are populated check the Command list and execute each in turn. http://downloads.jboss.com/drools/docs/4.0.7.19894.GA/html_single/index.h... Do we need to set some properties in RuleBaseConfiguration to make sure node memories are switched off? It would be great if we can somehow decrease the memory usage for stateless sessions? Has anybody come across such memory issues before? Correct me if I am wrong, but I think 35000 objects is not a lot of data for causing such issues. Thanks Malay ________________________________ NOTICE: If received in error, please destroy, and notify sender. Sender does not intend to waive confidentiality or privilege. Use of this email is prohibited when received in error. We may monitor and store emails to the extent permitted by applicable law. ________________________________ NOTICE: If received in error, please destroy, and notify sender. Sender does not intend to waive confidentiality or privilege. Use of this email is prohibited when received in error. We may monitor and store emails to the extent permitted by applicable law. _______________________________________________ rules-users mailing list rules-users(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/rules-users

Thanks W. Let me give you a rule that is causing us issues. Lets say we have classes: class A { int a; String id; } Class B { int b; String id; } Class C { int c; String id; } and we have 12000 objects each. These objects are connected using "id" property. If the rule is something like: Rule "ABC" When Obj_A: A(a!=1) //10000 object "A" instances have a!=1 Obj_B: B(b!=2,Obj_A.id = id) //10000 object "B" instances have b!=2 Obj_C: C(c!=3,Obj_B.id = id) //10000 object "C" instances have c!=3 Then System.out.println("Rule fired"); End This means there would be a join between A, B and C objects (10000 A's * 10000 B's * 10000 C's) to evaluate the rule. In such cases, we see very high org.drools.reteoo.LeftTuple object count while profiling the app. Is there a way to make the rete algo perform this evaluation serially, or in batches, or use some object swapping mechanism to limit creation of intermediate LeftTuple objects? Would the rete algo use indexes in this case? That would potentially decrease these intermediate objects. Currently, we use 5G heap and that is not enough for evaluating this rule. Regards, Malay -----Original Message----- From: rules-users-bounces(a)lists.jboss.org [mailto: rules-users-bounces(a)lists.jboss.org] On Behalf Of Wolfgang Laun Sent: Monday, March 22, 2010 12:39 PM To: Rules Users List Subject: Re: [rules-users] Drools stateless session question You have misunderstood the documentation. All eight items in the enumeration you cite from are steps performed internally by the Drools engine. The only thing you have to do is request sequential mode. Without an idea about the rule set you are using it's difficult to advise how to reduce memory consumption. -W 2010/3/22 Shah, Malay <Malay.Shah(a)morganstanley.com&gt;: > Hi All, > > We have an application that uses drools stateless session over a set > of 35000 objects (taking about 40 MB heap) of different types. During > profiling, I saw that for rules that perform joins between two huge > object types data sets, lots of org.drools.reteoo.LeftTuple objects > causing out of memory errors. We are using 5G heap, and have set > property drools.sequential=true too. Is there any way we can decrease > the memory usage while applying these rules? Drools documentation > mentions for sequential mode, we should do the following, some of > them I am not sure how to set: > > > Turn off all node memories, except the right-input Object memory. > > Disconnect the LeftInputAdapterNode propagation, and have the Object > plus the Node referenced in a Command object, which is added to a list > on the WorkingMemory for later execution. > > Assert all objects, when all assertions are finished and thus > right-input node memories are populated check the Command list and execute each in turn. > > http://downloads.jboss.com/drools/docs/4.0.7.19894.GA/html_single/inde > x.html#d0e1295 > > Do we need to set some properties in RuleBaseConfiguration to make > sure node memories are switched off? It would be great if we can > somehow decrease the memory usage for stateless sessions? Has anybody > come across such memory issues before? Correct me if I am wrong, but I > think 35000 objects is not a lot of data for causing such issues. > > Thanks > Malay > ________________________________ > > NOTICE: If received in error, please destroy, and notify sender. > Sender does not intend to waive confidentiality or privilege. Use of > this email is prohibited when received in error. We may monitor and > store emails to the extent permitted by applicable law. > > ________________________________ > > NOTICE: If received in error, please destroy, and notify sender. > Sender does not intend to waive confidentiality or privilege. Use of > this email is prohibited when received in error. We may monitor and > store emails to the extent permitted by applicable law. > > _______________________________________________ > rules-users mailing list > rules-users(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/rules-users > > _______________________________________________ rules-users mailing list rules-users(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/rules-users -------------------------------------------------------------------------- NOTICE: If received in error, please destroy, and notify sender. Sender does not intend to waive confidentiality or privilege. Use of this email is prohibited when received in error. We may monitor and store emails to the extent permitted by applicable law. _______________________________________________ rules-users mailing list rules-users(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/rules-users

It will be 1 trillion activations unless there is an index on the "id" property of the classes. If the right index is used, it should scan all objects of one class, and through indexes figure out the corresponding related objects of other classes. I am not sure if we can somehow tell rete to create indexes on certain attributes. I read a nice article on indexing in drools, which mentioned about Rete Beta node indexing. I am not sure if this is what drools is doing internally. Any suggestions/ideas? http://blog.decaresystems.ie/index.php/2008/02/24/rete-nodes-indexing-in-... Thanks, Malay ________________________________ From: rules-users-bounces(a)lists.jboss.org [mailto:rules-users-bounces@lists.jboss.org] On Behalf Of Wolfgang Laun Sent: Monday, March 22, 2010 3:17 PM To: Rules Users List Subject: Re: [rules-users] Drools stateless session question You have calculated it correctly: 10000 A's * 10000 B's * 10000 C's results in 1,000.000,000.000 activations, which is (almost) a mind-boggling figure. I can't imagine what benefit this trillion of activations is going to have for your application. Even if you have the memory for all that, assuming that firing of one of these activations and processing of the RHS statements can be done in one microsecond, this is going to take days to complete, and it'll create a huge data set of A/B/C combinations for just this one rule. Are you sure that you are on the right track?! -W On Mon, Mar 22, 2010 at 7:46 PM, Shah, Malay <Malay.Shah(a)morganstanley.com&gt; wrote: > > Thanks W. Let me give you a rule that is causing us issues. Lets say we > have classes: > > class A > { >        int a; >        String id; > } > Class B > { >        int  b; >        String id; > } > Class C > { >        int c; >        String id; > } > > and we have 12000 objects each. These objects are connected using "id" > property. > If the rule is something like: > > Rule "ABC" > When >        Obj_A: A(a!=1)                  //10000 object "A" instances have > a!=1 >        Obj_B: B(b!=2,Obj_A.id = id)            //10000 object "B" > instances have b!=2 >        Obj_C: C(c!=3,Obj_B.id = id)            //10000 object "C" > instances have c!=3 > Then >        System.out.println("Rule fired"); > End > > This means there would be a join between A, B and C objects (10000 A's * > 10000 B's * 10000 C's) to evaluate the rule. In such cases, we see very high > org.drools.reteoo.LeftTuple object count while profiling the app. Is there a > way to make the rete algo perform this evaluation serially, or in batches, > or use some object swapping mechanism to limit creation of intermediate > LeftTuple objects? Would the rete algo use indexes in this case? That would > potentially decrease these intermediate objects. Currently, we use 5G heap > and that is not enough for evaluating this rule. > > Regards, > Malay > > -----Original Message----- > From: rules-users-bounces(a)lists.jboss.org > [mailto:rules-users-bounces@lists.jboss.org] On Behalf Of Wolfgang Laun > Sent: Monday, March 22, 2010 12:39 PM > To: Rules Users List > Subject: Re: [rules-users] Drools stateless session question > > You have misunderstood the documentation. All eight items in the > enumeration you cite from are steps performed internally by the Drools > engine. The only thing you have to do is request sequential mode. > > Without an idea about the rule set you are using it's difficult to advise > how to reduce memory consumption. > > -W > > > 2010/3/22 Shah, Malay <Malay.Shah(a)morganstanley.com&gt;: > > Hi All, > > > > We have an application that uses drools stateless session over a set > > of 35000 objects (taking about 40 MB heap) of different types. During > > profiling, I saw that for rules that perform joins between two huge > > object types data sets, lots of org.drools.reteoo.LeftTuple objects > > causing out of memory errors. We are using 5G heap, and have set > > property drools.sequential=true too. Is there any way we can decrease > > the memory usage while applying these rules? Drools documentation > > mentions  for sequential mode, we should do the following, some of > > them I am not sure how to set: > > > > > > Turn off all node memories, except the right-input Object memory. > > > > Disconnect the LeftInputAdapterNode propagation, and have the Object > > plus the Node referenced in a Command object, which is added to a list > > on the WorkingMemory for later execution. > > > > Assert all objects, when all assertions are finished and thus > > right-input node memories are populated check the Command list and > > execute each in turn. > > > > http://downloads.jboss.com/drools/docs/4.0.7.19894.GA/html_single/inde > > x.html#d0e1295 > > > > Do we need to set some properties in RuleBaseConfiguration to make > > sure node memories are switched off? It would be great if we can > > somehow decrease the memory usage for stateless sessions? Has anybody > > come across such memory issues before? Correct me if I am wrong, but I > > think 35000 objects is not a lot of data for causing such issues. > > > > Thanks > > Malay > > ________________________________ > > > > NOTICE: If received in error, please destroy, and notify sender. > > Sender does not intend to waive confidentiality or privilege. Use of > > this email is prohibited when received in error. We may monitor and > > store emails to the extent permitted by applicable law. > > > > ________________________________ > > > > NOTICE: If received in error, please destroy, and notify sender. > > Sender does not intend to waive confidentiality or privilege. Use of > > this email is prohibited when received in error. We may monitor and > > store emails to the extent permitted by applicable law. > > > > _______________________________________________ > > rules-users mailing list > > rules-users(a)lists.jboss.org > > https://lists.jboss.org/mailman/listinfo/rules-users > > > > > > _______________________________________________ > rules-users mailing list > rules-users(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/rules-users > > -------------------------------------------------------------------------- > NOTICE: If received in error, please destroy, and notify sender. Sender > does not intend to waive confidentiality or privilege. Use of this email is > prohibited when received in error. We may monitor and store emails to the > extent permitted by applicable law. > > _______________________________________________ > rules-users mailing list > rules-users(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/rules-users ________________________________ NOTICE: If received in error, please destroy, and notify sender. Sender does not intend to waive confidentiality or privilege. Use of this email is prohibited when received in error. We may monitor and store emails to the extent permitted by applicable law. _______________________________________________ rules-users mailing list rules-users(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/rules-users