Re: [rules-users] Memory error when inserting facts in statefulsession

Hi Mike, Quick reply, thanks ! I think I can’t use « exists » because I want one and only one booking rule of a specific level (LEVEL_1, LEVEL_2, …) matching a deal, that’s why the bookingRuleModels list collected should only contain 1 booking rule. This explain why in the RHS I’m getting the first booking rule (bookingRuleModels.get(0)). If there is more than one booking rule (or none), I switch to the second Drools rule which tries to find a unique LEVEL_2 booking rule, then same thing if none or too many, switch to LEVEL_3 booking rule … Hope you get the idea. The matchXXX function compare the XXX criterion of a booking rule (ex Trader) with the corresponding Deal attribute (ex Trader). Here’s the implementation for comparing trader criterion, note that if a booking rule does not specify a criterion it is considered as matched. *private* *static* *boolean* matchCriterion(NorthIdEntity<?> ruleCriterionModel, NorthIdEntity<?> model) { *if* (ruleCriterionModel == *null*) { // no criterion => matching ok *return* *true*; } *return* ruleCriterionModel.equals(model); } I have read that eval is not performant but I thought it was only when executing rules not at facts insertion phase. Is this coorect or is eval impacting negatively facts insertion ? Another idea comes to my mind : can I use a stateless session since once a deal is classified/booked we don’t want to process it again ? To finish I forgot to give informations on my environment, it may be useful : - Drools version 5.01 - java 6 - launching my server with following memory options : -Xms256m -Xmx1024m Thanks again for your thought, Joel ------------------------------ *De :* rules-users-bounces(a)lists.jboss.org [mailto: rules-users-bounces(a)lists.jboss.org] *De la part de* Anstis, Michael (M.) *Envoyé :* vendredi 11 septembre 2009 10:35 *À :* Rules Users List *Objet :* Re: [rules-users] Memory error when inserting facts in stateful session Hi, As an aside, you shouldn't need to use a rule flow as the rules themselves determine which RHS is activated. Could you consider using "exists" instead of the "ArrayList( size == 1) from collect..." - it looks like you're not too bothered which Booking Rule matches as (I believe) you have no way to determine which is at index(0). Also, what do your "matchXXX" functions do? Could you investigate removing the "evals"? I don't have any knowledge of your domain - we're all newbies at some time or other. With kind regards, Mike ------------------------------ *From:* rules-users-bounces(a)lists.jboss.org [mailto: rules-users-bounces(a)lists.jboss.org] *On Behalf Of *Costigliola Joel (EXT) *Sent:* 11 September 2009 09:21 *To:* rules-users(a)lists.jboss.org *Subject:* [rules-users] Memory error when inserting facts in stateful session Hello, To be short I’m facing some performance/memory problems with Drools which leads to the error : java.lang.OutOfMemoryError: Java heap space. That was for the short story, let me now give you more details. First, I’m a Drools newbie so I certainly have made some “bad” choice. I’m using Drools 5.01 to classify automatically the deals made by the traders of my company (this is the functionnal problem I want to solve with Drools). I have written 4 rules, a rule-flow and start the deals classifying process with a stateful session. I have successfully (unit) tested different scenarios, so everything is ok on a functionnal point of view. Problems arise when I started to insert more deals in the session, which leads to an OutOfMemoryError before the call to ksession.fireAllRules. logs are below (after the drools rule). Number of facts I have tried to insert in my statefull session : - 222 booking rule (POJO expressing classification criteria) - 750 product index - 750 deals What drools does here is to find the correct booking rule to apply for each deal (we need some product index for that). Can you tell me if those numbers seems unrealistic ? What can I do to avoid the memory errors ? To be complete, I show you the drools rules : *rule* "Find and apply level 1 booking rule" *lock-on-active* *true* *ruleflow-group* "Level 1 booking rule group" *when* dealModel : MarketDealModel( $dealPortfolio : portfolio, $dealTrader : trader, $dealProduct : product) // retrieve the ProductRelatedIndexes corresponding to the deal product (only one by product) productRelatedIndexes : ProductRelatedIndexes( product.internalCode == $dealProduct.internalCode, $dealProductIndexes : relatedIndexes) // try to find one and only one level 1 matching rule (level 1 <=> all matching criteria are defined). bookingRuleModels : ArrayList( size == 1 ) *from* *collect* ( BTExecutionBookingRuleModel( priority == BTExecutionBookingRuleModel.LEVEL_1 && *eval*(matchTraderCriterion(traderCriterion, $dealTrader)) && *eval*(matchPortfolioCriterion(portfolioCriterion, $dealPortfolio)) && *eval*(matchIndexCriterion(listedIndexCriterion, $dealProductIndexes)) && *eval*(matchProductTypeCriterion(productTypeStringCriterion, $dealProduct)) ) ) *then* Logger log = LoggerFactory.getLogger("BOOKING RULE ENGINE LOGGER"); // get the unique collected bookingRuleModel. BTExecutionBookingRuleModel effectiveBookingRuleModel = (BTExecutionBookingRuleModel) bookingRuleModels.get(0); // log.info("Found matching level 1 booking rule --> " + effectiveBookingRuleModel); effectiveBookingRuleModel.applyRuleOnDeal(dealModel); *retract*( dealModel ); // only needed in use of stateful session to avoid processing this deal again. *end* I don’t put the 3 others, they are basically the same except the priority == BTExecutionBookingRuleModel.LEVEL_1 which is done against LEVEL_2, LEVEL_3 and LEVEL_4 rule. Each rule is in his own ruleflow-group, it is very basic, if first drools rule is not active then we try the second (with compares priority to BTExecutionBookingRuleModel.LEVEL_2). I also put some logs showing that inserting deals fact takes longer and longer : - the 222 booking rule are inserted in 16ms - the 750 booking rule are inserted in 46ms - the 750 facts are inserted at a pace of 5 by second, then it starts to deteriorate to several seconds for one insert to finish with the OutOfMemoryError. 2009-09-11 09:46:50 134 INFO [booking.impl.DealBookingProcessorImpl] 658 MarketDealModel inserted in Drools session 2009-09-11 09:46:50 244 INFO [booking.impl.DealBookingProcessorImpl] 659 MarketDealModel inserted in Drools session 2009-09-11 09:46:50 369 INFO [booking.impl.DealBookingProcessorImpl] 660 MarketDealModel inserted in Drools session 2009-09-11 09:46:51 197 INFO [booking.impl.DealBookingProcessorImpl] 661 MarketDealModel inserted in Drools session 2009-09-11 09:46:51 306 INFO [booking.impl.DealBookingProcessorImpl] 662 MarketDealModel inserted in Drools session 2009-09-11 09:46:51 540 INFO [booking.impl.DealBookingProcessorImpl] 663 MarketDealModel inserted in Drools session 2009-09-11 09:46:51 650 INFO [booking.impl.DealBookingProcessorImpl] 664 MarketDealModel inserted in Drools session 2009-09-11 09:46:51 775 INFO [booking.impl.DealBookingProcessorImpl] 665 MarketDealModel inserted in Drools session 2009-09-11 09:46:51 884 INFO [booking.impl.DealBookingProcessorImpl] 666 MarketDealModel inserted in Drools session 2009-09-11 09:46:52 009 INFO [booking.impl.DealBookingProcessorImpl] 667 MarketDealModel inserted in Drools session 2009-09-11 09:46:52 134 INFO [booking.impl.DealBookingProcessorImpl] 668 MarketDealModel inserted in Drools session 2009-09-11 09:46:52 243 INFO [booking.impl.DealBookingProcessorImpl] 669 MarketDealModel inserted in Drools session 2009-09-11 09:46:52 368 INFO [booking.impl.DealBookingProcessorImpl] 670 MarketDealModel inserted in Drools session 2009-09-11 09:47:32 784 INFO [booking.impl.DealBookingProcessorImpl] 671 MarketDealModel inserted in Drools session 2009-09-11 09:47:33 003 INFO [booking.impl.DealBookingProcessorImpl] 672 MarketDealModel inserted in Drools session 2009-09-11 09:47:33 128 INFO [booking.impl.DealBookingProcessorImpl] 673 MarketDealModel inserted in Drools session 2009-09-11 09:47:33 253 INFO [booking.impl.DealBookingProcessorImpl] 674 MarketDealModel inserted in Drools session 2009-09-11 09:47:33 362 INFO [booking.impl.DealBookingProcessorImpl] 675 MarketDealModel inserted in Drools session 2009-09-11 09:47:33 487 INFO [booking.impl.DealBookingProcessorImpl] 676 MarketDealModel inserted in Drools session 2009-09-11 09:47:33 596 INFO [booking.impl.DealBookingProcessorImpl] 677 MarketDealModel inserted in Drools session 2009-09-11 09:47:33 721 INFO [booking.impl.DealBookingProcessorImpl] 678 MarketDealModel inserted in Drools session 2009-09-11 09:47:33 831 INFO [booking.impl.DealBookingProcessorImpl] 679 MarketDealModel inserted in Drools session 2009-09-11 09:47:33 956 INFO [booking.impl.DealBookingProcessorImpl] 680 MarketDealModel inserted in Drools session 2009-09-11 09:47:34 128 INFO [booking.impl.DealBookingProcessorImpl] 681 MarketDealModel inserted in Drools session 2009-09-11 09:47:56 218 INFO [booking.impl.DealBookingProcessorImpl] 682 MarketDealModel inserted in Drools session 2009-09-11 09:47:56 374 INFO [booking.impl.DealBookingProcessorImpl] 683 MarketDealModel inserted in Drools session 2009-09-11 09:47:56 483 INFO [booking.impl.DealBookingProcessorImpl] 684 MarketDealModel inserted in Drools session 2009-09-11 09:47:56 608 INFO [booking.impl.DealBookingProcessorImpl] 685 MarketDealModel inserted in Drools session 2009-09-11 09:47:56 733 INFO [booking.impl.DealBookingProcessorImpl] 686 MarketDealModel inserted in Drools session 2009-09-11 09:47:56 858 INFO [booking.impl.DealBookingProcessorImpl] 687 MarketDealModel inserted in Drools session 2009-09-11 09:47:56 968 INFO [booking.impl.DealBookingProcessorImpl] 688 MarketDealModel inserted in Drools session 2009-09-11 09:47:57 093 INFO [booking.impl.DealBookingProcessorImpl] 689 MarketDealModel inserted in Drools session 2009-09-11 09:47:57 218 INFO [booking.impl.DealBookingProcessorImpl] 690 MarketDealModel inserted in Drools session 2009-09-11 09:47:57 421 INFO [booking.impl.DealBookingProcessorImpl] 691 MarketDealModel inserted in Drools session 2009-09-11 09:48:05 404 INFO [booking.impl.DealBookingProcessorImpl] 692 MarketDealModel inserted in Drools session 2009-09-11 09:48:05 529 INFO [booking.impl.DealBookingProcessorImpl] 693 MarketDealModel inserted in Drools session 2009-09-11 09:48:05 654 INFO [booking.impl.DealBookingProcessorImpl] 694 MarketDealModel inserted in Drools session 2009-09-11 09:48:05 763 INFO [booking.impl.DealBookingProcessorImpl] 695 MarketDealModel inserted in Drools session 2009-09-11 09:48:05 888 INFO [booking.impl.DealBookingProcessorImpl] 696 MarketDealModel inserted in Drools session 2009-09-11 09:48:05 998 INFO [booking.impl.DealBookingProcessorImpl] 697 MarketDealModel inserted in Drools session 2009-09-11 09:48:06 123 INFO [booking.impl.DealBookingProcessorImpl] 698 MarketDealModel inserted in Drools session 2009-09-11 09:48:06 357 INFO [booking.impl.DealBookingProcessorImpl] 699 MarketDealModel inserted in Drools session 2009-09-11 09:48:14 184 INFO [booking.impl.DealBookingProcessorImpl] 700 MarketDealModel inserted in Drools session 2009-09-11 09:48:14 293 INFO [booking.impl.DealBookingProcessorImpl] 701 MarketDealModel inserted in Drools session 2009-09-11 09:48:14 418 INFO [booking.impl.DealBookingProcessorImpl] 702 MarketDealModel inserted in Drools session 2009-09-11 09:48:14 543 INFO [booking.impl.DealBookingProcessorImpl] 703 MarketDealModel inserted in Drools session 2009-09-11 09:48:14 668 INFO [booking.impl.DealBookingProcessorImpl] 704 MarketDealModel inserted in Drools session 2009-09-11 09:48:14 856 INFO [booking.impl.DealBookingProcessorImpl] 705 MarketDealModel inserted in Drools session 2009-09-11 09:48:22 761 INFO [booking.impl.DealBookingProcessorImpl] 706 MarketDealModel inserted in Drools session 2009-09-11 09:48:22 886 INFO [booking.impl.DealBookingProcessorImpl] 707 MarketDealModel inserted in Drools session 2009-09-11 09:48:22 995 INFO [booking.impl.DealBookingProcessorImpl] 708 MarketDealModel inserted in Drools session 2009-09-11 09:48:23 120 INFO [booking.impl.DealBookingProcessorImpl] 709 MarketDealModel inserted in Drools session 2009-09-11 09:48:23 323 INFO [booking.impl.DealBookingProcessorImpl] 710 MarketDealModel inserted in Drools session 2009-09-11 09:48:32 166 INFO [booking.impl.DealBookingProcessorImpl] 711 MarketDealModel inserted in Drools session 2009-09-11 09:48:32 290 INFO [booking.impl.DealBookingProcessorImpl] 712 MarketDealModel inserted in Drools session 2009-09-11 09:48:32 400 INFO [booking.impl.DealBookingProcessorImpl] 713 MarketDealModel inserted in Drools session 2009-09-11 09:48:32 634 INFO [booking.impl.DealBookingProcessorImpl] 714 MarketDealModel inserted in Drools session 2009-09-11 09:48:40 570 INFO [booking.impl.DealBookingProcessorImpl] 715 MarketDealModel inserted in Drools session 2009-09-11 09:48:40 695 INFO [booking.impl.DealBookingProcessorImpl] 716 MarketDealModel inserted in Drools session 2009-09-11 09:48:40 899 INFO [booking.impl.DealBookingProcessorImpl] 717 MarketDealModel inserted in Drools session 2009-09-11 09:48:48 850 INFO [booking.impl.DealBookingProcessorImpl] 718 MarketDealModel inserted in Drools session 2009-09-11 09:48:48 975 INFO [booking.impl.DealBookingProcessorImpl] 719 MarketDealModel inserted in Drools session 2009-09-11 09:48:49 272 INFO [booking.impl.DealBookingProcessorImpl] 720 MarketDealModel inserted in Drools session 2009-09-11 09:48:57 209 INFO [booking.impl.DealBookingProcessorImpl] 721 MarketDealModel inserted in Drools session 2009-09-11 09:48:57 505 INFO [booking.impl.DealBookingProcessorImpl] 722 MarketDealModel inserted in Drools session 2009-09-11 09:49:05 598 INFO [booking.impl.DealBookingProcessorImpl] 723 MarketDealModel inserted in Drools session 2009-09-11 09:49:13 722 INFO [booking.impl.DealBookingProcessorImpl] 724 MarketDealModel inserted in Drools session 2009-09-11 09:49:21 752 INFO [booking.impl.DealBookingProcessorImpl] 725 MarketDealModel inserted in Drools session 2009-09-11 09:49:29 813 INFO [booking.impl.DealBookingProcessorImpl] 726 MarketDealModel inserted in Drools session 2009-09-11 09:49:37 921 INFO [booking.impl.DealBookingProcessorImpl] 727 MarketDealModel inserted in Drools session 2009-09-11 09:49:53 809 INFO [booking.impl.DealBookingProcessorImpl] 728 MarketDealModel inserted in Drools session 2009-09-11 09:50:26 507 INFO [booking.impl.DealBookingProcessorImpl] 729 MarketDealModel inserted in Drools session Exception in thread "Timer-0" java.lang.OutOfMemoryError: Java heap space at java.lang.Object.clone(*Native Method*) at java.util.LinkedList.clone(*LinkedList.java:830*) at com.mchange.v2.resourcepool.BasicResourcePool.cloneOfUnused(* BasicResourcePool.java:1661*) at com.mchange.v2.resourcepool.BasicResourcePool.cullExpired(* BasicResourcePool.java:1450*) at com.mchange.v2.resourcepool.BasicResourcePool.access$1900(* BasicResourcePool.java:32*) at com.mchange.v2.resourcepool.BasicResourcePool$CullTask.run(* BasicResourcePool.java:1937*) at java.util.TimerThread.mainLoop(*Timer.java:512*) at java.util.TimerThread.run(*Timer.java:462*) Exception in thread "Ice.ThreadPool.Server-1" java.lang.OutOfMemoryError: Java heap space at org.mvel2.integration.impl.ClassImportResolverFactory.<init>(* ClassImportResolverFactory.java:49*) at org.mvel2.compiler.CompiledExpression.getValue(* CompiledExpression.java:104*) at org.mvel2.MVEL.executeExpression(*MVEL.java:978*) at org.drools.base.mvel.MVELPredicateExpression.evaluate(* MVELPredicateExpression.java:75*) at org.drools.rule.PredicateConstraint.isAllowedCachedLeft(* PredicateConstraint.java:295*) at org.drools.common.SingleBetaConstraints.isAllowedCachedLeft(* SingleBetaConstraints.java:138*) at org.drools.reteoo.JoinNode.assertLeftTuple(*JoinNode.java:114*) at org.drools.reteoo.CompositeLeftTupleSinkAdapter.doPropagateAssertLeftTuple( *CompositeLeftTupleSinkAdapter.java:145*) at org.drools.reteoo.CompositeLeftTupleSinkAdapter.createAndPropagateAssertLeftTuple( *CompositeLeftTupleSinkAdapter.java:57*) at org.drools.reteoo.LeftInputAdapterNode.assertObject(* LeftInputAdapterNode.java:142*) at org.drools.reteoo.SingleObjectSinkAdapter.propagateAssertObject(* SingleObjectSinkAdapter.java:42*) at org.drools.reteoo.ObjectTypeNode.assertObject(* ObjectTypeNode.java:185*) at org.drools.reteoo.EntryPointNode.assertObject(* EntryPointNode.java:146*) at org.drools.common.AbstractWorkingMemory.insert(* AbstractWorkingMemory.java:1046*) at org.drools.common.AbstractWorkingMemory.insert(* AbstractWorkingMemory.java:1001*) at org.drools.common.AbstractWorkingMemory.insert(* AbstractWorkingMemory.java:788*) at org.drools.impl.StatefulKnowledgeSessionImpl.insert(* StatefulKnowledgeSessionImpl.java:216*) at north.stardust2.services.trading.booking.impl.DealBookingProcessorImpl.insertDealFacts( *DealBookingProcessorImpl.java:200*) So to summarize my questions : - how can I fix the facts insertion ? - is there some newbie mistakes in my approach ? Thanks in advance for your help, Regards, Joël Costigliola ------------------------------ Ce courriel et toutes les pièces jointes sont confidentiels et peuvent être couverts par un privilège ou une protection légale. Il est établi à l’attention exclusive de ses destinataires. Toute utilisation de ce courriel non conforme à sa destination, toute diffusion ou toute publication, totale ou partielle, est interdite, sauf autorisation expresse préalable. Toutes opinions exprimées dans ce courriel ne sauraient nécessairement refléter celle de Natixis, de ses filiales. Elles sont aussi susceptibles de modification sans notification préalable. Si vous recevez ce courriel par erreur, merci de le détruire et d’en avertir immédiatement l’expéditeur. L’Internet ne permettant pas d’assurer l’intégrité de ce courriel, Natixis décline toute responsabilité s’il a été altéré, déformé ou falsifié et chaque destinataire qui utilise ce mode de communication est supposé en accepter les risques. This email and any attachment are confidential and may be legally privileged or otherwise protected from disclosure. It is intended only for the stated addressee(s) and access to it by any other person(s) is unauthorised. Any use, dissemination or disclosure not in accordance with its purpose, either in whole or in part, is prohibited without our prior formal approval. Any opinion expressed in this email may not necessarily reflect the opinion of Natixis, its affiliates. It may also be subject to change without prior notice. If you are not an addressee, you must not disclose, copy, circulate or in any other way use or rely on the information contained in this email. If you have received it in error, please inform us immediately and delete all copies. The Internet can not guarantee the integrity of this email therefore Natixis shall not be liable for the email if altered, changed or falsified and anyone who communicates with us by e-mail is taken to accept these risks. ------------------------------ ------------------------------ Ce courriel et toutes les pièces jointes sont confidentiels et peuvent être couverts par un privilège ou une protection légale. Il est établi à l’attention exclusive de ses destinataires. Toute utilisation de ce courriel non conforme à sa destination, toute diffusion ou toute publication, totale ou partielle, est interdite, sauf autorisation expresse préalable. Toutes opinions exprimées dans ce courriel ne sauraient nécessairement refléter celle de Natixis, de ses filiales. Elles sont aussi susceptibles de modification sans notification préalable. Si vous recevez ce courriel par erreur, merci de le détruire et d’en avertir immédiatement l’expéditeur. L’Internet ne permettant pas d’assurer l’intégrité de ce courriel, Natixis décline toute responsabilité s’il a été altéré, déformé ou falsifié et chaque destinataire qui utilise ce mode de communication est supposé en accepter les risques. This email and any attachment are confidential and may be legally privileged or otherwise protected from disclosure. It is intended only for the stated addressee(s) and access to it by any other person(s) is unauthorised. Any use, dissemination or disclosure not in accordance with its purpose, either in whole or in part, is prohibited without our prior formal approval. Any opinion expressed in this email may not necessarily reflect the opinion of Natixis, its affiliates. It may also be subject to change without prior notice. If you are not an addressee, you must not disclose, copy, circulate or in any other way use or rely on the information contained in this email. If you have received it in error, please inform us immediately and delete all copies. The Internet can not guarantee the integrity of this email therefore Natixis shall not be liable for the email if altered, changed or falsified and anyone who communicates with us by e-mail is taken to accept these risks. ------------------------------ _______________________________________________ rules-users mailing list rules-users(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/rules-users