Hello,

 

To be short I’m facing some performance/memory problems with Drools which leads to the error : java.lang.OutOfMemoryError: Java heap space.

 

That was for the short story, let me now give you more details.

First, I’m a Drools newbie so I certainly have made some “bad” choice.

 

I’m using Drools 5.01 to classify automatically the deals made by the traders of my company (this is the functionnal problem I want to solve with Drools).

I have written 4 rules, a rule-flow and start the deals classifying process with a stateful session.

I have successfully (unit) tested different scenarios, so everything is ok on a functionnal point of view.

 

Problems arise when I started to insert more deals in the session, which leads to an OutOfMemoryError before the call to ksession.fireAllRules. logs are below (after the drools rule).

Number of facts I have tried to insert in my statefull session :

-         222 booking rule (POJO expressing classification criteria)

-         750 product index

-         750 deals

 

What drools does here is to find the correct booking rule to apply for each deal (we need some product index for that).

 

Can you tell me if those numbers seems unrealistic ?

What can I do to avoid the memory errors ?

 

To be complete, I show you the drools rules :

 

rule "Find and apply level 1 booking rule"

      lock-on-active true

      ruleflow-group "Level 1 booking rule group"

      when

            dealModel : MarketDealModel( $dealPortfolio : portfolio, $dealTrader : trader, $dealProduct : product)

            // retrieve the ProductRelatedIndexes corresponding to the deal product (only one by product) 

            productRelatedIndexes : ProductRelatedIndexes( product.internalCode == $dealProduct.internalCode, $dealProductIndexes : relatedIndexes)

            // try to find one and only one level 1 matching rule (level 1 <=> all matching criteria are defined).

            bookingRuleModels : ArrayList( size == 1 ) from collect (

                  BTExecutionBookingRuleModel(

                        priority == BTExecutionBookingRuleModel.LEVEL_1

                        && eval(matchTraderCriterion(traderCriterion, $dealTrader))

                        && eval(matchPortfolioCriterion(portfolioCriterion, $dealPortfolio))

                        && eval(matchIndexCriterion(listedIndexCriterion, $dealProductIndexes))

                        && eval(matchProductTypeCriterion(productTypeStringCriterion, $dealProduct))

                  )

            )

      then

            Logger log = LoggerFactory.getLogger("BOOKING RULE ENGINE LOGGER");

            // get the unique collected bookingRuleModel.

            BTExecutionBookingRuleModel effectiveBookingRuleModel = (BTExecutionBookingRuleModel) bookingRuleModels.get(0);

            // log.info("Found matching level 1 booking rule --> " + effectiveBookingRuleModel);

            effectiveBookingRuleModel.applyRuleOnDeal(dealModel);

            retract( dealModel );  // only needed in use of stateful session to avoid processing this deal again.

end

 

I don’t put the 3 others, they are basically the same except the priority == BTExecutionBookingRuleModel.LEVEL_1 which is done against LEVEL_2, LEVEL_3 and LEVEL_4 rule.

Each rule is in his own ruleflow-group, it is very basic, if first drools rule is not active then we try the second (with compares priority to BTExecutionBookingRuleModel.LEVEL_2).

 

I also put some logs showing that inserting deals fact takes longer and longer :

- the 222 booking rule are inserted in 16ms

- the 750 booking rule are inserted in 46ms

- the 750 facts are inserted at a pace of 5 by second, then it starts to deteriorate to several seconds for one insert to finish with the OutOfMemoryError.

 

2009-09-11 09:46:50 134 INFO  [booking.impl.DealBookingProcessorImpl] 658 MarketDealModel inserted in Drools session

2009-09-11 09:46:50 244 INFO  [booking.impl.DealBookingProcessorImpl] 659 MarketDealModel inserted in Drools session

2009-09-11 09:46:50 369 INFO  [booking.impl.DealBookingProcessorImpl] 660 MarketDealModel inserted in Drools session

2009-09-11 09:46:51 197 INFO  [booking.impl.DealBookingProcessorImpl] 661 MarketDealModel inserted in Drools session

2009-09-11 09:46:51 306 INFO  [booking.impl.DealBookingProcessorImpl] 662 MarketDealModel inserted in Drools session

2009-09-11 09:46:51 540 INFO  [booking.impl.DealBookingProcessorImpl] 663 MarketDealModel inserted in Drools session

2009-09-11 09:46:51 650 INFO  [booking.impl.DealBookingProcessorImpl] 664 MarketDealModel inserted in Drools session

2009-09-11 09:46:51 775 INFO  [booking.impl.DealBookingProcessorImpl] 665 MarketDealModel inserted in Drools session

2009-09-11 09:46:51 884 INFO  [booking.impl.DealBookingProcessorImpl] 666 MarketDealModel inserted in Drools session

2009-09-11 09:46:52 009 INFO  [booking.impl.DealBookingProcessorImpl] 667 MarketDealModel inserted in Drools session

2009-09-11 09:46:52 134 INFO  [booking.impl.DealBookingProcessorImpl] 668 MarketDealModel inserted in Drools session

2009-09-11 09:46:52 243 INFO  [booking.impl.DealBookingProcessorImpl] 669 MarketDealModel inserted in Drools session

2009-09-11 09:46:52 368 INFO  [booking.impl.DealBookingProcessorImpl] 670 MarketDealModel inserted in Drools session

2009-09-11 09:47:32 784 INFO  [booking.impl.DealBookingProcessorImpl] 671 MarketDealModel inserted in Drools session

2009-09-11 09:47:33 003 INFO  [booking.impl.DealBookingProcessorImpl] 672 MarketDealModel inserted in Drools session

2009-09-11 09:47:33 128 INFO  [booking.impl.DealBookingProcessorImpl] 673 MarketDealModel inserted in Drools session

2009-09-11 09:47:33 253 INFO  [booking.impl.DealBookingProcessorImpl] 674 MarketDealModel inserted in Drools session

2009-09-11 09:47:33 362 INFO  [booking.impl.DealBookingProcessorImpl] 675 MarketDealModel inserted in Drools session

2009-09-11 09:47:33 487 INFO  [booking.impl.DealBookingProcessorImpl] 676 MarketDealModel inserted in Drools session

2009-09-11 09:47:33 596 INFO  [booking.impl.DealBookingProcessorImpl] 677 MarketDealModel inserted in Drools session

2009-09-11 09:47:33 721 INFO  [booking.impl.DealBookingProcessorImpl] 678 MarketDealModel inserted in Drools session

2009-09-11 09:47:33 831 INFO  [booking.impl.DealBookingProcessorImpl] 679 MarketDealModel inserted in Drools session

2009-09-11 09:47:33 956 INFO  [booking.impl.DealBookingProcessorImpl] 680 MarketDealModel inserted in Drools session

2009-09-11 09:47:34 128 INFO  [booking.impl.DealBookingProcessorImpl] 681 MarketDealModel inserted in Drools session

2009-09-11 09:47:56 218 INFO  [booking.impl.DealBookingProcessorImpl] 682 MarketDealModel inserted in Drools session

2009-09-11 09:47:56 374 INFO  [booking.impl.DealBookingProcessorImpl] 683 MarketDealModel inserted in Drools session

2009-09-11 09:47:56 483 INFO  [booking.impl.DealBookingProcessorImpl] 684 MarketDealModel inserted in Drools session

2009-09-11 09:47:56 608 INFO  [booking.impl.DealBookingProcessorImpl] 685 MarketDealModel inserted in Drools session

2009-09-11 09:47:56 733 INFO  [booking.impl.DealBookingProcessorImpl] 686 MarketDealModel inserted in Drools session

2009-09-11 09:47:56 858 INFO  [booking.impl.DealBookingProcessorImpl] 687 MarketDealModel inserted in Drools session

2009-09-11 09:47:56 968 INFO  [booking.impl.DealBookingProcessorImpl] 688 MarketDealModel inserted in Drools session

2009-09-11 09:47:57 093 INFO  [booking.impl.DealBookingProcessorImpl] 689 MarketDealModel inserted in Drools session

2009-09-11 09:47:57 218 INFO  [booking.impl.DealBookingProcessorImpl] 690 MarketDealModel inserted in Drools session

2009-09-11 09:47:57 421 INFO  [booking.impl.DealBookingProcessorImpl] 691 MarketDealModel inserted in Drools session

2009-09-11 09:48:05 404 INFO  [booking.impl.DealBookingProcessorImpl] 692 MarketDealModel inserted in Drools session

2009-09-11 09:48:05 529 INFO  [booking.impl.DealBookingProcessorImpl] 693 MarketDealModel inserted in Drools session

2009-09-11 09:48:05 654 INFO  [booking.impl.DealBookingProcessorImpl] 694 MarketDealModel inserted in Drools session

2009-09-11 09:48:05 763 INFO  [booking.impl.DealBookingProcessorImpl] 695 MarketDealModel inserted in Drools session

2009-09-11 09:48:05 888 INFO  [booking.impl.DealBookingProcessorImpl] 696 MarketDealModel inserted in Drools session

2009-09-11 09:48:05 998 INFO  [booking.impl.DealBookingProcessorImpl] 697 MarketDealModel inserted in Drools session

2009-09-11 09:48:06 123 INFO  [booking.impl.DealBookingProcessorImpl] 698 MarketDealModel inserted in Drools session

2009-09-11 09:48:06 357 INFO  [booking.impl.DealBookingProcessorImpl] 699 MarketDealModel inserted in Drools session

2009-09-11 09:48:14 184 INFO  [booking.impl.DealBookingProcessorImpl] 700 MarketDealModel inserted in Drools session

2009-09-11 09:48:14 293 INFO  [booking.impl.DealBookingProcessorImpl] 701 MarketDealModel inserted in Drools session

2009-09-11 09:48:14 418 INFO  [booking.impl.DealBookingProcessorImpl] 702 MarketDealModel inserted in Drools session

2009-09-11 09:48:14 543 INFO  [booking.impl.DealBookingProcessorImpl] 703 MarketDealModel inserted in Drools session

2009-09-11 09:48:14 668 INFO  [booking.impl.DealBookingProcessorImpl] 704 MarketDealModel inserted in Drools session

2009-09-11 09:48:14 856 INFO  [booking.impl.DealBookingProcessorImpl] 705 MarketDealModel inserted in Drools session

2009-09-11 09:48:22 761 INFO  [booking.impl.DealBookingProcessorImpl] 706 MarketDealModel inserted in Drools session

2009-09-11 09:48:22 886 INFO  [booking.impl.DealBookingProcessorImpl] 707 MarketDealModel inserted in Drools session

2009-09-11 09:48:22 995 INFO  [booking.impl.DealBookingProcessorImpl] 708 MarketDealModel inserted in Drools session

2009-09-11 09:48:23 120 INFO  [booking.impl.DealBookingProcessorImpl] 709 MarketDealModel inserted in Drools session

2009-09-11 09:48:23 323 INFO  [booking.impl.DealBookingProcessorImpl] 710 MarketDealModel inserted in Drools session

2009-09-11 09:48:32 166 INFO  [booking.impl.DealBookingProcessorImpl] 711 MarketDealModel inserted in Drools session

2009-09-11 09:48:32 290 INFO  [booking.impl.DealBookingProcessorImpl] 712 MarketDealModel inserted in Drools session

2009-09-11 09:48:32 400 INFO  [booking.impl.DealBookingProcessorImpl] 713 MarketDealModel inserted in Drools session

2009-09-11 09:48:32 634 INFO  [booking.impl.DealBookingProcessorImpl] 714 MarketDealModel inserted in Drools session

2009-09-11 09:48:40 570 INFO  [booking.impl.DealBookingProcessorImpl] 715 MarketDealModel inserted in Drools session

2009-09-11 09:48:40 695 INFO  [booking.impl.DealBookingProcessorImpl] 716 MarketDealModel inserted in Drools session

2009-09-11 09:48:40 899 INFO  [booking.impl.DealBookingProcessorImpl] 717 MarketDealModel inserted in Drools session

2009-09-11 09:48:48 850 INFO  [booking.impl.DealBookingProcessorImpl] 718 MarketDealModel inserted in Drools session

2009-09-11 09:48:48 975 INFO  [booking.impl.DealBookingProcessorImpl] 719 MarketDealModel inserted in Drools session

2009-09-11 09:48:49 272 INFO  [booking.impl.DealBookingProcessorImpl] 720 MarketDealModel inserted in Drools session

2009-09-11 09:48:57 209 INFO  [booking.impl.DealBookingProcessorImpl] 721 MarketDealModel inserted in Drools session

2009-09-11 09:48:57 505 INFO  [booking.impl.DealBookingProcessorImpl] 722 MarketDealModel inserted in Drools session

2009-09-11 09:49:05 598 INFO  [booking.impl.DealBookingProcessorImpl] 723 MarketDealModel inserted in Drools session

2009-09-11 09:49:13 722 INFO  [booking.impl.DealBookingProcessorImpl] 724 MarketDealModel inserted in Drools session

2009-09-11 09:49:21 752 INFO  [booking.impl.DealBookingProcessorImpl] 725 MarketDealModel inserted in Drools session

2009-09-11 09:49:29 813 INFO  [booking.impl.DealBookingProcessorImpl] 726 MarketDealModel inserted in Drools session

2009-09-11 09:49:37 921 INFO  [booking.impl.DealBookingProcessorImpl] 727 MarketDealModel inserted in Drools session

2009-09-11 09:49:53 809 INFO  [booking.impl.DealBookingProcessorImpl] 728 MarketDealModel inserted in Drools session

2009-09-11 09:50:26 507 INFO  [booking.impl.DealBookingProcessorImpl] 729 MarketDealModel inserted in Drools session

Exception in thread "Timer-0" java.lang.OutOfMemoryError: Java heap space

      at java.lang.Object.clone(Native Method)

      at java.util.LinkedList.clone(LinkedList.java:830)

      at com.mchange.v2.resourcepool.BasicResourcePool.cloneOfUnused(BasicResourcePool.java:1661)

      at com.mchange.v2.resourcepool.BasicResourcePool.cullExpired(BasicResourcePool.java:1450)

      at com.mchange.v2.resourcepool.BasicResourcePool.access$1900(BasicResourcePool.java:32)

      at com.mchange.v2.resourcepool.BasicResourcePool$CullTask.run(BasicResourcePool.java:1937)

      at java.util.TimerThread.mainLoop(Timer.java:512)

      at java.util.TimerThread.run(Timer.java:462)

Exception in thread "Ice.ThreadPool.Server-1" java.lang.OutOfMemoryError: Java heap space

      at org.mvel2.integration.impl.ClassImportResolverFactory.<init>(ClassImportResolverFactory.java:49)

      at org.mvel2.compiler.CompiledExpression.getValue(CompiledExpression.java:104)

      at org.mvel2.MVEL.executeExpression(MVEL.java:978)

      at org.drools.base.mvel.MVELPredicateExpression.evaluate(MVELPredicateExpression.java:75)

      at org.drools.rule.PredicateConstraint.isAllowedCachedLeft(PredicateConstraint.java:295)

      at org.drools.common.SingleBetaConstraints.isAllowedCachedLeft(SingleBetaConstraints.java:138)

      at org.drools.reteoo.JoinNode.assertLeftTuple(JoinNode.java:114)

      at org.drools.reteoo.CompositeLeftTupleSinkAdapter.doPropagateAssertLeftTuple(CompositeLeftTupleSinkAdapter.java:145)

      at org.drools.reteoo.CompositeLeftTupleSinkAdapter.createAndPropagateAssertLeftTuple(CompositeLeftTupleSinkAdapter.java:57)

      at org.drools.reteoo.LeftInputAdapterNode.assertObject(LeftInputAdapterNode.java:142)

      at org.drools.reteoo.SingleObjectSinkAdapter.propagateAssertObject(SingleObjectSinkAdapter.java:42)

      at org.drools.reteoo.ObjectTypeNode.assertObject(ObjectTypeNode.java:185)

      at org.drools.reteoo.EntryPointNode.assertObject(EntryPointNode.java:146)

      at org.drools.common.AbstractWorkingMemory.insert(AbstractWorkingMemory.java:1046)

      at org.drools.common.AbstractWorkingMemory.insert(AbstractWorkingMemory.java:1001)

      at org.drools.common.AbstractWorkingMemory.insert(AbstractWorkingMemory.java:788)

      at org.drools.impl.StatefulKnowledgeSessionImpl.insert(StatefulKnowledgeSessionImpl.java:216)

      at north.stardust2.services.trading.booking.impl.DealBookingProcessorImpl.insertDealFacts(DealBookingProcessorImpl.java:200)

 

 

So to summarize my questions :

-         how can I fix the facts insertion ?

-         is there some newbie mistakes in my approach ?

 

Thanks in advance for your help,

Regards,

 

Joël Costigliola

 

 
Ce courriel et toutes les pièces jointes sont confidentiels et peuvent être couverts par un privilège ou une protection légale. Il est établi à l’attention exclusive de ses destinataires. Toute utilisation de ce courriel non conforme à sa destination, toute diffusion ou toute publication, totale ou partielle, est interdite, sauf autorisation expresse préalable. Toutes opinions exprimées dans ce courriel ne sauraient nécessairement refléter celle de Natixis, de ses filiales. Elles sont aussi susceptibles de modification sans notification préalable. Si vous recevez ce courriel par erreur, merci de le détruire et d’en avertir immédiatement l’expéditeur. L’Internet ne permettant pas d’assurer l’intégrité de ce courriel, Natixis décline toute responsabilité s’il a été altéré, déformé ou falsifié et chaque destinataire qui utilise ce mode de communication est supposé en accepter les risques.
 
This email and any attachment are confidential and may be legally privileged or otherwise protected from disclosure. It is intended only for the stated addressee(s) and access to it by any other person(s) is unauthorised. Any use, dissemination or disclosure not in accordance with its purpose, either in whole or in part, is prohibited without our prior formal approval. Any opinion expressed in this email may not necessarily reflect the opinion of Natixis, its affiliates. It may also be subject to change without prior notice. If you are not an addressee, you must not disclose, copy, circulate or in any other way use or rely on the information contained in this email. If you have received it in error, please inform us immediately and delete all copies. The Internet can not guarantee the integrity of this email therefore Natixis shall not be liable for the email if altered, changed or falsified and anyone who communicates with us by e-mail is taken to accept these risks.