Re: [rules-users] Rule set execution performance and memory consumption issues

Hi Wolfgang On 08/04/2013 03:10 PM, Wolfgang Laun wrote: > A simple test shows that, with the rules as shown, you can insert 100k > Assignments and an equal number of SourceAssignments (with ~300 > InstrumentList facts) in a matter of seconds, irrespective of the > number of hits and misses, either way. I suspect that there is some > other fly in the ointment. I suspect this too, but am unable to lay my fingers on it, unfortunately. That's why I started to question my basic pattern. But all my test trying to get to the core of the problem showed the same as your quick tests - there is absolutely no fundamental problem handling a couple of 100'000 objects. > > There is no "Instrument" pattern in any of the rules, which makes your > post slightly inconsistent in itself. Right - I was trying to explain the pattern which I suspected to be the culprit, without posting a dozen Java classes and a couple of dozens of rules. Always a difficult thing to do. The real rules have many more patterns, after matching the basic objects, e.g. rule "R1110: MatchExistingAssignment" when sa : SourcingAssignment( ) a : Assignment( instrumentId == sa.instrumentId, instrumentListId == sa.instrumentListId ) i : Instrument( id == instrumentId, foType == FOTYPE.Stock, statusCd == INSTRUMENT_STATUS.Draft ) not DQAttribute ( rootId == i.id, attributePath == Instrument._ATTR_statusCd ) .... then I think I have in the meantime found the main issue - it was one single rule that caused the majority of the problem. It was written really stupidly ... The time is now down to ~ 2 minutes. Still I see in the log 2013-08-05 00:36:56.224 [ain] DEBUG: Check: start inserting objects into working memory {|||||| - |} 2013-08-05 00:36:56.225 [ain] DEBUG: Check: next object type: SourcingInstrument, (12668 instances) {|||||| - |} 2013-08-05 00:36:56.701 [ain] DEBUG: Check: next object type: SourcingInstrumentList, (38 instances) {|||||| - |} 2013-08-05 00:36:56.707 [ain] DEBUG: Check: next object type: SourcingAssignment, (194693 instances) {|||||| - |} 2013-08-05 00:36:57.573 [ain] DEBUG: Check: next object type: Instrument, (17876 instances) {|||||| - |} 2013-08-05 00:37:41.009 [ain] DEBUG: Check: next object type: InstrumentList, (49 instances) {|||||| - |} 2013-08-05 00:37:41.359 [ain] DEBUG: Check: finished inserting objects into working memory {|||||| - |} 2013-08-05 00:37:41.360 [ain] DEBUG: Check: fire validate staging rules {|||||| - |} 2013-08-05 00:37:43.502 [ain] DEBUG: Check: validate staging rules finished, 147674 executed {|||||| - |} 2013-08-05 00:37:43.502 [ain] DEBUG: Check: start inserting objects into working memory {|||||| - |} 2013-08-05 00:37:43.502 [ain] DEBUG: Check: next object type: Assignment, (120006 instances) {|||||| - |} 2013-08-05 00:37:50.114 [ain] DEBUG: Check: finished inserting objects into working memory {|||||| - |} And I'm wondering a bit why inserting 17876 Instruments takes 45 seconds, while inserting 194693 SourcingAssignments takes less than a second. I must admit that I have absolutely no clue how I would systematically debug such a situation and try to find out which rules are causing the "problem". I stumbled over this one stupid rule just by going through them one by one. I'm trying to get some experience to cope with such problems - since I'm not the only one writing the rules, I continue to see such "unexplicable" slowdowns. Sigh - I guess I still have A LOT to learn about Drools :-) Thanks for your help Wolfgang - and BTW: your whitepaper about rule patterns is a major reference for my learning. -- CU, Joe _______________________________________________ rules-users mailing list rules-users(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/rules-users