<html>
<head>
<meta http-equiv="content-type" content="text/html; charset=ISO-8859-1">
</head>
<body text="#000000" bgcolor="#FFFFFF">
Hi all (sorry for the lenghty post, I'm trying to describe a basic
pattern)<br>
<br>
I have a set of similar rules which work ok, but due to increasing
number of facts are starting to create performance headaches. They
all share a basic pattern, and I wanted to ask if you think that my
approach is feasible - or if I should choose a totally different
approach.<br>
<br>
The basic pattern is that I have a set of facts of one type, where
every fact "owns" a collection of facts of another type. The
association is not a simple one, rather it has some attributes on
its own. The ruleset is triggered when a new file delivery from a
financial data provider arrives, and we want to check the quality of
the new data.<br>
<br>
Examples:<br>
<br>
FinancialInstrument -> Recommendation/Rating (where the
recommendation has a date, a rating code, etc.)<br>
InstrumentList -> FinancialInstrumentAssignment (where the
assignment has a priority, a validFrom date, etc.)<br>
<br>
So we have existing assignments in our database (say 100
InstrumentLists, 20'000 Instruments, 200'000 individual list
assignments), and a new file arrives. The data in this file is
converted and loaded into staging tables, and ends up in special
Java objects we call "SourcingXXX". So a "Instrument" always has a
corresponding "SourcingInstrument" class.<br>
<br>
The rules must validate the incoming data and modify (insert,
update, delete) the existing objects. My current basic rule pattern
is that for every such "association", I have 3 or more rules like:<br>
<br>
<blockquote><tt># the mappings that already exist</tt><br>
<tt>rule "R1110: Match</tt><tt><tt>Existing</tt>Assignment"</tt><br>
<tt>when</tt><br>
<tt> sa : SourcingAssignment( )</tt><br>
<tt> a : Assignment( instrumentId == sa.instrumentId, </tt><tt>instrumentListId
== sa.instrumentListId )</tt><br>
<tt>then</tt><br>
<tt> # apply the logic</tt><br>
<tt> retract(a);</tt><br>
<tt> retract(sa);</tt><br>
<tt>end</tt><br>
<br>
<tt># new mappings must be created</tt><br>
<tt>rule "R1111: MatchNewAssignment"</tt><br>
<tt>when</tt><br>
<tt> sa : </tt><tt><tt>SourcingAssignment</tt>( )</tt><br>
<tt> not a : </tt><tt><tt>Assignment</tt>( instrumentId ==
sa.instrumentId, </tt><tt>instrumentListId ==
sa.instrumentListId )</tt><br>
<tt> il : InstrumentList( id == sa.instrumentListId )</tt><br>
<tt>then</tt><br>
<tt> # create new assignment</tt><br>
<tt> Assignment a = ...</tt><br>
<tt> retract(sa);</tt><br>
<tt>end</tt><br>
<br>
<tt># missing mappings must be removed</tt><br>
<tt># execute late when the above have had a chance</tt><br>
<tt>rule "R1112: RemoveAssignment"</tt><br>
<tt>salience -100</tt><br>
<tt>when</tt><br>
<tt> a : </tt><tt><tt><tt>Assignment</tt></tt>( )</tt><br>
<tt>then</tt><br>
<tt> # remove the assignment</tt><br>
<tt> retract(a);</tt><br>
<tt>end</tt><br>
</blockquote>
I shortened these rules to emphasize the basic pattern (hopefully no
typos made..). Normally the rules have more conditions than just the
mapping conditions shown here, and typically there is more than one
rule for the "new case" and the "already exists case".<br>
<br>
Typical fact numbers are<br>
- a few 100-1000 of the "owning" object (InstrumentList above)<br>
- a few 10'000 of the "owned" object (Instrument above)<br>
- a few 100'000 of the "assignment" object (Assignment above)<br>
<br>
I'm starting to see problematic runtimes (~20 minutes) and memory
consumption (1-2 GB), of course mainly with the "assignment object"
rules, where 100'000 Sourcing objects are matched against 100'000 or
so existing objects. Most of that is of course used during the
insertion of the facts, the fireAllRules() is fast (1-2 minutes).<br>
<br>
Is my approach totally flawed? Is there a fundamentally different
approach? I see that some simple cases could be done without Drools
in SQL or Java, but IMHO the more complex cases result in
unmanageable Java/SQL code (we already have one example of a 200
line SQL statement that nobody dares to touch...)<br>
<pre class="moz-signature" cols="72">--
CU, Joe</pre>
</body>
</html>