6:34 a.m.
Hi all (sorry for the lenghty post, I'm trying to describe a basic pattern)
I have a set of similar rules which work ok, but due to increasing
number of facts are starting to create performance headaches. They all
share a basic pattern, and I wanted to ask if you think that my approach
is feasible - or if I should choose a totally different approach.
The basic pattern is that I have a set of facts of one type, where every
fact "owns" a collection of facts of another type. The association is
not a simple one, rather it has some attributes on its own. The ruleset
is triggered when a new file delivery from a financial data provider
arrives, and we want to check the quality of the new data.
Examples:
FinancialInstrument -> Recommendation/Rating (where the recommendation
has a date, a rating code, etc.)
InstrumentList -> FinancialInstrumentAssignment (where the assignment
has a priority, a validFrom date, etc.)
So we have existing assignments in our database (say 100
InstrumentLists, 20'000 Instruments, 200'000 individual list
assignments), and a new file arrives. The data in this file is converted
and loaded into staging tables, and ends up in special Java objects we
call "SourcingXXX". So a "Instrument" always has a corresponding
"SourcingInstrument" class.
The rules must validate the incoming data and modify (insert, update,
delete) the existing objects. My current basic rule pattern is that for
every such "association", I have 3 or more rules like:
# the mappings that already exist
rule "R1110: MatchExistingAssignment"
when
sa : SourcingAssignment( )
a : Assignment( instrumentId == sa.instrumentId,
instrumentListId == sa.instrumentListId )
then
# apply the logic
retract(a);
retract(sa);
end
# new mappings must be created
rule "R1111: MatchNewAssignment"
when
sa : SourcingAssignment( )
not a : Assignment( instrumentId == sa.instrumentId,
instrumentListId == sa.instrumentListId )
il : InstrumentList( id == sa.instrumentListId )
then
# create new assignment
Assignment a = ...
retract(sa);
end
# missing mappings must be removed
# execute late when the above have had a chance
rule "R1112: RemoveAssignment"
salience -100
when
a : Assignment( )
then
# remove the assignment
retract(a);
end
I shortened these rules to emphasize the basic pattern (hopefully no
typos made..). Normally the rules have more conditions than just the
mapping conditions shown here, and typically there is more than one rule
for the "new case" and the "already exists case".
Typical fact numbers are
- a few 100-1000 of the "owning" object (InstrumentList above)
- a few 10'000 of the "owned" object (Instrument above)
- a few 100'000 of the "assignment" object (Assignment above)
I'm starting to see problematic runtimes (~20 minutes) and memory
consumption (1-2 GB), of course mainly with the "assignment object"
rules, where 100'000 Sourcing objects are matched against 100'000 or so
existing objects. Most of that is of course used during the insertion of
the facts, the fireAllRules() is fast (1-2 minutes).
Is my approach totally flawed? Is there a fundamentally different
approach? I see that some simple cases could be done without Drools in
SQL or Java, but IMHO the more complex cases result in unmanageable
Java/SQL code (we already have one example of a 200 line SQL statement
that nobody dares to touch...)
--
CU, Joe
8:10 a.m.
A simple test shows that, with the rules as shown, you can insert 100k
Assignments and an equal number of SourceAssignments (with ~300
InstrumentList facts) in a matter of seconds, irrespective of the
number of hits and misses, either way. I suspect that there is some
other fly in the ointment.
There is no "Instrument" pattern in any of the rules, which makes your
post slightly inconsistent in itself.
-W
On 04/08/2013, Joe Ammann <joe(a)pyx.ch> wrote:
Hi all (sorry for the lenghty post, I'm trying to describe a
basic pattern)
I have a set of similar rules which work ok, but due to increasing
number of facts are starting to create performance headaches. They all
share a basic pattern, and I wanted to ask if you think that my approach
is feasible - or if I should choose a totally different approach.
The basic pattern is that I have a set of facts of one type, where every
fact "owns" a collection of facts of another type. The association is
not a simple one, rather it has some attributes on its own. The ruleset
is triggered when a new file delivery from a financial data provider
arrives, and we want to check the quality of the new data.
Examples:
FinancialInstrument -> Recommendation/Rating (where the recommendation
has a date, a rating code, etc.)
InstrumentList -> FinancialInstrumentAssignment (where the assignment
has a priority, a validFrom date, etc.)
So we have existing assignments in our database (say 100
InstrumentLists, 20'000 Instruments, 200'000 individual list
assignments), and a new file arrives. The data in this file is converted
and loaded into staging tables, and ends up in special Java objects we
call "SourcingXXX". So a "Instrument" always has a corresponding
"SourcingInstrument" class.
The rules must validate the incoming data and modify (insert, update,
delete) the existing objects. My current basic rule pattern is that for
every such "association", I have 3 or more rules like:
# the mappings that already exist
rule "R1110: MatchExistingAssignment"
when
sa : SourcingAssignment( )
a : Assignment( instrumentId == sa.instrumentId,
instrumentListId == sa.instrumentListId )
then
# apply the logic
retract(a);
retract(sa);
end
# new mappings must be created
rule "R1111: MatchNewAssignment"
when
sa : SourcingAssignment( )
not a : Assignment( instrumentId == sa.instrumentId,
instrumentListId == sa.instrumentListId )
il : InstrumentList( id == sa.instrumentListId )
then
# create new assignment
Assignment a = ...
retract(sa);
end
# missing mappings must be removed
# execute late when the above have had a chance
rule "R1112: RemoveAssignment"
salience -100
when
a : Assignment( )
then
# remove the assignment
retract(a);
end
I shortened these rules to emphasize the basic pattern (hopefully no
typos made..). Normally the rules have more conditions than just the
mapping conditions shown here, and typically there is more than one rule
for the "new case" and the "already exists case".
Typical fact numbers are
- a few 100-1000 of the "owning" object (InstrumentList above)
- a few 10'000 of the "owned" object (Instrument above)
- a few 100'000 of the "assignment" object (Assignment above)
I'm starting to see problematic runtimes (~20 minutes) and memory
consumption (1-2 GB), of course mainly with the "assignment object"
rules, where 100'000 Sourcing objects are matched against 100'000 or so
existing objects. Most of that is of course used during the insertion of
the facts, the fireAllRules() is fast (1-2 minutes).
Is my approach totally flawed? Is there a fundamentally different
approach? I see that some simple cases could be done without Drools in
SQL or Java, but IMHO the more complex cases result in unmanageable
Java/SQL code (we already have one example of a 200 line SQL statement
that nobody dares to touch...)
--
CU, Joe
6:10 p.m.
Hi Wolfgang
On 08/04/2013 03:10 PM, Wolfgang Laun wrote:
A simple test shows that, with the rules as shown, you can insert
100k
Assignments and an equal number of SourceAssignments (with ~300
InstrumentList facts) in a matter of seconds, irrespective of the
number of hits and misses, either way. I suspect that there is some
other fly in the ointment.
I suspect this too, but am unable to lay my fingers on
it,
unfortunately. That's why I started to question my basic pattern. But
all my test trying to get to the core of the problem showed the same as
your quick tests - there is absolutely no fundamental problem handling a
couple of 100'000 objects.
There is no "Instrument" pattern in any of the rules, which makes your
post slightly inconsistent in itself.
Right - I was trying to explain the pattern
which I suspected to be the
culprit, without posting a dozen Java classes and a couple of dozens of
rules. Always a difficult thing to do. The real rules have many more
patterns, after matching the basic objects, e.g.
rule "R1110: MatchExistingAssignment"
when
sa : SourcingAssignment( )
a : Assignment( instrumentId == sa.instrumentId, instrumentListId ==
sa.instrumentListId )
i : Instrument( id == instrumentId, foType == FOTYPE.Stock,
statusCd == INSTRUMENT_STATUS.Draft )
not DQAttribute ( rootId == i.id, attributePath ==
Instrument._ATTR_statusCd )
....
then
I think I have in the meantime found the main issue - it was one single
rule that caused the majority of the problem. It was written really
stupidly ... The time is now down to ~ 2 minutes. Still I see in the log
2013-08-05 00:36:56.224 [ain] DEBUG: Check:
start inserting objects into working memory {|||||| - |}
2013-08-05 00:36:56.225 [ain] DEBUG: Check: next
object type: SourcingInstrument, (12668 instances) {|||||| - |}
2013-08-05 00:36:56.701 [ain] DEBUG: Check: next
object type: SourcingInstrumentList, (38 instances) {|||||| - |}
2013-08-05 00:36:56.707 [ain] DEBUG: Check: next
object type: SourcingAssignment, (194693 instances) {|||||| - |}
2013-08-05 00:36:57.573 [ain] DEBUG: Check: next
object type: Instrument, (17876 instances) {|||||| - |}
2013-08-05 00:37:41.009 [ain] DEBUG: Check: next
object type: InstrumentList, (49 instances) {|||||| - |}
2013-08-05 00:37:41.359 [ain] DEBUG: Check:
finished inserting objects into working memory {|||||| - |}
2013-08-05 00:37:41.360 [ain] DEBUG: Check: fire
validate staging rules {|||||| - |}
2013-08-05 00:37:43.502 [ain] DEBUG: Check:
validate staging rules finished, 147674 executed {|||||| - |}
2013-08-05 00:37:43.502 [ain] DEBUG: Check:
start inserting objects into working memory {|||||| - |}
2013-08-05 00:37:43.502 [ain] DEBUG: Check: next
object type: Assignment, (120006 instances) {|||||| - |}
2013-08-05 00:37:50.114 [ain] DEBUG: Check:
finished inserting objects into working memory {|||||| - |}
And I'm wondering a bit why inserting 17876 Instruments takes 45
seconds, while inserting 194693 SourcingAssignments takes less than a
second.
I must admit that I have absolutely no clue how I would systematically
debug such a situation and try to find out which rules are causing the
"problem". I stumbled over this one stupid rule just by going through
them one by one. I'm trying to get some experience to cope with such
problems - since I'm not the only one writing the rules, I continue to
see such "unexplicable" slowdowns.
Sigh - I guess I still have A LOT to learn about Drools :-)
Thanks for your help Wolfgang - and BTW: your whitepaper about rule
patterns is a major reference for my learning.
--
CU, Joe
11:29 p.m.
Guete Morge Joe,
from that other thread I can recall that you are using "from
<expression>", which (not knowing the rest) would be high up on my
suspect list.
And, in order to learn more, some monitoring should be considered. The
number of activation creations and deletions (not by firing) can
indicate wasted work that might be avoided.
Cheers
-W
On 05/08/2013, Joe Ammann <joe(a)pyx.ch> wrote:
Hi Wolfgang
On 08/04/2013 03:10 PM, Wolfgang Laun wrote:
> A simple test shows that, with the rules as shown, you can insert 100k
> Assignments and an equal number of SourceAssignments (with ~300
> InstrumentList facts) in a matter of seconds, irrespective of the
> number of hits and misses, either way. I suspect that there is some
> other fly in the ointment.
I suspect this too, but am unable to lay my fingers on it,
unfortunately. That's why I started to question my basic pattern. But
all my test trying to get to the core of the problem showed the same as
your quick tests - there is absolutely no fundamental problem handling a
couple of 100'000 objects.
>
> There is no "Instrument" pattern in any of the rules, which makes your
> post slightly inconsistent in itself.
Right - I was trying to explain the pattern which I suspected to be the
culprit, without posting a dozen Java classes and a couple of dozens of
rules. Always a difficult thing to do. The real rules have many more
patterns, after matching the basic objects, e.g.
rule "R1110: MatchExistingAssignment"
when
sa : SourcingAssignment( )
a : Assignment( instrumentId == sa.instrumentId, instrumentListId ==
sa.instrumentListId )
i : Instrument( id == instrumentId, foType == FOTYPE.Stock,
statusCd == INSTRUMENT_STATUS.Draft )
not DQAttribute ( rootId == i.id, attributePath ==
Instrument._ATTR_statusCd )
....
then
I think I have in the meantime found the main issue - it was one single
rule that caused the majority of the problem. It was written really
stupidly ... The time is now down to ~ 2 minutes. Still I see in the log
2013-08-05 00:36:56.224 [ain] DEBUG: Check:
start inserting objects into working memory {|||||| - |}
2013-08-05 00:36:56.225 [ain] DEBUG: Check: next
object type: SourcingInstrument, (12668 instances) {|||||| - |}
2013-08-05 00:36:56.701 [ain] DEBUG: Check: next
object type: SourcingInstrumentList, (38 instances) {|||||| - |}
2013-08-05 00:36:56.707 [ain] DEBUG: Check: next
object type: SourcingAssignment, (194693 instances) {|||||| - |}
2013-08-05 00:36:57.573 [ain] DEBUG: Check: next
object type: Instrument, (17876 instances) {|||||| - |}
2013-08-05 00:37:41.009 [ain] DEBUG: Check: next
object type: InstrumentList, (49 instances) {|||||| - |}
2013-08-05 00:37:41.359 [ain] DEBUG: Check:
finished inserting objects into working memory {|||||| - |}
2013-08-05 00:37:41.360 [ain] DEBUG: Check: fire
validate staging rules {|||||| - |}
2013-08-05 00:37:43.502 [ain] DEBUG: Check:
validate staging rules finished, 147674 executed {|||||| - |}
2013-08-05 00:37:43.502 [ain] DEBUG: Check:
start inserting objects into working memory {|||||| - |}
2013-08-05 00:37:43.502 [ain] DEBUG: Check: next
object type: Assignment, (120006 instances) {|||||| - |}
2013-08-05 00:37:50.114 [ain] DEBUG: Check:
finished inserting objects into working memory {|||||| - |}
And I'm wondering a bit why inserting 17876 Instruments takes 45
seconds, while inserting 194693 SourcingAssignments takes less than a
second.
I must admit that I have absolutely no clue how I would systematically
debug such a situation and try to find out which rules are causing the
"problem". I stumbled over this one stupid rule just by going through
them one by one. I'm trying to get some experience to cope with such
problems - since I'm not the only one writing the rules, I continue to
see such "unexplicable" slowdowns.
Sigh - I guess I still have A LOT to learn about Drools :-)
Thanks for your help Wolfgang - and BTW: your whitepaper about rule
patterns is a major reference for my learning.
--
CU, Joe
_______________________________________________
rules-users mailing list
rules-users(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/rules-users
1:01 a.m.
On 08/05/2013 06:29 AM, Wolfgang Laun wrote:
Guete Morge Joe,
:-) Guete Morge Wolfgang
from that other thread I can recall that you are using "from
<expression>", which (not knowing the rest) would be high up on my
suspect list.
That's one thing I already learnt :-) I'm not using any
"from" clauses
in my rule sets which are used against the "big" fact sets.
That other rule set from the other thread was used in a different
context, where the number of facts is much smaller (in the hundreds).
And, in order to learn more, some monitoring should be considered. The
number of activation creations and deletions (not by firing) can
indicate wasted work that might be avoided.
I tried to implement something like
that, but must admit that I don't
yet understand how I would do such a monitoring?! All I currently have
are custom AgendaEventListener and WorkingMemoryEventListener. I would
suspect that to implement such a monitoring, I would use
void activationCreated(ActivationCreatedEvent event);
void activationCancelled(ActivationCancelledEvent event);
of the AgendaEventListener. Could you point me to some documentation
that explains when exactly those events are triggered? Until now, I
haven't found too much doc on these aspects of Drools :-/
--
CU, Joe
2:04 a.m.
Grüezi Joe,
On 05/08/2013, Joe Ammann <joe(a)pyx.ch> wrote:
On 08/05/2013 06:29 AM, Wolfgang Laun wrote:
That's one thing I already learnt :-) I'm not using any "from" clauses
in my rule sets which are used against the "big" fact sets.
Fine. Although occasionally useful, "from <expression>" is often
used
as a cover-up for inadequate design of fact types. (from
collect/accumulate is a different matter.)
> And, in order to learn more, some monitoring should be
considered. The
> number of activation creations and deletions (not by firing) can
> indicate wasted work that might be avoided.
I tried to implement something like that, but must admit that I don't
yet understand how I would do such a monitoring?! All I currently have
are custom AgendaEventListener and WorkingMemoryEventListener. I would
suspect that to implement such a monitoring, I would use
void activationCreated(ActivationCreatedEvent event);
void activationCancelled(ActivationCancelledEvent event);
of the AgendaEventListener. Could you point me to some documentation
that explains when exactly those events are triggered? Until now, I
haven't found too much doc on these aspects of Drools :-/
Hmm, there isn't much. Perhaps I should consider writing another white
paper on monitoring.
Activations are created whenever WM changes (or rule base changes)
make LHSs true for some fact set. At first, an activation is added to
the agenda. Firing removes an activation, but so does any change to
the set of facts in the WM. For instance:
rule AB when
A( $aid: id )
not B( id == $aid )
then ... end
Inserting 100k As and *then* 100k matching Bs before calling
fireAllRules creates and deletes 100k activations; with Bs before As
nothing is wasted.
Activations can be destroyed by calling clear() on the Agenda, and
they can "evaporate" by being filtered away by an AgendaFilter that's
passed to the fireXY method.
Cheers
Wolfgang
--
CU, Joe
_______________________________________________
rules-users mailing list
rules-users(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/rules-users
3:30 a.m.
On 08/05/2013 09:04 AM, Wolfgang Laun wrote:
Fine. Although occasionally useful, "from
<expression>" is often used
as a cover-up for inadequate design of fact types. (from
collect/accumulate is a different matter.)
That was exactly the case before we
found out how to make fact types
"more rule friendly". Luckily we are generating the fact classes from a
model, so we could implement some "rule helper functionality" quite
easily. For example, we generate all fact classes in 2 incarnations:
once as fully featured implementation classes with Hibernate, JAXB and
god-knows-what dependencies for the runtime system, and once as "almost
pure Java beans" for using them as models in Guvnor.
Hmm, there isn't much. Perhaps I should consider writing another
white
paper on monitoring.
That would be an excellent idea, me thinks ! I think with the
Annotations coming in Drools 6 one could put together a really useful
set of monitoring annotations (I have some experience doing this for
Java/Spring, but not for Drools, obviously. But I could be a good tester :-)
Activations are created whenever WM changes (or rule base changes)
make LHSs true for some fact set. At first, an activation is added to
the agenda. Firing removes an activation, but so does any change to
the set of facts in the WM. For instance:
rule AB when
A( $aid: id )
not B( id == $aid )
then ... end
Inserting 100k As and *then* 100k matching Bs before calling
fireAllRules creates and deletes 100k activations; with Bs before As
nothing is wasted.
Aha! That could be my problem, I'm doing exactly that in one
case.
Wasn't really aware the the sequence of fact insertion can have such a
great impact...
Activations can be destroyed by calling clear() on the Agenda, and
they can "evaporate" by being filtered away by an AgendaFilter that's
passed to the fireXY method.
I can see use cases for using an AgendaFilter (like "eliminating"
certain rules without changing the rule set and without clobbering the
LHS with guard facts.
But what would be a good example where I would use clear() on the Agenda
? Can't really think of one, but maybe I still did not understand the
concept fully...
--
CU, Joe
4:16 a.m.
On 05/08/2013, Joe Ammann <joe(a)pyx.ch> wrote:
But what would be a good example where I would use clear() on the
Agenda
? Can't really think of one, but maybe I still did not understand the
concept fully...
It is some sort of "cut". You might use it when you have found one
solution and aren't interested in others.
-W
4131
days inactive
4132
days old
7 comments
2 participants
participants (2)
-
Joe Ammann -
Wolfgang Laun