9:09 a.m.
Hi all,
With my set of rules and data, I get OutOfMemory issues. It works fine with
smaller sets of Data.
Looking with a HeapDump analyzer, I found that there was thousands of
RightTuples. In a previous run, before some rules fixes, I also add tons of
JoinNodeLeftTuple's.
They seem to come from poorly written rules; unfortunately, I find it quite
difficult to find and correct the conditions/rules causing these tons of
RightTuple.
Is there any analysis option/tool that can give information on the Left Hand
Side (tuples and sinks), or is there any BestPractices on how to correct the
rules to limit this behavior?
I use Drools 6.1.0.Beta4, with Phreak enabled, but had similary issues with
6.0.0 or 6.0.1
Regards,
Benoît
--
View this message in context:
http://drools.46999.n3.nabble.com/Analysing-OutOfMemory-too-many-RightTup...
Sent from the Drools: User forum mailing list archive at Nabble.com.
9:21 a.m.
Most of the time, a look on a (representative) sample of the rules is
enough for some watchers of this list to drop a few pointers.
There are some Best Practices, but people tend to come up with new
counter-examples ;-)
-W
On 11/06/2014, bvoisin <benoit.voisin(a)igc-erp.com> wrote:
Hi all,
With my set of rules and data, I get OutOfMemory issues. It works fine with
smaller sets of Data.
Looking with a HeapDump analyzer, I found that there was thousands of
RightTuples. In a previous run, before some rules fixes, I also add tons of
JoinNodeLeftTuple's.
They seem to come from poorly written rules; unfortunately, I find it quite
difficult to find and correct the conditions/rules causing these tons of
RightTuple.
Is there any analysis option/tool that can give information on the Left Hand
Side (tuples and sinks), or is there any BestPractices on how to correct the
rules to limit this behavior?
I use Drools 6.1.0.Beta4, with Phreak enabled, but had similary issues with
6.0.0 or 6.0.1
Regards,
Benoît
--
View this message in context:
http://drools.46999.n3.nabble.com/Analysing-OutOfMemory-too-many-RightTup...
Sent from the Drools: User forum mailing list archive at Nabble.com.
_______________________________________________
rules-users mailing list
rules-users(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/rules-users
9:47 a.m.
Most of the RightTuple's sinks are AccumulatorNodes. Here is one of the 80
rules that I believe cause my problem. The other 80 rules are similar, but
will change in the places I formatted in /italic/.
Some information on this rule:
* The EvemetierData is the business class. They are 10-100k instances in the
VM.
* It has a accompaning object EvemetierCalculDHF that store some temporary
rules data. In particular, the calculationMode to use for a given
field=attribute
* The AttributEvemetierDHF is a Meta information class with a dozen
instances (one per field to calculate in EvemetierData) in WM.
* The UNWRAP_NULL it there to enable Null values to passe the accumulator
(an Option would have been clearer)
rule "valo_2a_applyCalcMode_evemetiers_som_nivinf_1"
salience -1110
when
$attDHF:AttributEvemetierDHF()
$evCalcDHF:EvemetierCalculDHF(calcModes.get($attDHF.calculationModeKey)=="/som_nivinf_1/")
$obj:EvemetierData() from $evCalcDHF.evemetier
accumulate (
$src: EvemetierData (
/pere==$obj.niveau,
affperiode==$obj.affperiode,
typeeve_id==$obj.typeeve_id,
dteve==$obj.dteve,
suffixe==$obj.suffixe,/
true);
$acc1 :
sumWithNulls(/DIVIDE($src.getAttributeValue($attDHF.safeGetAttName("valo_2a_applyCalcMode_evemetiers_som_nivinf_1")),
$src.getAttributeValue($attDHF.safeGetCoefinterniv("valo_2a_applyCalcMode_evemetiers_som_nivinf_1")))/
)
);
$targetValue:Object(EQUALS(UNWRAP_NULL(this),$obj.getAttributeValue($attDHF.attName))==false)
from WRAP_NULL($acc1)
then
$obj.setAttributeValue($attDHF.getAttName(), UNWRAP_NULL($targetValue));
end
--
View this message in context:
http://drools.46999.n3.nabble.com/Analysing-OutOfMemory-too-many-RightTup...
Sent from the Drools: User forum mailing list archive at Nabble.com.
10:44 a.m.
To help me understand what is happening in the WM, I've just made a analysis
method (see below), scrutating all the RightTuples from the WM facts and
showing some stats on the 5 biggest ones :
I call it every 100 matchFired and it shows me info like:
Biggest RightTuple types :
- EvemetierData: RightTuple/NotNode[niveau == $niveau, suffixe == $suffixe]
x *209346*
- SuffixeMapDHF: RightTuple/JoinNode[id == $ev.suffixe] x 17610
- EvemetierData: RightTuple/AccumulateNode[this == $obj.father] x 14145
- EvemetierData: RightTuple/AccumulateNode[niveau == $obj.niveau,
affperiode == $obj.affperiode, typeeve_id == $obj.typeeve_id, dteve ==
$obj.dteve, $suffixeMap.toPhaseTOT != null && ( $suffixeMap.toPhaseTOT !=
null && suffixe == $suffixeMap.toPhaseTOT.id )] x 14145
- EvemetierData: RightTuple/NotNode[niveau == $niveau, suffixe == $suffixe,
sources contains $source3] x 11316
I clearly found my suspect N°1. Is there any pre-existing and better written
methods to do this kind of research in drools ?
Regards,
Benoît
Method to be called after each/N match fired or ... It uses Guava for the
collections :
private void analyzeWorkingMemoryTuples()
{
Multiset<String> keys = HashMultiset.create();
for(FactHandle handle:kieSession.getFactHandles())
{
DefaultFactHandle factHandle = (DefaultFactHandle)handle;
String
factType=factHandle.getObject().getClass().getSimpleName();
RightTuple rightTuple = factHandle.getFirstRightTuple();
while(rightTuple!=null)
{
String key = factType + ": RightTuple";
BetaNode sink =
(BetaNode)rightTuple.getRightTupleSink();
if (sink!=null) {
key+= "/" + sink.getClass().getSimpleName();
key+=Arrays.toString(sink.getConstraints());
}
keys.add(key);
rightTuple = rightTuple.getHandleNext();
}
}
ArrayList<Multiset.Entry<String>> entries =
Lists.newArrayList(keys.entrySet());
Collections.sort(entries, new
Comparator<Multiset.Entry<String>>(){
@Override
public int compare(Entry<String> o1, Entry<String> o2)
{
return -Integer.compare(o1.getCount(), o2.getCount());
}
});
String analysis = Joiner.on("\n -
").join(Iterables.limit(entries, 5));
if (!analysis.equals(previousAnalysis))
{
logger.info("Biggest RightTuple types :\n - " + analysis);
previousAnalysis = analysis;
}
}
--
View this message in context:
http://drools.46999.n3.nabble.com/Analysing-OutOfMemory-too-many-RightTup...
Sent from the Drools: User forum mailing list archive at Nabble.com.
2:50 a.m.
Before I say something, I should know:
* If an EvemetierCalculDHF "accompanies" an EvemetierData object: is
this a 1:1 relationship or are there several EvemetierData objects
tied to a single EvemetierCalculDHF?
* Is EvemetierCalculDHF.evemetier a Collection? (Probably "yes" if the
answer to the first question is "several".
* Why is the (custom) evaluator sumWithNulls used here? One would
think that DIVIDE can be written so as never to return null.
* The extra CE "Object(...) from WRAP_NULL($acc1)" looks contrived. A
constraint on $acc1/sumWithNulls can be written within the accumulate.
* Personally, I abhor grappling with null (UN/WRAP_NULL). One should
think that it is possible to design objects in a way that avoids null
being returned by its getters. (Frequently, it is just a matter of a
little extra effort to think up a reasonable default value and write
the object initialisation accordingly.)
I can say one thing right away:
$obj: EvemetierData() from $evCalcDHF.evemetier
accumulate ( $src: EvemetierData (
pere == $obj.niveau,... )
The first CE picks out *some* EvemetierData object (from a Collection
(?) see Q1). The accumulate collects lots of partially equal
EvemetierData objects. Now, if there is more than one EvemetierData in
that collection, a similar thing will happen, very likely, over and
over again, with the same subset of EvemetierData facts.
If a "representative" is required for matching a set of similar facts
in an accumulate, it is best practice to pick the representative using
a restrictive constraint, e.g., if it has a unique integer id
attribute, pick the one with the smallest id.
-W
On 11/06/2014, bvoisin <benoit.voisin(a)igc-erp.com> wrote:
To help me understand what is happening in the WM, I've just made
a analysis
method (see below), scrutating all the RightTuples from the WM facts and
showing some stats on the 5 biggest ones :
I call it every 100 matchFired and it shows me info like:
Biggest RightTuple types :
- EvemetierData: RightTuple/NotNode[niveau == $niveau, suffixe == $suffixe]
x *209346*
- SuffixeMapDHF: RightTuple/JoinNode[id == $ev.suffixe] x 17610
- EvemetierData: RightTuple/AccumulateNode[this == $obj.father] x 14145
- EvemetierData: RightTuple/AccumulateNode[niveau == $obj.niveau,
affperiode == $obj.affperiode, typeeve_id == $obj.typeeve_id, dteve ==
$obj.dteve, $suffixeMap.toPhaseTOT != null && ( $suffixeMap.toPhaseTOT !=
null && suffixe == $suffixeMap.toPhaseTOT.id )] x 14145
- EvemetierData: RightTuple/NotNode[niveau == $niveau, suffixe == $suffixe,
sources contains $source3] x 11316
I clearly found my suspect N°1. Is there any pre-existing and better written
methods to do this kind of research in drools ?
Regards,
Benoît
Method to be called after each/N match fired or ... It uses Guava for the
collections :
private void analyzeWorkingMemoryTuples()
{
Multiset<String> keys = HashMultiset.create();
for(FactHandle handle:kieSession.getFactHandles())
{
DefaultFactHandle factHandle = (DefaultFactHandle)handle;
String
factType=factHandle.getObject().getClass().getSimpleName();
RightTuple rightTuple = factHandle.getFirstRightTuple();
while(rightTuple!=null)
{
String key = factType + ": RightTuple";
BetaNode sink =
(BetaNode)rightTuple.getRightTupleSink();
if (sink!=null) {
key+= "/" + sink.getClass().getSimpleName();
key+=Arrays.toString(sink.getConstraints());
}
keys.add(key);
rightTuple = rightTuple.getHandleNext();
}
}
ArrayList<Multiset.Entry<String>> entries =
Lists.newArrayList(keys.entrySet());
Collections.sort(entries, new
Comparator<Multiset.Entry<String>>(){
@Override
public int compare(Entry<String> o1, Entry<String> o2)
{
return -Integer.compare(o1.getCount(), o2.getCount());
}
});
String analysis = Joiner.on("\n -
").join(Iterables.limit(entries, 5));
if (!analysis.equals(previousAnalysis))
{
logger.info("Biggest RightTuple types :\n - " + analysis);
previousAnalysis = analysis;
}
}
--
View this message in context:
http://drools.46999.n3.nabble.com/Analysing-OutOfMemory-too-many-RightTup...
Sent from the Drools: User forum mailing list archive at Nabble.com.
_______________________________________________
rules-users mailing list
rules-users(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/rules-users
3:34 a.m.
Thanks for your time and answers, and sorry for the incomplete information.
There is a 1:1 relationship
between EvemetierCalculDHF and EvemetierData objects. The reason for
the EvemetierCalculDHF is that we want the EvemetierData structure to be
independent from the rules, and therefore needed a declared fact
(EvemetierCalculDHF) to type safely store some information. Ideally, we
would like to add a trait to the EvemetierData instance, but is it possible
on externally managed beans ?
=> The $evCalcDHF.evemetier returns a single object (and Drools correctly
sees it a a single value and uses this value for the $obj). We could have
written $obj:EvemetierData(this==$evCalcDHF.evemetier) but this created
extra Tuples we want to avoid. And each time the EvemetierData is updated
so is the EvemetierCalculDHF ; therefore the modification propagation still
works despite the from. A binding variable would (correct me if I'm wrong)
have done the same thing but in a possibly clearer way.
The "null wrapping" is mostly historical and will be changed to an Option
in a near future. We functionally want that null + null = null and that
0+null=0. Similarly null/any should return null. But this is not related to
this "too many RightTuple" problem, sorry I did not clean up the rule
before sending it over.
I did not quite understand your last paragraph on a "representative". Do
you have an example I could study ?
In a more general way, if I have a LHS with the following conditions:
$x: A(someConditions)
accumulate ($y:A(someField==$x.somefield); ....)
It seems that the Drools engine will create one RightTuple per instance of
A (and per rule with a similar accumulate), even if no instance of A
matches the someConditions.
If I have many instances of A in the WM, is there a way to prevent Drools
to create the RightTuples for $y until it has found a valid $x, and
therefore create RightTuples only for the A matching someField ? I could
use a "from" in the accumulate, but then the engine will not see updates on
$y instances and won't recalculate the accumulate.
Regards,
Benoît
--
View this message in context:
http://drools.46999.n3.nabble.com/Analysing-OutOfMemory-too-many-RightTup...
Sent from the Drools: User forum mailing list archive at Nabble.com.
8 a.m.
On 12/06/2014, bvoisin <benoit.voisin(a)igc-erp.com> wrote:
I did not quite understand your last paragraph on a
"representative". Do
you have an example I could study ?
Given a set of facts that match some constraints, and you want to
accumulate them in some way, this approach appears to achieve it:
Foo( $bar: bar == "bar-x", ... )
accumulate( Foo( bar == $bar,... ); ...; ... )
But, obviously, there is more than one such Foo where bar == "bar-x",
and hence the first CE matches more than once, while the accumulate
may not collect a set that's different each time. Details may vary, of
course, but there's no guarantee that each accumulate will be unique.
In order to have each set of Foo-facts accumulated only once, some
constraint must be exercised on the "trigger" Foo, e.g., making it
unique among all the candidates:
Foo( $id: id, $bar: bar == "bar-x", ... )
not Foo( id < $id, $bar: bar == "bar-x", ... )
accumulate( Foo( bar == $bar,... ); ...; ... )
Now, this particular accumulation occurs only once.
-W
11:58 a.m.
Thanks I understand now. It is quite clear.
In my case however, the accumulated Foo's will be different for each source
Foo. The two classes could have been different. A more precise sample
conditions would be :
$a: A(someConditions)
accumulate ($b:B(someField==$a.somefield); ....)
My problem being that the Drools engine will create one RightTuple per
instance of B (and per rule with a similar accumulate), even if no instance
of A matches the someConditions.
I will try to add alpha conditions (not related to $a or other CI) to $b,
to reduce the number of joinable instances, but it would be great to be
able to ask drools to wait for the $a to match before looking for the $b's.
Was'nt it something that Phreak enabled ?
And about the diagnostic function, listing the most numerous Tuples. Does a
similar it exist somewhere? If not do you think it could help others ? It
sure helps me finding the related conditions.
Regards,
Benoît
--
View this message in context:
http://drools.46999.n3.nabble.com/Analysing-OutOfMemory-too-many-RightTup...
Sent from the Drools: User forum mailing list archive at Nabble.com.
2:56 a.m.
Hi,
This occurs mainly when if you not executing retract() ot dispose() on
particular object.
If you have employee as an object , and you are doing a transition using
employee object its better to use retract (employee ), which will remove or
clear an object.
regards
Ganesh Neelekani
--
View this message in context:
http://drools.46999.n3.nabble.com/Analysing-OutOfMemory-too-many-RightTup...
Sent from the Drools: User forum mailing list archive at Nabble.com.
3:37 a.m.
Hi Ganesh,
I'm not sure I understand your answer. My problem is on the network of
Tuples that the engine creates on objects in memory. Of course, if I could
retract the EvemetierData and have only a limited set the WM memory would
be smaller and my problem not one any more, but currently this is not an
option.
Regards,
Benoît
2014-06-12 9:56 GMT+02:00 ganeshneelekani [via Drools] <
ml-node+s46999n4030006h53(a)n3.nabble.com>:
Hi,
This occurs mainly when if you not executing retract() ot dispose() on
particular object.
If you have employee as an object , and you are doing a transition using
employee object its better to use retract (employee ), which will remove or
clear an object.
regards
Ganesh Neelekani
------------------------------
If you reply to this email, your message will be added to the discussion
below:
http://drools.46999.n3.nabble.com/Analysing-OutOfMemory-too-many-RightTup...
To unsubscribe from Analysing OutOfMemory, too many RightTuple, click
here
<http://drools.46999.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscr...
.
NAML
<http://drools.46999.n3.nabble.com/template/NamlServlet.jtp?macro=macro_vi...
--
View this message in context:
http://drools.46999.n3.nabble.com/Analysing-OutOfMemory-too-many-RightTup...
Sent from the Drools: User forum mailing list archive at Nabble.com.
3820
days inactive
3821
days old
9 comments
3 participants
participants (3)
-
bvoisin -
ganeshneelekani -
Wolfgang Laun