Rule set execution performance and memory consumption issues
by Joe Ammann
Hi all (sorry for the lenghty post, I'm trying to describe a basic pattern)
I have a set of similar rules which work ok, but due to increasing
number of facts are starting to create performance headaches. They all
share a basic pattern, and I wanted to ask if you think that my approach
is feasible - or if I should choose a totally different approach.
The basic pattern is that I have a set of facts of one type, where every
fact "owns" a collection of facts of another type. The association is
not a simple one, rather it has some attributes on its own. The ruleset
is triggered when a new file delivery from a financial data provider
arrives, and we want to check the quality of the new data.
Examples:
FinancialInstrument -> Recommendation/Rating (where the recommendation
has a date, a rating code, etc.)
InstrumentList -> FinancialInstrumentAssignment (where the assignment
has a priority, a validFrom date, etc.)
So we have existing assignments in our database (say 100
InstrumentLists, 20'000 Instruments, 200'000 individual list
assignments), and a new file arrives. The data in this file is converted
and loaded into staging tables, and ends up in special Java objects we
call "SourcingXXX". So a "Instrument" always has a corresponding
"SourcingInstrument" class.
The rules must validate the incoming data and modify (insert, update,
delete) the existing objects. My current basic rule pattern is that for
every such "association", I have 3 or more rules like:
# the mappings that already exist
rule "R1110: MatchExistingAssignment"
when
sa : SourcingAssignment( )
a : Assignment( instrumentId == sa.instrumentId,
instrumentListId == sa.instrumentListId )
then
# apply the logic
retract(a);
retract(sa);
end
# new mappings must be created
rule "R1111: MatchNewAssignment"
when
sa : SourcingAssignment( )
not a : Assignment( instrumentId == sa.instrumentId,
instrumentListId == sa.instrumentListId )
il : InstrumentList( id == sa.instrumentListId )
then
# create new assignment
Assignment a = ...
retract(sa);
end
# missing mappings must be removed
# execute late when the above have had a chance
rule "R1112: RemoveAssignment"
salience -100
when
a : Assignment( )
then
# remove the assignment
retract(a);
end
I shortened these rules to emphasize the basic pattern (hopefully no
typos made..). Normally the rules have more conditions than just the
mapping conditions shown here, and typically there is more than one rule
for the "new case" and the "already exists case".
Typical fact numbers are
- a few 100-1000 of the "owning" object (InstrumentList above)
- a few 10'000 of the "owned" object (Instrument above)
- a few 100'000 of the "assignment" object (Assignment above)
I'm starting to see problematic runtimes (~20 minutes) and memory
consumption (1-2 GB), of course mainly with the "assignment object"
rules, where 100'000 Sourcing objects are matched against 100'000 or so
existing objects. Most of that is of course used during the insertion of
the facts, the fireAllRules() is fast (1-2 minutes).
Is my approach totally flawed? Is there a fundamentally different
approach? I see that some simple cases could be done without Drools in
SQL or Java, but IMHO the more complex cases result in unmanageable
Java/SQL code (we already have one example of a 200 line SQL statement
that nobody dares to touch...)
--
CU, Joe
10 years, 8 months
Jbpm-standalone-designer with postgres
by Naman Shah
Hi,
I need to use the standalone jbpm-desginer-standalone with postgresql.
I check the code and read some where that i need to use the Repository for
further modification.
Is there any documentation out there or some example on this, so that i can
use it for reference.
Thanks
Naman Shah
10 years, 8 months
OptaPlanner: score corruption when using insertLogical with custom objects
by pvandenbrink
Hi,
I've been using Drools Planner with success for a while for our Conversation
planning tool. It's used by schools to plan conversations between teachers
and parents of schoolgoing children.
Recently I checked on the new developments on Drools (the current version
we're using in production is 5.4.0.FINAL) and started to upgrade to
OptaPlanner 6.0.0.Beta5.
Everything looked fine, and seeing how it's now easier to check which
constraints are broken in a given solution, I decided to have a look at that
as well.
And that's where I noticed that we actually had an issue with score
corruption in one of our rules...
The goal of the rule in case is to minimize the amount of timeslots between
the first and last Conversation of a family (they can have conversations for
multiple children.)
The rule uses custom helper objects, like in the NurseRostering example,
which are added with insertLogical.
Conversation is the planning entity, which has the planning variable:
timeslot. FamilyStart and FamilyEnd are custom helper objects to register
the timeslot of the first and last conversation for a family.
The relevant rules are as follows:
rule "familyStart"
salience 1
when
Conversation(
family != null, $family: family,
timeslot != null, $timeslot: timeslot);
not Conversation(
family == $family,
timeslot < $timeslot);
then
insertLogical(new FamilyStart($family, $timeslot));
end
rule "familyEnd"
salience 1
when
Conversation(
family != null, $family: family,
timeslot != null, $timeslot: timeslot);
not Conversation(
family == $family,
timeslot > $timeslot);
then
insertLogical(new FamilyEnd($family, $timeslot));
end
rule "familyCompact"
when
$start: FamilyStart(
family != null, $family: family,
timeslot != null, $timeslotBegin: timeslot);
$end: FamilyEnd(
family == $family,
(timeslot.id - $timeslotBegin.id > family.minimumTimeslotsRequired),
timeslot != null, $timeslotEnd: timeslot);
then
scoreHolder.addSoftConstraintMatch(kcontext, -(($timeslotEnd.getId() -
$timeslotBegin.getId()) - $family.getMinimumTimeslotsRequired()));
end
When running one of my example schedules in FULL_ASSERT mode, the following
error appears during the LocalSearch phase:
Exception in thread "main" java.lang.IllegalStateException: Score
corruption: the workingScore (-2hard/-119soft) is not the uncorruptedScore
(-2hard/-118soft) after completedAction (Teacher 3 family 11 timeslot 2 =>
12):
The corrupted scoreDirector has 1 ConstraintMatch(s) which are in excess
(and should not be there):
defaultpkg/familyCompact/level1/[FamilyEnd Family 11, Timeslot 12,
FamilyStart Family 11, Timeslot 7]=-1
The corrupted scoreDirector has no ConstraintMatch(s) which are missing.
Check your score constraints.
I use a custom ChangeMove and SwapMove implementation. I've also tried using
the new generic Moves, these have the same problem.
It seems like when a move happens that should update an inserted FamilyStart
or FamilyEnd because of a change in timeslot, this change isn't properly
reflected in the scoreDirector.
Do you have any idea what could cause this corruption? The moves do call
beforeVariableChanged and afterVariableChanged on the scoreDirector, and the
FamilyStart and FamilyEnd classes implement an equals and hashcode based on
the Family and Timeslot fields.
(Actually, I've also tried using the default equals and hashcode of Object
for these 2 classes, which causes corruption much faster and with more
ConstraintOccurrences.)
Thanks for any insights,
Pieter
--
View this message in context: http://drools.46999.n3.nabble.com/OptaPlanner-score-corruption-when-using...
Sent from the Drools: User forum mailing list archive at Nabble.com.
10 years, 8 months
Guvnor assets storinf in DB as well as in Repository directory on file-system
by 706826
I am facing the issue that, even when external RDBMS is configured in
repository.xml, the "repository" directory is still created in the file
system and its size is increasing. So when external RDBMS is configured then
there shouldn't be anything created/written in FileSystem. All the data
(except for the indexes) should go to the external RDBMS (in my case Oracle
11).
What I assume is that only the indexes get created in the file system, but
this is not correct. I followed the following steps to check this,
1. I have two Guvnor instances (Guvnor A, Guvnor B) pointing to the same
database (db_Oracle11)
2. Start 'Guvnor A' and generate repository.xml for db_oracle11. Also
configure <DataStore> in repository.xml
i. Persistence managers tested are as follows
a.
org.apache.jackrabbit.core.persistence.bundle.OraclePersistenceManager
b.
org.apache.jackrabbit.core.persistence.pool.OraclePersistenceManager
ii. <FileSystem
class="org.apache.jackrabbit.core.fs.db.OracleFileSystem">
3. copy paste the generated configs in repository.xml of 'Guvnor A' and
'Guvnor B'.
4. Re-Start 'Guvnor A'. 'Guvnor B' is not started. Only 'Guvnor A' is
started.
5. Create and Import assets in 'Guvnor A' using guvnor web interface. These
should be saved in db_Oracle11.
6. Now start 'Guvnor B'. And check all assets created/imported in 'Guvnor A'
should be visible in 'Guvnor B', as both the Guvnors have the same database.
7. Only packages of 'Guvnor A' are visible in 'Guvnor B'. But assets inside
the packages are not available.
Repository.xml is attached. I have also posted the same in the following
thread.
https://community.jboss.org/thread/230419
repository-primary_server.xml
<http://drools.46999.n3.nabble.com/file/n4025303/repository-primary_server...>
repository-secondary_server.xml
<http://drools.46999.n3.nabble.com/file/n4025303/repository-secondary_serv...>
workspace.xml
<http://drools.46999.n3.nabble.com/file/n4025303/workspace.xml>
--
View this message in context: http://drools.46999.n3.nabble.com/Guvnor-assets-storinf-in-DB-as-well-as-...
Sent from the Drools: User forum mailing list archive at Nabble.com.
10 years, 8 months
Testing for non-existance of an object in list
by Joe Ammann
Hi all
I'm still struggling with the handling of lists and membership in the
LHS of rule. I'm trying to formulate a rule that only fires if a certain
object is not yet a member of a list.
Here's what I currently have (Drools 5.5.0 btw):
rule "R2030: UpdateDQAttribute"
dialect "mvel"
agenda-group "PROCESS_CHECKRESULTS"
when
pv : PriceViolation ( p : changedPrice.price, dqc : check )
ps : PriceSeries ( id == p.seriesId )
i : Instrument ( id == ps.instrument.id )
dqa : DQAttribute ( rootId == i.id, objectId == p.id )
not dqcv : DQCheckViolation ( check.id == dqc.id ) from
dqa.checkViolations
then
# debug
for (DQCheckViolation dqcv : dqa.checkViolations) {
logger.debug("dqc: {}, dqcv: {}", dqc.id, dqcv.check.id);
}
end
The rule fires and the log statement produces
PriceCheck: dqc: 6fc6636b-9b7c-4302-bce7-1680eec57830, dqcv:
6fc6636b-9b7c-4302-bce7-1680eec57830
The ids are the same, so I would have expected that last LHS condition
to avoid that the rule fires.
--
CU, Joe
10 years, 8 months
Guvnor seems to copy assets across External RDBMS and file system
by Yuri
I set up SQL Server as the external repository, and left everything else in
guvnor and jboss AS7 configured pretty much according to defaults.
When I create assets in guvnor, I see that changes are being made in the
MSSQL db. Despite changes to the db, however, I'm still seeing changes to
the Repository and Workspace folders. It gets more interesting when I
truncate the tables in the external guvnor db, and restart guvnor, it seems
to get all of its assets back from those two folders.
Am I doing something wrong? Is it reconstructing the assets from lucene
indexes? If MSSQL is already indexing things, is the lucene index still
necessary? Are these basic JCR concepts that I am not grasping?
Thanks, Yuri
10 years, 8 months
Scheduling Application and Optaplanner
by marchias
Just started researching OptaPlanner. Loaded examples in Eclipse and got them
running. I'm interested in extending the Nurse rostering example to work
with a scheduling application I have written. I have some interesting
constraints though that wanted to reach out to the community with to make
sure they can be accomodated. My scheduling app has a hierachical entity
setup with Organizations and child locations. An employee can be configured
to work at multiple locations and are also assigned a "Primary" location
which means they should be picked up on shifts at that location before an
employee that does not have that location as primary. Also I have a priority
group or ranking (1-10) which would be considered in sorting employees as
well. Shifts are location, date/time and skill specific with a configured
amount of resources needed.
Shift Examples
Location 1 = 2 nurses, (Monday Wednesay Friday), 7AM - 11PM
Location 1 = 1 nurse, (Sunday,Tuesday,Thursday,Saturday), 7AM - 11PM
Constraints for an employee
Max Hours/Week: 0.00
Min Hours/Week: 0.00
Max Hours/Day: 0.00
Min Hours/Day: 0.00
Max Days/Week: 0.00
Max Consecutive Days: 0.00
Max Shifts/Day: 0.00
Max Shifts/Month: 2.00
Max Weekend Shifts/Month 0.00
Max Consecutive Shifts of Duration 0.00 consecutive 0.00 hr shifts
--
View this message in context: http://drools.46999.n3.nabble.com/Scheduling-Application-and-Optaplanner-...
Sent from the Drools: User forum mailing list archive at Nabble.com.
10 years, 8 months
