[rules-users] Query on Performance Improvement in Drools 5 while dealing with huge data

[senthij]

Hi There,
 
We are using Drools 5 in our project, the requirement was to have huge
database, the data is being fetched from DB and inserted into drools working
memory.
 
For eg:
Cell - 50k  
ExternalCell - 50k
CellRelation - 2500000 
 
Topology hiearchy:
------------------------
 -Subnetwork  
    - PLMN  (Contained in Subnetwork) 
     - RNC  (Contained in PLMN) 
       - Cells  (Contained in RNC) 
 
 
Problem statement:
Check if bidirectional relation not exist between cells in different
subnetwork.
 
How to prove?
Prove that,
First check:
- CellObA and ExternalCellObB in subnetwork 1, both are related using
CellRelationX.  - Unidirectional 
  note: CellObA will have List property contains all ExternalCellOb names,
so ExternalCellObB name should be part of List if
  there is an CellRelation relating CellObA and ExternalCellObB.
Second check:
- CellObB and ExternalCellObA in subnetwork 2, both are related using
CellRelationY.  - Unidirectional (inturn proves that bi-directional relation
exist)
  note: CellObB will have List property contains all ExternalCellOb names,
so similarly ExternalCellObA should be part of List if
  there is an CellRelation relating CellObB and ExternalCellObA.
 
 Check if there is missing/no relation exist between CellObB and
ExteranlCellObA.
 
 
Cell table has oneToMany relation to CellRelation table. We are using
eclipselink for ORM, since Cell and CellRelation tables are related, while
reteriving Cell, CellRelation are also part of the it (using fetch.type =
LAZY). Since Cell table is kind of master table, we are creating the
required objects by from cell entities and inserting the newly created
objects into working memory. 
 
 
Cell table contains master data and ExternalCell table contains proxy
(duplicate copy of master) data.
 
Mapping from Entities to Objects
Entity                 -  Objects
----------------------------------
Cell                    -   CellOb
ExternalCell       -   ExternalCellOb
CellRelation       -   CellRelation
 
CellOb & ExternalCellOb contains following attributes which are used for
comparison. 
- mcc,mnc,rncId,cellId  and nodeId are all of type integer used for
comparsion.   
- mcc, mnc, rncId and cellId are used to uniquely identify CellOb and its
ExternalCellOb through object equality. These fours attributes are used in
overrided equals() and hashcode() method.
 
--------------------------------------------------------------------------------------------------------------------------------------------------  
Rule file:
rule "Cell Selection Rule"
salience 50
dialect "java"
when
  $ucell:CellOb() from TopologyService.getInstance().getCellObs()
then
  insert($ucell);   // inserts each CellOb into working memory.
end
 
rule "External Cell Selection Rule"
salience 49
dialect "java"
when
  $extucell:ExternalCellOb() from
TopologyService.getInstance().getExternalCellObs()
then
  insert($extucell); // inserts each ExternalCellOb into working memory.
end
 
rule "Bidirectional Relations Evaluation Rule"
salience 10
dialect "java"
when
 //Takes ExternalCellObB from Subnetwork 1
  $extuccell : ExternalCellOb(subnetworkMgrId==1)  
 
 // Takes CellObA from Subnetwork 1, not matching cgiParam (is
mcc+mnc+rncId+cellId) of ExternalCellObB, and seeing if there is
CellRelation exist
 // between  ExternalCellObB and CellObA using contains check
  $cell :
CellObA(subnetworkMgrId==$extuccell.subnetworkMgrId,cgiParam!=$extuccell.cgiParam,adjacentCells
contains $extuccell.fdnReverse) 
 
 //Takes ExternalCellObA from Subnetwork 2, matching CellObA cgiParam (is
mcc+mnc+rncId+cellId) 
  $adjextcell : ExternalCellOb (subnetworkMgrId!=$cell.subnetworkMgrId,
cgiParam==$cell.cgiParam)
 
 //Takes CellObB from Subnetwork 2, matching ExternalCellObB, and seeing if
there is no CellRelation exist between CellObB and
 //ExternalCellObA
  $adjcell : CellOb ( subnetworkMgrId==$adjextcell.subnetworkMgrId,
cgiParam==$extuccell.cgiParam,adjacentCells 
    not contains $adjextcell.fdnReverse)
then 
// FDN format looks like the one mentioned below.
//
SubNetwork=ROOT_OB,SubNetwork=1,Context=01,MElement=1,Rnc=1,Cell=RNC01-104 
  System.out.println("Missing Cell Relation from " + $cell.getFdn() + " to "
+ $adjcell.getFdn());  
end
--------------------------------------------------------------------------------------------------------------------------------------------------
 
Tried following and gave performance imporvements after each change:
- Initially, we use to check the following attributes individually, like
mcc, mnc, rncId and cellId but later we combined and it 
   gave some performance improvements.
- Used subnetworkId to  search CellOb/ExternalCellOb with in a appropriate
subnetwork.
- Java string search trick, cellNames are very lengthy but they vary at the
end of the string, so reverse the string and it improved in the 
  contains/not contains check in the List of strings.   
 
What we wanted to know is from Drools part any scope of improvement/tuning
is still possible to deal with huge data and give much better performance??

Your help and guidance is much appreciated.
 
Regards,
Senthil Jayakumar

-- 
View this message in context: http://drools-java-rules-engine.46999.n3.nabble.com/Query-on-Performance-Improvement-in-Drools-5-while-dealing-with-huge-data-tp1003680p1003680.html
Sent from the Drools - User mailing list archive at Nabble.com.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/rules-users/attachments/20100729/0ca322c6/attachment.html