[rules-users] Drools 5.3 partitioned rule base

gboro54 gboro54 at gmail.com
Wed Mar 7 07:57:57 EST 2012 

I have been reevaluating our use cases and have decided to make a couple
modifications:

1.Remove jBPM and Ruleflow groups. While it certainly fit a need I have
derived a new way to do the same thing.
2. After reviewing our current ruleset I decided to modify what the rules
actually do. At the end of the day the rules are written to infer certain
things about the order. With this in mind I have decided to let the rules 
generate the billing meta-data about the Order(i.e I am may be eligible for
charge A and for cap B) and have post processing which will use this
information accordingly. The post processing is where the order of the
Orders will matter and I am currently thinking of a couple ways to handle
this.
3. Partitioning the data by account will allow us to multithread easily when
running rules. However rather then have 1 session per account I was thinking
about having sessions that represent a group of accounts. 
4. With 3 in mind I am now trying to determine the best way to do this. My
thinking atm is to create x sessions(for this example let's say 5), each
associated with a thread via fireUntilHalt(). Since orders are streamed in
by another thread reading from the DB my thinking is this is the most
efficient way. However these leads me to a couple questions. How would I
determine that an order has been processes through the ruleset and is ready
for post processing(with fireAllRules I know that once the rules have fired
I can retract the Order and move it to post processing and perhaps this is
the answer that will be provided).  With fireUntilHalt I thought about
raising an event which I would use CDI to listen for and then retract the
Order and pass it along for post processing. Is this a good approach? I am
open to any suggestion here.
5. After watching a video posted by salaboy(http://vimeo.com/27209589) I am
looking into some of the technologies he used. Particularly Kryo. The idea
would be that if an Order was eligible for certain post processing it would
be serialized to disc with a ranking(i.e key). Another part of the process
would then read one at a time and process them in order. Any thoughts around
this(I know this is not a drools question but I figured I would put it out
there anyway). 

Thanks Again
-Mike



laune wrote
> 
> A few random observations.
> 
> A recent experiment has shown that insert takes 0.82 seconds for
> 100000 facts with one very simple rule with a CPU G620 @ 2.60GHz and:
> java version "1.7.0_02"
> Java(TM) SE Runtime Environment (build 1.7.0_02-b13)
> Java HotSpot(TM) 64-Bit Server VM (build 22.0-b10, mixed mode)
> 
> There is, of course, no upper limit for insert as this depends on the
> number and complexity of the rules. And if you apply the jBPM Process
> paradigm, I'm not prepared to make any guesses. (It stands to reason
> that this will incur more overhead than straightforward processing in
> the Expert engine.)
> 
> Estimating 1K per order, I'd say that i/o time for 150M orders is not
> to be neglected.
> 
> I'd investigate batch (but not bulk) insertion of quantities, so that
> inserts and fire calls remain in good proportion. Also, balancing i/o
> threads and processing threads in relation to available CPUs shouldn't
> be different for an application using Drools.
> 
> I've been re-reading some of this thread, and the requirement of
> ordered processing is going to be very difficult if the order this to
> be maintained over 150M facts. I mentioned dynamic property, but many
> different salience levels for a large number of activations is not
> efficient, since this requires a sequential search.
> 
> -W
> 
> 
> On 3 March 2012 21:30, gboro54 <gboro54@> wrote:
>>
>> My previous statement was poorly written(that is what I get for trying to
>> respond via my cell phone ). I agree that the Order fact is simple enough
>> to
>> have a rule retract it once processing is completed. As far as the
>> threading, this is an ejb application being deployed on JBoss 7.1.
>> Because
>> of this I was hoping to use @Asyn and allow the container to really
>> manage
>> the threads. This method would simply request the cached session insert
>> the
>> Order fact, and start the jBPM process to process the Order. Perhaps this
>> part of the design is still flawed as I have only recently been thinking
>> about this and I would welcome all suggestions.
>>
>> Additionally is there any documentation on utilizing drools in a high
>> throughput environment(i.e white papers,etc.) This billing project is the
>> first of many potential projects we are looking to utilize drools for and
>> I
>> would love to share some more documentation on this(as well as read on
>> myself).
>>
>>
>>
>> gboro54 wrote
>> >
>> > Agreed that I can have a rule to retract the order....i was just
>> thinking
>> > if the overhead of a reevaluation but can live with that...As far as
>> the
>> > threading...this logic is going to be deployed on a jboss instance
>> using
>> > ejb so I was thinking about, using the jboss thread pool...so when I
>> say
>> > new thread it may not, be new put, pulled from a pool....does that
>> sound
>> > reasonable?
>> >
>> >
>> >
>> > laune wrote
>> >>
>> >> Starting a new thread (if that's what you mean by process) for reusing
>> >> an existing session to process another order is likely to create more
>> >> overhead
>> >> and it'll just make threads compete for this resources.
>> >>
>> >> Multiple threads each dedicated to a single session object might be a
>> >> better way to go.
>> >>
>> >> I don't see any benefit to make simple order facts into events just
>> for
>> >> the
>> >> sake
>> >> of making them expire automatically. There ought to be a well defined
>> >> state (or states) when processed orders are retracted by some rule.
>> >>
>> >> -W
>> >>
>> >>
>> >> On 3 March 2012 14:11, gboro54 <gboro54@> wrote:
>> >>
>> >>> Mark,
>> >>>
>> >>>  I really appreciate the help and think I have come to a solution.
>> Let
>> >>> me
>> >>> know if this sounds reasonable.
>> >>> 1. We will continue to use jBPM to coordinate the rules and business
>> >>> logic
>> >>> that need to occur in calculating charges for orders. However we will
>> >>> work
>> >>> the process to only work on a per order level.
>> >>> 2. All orders are associated with accounts. We will keep one
>> >>> knowledgebase
>> >>> as the rule sets are the same and we will partition sessions by
>> >>> accounts.
>> >>> The flow will go as follows:
>> >>> a. If the session exists insert the order, start a new process
>> instance
>> >>> and
>> >>> fire all rules
>> >>> b. If the session has not been created: create the session, insert
>> all
>> >>> reference data that will be used by all orders in executions of the
>> rule
>> >>> set, insert the order, start a process, and fire all rules. This
>> session
>> >>> is
>> >>> then cached(via a hashmap more then likely)
>> >>> c. This process will be invoked asyn from the main thread, allowing
>> us
>> >>> to
>> >>> control the multithreading
>> >>> 3. Orders will be treated as an event and will be set to expire in a
>> >>> certain
>> >>> amount of time, allowing us to mange the memory footprint of each
>> >>> session.
>> >>>
>> >>> Does this sound reasonable based on what you know of our usecase?
>> >>> Additionally with expiring Orders does this cause a reevaluation of
>> the
>> >>> rules when the expiration occurs? The only other question I have is
>> does
>> >>> the
>> >>> expiration clock start when no more activation's get created for the
>> >>> given
>> >>> event?
>> >>>
>> >>> Thanks again.
>> >>>
>> >>> --
>> >>> View this message in context:
>> >>>
>> http://drools.46999.n3.nabble.com/Drools-5-3-partitioned-rule-base-tp3793558p3795920.html
>> >>> Sent from the Drools: User forum mailing list archive at Nabble.com.
>> >>> _______________________________________________
>> >>> rules-users mailing list
>> >>> rules-users at .jboss
>> >>> https://lists.jboss.org/mailman/listinfo/rules-users
>> >>>
>> >>
>> >> _______________________________________________
>> >> rules-users mailing list
>> >> rules-users at .jboss
>> >> https://lists.jboss.org/mailman/listinfo/rules-users
>> >>
>> >
>>
>>
>> --
>> View this message in context:
>> http://drools.46999.n3.nabble.com/Drools-5-3-partitioned-rule-base-tp3793558p3796689.html
>> Sent from the Drools: User forum mailing list archive at Nabble.com.
>> _______________________________________________
>> rules-users mailing list
>> rules-users at .jboss
>> https://lists.jboss.org/mailman/listinfo/rules-users
> 
> _______________________________________________
> rules-users mailing list
> rules-users at .jboss
> https://lists.jboss.org/mailman/listinfo/rules-users
> 

--
View this message in context: http://drools.46999.n3.nabble.com/Drools-5-3-partitioned-rule-base-tp3793558p3806627.html
Sent from the Drools: User forum mailing list archive at Nabble.com.

More information about the rules-users mailing list