[rules-users] Drools 5.3 partitioned rule base

Wed Mar 7 09:35:19 EST 2012

On 07/03/2012, gboro54 <gboro54 at gmail.com> wrote:
> I have been reevaluating our use cases and have decided to make a couple
> modifications:

> 4. With 3 in mind I am now trying to determine the best way to do this. My
> thinking atm is to create x sessions(for this example let's say 5), each
> associated with a thread via fireUntilHalt(). Since orders are streamed in
> by another thread reading from the DB my thinking is this is the most
> efficient way. However these leads me to a couple questions. How would I
> determine that an order has been processes through the ruleset and is ready
> for post processing(with fireAllRules I know that once the rules have fired
> I can retract the Order and move it to post processing and perhaps this is
> the answer that will be provided).  With fireUntilHalt I thought about
> raising an event which I would use CDI to listen for and then retract the
> Order and pass it along for post processing. Is this a good approach? I am
> open to any suggestion here.

A low-salience rule ought to detect orders for which no other rules
fire any more - that's the moment when it has to be kicked into post
processing.

> 5. After watching a video posted by salaboy(http://vimeo.com/27209589) I am
> looking into some of the technologies he used. Particularly Kryo. The idea
> would be that if an Order was eligible for certain post processing it would
> be serialized to disc with a ranking(i.e key). Another part of the process
> would then read one at a time and process them in order. Any thoughts around
> this(I know this is not a drools question but I figured I would put it out
> there anyway).

For the session, provide an abstract service via a global that accepts
the retracted order fact. Make postprocessing as independent as
possible, could even RMI to another system if cycles are a bottleneck.
Go to disk only if there's no other way, eg. when it is too slow and
you need to spill to disk.

-W

>
> Thanks Again
> -Mike
>
>
>
> laune wrote
>>
>> A few random observations.
>>
>> A recent experiment has shown that insert takes 0.82 seconds for
>> 100000 facts with one very simple rule with a CPU G620 @ 2.60GHz and:
>> java version "1.7.0_02"
>> Java(TM) SE Runtime Environment (build 1.7.0_02-b13)
>> Java HotSpot(TM) 64-Bit Server VM (build 22.0-b10, mixed mode)
>>
>> There is, of course, no upper limit for insert as this depends on the
>> number and complexity of the rules. And if you apply the jBPM Process
>> paradigm, I'm not prepared to make any guesses. (It stands to reason
>> that this will incur more overhead than straightforward processing in
>> the Expert engine.)
>>
>> Estimating 1K per order, I'd say that i/o time for 150M orders is not
>> to be neglected.
>>
>> I'd investigate batch (but not bulk) insertion of quantities, so that
>> inserts and fire calls remain in good proportion. Also, balancing i/o
>> threads and processing threads in relation to available CPUs shouldn't
>> be different for an application using Drools.
>>
>> I've been re-reading some of this thread, and the requirement of
>> ordered processing is going to be very difficult if the order this to
>> be maintained over 150M facts. I mentioned dynamic property, but many
>> different salience levels for a large number of activations is not
>> efficient, since this requires a sequential search.
>>
>> -W
>>
>>
>> On 3 March 2012 21:30, gboro54 &lt;gboro54@&gt; wrote:
>>>
>>> My previous statement was poorly written(that is what I get for trying to
>>> respond via my cell phone ). I agree that the Order fact is simple enough
>>> to
>>> have a rule retract it once processing is completed. As far as the
>>> threading, this is an ejb application being deployed on JBoss 7.1.
>>> Because
>>> of this I was hoping to use @Asyn and allow the container to really
>>> manage
>>> the threads. This method would simply request the cached session insert
>>> the
>>> Order fact, and start the jBPM process to process the Order. Perhaps this
>>> part of the design is still flawed as I have only recently been thinking
>>> about this and I would welcome all suggestions.
>>>
>>> Additionally is there any documentation on utilizing drools in a high
>>> throughput environment(i.e white papers,etc.) This billing project is the
>>> first of many potential projects we are looking to utilize drools for and
>>> I
>>> would love to share some more documentation on this(as well as read on
>>> myself).
>>>
>>>
>>>
>>> gboro54 wrote
>>> >
>>> > Agreed that I can have a rule to retract the order....i was just
>>> thinking
>>> > if the overhead of a reevaluation but can live with that...As far as
>>> the
>>> > threading...this logic is going to be deployed on a jboss instance
>>> using
>>> > ejb so I was thinking about, using the jboss thread pool...so when I
>>> say
>>> > new thread it may not, be new put, pulled from a pool....does that
>>> sound
>>> > reasonable?
>>> >
>>> >
>>> >
>>> > laune wrote
>>> >>
>>> >> Starting a new thread (if that's what you mean by process) for reusing
>>> >> an existing session to process another order is likely to create more
>>> >> overhead
>>> >> and it'll just make threads compete for this resources.
>>> >>
>>> >> Multiple threads each dedicated to a single session object might be a
>>> >> better way to go.
>>> >>
>>> >> I don't see any benefit to make simple order facts into events just
>>> for
>>> >> the
>>> >> sake
>>> >> of making them expire automatically. There ought to be a well defined
>>> >> state (or states) when processed orders are retracted by some rule.
>>> >>
>>> >> -W
>>> >>
>>> >>
>>> >> On 3 March 2012 14:11, gboro54 &lt;gboro54@&gt; wrote:
>>> >>
>>> >>> Mark,
>>> >>>
>>> >>>  I really appreciate the help and think I have come to a solution.
>>> Let
>>> >>> me
>>> >>> know if this sounds reasonable.
>>> >>> 1. We will continue to use jBPM to coordinate the rules and business
>>> >>> logic
>>> >>> that need to occur in calculating charges for orders. However we will
>>> >>> work
>>> >>> the process to only work on a per order level.
>>> >>> 2. All orders are associated with accounts. We will keep one
>>> >>> knowledgebase
>>> >>> as the rule sets are the same and we will partition sessions by
>>> >>> accounts.
>>> >>> The flow will go as follows:
>>> >>> a. If the session exists insert the order, start a new process
>>> instance
>>> >>> and
>>> >>> fire all rules
>>> >>> b. If the session has not been created: create the session, insert
>>> all
>>> >>> reference data that will be used by all orders in executions of the
>>> rule
>>> >>> set, insert the order, start a process, and fire all rules. This
>>> session
>>> >>> is
>>> >>> then cached(via a hashmap more then likely)
>>> >>> c. This process will be invoked asyn from the main thread, allowing
>>> us
>>> >>> to
>>> >>> control the multithreading
>>> >>> 3. Orders will be treated as an event and will be set to expire in a
>>> >>> certain
>>> >>> amount of time, allowing us to mange the memory footprint of each
>>> >>> session.
>>> >>>
>>> >>> Does this sound reasonable based on what you know of our usecase?
>>> >>> Additionally with expiring Orders does this cause a reevaluation of
>>> the
>>> >>> rules when the expiration occurs? The only other question I have is
>>> does
>>> >>> the
>>> >>> expiration clock start when no more activation's get created for the
>>> >>> given
>>> >>> event?
>>> >>>
>>> >>> Thanks again.
>>> >>>
>>> >>> --
>>> >>> View this message in context:
>>> >>>
>>> http://drools.46999.n3.nabble.com/Drools-5-3-partitioned-rule-base-tp3793558p3795920.html
>>> >>> Sent from the Drools: User forum mailing list archive at Nabble.com.
>>> >>> _______________________________________________
>>> >>> rules-users mailing list
>>> >>> rules-users at .jboss
>>> >>> https://lists.jboss.org/mailman/listinfo/rules-users
>>> >>>
>>> >>
>>> >> _______________________________________________
>>> >> rules-users mailing list
>>> >> rules-users at .jboss
>>> >> https://lists.jboss.org/mailman/listinfo/rules-users
>>> >>
>>> >
>>>
>>>
>>> --
>>> View this message in context:
>>> http://drools.46999.n3.nabble.com/Drools-5-3-partitioned-rule-base-tp3793558p3796689.html
>>> Sent from the Drools: User forum mailing list archive at Nabble.com.
>>> _______________________________________________
>>> rules-users mailing list
>>> rules-users at .jboss
>>> https://lists.jboss.org/mailman/listinfo/rules-users
>>
>> _______________________________________________
>> rules-users mailing list
>> rules-users at .jboss
>> https://lists.jboss.org/mailman/listinfo/rules-users
>>
>
> --
> View this message in context:
> http://drools.46999.n3.nabble.com/Drools-5-3-partitioned-rule-base-tp3793558p3806627.html
> Sent from the Drools: User forum mailing list archive at Nabble.com.
>
> _______________________________________________
> rules-users mailing list
> rules-users at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/rules-users
>