10:41 p.m.
I have to run rules(1000 rules) against 20-30 million of record each day.
I guess I would have memory issue If I insert all of records into working
memory.
so I'm thinking to have multiple rule base instance and do batch processing
as given below.so that I can avoid memory issue and quick rule processing
a.Take first 1 million records and distribute 1 millon across working
memory of 5 rule base.
b.fire rules in all 5 rule base
c.clear working memory of all rule base
d. take next 1 million record and start from step-a till I process all of
the records.
Please advise whether this is possible/recommended way or not.
thanks
--
View this message in context:
http://www.nabble.com/Rule-processing-for-High-volume-input-tp20370272p20...
Sent from the drools - user mailing list archive at Nabble.com.
4:04 a.m.
I'm assuming you'll be running each rule base in a separate thread simultaneously?
There are no dependencies between the rules in the five rule bases? If so, that will
work just fine.
If your rules are simple enough you can use sequential mode. It can speed up execution
quite a bit.
http://downloads.jboss.com/drools/docs/4.0.7.19894.GA/html_single/index.h...
This could be used in conjunction with the simultaneous execution above.
Finally, you can get some multithreaded benefit in a single rule session if you're
using drools 5 and turn on rule base partitioning. As I understand it this works best
when the rules have less join conditions in them. (i.e. mostly single object conditions,
and less multi-object conditions.)
--- On Thu, 11/6/08, techy <techluver007(a)gmail.com> wrote:
From: techy <techluver007(a)gmail.com>
Subject: [rules-users] Rule processing for High volume input
To: rules-users(a)lists.jboss.org
Date: Thursday, November 6, 2008, 3:41 PM
I have to run rules(1000 rules) against 20-30 million of
record each day.
I guess I would have memory issue If I insert all of
records into working
memory.
so I'm thinking to have multiple rule base instance and
do batch processing
as given below.so that I can avoid memory issue and quick
rule processing
a.Take first 1 million records and distribute 1 millon
across working
memory of 5 rule base.
b.fire rules in all 5 rule base
c.clear working memory of all rule base
d. take next 1 million record and start from step-a till I
process all of
the records.
Please advise whether this is possible/recommended way or
not.
thanks
--
View this message in context:
http://www.nabble.com/Rule-processing-for-High-volume-input-tp20370272p20...
Sent from the drools - user mailing list archive at
Nabble.com.
_______________________________________________
rules-users mailing list
rules-users(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/rules-users
3:29 p.m.
Thanks Greg. Since I'll do update on the facts to control the order of
execution, I don't think i can go for sequential mode.
1. I can't have multiple session from same rulebase and assign each session
to single thread. right?
2. Can I use drools 5 in production environment? if not, when is first prod
release expected?
3. where can I do look for rule base partitioning examples/info?
Greg Barton wrote:
I'm assuming you'll be running each rule base in a separate thread
simultaneously? There are no dependencies between the rules in the five
rule bases? If so, that will work just fine.
If your rules are simple enough you can use sequential mode. It can speed
up execution quite a bit.
http://downloads.jboss.com/drools/docs/4.0.7.19894.GA/html_single/index.h...
This could be used in conjunction with the simultaneous execution above.
Finally, you can get some multithreaded benefit in a single rule session
if you're using drools 5 and turn on rule base partitioning. As I
understand it this works best when the rules have less join conditions in
them. (i.e. mostly single object conditions, and less multi-object
conditions.)
--- On Thu, 11/6/08, techy <techluver007(a)gmail.com> wrote:
> From: techy <techluver007(a)gmail.com>
> Subject: [rules-users] Rule processing for High volume input
> To: rules-users(a)lists.jboss.org
> Date: Thursday, November 6, 2008, 3:41 PM
> I have to run rules(1000 rules) against 20-30 million of
> record each day.
> I guess I would have memory issue If I insert all of
> records into working
> memory.
>
> so I'm thinking to have multiple rule base instance and
> do batch processing
> as given below.so that I can avoid memory issue and quick
> rule processing
>
>
> a.Take first 1 million records and distribute 1 millon
> across working
> memory of 5 rule base.
> b.fire rules in all 5 rule base
> c.clear working memory of all rule base
> d. take next 1 million record and start from step-a till I
> process all of
> the records.
>
>
> Please advise whether this is possible/recommended way or
> not.
>
> thanks
> --
> View this message in context:
> http://www.nabble.com/Rule-processing-for-High-volume-input-tp20370272p20...
> Sent from the drools - user mailing list archive at
> Nabble.com.
>
> _______________________________________________
> rules-users mailing list
> rules-users(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/rules-users
_______________________________________________
rules-users mailing list
rules-users(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/rules-users
--
View this message in context:
http://www.nabble.com/Rule-processing-for-High-volume-input-tp20370272p20...
Sent from the drools - user mailing list archive at Nabble.com.
5656
days inactive
5657
days old
2 comments
2 participants
participants (2)
-
Greg Barton -
techy