6:55 a.m.
I've a use case where I need to create a rule dynamically every time the user
sends a request.
My current understanding of Drools is that you need to create a
KnowledgeBuilder and then add the rules
I'm creating the KnowledgeBase as follows.
private static KnowledgeBase readKnowledgeBase() throws Exception {
long t1 = System.currentTimeMillis();
KnowledgeBuilder kbuilder =
KnowledgeBuilderFactory.newKnowledgeBuilder();
long t11 = System.currentTimeMillis();
kbuilder.add(
org.drools.io.ResourceFactory.newByteArrayResource(getRule()),
ResourceType.DRL);
long t2 = System.currentTimeMillis();
KnowledgeBuilderErrors errors = kbuilder.getErrors();
if (errors.size() > 0) {
for (KnowledgeBuilderError error: errors) {
System.err.println(error);
}
throw new IllegalArgumentException("Could not parse
knowledge.");
}
long t3 = System.currentTimeMillis();
KnowledgeBase kbase = KnowledgeBaseFactory.newKnowledgeBase();
long t4 = System.currentTimeMillis();
kbase.addKnowledgePackages(kbuilder.getKnowledgePackages());
long t5 = System.currentTimeMillis();
Collection<KnowledgePackage> kpackages =
kbase.getKnowledgePackages();
for (KnowledgePackage knowledgePackage : kpackages) {
System.out.println("Package -------- " + knowledgePackage.getName());
Collection<Rule> rules = knowledgePackage.getRules();
for (Rule rule : rules) {
System.out.println("****" + rule.getName());
}
}
long t6 = System.currentTimeMillis();
System.out.println( (t11-t1) + " " + (t2-t11) + " " + ( t3-t2) +
" " +
( t4-t3) + " " + ( t5-t4) + " " + ( t6-t5)+ " " );
return kbase;
}
Based on the timing logs the code take majority (more than 80%) of time in
only these two operations. Is there a way to make it faster ?
KnowledgeBuilder kbuilder = KnowledgeBuilderFactory.newKnowledgeBuilder();
kbuilder.add( org.drools.io.ResourceFactory.newByteArrayResource(getRule()),
ResourceType.DRL);
thanks.
--
View this message in context:
http://drools.46999.n3.nabble.com/Is-there-a-faster-way-of-doing-this-in-...
Sent from the Drools: User forum mailing list archive at Nabble.com.
6:58 a.m.
You can do that step once and then reload the compiled rules from the disk,
or use guvnor which will compile the rules for you.
Cheers
On Wed, May 9, 2012 at 12:55 PM, soumya_sd <soumya_sd(a)yahoo.com> wrote:
I've a use case where I need to create a rule dynamically every
time the
user
sends a request.
My current understanding of Drools is that you need to create a
KnowledgeBuilder and then add the rules
I'm creating the KnowledgeBase as follows.
private static KnowledgeBase readKnowledgeBase() throws Exception {
long t1 = System.currentTimeMillis();
KnowledgeBuilder kbuilder =
KnowledgeBuilderFactory.newKnowledgeBuilder();
long t11 = System.currentTimeMillis();
kbuilder.add(
org.drools.io.ResourceFactory.newByteArrayResource(getRule()),
ResourceType.DRL);
long t2 = System.currentTimeMillis();
KnowledgeBuilderErrors errors = kbuilder.getErrors();
if (errors.size() > 0) {
for (KnowledgeBuilderError error: errors) {
System.err.println(error);
}
throw new IllegalArgumentException("Could not parse
knowledge.");
}
long t3 = System.currentTimeMillis();
KnowledgeBase kbase = KnowledgeBaseFactory.newKnowledgeBase();
long t4 = System.currentTimeMillis();
kbase.addKnowledgePackages(kbuilder.getKnowledgePackages());
long t5 = System.currentTimeMillis();
Collection<KnowledgePackage> kpackages =
kbase.getKnowledgePackages();
for (KnowledgePackage knowledgePackage : kpackages) {
System.out.println("Package -------- " +
knowledgePackage.getName());
Collection<Rule> rules =
knowledgePackage.getRules();
for (Rule rule : rules) {
System.out.println("****" + rule.getName());
}
}
long t6 = System.currentTimeMillis();
System.out.println( (t11-t1) + " " + (t2-t11) + " " + ( t3-t2)
+ "
" +
( t4-t3) + " " + ( t5-t4) + " " + ( t6-t5)+ " " );
return kbase;
}
Based on the timing logs the code take majority (more than 80%) of time in
only these two operations. Is there a way to make it faster ?
KnowledgeBuilder kbuilder = KnowledgeBuilderFactory.newKnowledgeBuilder();
kbuilder.add(
org.drools.io.ResourceFactory.newByteArrayResource(getRule()),
ResourceType.DRL);
thanks.
--
View this message in context:
http://drools.46999.n3.nabble.com/Is-there-a-faster-way-of-doing-this-in-...
Sent from the Drools: User forum mailing list archive at Nabble.com.
_______________________________________________
rules-users mailing list
rules-users(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/rules-users
--
- MyJourney @ http://salaboy.wordpress.com
- Co-Founder @ http://www.jugargentina.org
- Co-Founder @ http://www.jbug.com.ar
- Salatino "Salaboy" Mauricio -
7:07 a.m.
salaboy wrote
You can do that step once and then reload the compiled rules from the
disk,
or use guvnor which will compile the rules for you.
Cheers
Thanks for responding so quickly. I tried using Guvnor. Do you think it will
be faster than this ? In the worse case, the rules that I need to apply can
change with every iteration i.e., they are completely dynamic and defined at
runtime. So, storing the rules is not an options.
--
View this message in context:
http://drools.46999.n3.nabble.com/Is-there-a-faster-way-of-doing-this-in-...
Sent from the Drools: User forum mailing list archive at Nabble.com.
7:15 a.m.
Can you explain a little bit more about your context? having completely
dynamic rules is not an usual use case, rules tends to be static because
they define the business logic that you want to apply. So please elaborate
on the context.
On Wed, May 9, 2012 at 1:07 PM, soumya_sd <soumya_sd(a)yahoo.com> wrote:
salaboy wrote
>
> You can do that step once and then reload the compiled rules from the
> disk,
> or use guvnor which will compile the rules for you.
> Cheers
>
>
>
Thanks for responding so quickly. I tried using Guvnor. Do you think it
will
be faster than this ? In the worse case, the rules that I need to apply can
change with every iteration i.e., they are completely dynamic and defined
at
runtime. So, storing the rules is not an options.
--
View this message in context:
http://drools.46999.n3.nabble.com/Is-there-a-faster-way-of-doing-this-in-...
Sent from the Drools: User forum mailing list archive at Nabble.com.
_______________________________________________
rules-users mailing list
rules-users(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/rules-users
--
- MyJourney @ http://salaboy.wordpress.com
- Co-Founder @ http://www.jugargentina.org
- Co-Founder @ http://www.jbug.com.ar
- Salatino "Salaboy" Mauricio -
7:24 a.m.
Could you please provide a more detailed example?
The usecase you propose is really interesting and the optimization of
something very similar is on the todo-list (@salaboy: "request-when" rings a
bell?)
Now, depending on your exact requirements, there could be various options,
more or less "clean" ;)
First of all, we should determine whether a) you need a new rule or b) you
can just use a parametric rule and carry the parameters in a fact, e.g.:
when
ParamHolder( $x : threshold, $bid : beanId )
Bean( id == $bid, value > $x )
then
...
end
If you do need new rules, generating the drl on the fly is feasible but the
parsing and compilation is a bit expensive, so maybe one should try to build
the internal representation, to save at least the parsing process and "jump"
to the RETE construction. Btw, you mentioned that you'd have to generate a
rule for each transaction: would that rule "lifecycle" be limited to the
transaction too? i.e. would it be possible/necessary to remove the rule once
it has been used?
Davide
--
View this message in context:
http://drools.46999.n3.nabble.com/Is-there-a-faster-way-of-doing-this-in-...
Sent from the Drools: User forum mailing list archive at Nabble.com.
9:29 a.m.
Davide Sottara wrote
Could you please provide a more detailed example?
The usecase you propose is really interesting and the optimization of
something very similar is on the todo-list (@salaboy: "request-when" rings
a bell?)
Now, depending on your exact requirements, there could be various options,
more or less "clean" ;)
First of all, we should determine whether a) you need a new rule or b) you
can just use a parametric rule and carry the parameters in a fact, e.g.:
when
ParamHolder( $x : threshold, $bid : beanId )
Bean( id == $bid, value > $x )
then
...
end
If you do need new rules, generating the drl on the fly is feasible but
the parsing and compilation is a bit expensive, so maybe one should try to
build the internal representation, to save at least the parsing process
and "jump" to the RETE construction. Btw, you mentioned that you'd have to
generate a rule for each transaction: would that rule "lifecycle" be
limited to the transaction too? i.e. would it be possible/necessary to
remove the rule once it has been used?
Davide
Both of you are correct. This is not the typical use case Drools is designed
for.
Here are more details about the the context and use case.
Most of these rules will be used for filtering facts.
For example,
MyBean
{
var 1;
var 2;
var n;
}
We want to give the user the ability to define rules that can then be used
to filter a collection of face (Collection<MyBean>) at runtime. So you can
see how this can change with every request. In one case, the user may only
want to see a subset that satisfies the condition
var 1 > 2 and var 2.equals("something")
and the same user may may want to use a rule
var 3 > 10 and var 2.equals("something else").
I hope you get the idea. So, even if my business model is fixed the rules
are not. Although the rules are not very complicated, you can imagine that
the user can defined pretty much anything at runtime.
Can I use a "parameteric" rule for each of fact type in my business model?
--
View this message in context:
http://drools.46999.n3.nabble.com/Is-there-a-faster-way-of-doing-this-in-...
Sent from the Drools: User forum mailing list archive at Nabble.com.
10:20 a.m.
Users changing rules IS the correct use-case of drools (I would even say that if rules
never change, drools is likely to be worse than pure java).
What others said is "changing the rules at each request by regenerating and
recompiling" is not.
I can't imagine that users may change rules at each request, so you must have, at
least for some time (thus a lot of requests), a stable rule base.
What is costly is to compile rules, but you already know that.
So the idea is to keep in some kind of cache the actual rules, and only recompile them
when they change.
The source of rules does not matter : you can generate the DRL according to your
configuration, or make you users directly write rules (with guvnor, which will fits well
if your rules are only basic filters).
The optimisation you have to do is to keep your compiled rules somewhere (by serializing
your KnowlegdePackages or simply keep it in a kind of map), and reuse it until your source
change : you have to find a way to detect that rules (or the config that is used to
generate them) have changed, and then recompile a new up-to-date KnowledgePackage.
10:41 a.m.
Vincent Legendre wrote
Users changing rules IS the correct use-case of drools (I would even say
that if rules never change, drools is likely to be worse than pure java).
What others said is "changing the rules at each request by regenerating
and recompiling" is not.
I can't imagine that users may change rules at each request, so you must
have, at least for some time (thus a lot of requests), a stable rule base.
What is costly is to compile rules, but you already know that.
So the idea is to keep in some kind of cache the actual rules, and only
recompile them when they change.
The source of rules does not matter : you can generate the DRL according
to your configuration, or make you users directly write rules (with
guvnor, which will fits well if your rules are only basic filters).
The optimisation you have to do is to keep your compiled rules somewhere
(by serializing your KnowlegdePackages or simply keep it in a kind of
map), and reuse it until your source change : you have to find a way to
detect that rules (or the config that is used to generate them) have
changed, and then recompile a new up-to-date KnowledgePackage.
Thank you for your insights.
At this point I'm designing for the worst case i.e., I'm assuming the rules
with change with every request.
Moreover, this can happen on a regular basis in the system I'm trying to
design.
Product
{
price
ave_user_rating
num_ratings
category
}
User wants to see Product with with the following rules
at t=t_1 : If product.category in {A,B}.
at t=t_2 : If product.category in {A,B} AND price < 100
at t=t_3 : If product.category in {A,B} AND price < 100 AND num_ratings >
100
You can imagine the user changing these rules in a span of less than a
minute.
I understand that Drools is designed for cases when rules changes often. But
can it support something like the one I described above where rules are
changing at a much faster rate?
Thanks.
--
View this message in context:
http://drools.46999.n3.nabble.com/Is-there-a-faster-way-of-doing-this-in-...
Sent from the Drools: User forum mailing list archive at Nabble.com.
1:22 p.m.
so you are using drools to build on-the-fly queries based on user input.
You could probably do this via "static" rules. Since you basically have 4
variables (price, avg_user_rating, num_ratings, category), you could write
4 rules, one each that filters based on one of the 4 variables and then
chain the execution through salience:
rule price:
salience 1
when
$p: Product(price in param)
then
add $p to result
rule avg_user_rating:
salience 2
when
$p: Product(avg_user_rating > param)
then
add $p to result
and so on
From: soumya_sd <soumya_sd(a)yahoo.com To:
rules-users(a)lists.jboss.org,
Date: 05/09/2012 08:43 AM
Subject: Re: [rules-users] Is there a faster way of doing this in
Drools ?
Sent by: rules-users-bounces(a)lists.jboss.org
Vincent Legendre wrote
100
You can imagine the user changing these rules in a span of less than a
minute.
I understand that Drools is designed for cases when rules changes often.
But
can it support something like the one I described above where rules are
changing at a much faster rate?
Thanks.
--
View this message in context:
http://drools.46999.n3.nabble.com/Is-there-a-faster-way-of-doing-this-in-...
Sent from the Drools: User forum mailing list archive at Nabble.com.
_______________________________________________
rules-users mailing list
rules-users(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/rules-users
-----------------------------------------
The information contained in this communication (including any
attachments hereto) is confidential and is intended solely for the
personal and confidential use of the individual or entity to whom
it is addressed. If the reader of this message is not the intended
recipient or an agent responsible for delivering it to the intended
recipient, you are hereby notified that you have received this
communication in error and that any review, dissemination, copying,
or unauthorized use of this information, or the taking of any
action in reliance on the contents of this information is strictly
prohibited. If you have received this communication in error,
please notify us immediately by e-mail, and delete the original
message. Thank you
Users changing rules IS the correct use-case of drools (I would even say
that if rules never change, drools is likely to be worse than pure
java).
What others said is "changing the rules at each request by
regenerating
and recompiling" is not.
I can't imagine that users may change rules at each request, so you must
have, at least for some time (thus a lot of requests), a stable rule
base.
What is costly is to compile rules, but you already know that.
So the idea is to keep in some kind of cache the actual rules, and only
recompile them when they change.
The source of rules does not matter : you can generate the DRL according
to your configuration, or make you users directly write rules (with
guvnor, which will fits well if your rules are only basic filters).
The optimisation you have to do is to keep your compiled rules somewhere
(by serializing your KnowlegdePackages or simply keep it in a kind of
map), and reuse it until your source change : you have to find a way to
detect that rules (or the config that is used to generate them) have
changed, and then recompile a new up-to-date KnowledgePackage.
Thank you for your insights.
At this point I'm designing for the worst case i.e., I'm assuming the
rules
with change with every request.
Moreover, this can happen on a regular basis in the system I'm trying to
design.
Product
{
price
ave_user_rating
num_ratings
category
}
User wants to see Product with with the following rules
at t=t_1 : If product.category in {A,B}.
at t=t_2 : If product.category in {A,B} AND price < 100
at t=t_3 : If product.category in {A,B} AND price < 100 AND num_ratings
10:43 a.m.
Vincent Legendre wrote
Users changing rules IS the correct use-case of drools (I would even say
that if rules never change, drools is likely to be worse than pure java).
What others said is "changing the rules at each request by regenerating
and recompiling" is not.
I can't imagine that users may change rules at each request, so you must
have, at least for some time (thus a lot of requests), a stable rule base.
What is costly is to compile rules, but you already know that.
So the idea is to keep in some kind of cache the actual rules, and only
recompile them when they change.
The source of rules does not matter : you can generate the DRL according
to your configuration, or make you users directly write rules (with
guvnor, which will fits well if your rules are only basic filters).
The optimisation you have to do is to keep your compiled rules somewhere
(by serializing your KnowlegdePackages or simply keep it in a kind of
map), and reuse it until your source change : you have to find a way to
detect that rules (or the config that is used to generate them) have
changed, and then recompile a new up-to-date KnowledgePackage.
Thank you for your insights.
At this point I'm designing for the worst case i.e., I'm assuming the rules
with change with every request.
Moreover, this can happen on a regular basis in the system I'm trying to
design.
Product
{
price
ave_user_rating
num_ratings
category
}
User wants to see Product with with the following rules
at t=t_1 : If product.category in {A,B}.
at t=t_2 : If product.category in {A,B} AND price < 100
at t=t_3 : If product.category in {A,B} AND price < 100 AND num_ratings >
100
You can imagine the user changing these rules in a span of less than a
minute.
I understand that Drools is designed for cases when rules changes often. But
can it support something like the one I described above where rules are
changing at a much faster rate?
Thanks.
--
View this message in context:
http://drools.46999.n3.nabble.com/Is-there-a-faster-way-of-doing-this-in-...
Sent from the Drools: User forum mailing list archive at Nabble.com.
12:12 p.m.
At this point I'm designing for the worst case i.e., I'm
assuming the rules
with change with every request.
Keeping a cache will work for the worst case, but will work faster for "normal"
(see below) cases.
You can imagine the user changing these rules in a span of less than
a
minute.
Well, no ... it is really hard to imagine that kind of frequence as a normal use over a
long time.
For all systems I made, rules are stable for much more than that. When changing rules, you
may change them every minutes (to test the outcomes, just like debuging a java code), but
the objective is to produce something stable that will be used in a production environment
...
Changing rules every minutes or less in a production environment is definitely not a usual
use-case ...
Why your rules should change every minute ? What is the system your are building ?
I understand that Drools is designed for cases when rules changes
often. But
can it support something like the one I described above where rules are
changing at a much faster rate?
You won't be able to speed up the compile time.
You can only act on the volume and organization of what you are compiling.
So, you can use cache (as said before), and may be you can also cut your rules in smaller
packages and only recompile the package where some modifications take place (if it take
really some mesureable time, it is likely also because you have plenty of rules to
compile).
If you have a single package with few rules, compile time should not be a problem..
And a last thing : In your first post, you said :
Based on the timing logs the code take majority (more than 80%) of
time in
only these two operations. Is there a way to make it faster ?
KnowledgeBuilder kbuilder = KnowledgeBuilderFactory.newKnowledgeBuilder();
kbuilder.add( org.drools.io.ResourceFactory.newByteArrayResource(getRule()),
ResourceType.DRL);
You first have to make the distinction between KnowledgeBuilder compilation time (up to
drools code, and you can't change anything, at least easyly),
and your getRule() method (guess one of your custom code), with may be speeded up.
For instance, if you can't globally cut your compiled rules into smaller compiled
packages, may be you can cut your big resulting DRL in smaller pieces, regenerating only
what have changed ...
12:36 p.m.
Vincent Legendre wrote
> At this point I'm designing for the worst case i.e., I'm assuming the
> rules
> with change with every request.
Keeping a cache will work for the worst case, but will work faster for
"normal" (see below) cases.
I don't see how can a cache would work in the worse case. I mean if the rule
is not there in the cache then I've to figure out a way to load it. So in
the worse case, you will "always" have a cache miss. Is this not correct ?
Vincent Legendre wrote
> You can imagine the user changing these rules in a span of less than a
> minute.
Well, no ... it is really hard to imagine that kind of frequence as a
normal use over a long time.
For all systems I made, rules are stable for much more than that. When
changing rules, you may change them every minutes (to test the outcomes,
just like debuging a java code), but the objective is to produce something
stable that will be used in a production environment ...
Changing rules every minutes or less in a production environment is
definitely not a usual use-case ...
Why your rules should change every minute ? What is the system your are
building ?
I just gave you an example of the type of use case I want to implement -
realtime filtering of data. In this case, the data is product data and the
rules are filtering rules that the user has defined. Imagine if you want to
implement a website where the user can control the products filters. Again,
with all due respect, if you haven't seen a use case doesn't mean it doesn't
exist or is not relevant in other situations. I'm building an experimental
prototype, so for now if you can just assume that such a use case exists I
would appreciate that.
Vincent Legendre wrote
> I understand that Drools is designed for cases when rules changes often.
> But
> can it support something like the one I described above where rules are
> changing at a much faster rate?
You won't be able to speed up the compile time.
You can only act on the volume and organization of what you are compiling.
So, you can use cache (as said before), and may be you can also cut your
rules in smaller packages and only recompile the package where some
modifications take place (if it take really some mesureable time, it is
likely also because you have plenty of rules to compile).
If you have a single package with few rules, compile time should not be a
problem..
For now, I cannot imagine these rules to very complex. And I think I can
limit them to a single package and a few rules (less than 10.)
Vincent Legendre wrote
And a last thing : In your first post, you said :
>Based on the timing logs the code take majority (more than 80%) of time in
>only these two operations. Is there a way to make it faster ?
>
>KnowledgeBuilder kbuilder = KnowledgeBuilderFactory.newKnowledgeBuilder();
>kbuilder.add(
org.drools.io.ResourceFactory.newByteArrayResource(getRule()),
>ResourceType.DRL);
You first have to make the distinction between KnowledgeBuilder
compilation time (up to drools code, and you can't change anything, at
least easyly),
and your getRule() method (guess one of your custom code), with may be
speeded up.
For instance, if you can't globally cut your compiled rules into smaller
compiled packages, may be you can cut your big resulting DRL in smaller
pieces, regenerating only what have changed ...
My getRule() just reads a drl file and returns it as a byte[]. Currently
it's taking around 0ms. So I don't think this is a bottleneck.
--
View this message in context:
http://drools.46999.n3.nabble.com/Is-there-a-faster-way-of-doing-this-in-...
Sent from the Drools: User forum mailing list archive at Nabble.com.
1:25 p.m.
I did not assume that I've seen all use cases, nor that what I am a telling is the
absolute truth. See no offense in my previous post.
Yes, I was supposing some use case of my own experience, but please note it is the first
post when you describe what you are really trying to do, ie implementing a dymanic filter
configured with a web form that generates rules to do the filtering job.
This is indeed highly dynamic, and for 10 rules, the cache will not be very relevant.
And you are perfectly correct when you say that if rules change at each request, using a
cache won't speed the process.
The cache is only usefull if the rules are stable for a time, or if only some few rules
change from one time to the next one.
But if you have 10 rules, compile time should not be so long, so I guess that the guy
writing filtering rules is expecting very quick results (so I guess that you have few data
too, otherwise the compile time would be far much smaller that processing time, especially
for 10 rules ...).
You want to test the adequation of drools for a use case where a user can set a set of
filtering rules via a web form to filter some data somewhere ...
I would say that :
- If there is only simple filters you should not use drools (an in-memory DB is not so
costly, at least not more than a drools' WM, and with hibernate you can write queries
using a POJO-like syntax which is very close to rules' LHS)
- You said that you are looking to drools to allow writing more complex rules than
simple filter rules (like chaining and so on). In that case, SQL won't fit, and then
you have to accept a performance overcome just because drools does much more than SQL. And
to deal with that overcome, you have to think about partitionning/caching your rules (but
for 10 rules, it will be difficult)
1:35 p.m.
And I did not see all pending posts ...
By reading them, there is another fact that lead to "don't use drools".
You said that you query a distant DB. So, if you write rules to filter your data, you will
have first to get ALL data from your distant DB, add ALL data in the drools WM, and only
then fire rules on them to filter the data and get only a subset of them.
To me (my opinion), it is an anti-pattern. Filtering a large amount of data and return
only a subset of them is the database job.
Oh, yes, you intend to do complex rules that cannot be set as SQL like syntax ...
You should think of cutting you system in two, by making each tool doing what it is
designed for :
- use SQL like for basic filtering, by sending the query to the distant DB and thus
getting ONLY the data needed
- make drools running of this subset of data to implement more complex things
1:47 p.m.
Vincent Legendre wrote
And I did not see all pending posts ...
By reading them, there is another fact that lead to "don't use drools".
You said that you query a distant DB. So, if you write rules to filter
your data, you will have first to get ALL data from your distant DB, add
ALL data in the drools WM, and only then fire rules on them to filter the
data and get only a subset of them.
To me (my opinion), it is an anti-pattern. Filtering a large amount of
data and return only a subset of them is the database job.
The filtering options provided by the remote databases (most of them are
exposed as REST APIs) are "limited". I uses these options whenever I can to
filter data at the source. However, sometimes it's not possible to filter
everything at the source. In such cases I don't have any other option but to
get all the data and then apply my filters "locally."
Vincent Legendre wrote
Oh, yes, you intend to do complex rules that cannot be set as SQL like
syntax ...
You should think of cutting you system in two, by making each tool doing
what it is designed for :
- use SQL like for basic filtering, by sending the query to the distant
DB and thus getting ONLY the data needed
- make drools running of this subset of data to implement more complex
things
Now as discussed below, I can apply "simple" filters by using a in-memory
database (or inserting the data into a SQL database and then using SQL
queries) OR load everything into a WM and then apply these "dynamic" rules.
I think both approaches can work, I just need to figure out the tradeoffs.
Thanks again !
--
View this message in context:
http://drools.46999.n3.nabble.com/Is-there-a-faster-way-of-doing-this-in-...
Sent from the Drools: User forum mailing list archive at Nabble.com.
2:12 p.m.
... OR load everything into a WM and then apply these
"dynamic" rules.
Beware, I am not sure whether we can add/remove rules in an existing session (I don't
think so but I may be wrong).
Basically, a WM is built from a kBase, not the inverse ...
So you will have to recompile rules, that re-add all data in the WM ... really not a good
idea then ...
But, if you have your own GUI to set the filters, you should have some masks to speficify
your filtering values, or at least some objects to store the filters/user's choices to
display them in some screens, right ?
In that case, you can add these objects directly in the WM, and write generic rules that
use these objects to filter the raw data (as 2 or 3 previous post suggest). This way, you
build your KB once, create a 'deamon' session, add all raw data once, and then
only insert/retract filtering objects, with the results updated dynamically too. And if
you can only work on differences (ie a result entering or getting out the global result
list by invoking a gui callback), it can even be faster than a DB ... because RETE is good
to propagate small diffs over a large amount of data and deduce related diffs on the
outcome (this is drools-planner raw idea too).
2:29 p.m.
Vincent Legendre wrote
> ... OR load everything into a WM and then apply these "dynamic" rules.
Beware, I am not sure whether we can add/remove rules in an existing
session (I don't think so but I may be wrong).
Basically, a WM is built from a kBase, not the inverse ...
So you will have to recompile rules, that re-add all data in the WM ...
really not a good idea then ...
But, if you have your own GUI to set the filters, you should have some
masks to speficify your filtering values, or at least some objects to
store the filters/user's choices to display them in some screens, right ?
In that case, you can add these objects directly in the WM, and write
generic rules that use these objects to filter the raw data (as 2 or 3
previous post suggest). This way, you build your KB once, create a
'deamon' session, add all raw data once, and then only insert/retract
filtering objects, with the results updated dynamically too. And if you
can only work on differences (ie a result entering or getting out the
global result list by invoking a gui callback), it can even be faster than
a DB ... because RETE is good to propagate small diffs over a large amount
of data and deduce related diffs on the outcome (this is drools-planner
raw idea too).
thanks again everyone ! I think I've enough information at this point to do
some more prototypes to actually get a better idea of all the important
things people have shared in this post.
I'm sure I'll have more "implementation" or "drools" specific
questions once
I start exploring a little more.
Thanks.
--
View this message in context:
http://drools.46999.n3.nabble.com/Is-there-a-faster-way-of-doing-this-in-...
Sent from the Drools: User forum mailing list archive at Nabble.com.
4:50 p.m.
@Vincent: it should indeed be possible to hot-plug rules directly into a
knowledge BASE (meaning all sessions derived from that KB will be affected:
existing facts will be matched against the new rules). More typically, this
is done using a Knowledge Agent to manage the Knowledge Base.
Whether this is appropriate for this use case or not, is a totally different
matter, as thoroughly discussed :)
--
View this message in context:
http://drools.46999.n3.nabble.com/Is-there-a-faster-way-of-doing-this-in-...
Sent from the Drools: User forum mailing list archive at Nabble.com.
4:50 p.m.
@Vincent: it should indeed be possible to hot-plug rules directly into a
knowledge BASE (meaning all sessions derived from that KB will be affected:
existing facts will be matched against the new rules). More typically, this
is done using a Knowledge Agent to manage the Knowledge Base.
Whether this is appropriate for this use case or not, is a totally different
matter, as thoroughly discussed :)
--
View this message in context:
http://drools.46999.n3.nabble.com/Is-there-a-faster-way-of-doing-this-in-...
Sent from the Drools: User forum mailing list archive at Nabble.com.
3:03 a.m.
ok thanks.
seems dangerous anyway ..
and you said "should be possible", which is (according to what I know in
english) not really sure ?
----- Mail original -----
De: "Davide Sottara" <dsotty(a)gmail.com>
À: rules-users(a)lists.jboss.org
Envoyé: Mercredi 9 Mai 2012 23:50:54
Objet: Re: [rules-users] Is there a faster way of doing this in Drools ?
@Vincent: it should indeed be possible to hot-plug rules directly into a
knowledge BASE (meaning all sessions derived from that KB will be affected:
existing facts will be matched against the new rules). More typically, this
is done using a Knowledge Agent to manage the Knowledge Base.
Whether this is appropriate for this use case or not, is a totally different
matter, as thoroughly discussed :)
--
View this message in context:
http://drools.46999.n3.nabble.com/Is-there-a-faster-way-of-doing-this-in-...
Sent from the Drools: User forum mailing list archive at Nabble.com.
_______________________________________________
rules-users mailing list
rules-users(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/rules-users
3:12 a.m.
"should" as in "bugs/regressions apart" :)
--
View this message in context:
http://drools.46999.n3.nabble.com/Is-there-a-faster-way-of-doing-this-in-...
Sent from the Drools: User forum mailing list archive at Nabble.com.
1:39 p.m.
Vincent Legendre wrote
I did not assume that I've seen all use cases, nor that what I am a
telling is the absolute truth. See no offense in my previous post.
No offense taken at all. In fact, I would like to thank you for asking
important questions. These questions help me validate my design decisions.
Also, given that I'm new to Drools it also helps me understand it's
strengths and weaknesses.
Vincent Legendre wrote
Yes, I was supposing some use case of my own experience, but please note
it is the first post when you describe what you are really trying to do,
ie implementing a dymanic filter configured with a web form that generates
rules to do the filtering job.
This is indeed highly dynamic, and for 10 rules, the cache will not be
very relevant.
And you are perfectly correct when you say that if rules change at each
request, using a cache won't speed the process.
My example was to provide with you will enough context regarding a dynamic
filter where are highly dynamic. My user interface will be designed in such
a way to such that the user can change input values (which will then
generate the appropriate underlying rules.) So the user is not directly
writing rules.
Vincent Legendre wrote
You want to test the adequation of drools for a use case where a user can
set a set of filtering rules via a web form to filter some data somewhere
...
I would say that :
- If there is only simple filters you should not use drools (an
in-memory DB is not so costly, at least not more than a drools' WM, and
with hibernate you can write queries using a POJO-like syntax which is
very close to rules' LHS)
This looks like a good alternative. Let me explore this option. Thanks.
Vincent Legendre wrote
- You said that you are looking to drools to allow writing more complex
rules than simple filter rules (like chaining and so on). In that case,
SQL won't fit, and then you have to accept a performance overcome just
because drools does much more than SQL. And to deal with that overcome,
you have to think about partitionning/caching your rules (but for 10
rules, it will be difficult)
At this point I think I can accept the performance overhead. Being new to
Drools I was wondering if there is a way to make this stuff run faster.
--
View this message in context:
http://drools.46999.n3.nabble.com/Is-there-a-faster-way-of-doing-this-in-...
Sent from the Drools: User forum mailing list archive at Nabble.com.
12:26 p.m.
If the structure of the constraints/filters the user can define remains
static, and only the values are changing, then a single rule with
parametric joins will be enough. E.g.:
Params( $cat : categorySet, $minPrice : minPrice, $maxPrice : maxPrice, ....
)
Product( category in $cat, price >= $minPrice && <= $maxPrice, num_ratings
... )
You'd then have to update the "Params" according to what the user e.g.
selects from a checkbox.
A "null" selection would then be converted e.g. categorySet = all
categories, minPrice = -infinity, maxPrice = +infinity etc...
If, on the other hand, the user is completely free to create their own
rules, this strategy may not be applicable. Wouldn't then the bottleneck be
the user writing/picking the rules (seconds) rather than their compilation
(a few milliseconds)? If so, you might have "horizontal" scalability issues
- allowing more and more users to do this - rather than "vertical" ones -
responding faster to a single user.
If, instead, the criteria are built automatically by a software agent of
some sort, then the direct creation might be taken into consideration. You
can skip the parsing step by creating the Abstract Syntax Tree directly -
the compiler APIs ("Descrs") have a nice fluent structure.
Davide
btw, what are your rules supposed to do as a consequence? Do you have
chaining? or actions?
If all you need is selection and filtering, I wonder if a production rule
system is the best option
for your application, or at least this sub-module...
--
View this message in context:
http://drools.46999.n3.nabble.com/Is-there-a-faster-way-of-doing-this-in-...
Sent from the Drools: User forum mailing list archive at Nabble.com.
12:37 p.m.
Merely filtering a subset of data items from a collection according to
user's changing needs and whims is NOT the primary use case of a production
rule system. This kind of problem has been solved adequately and
efficiently by SQL queries or some similar DB query technique.
It is true that such filtering rules, if compiled and stored in a
KnowledgeBase, can achieve the same thing, but, since rules are targeted at
a more sophisticated use cases, this will never be as efficient as a simple
query system.
-W
12:52 p.m.
laune wrote
Merely filtering a subset of data items from a collection according to
user's changing needs and whims is NOT the primary use case of a
production
rule system. This kind of problem has been solved adequately and
efficiently by SQL queries or some similar DB query technique.
It is true that such filtering rules, if compiled and stored in a
KnowledgeBase, can achieve the same thing, but, since rules are targeted
at
a more sophisticated use cases, this will never be as efficient as a
simple
query system.
-W
you bring up an excellent point.
What if the data is streaming in nature and is not stored in a database and
outside of your control.
Another approach is to cache the data locally (in a database) and then use
SQL to filter it.
I decided not to use the database approach because of two reasons.
1. The database write operation may be expensive.
2. In the future I may decide apply more complex consequence to my rules
than just filtering.
Thanks.
--
View this message in context:
http://drools.46999.n3.nabble.com/Is-there-a-faster-way-of-doing-this-in-...
Sent from the Drools: User forum mailing list archive at Nabble.com.
1:09 p.m.
On 9 May 2012 19:52, soumya_sd <soumya_sd(a)yahoo.com> wrote:
laune wrote
>
> Merely filtering a subset of data items from a collection according to
> user's changing needs and whims is NOT the primary use case of a
> production
> rule system. This kind of problem has been solved adequately and
> efficiently by SQL queries or some similar DB query technique.
>
> It is true that such filtering rules, if compiled and stored in a
> KnowledgeBase, can achieve the same thing, but, since rules are targeted
> at
> a more sophisticated use cases, this will never be as efficient as a
> simple
> query system.
>
> -W
>
>
you bring up an excellent point.
What if the data is streaming in nature and is not stored in a database and
outside of your control.
It you can insert it into a WM, it can't be "outside of your control". If
both data
and selection rules arrive at random, it's going to be chaotic. Some sort
of higher order control pattern will have to take care of that...
Another approach is to cache the data locally (in a database) and then use
SQL to filter it.
I decided not to use the database approach because of two reasons.
1. The database write operation may be expensive.
Storing a record in a DB is not really expensive.
2. In the future I may decide apply more complex consequence to my
rules
than just filtering.
The effort for processing of selected entities is the same, irrespective of
their source, be it
a DB or a WM. Compiling some processing statements could be done by
invoking the
Java compiler, without the additional overhead of compiling DRL first.
-W
Thanks.
--
View this message in context:
http://drools.46999.n3.nabble.com/Is-there-a-faster-way-of-doing-this-in-...
Sent from the Drools: User forum mailing list archive at Nabble.com.
_______________________________________________
rules-users mailing list
rules-users(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/rules-users
1:21 p.m.
laune wrote
On 9 May 2012 19:52, soumya_sd <soumya_sd@> wrote:
>
> laune wrote
> >
> > Merely filtering a subset of data items from a collection according to
> > user's changing needs and whims is NOT the primary use case of a
> > production
> > rule system. This kind of problem has been solved adequately and
> > efficiently by SQL queries or some similar DB query technique.
> >
> > It is true that such filtering rules, if compiled and stored in a
> > KnowledgeBase, can achieve the same thing, but, since rules are
> targeted
> > at
> > a more sophisticated use cases, this will never be as efficient as a
> > simple
> > query system.
> >
> > -W
> >
> >
>
> you bring up an excellent point.
>
> What if the data is streaming in nature and is not stored in a database
> and
> outside of your control.
>
It you can insert it into a WM, it can't be "outside of your control". If
both data
and selection rules arrive at random, it's going to be chaotic. Some sort
of higher order control pattern will have to take care of that...
Correct. Pardon my language. What I really meant by "outside of my control"
is that source databases are not controlled by me. I query external
databases at runtime to get the results I need and then filter a subset
based on the user defined needs.
Currently I'm using Apache Camel for getting my data from remote sources.
laune wrote
>
> Another approach is to cache the data locally (in a database) and then
> use
> SQL to filter it.
>
> I decided not to use the database approach because of two reasons.
>
> 1. The database write operation may be expensive.
>
Storing a record in a DB is not really expensive.
> 2. In the future I may decide apply more complex consequence to my rules
> than just filtering.
>
The effort for processing of selected entities is the same, irrespective
of
their source, be it
a DB or a WM. Compiling some processing statements could be done by
invoking the
Java compiler, without the additional overhead of compiling DRL first.
I didn't understand what you mean by "Compiling some processing statements
could be done by
invoking the Java compiler, without the additional overhead of compiling DRL
first.".
--
View this message in context:
http://drools.46999.n3.nabble.com/Is-there-a-faster-way-of-doing-this-in-...
Sent from the Drools: User forum mailing list archive at Nabble.com.
9:42 p.m.
You can compile Java by invoking javac from a Java program
and load and execute the resulting class file. 1.7 has an API for
calling the Java compiler.
-W
On 9 May 2012 20:21, soumya_sd <soumya_sd(a)yahoo.com> wrote:
laune wrote
>
> On 9 May 2012 19:52, soumya_sd <soumya_sd@> wrote:
>
I didn't understand what you mean by "Compiling some processing statements
could be done by
invoking the Java compiler, without the additional overhead of compiling
DRL
first.".
--
View this message in context:
http://drools.46999.n3.nabble.com/Is-there-a-faster-way-of-doing-this-in-...
Sent from the Drools: User forum mailing list archive at Nabble.com.
_______________________________________________
rules-users mailing list
rules-users(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/rules-users
1:22 p.m.
@ laune - so are you suggesting that I don't use Drools ?
--
View this message in context:
http://drools.46999.n3.nabble.com/Is-there-a-faster-way-of-doing-this-in-...
Sent from the Drools: User forum mailing list archive at Nabble.com.
12:45 p.m.
Davide, thanks for your insights and questions. All of these are good
questions. Please see my responses below.
Davide Sottara wrote
If the structure of the constraints/filters the user can define remains
static, and only the values are changing, then a single rule with
parametric joins will be enough. E.g.:
Params( $cat : categorySet, $minPrice : minPrice, $maxPrice : maxPrice,
.... )
Product( category in $cat, price >= $minPrice && <= $maxPrice, num_ratings
... )
You'd then have to update the "Params" according to what the user e.g.
selects from a checkbox.
A "null" selection would then be converted e.g. categorySet = all
categories, minPrice = -infinity, maxPrice = +infinity etc...
This may work but I can see this a limitation in many cases.
Davide Sottara wrote
If, on the other hand, the user is completely free to create their own
rules, this strategy may not be applicable.
This is closer to what I want to do.
Davide Sottara wrote
Wouldn't then the bottleneck be the user writing/picking the rules
(seconds) rather than their compilation (a few milliseconds)? If so, you
might have "horizontal" scalability issues - allowing more and more users
to do this - rather than "vertical" ones - responding faster to a single
user.
Why do you think horizontal scalability is an issue ?
Davide Sottara wrote
If, instead, the criteria are built automatically by a software agent of
some sort, then the direct creation might be taken into consideration. You
can skip the parsing step by creating the Abstract Syntax Tree directly -
the compiler APIs ("Descrs") have a nice fluent structure.
Is parsing really expensive ? Do you have any idea of how much improvements
one can get by directly putting the rules into as AST.
Davide Sottara wrote
btw, what are your rules supposed to do as a consequence? Do you have
chaining? or actions?
If all you need is selection and filtering, I wonder if a production rule
system is the best option
for your application, or at least this sub-module...
For now I'm just using it for filtering. But in the future I can see how the
system can be extended to support other use cases with complex consequences.
Do you have any other solution in mind that can help address the use case
I'm trying to implement?
thanks again !
--
View this message in context:
http://drools.46999.n3.nabble.com/Is-there-a-faster-way-of-doing-this-in-...
Sent from the Drools: User forum mailing list archive at Nabble.com.
4580
days inactive
4581
days old
29 comments
6 participants
participants (6)
-
Davide Sottara -
GPatel＠tsys.com -
Mauricio Salatino -
soumya_sd -
Vincent LEGENDRE -
Wolfgang Laun