5:53 p.m.
I've been working to load test Drools 5.3 to determine if its a fit for a
project. As part of the test I programmatically insert events as rapidly
as possible; an example, my earlier test inserted 10k events in about
300ms. There is currently a single rule which reads the event and stores
the event's value into the only fact in Drools. I'm very happy to report,
and I'm sure it will be no surprise to anyone, that the engine processes
all the events in roughly 1 sec. However I have noticed that any large
number of events (~>1000) usually sees that a small number of events don't
get processed. I think after 10k events as many as 7 appear to have gone
unprocessed. If 100 events are inserted, rather than 10k, no events get
disregarded. Being new to Drools I can easily accept that my java code or
DRL code is off or unoptimized in some way. However not knowing how Drools
does its magic I'm currently inclined to think that Drools gets swamped
(10k in 300ms is a lot) and a few events get dropped so Drools can keep
operating. Is this a known or expected behavior from Drools? If not I am
happy to post my code, it is similar to the other code sets I've posted in
the last few days. I'm still new to Drools and trying to learn its behavior
so any insight or enlightenment is greatly appreciated.
Sincerely,
Jason
10:25 p.m.
What you describe is definitely not "expected behaviour" - it's more
like that race condition (or another one) already being in 5.3. Posting
your code might be a good idea ;-)
-W
On 16/05/2013, Jason Barto <jason.p.barto(a)gmail.com> wrote:
I've been working to load test Drools 5.3 to determine if its a
fit for a
project. As part of the test I programmatically insert events as rapidly
as possible; an example, my earlier test inserted 10k events in about
300ms. There is currently a single rule which reads the event and stores
the event's value into the only fact in Drools. I'm very happy to report,
and I'm sure it will be no surprise to anyone, that the engine processes
all the events in roughly 1 sec. However I have noticed that any large
number of events (~>1000) usually sees that a small number of events don't
get processed. I think after 10k events as many as 7 appear to have gone
unprocessed. If 100 events are inserted, rather than 10k, no events get
disregarded. Being new to Drools I can easily accept that my java code or
DRL code is off or unoptimized in some way. However not knowing how Drools
does its magic I'm currently inclined to think that Drools gets swamped
(10k in 300ms is a lot) and a few events get dropped so Drools can keep
operating. Is this a known or expected behavior from Drools? If not I am
happy to post my code, it is similar to the other code sets I've posted in
the last few days. I'm still new to Drools and trying to learn its behavior
so any insight or enlightenment is greatly appreciated.
Sincerely,
Jason
1:12 a.m.
Wolfgang,
the code to reproduce is below. I'm hoping to process between 20k and 50k
events through Drools per second thus the extreme high-throughput testing.
I could settle for a single Drools node handling only say 5K per second IFF
I could cluster Drools but I've not yet found a way to distribute workload
across an active-active Drools cluster (seems there is no such thing?).
Since you're recommendation I've shifted to using Drools 5.3 just FYI:
### Average.java ###
package drools53fusioneval;
import java.io.IOException;
import java.util.Random;
import org.drools.KnowledgeBase;
import org.drools.KnowledgeBaseConfiguration;
import org.drools.KnowledgeBaseFactory;
import org.drools.builder.KnowledgeBuilder;
import org.drools.builder.KnowledgeBuilderFactory;
import org.drools.builder.ResourceType;
import org.drools.conf.EventProcessingOption;
import org.drools.io.ResourceFactory;
import org.drools.runtime.StatefulKnowledgeSession;
import org.drools.runtime.rule.WorkingMemoryEntryPoint;
class AvgDFEChannel implements org.drools.runtime.Channel {
@Override
public void send(Object o) {
System.err.println("Recieved channel message: " + o);
}
}
public class Average {
public static void main(String[] args) throws InterruptedException,
IOException {
KnowledgeBaseConfiguration kbconfig =
KnowledgeBaseFactory.newKnowledgeBaseConfiguration();
kbconfig.setOption(EventProcessingOption.STREAM);
KnowledgeBase kbase =
KnowledgeBaseFactory.newKnowledgeBase(kbconfig);
KnowledgeBuilder kbuilder =
KnowledgeBuilderFactory.newKnowledgeBuilder();
kbuilder.add(ResourceFactory.newClassPathResource("drools53fusioneval/basic.drl"),
ResourceType.DRL);
if (kbuilder.hasErrors()) {
System.err.println(kbuilder.getErrors().toString());
}
kbase.addKnowledgePackages(kbuilder.getKnowledgePackages());
final StatefulKnowledgeSession session =
kbase.newStatefulKnowledgeSession();
session.registerChannel("heartbeat", new AvgDFEChannel());
WorkingMemoryEntryPoint ep01 =
session.getWorkingMemoryEntryPoint("ep01");
new Thread() {
public void run() {
session.fireUntilHalt();
}
}.start();
Thread.sleep(5000); // give the engine time to get setup
Server hiwaesdk = new Server("hiwaesdk");
session.insert(hiwaesdk);
long LIMIT = 10000;
long sentCount = 0;
int batchSize = 10000;
Random rnd = new Random(System.nanoTime());
int temp = 0;
long startTS = System.nanoTime();
while (sentCount < LIMIT) {
for (int i = 0; i < batchSize; i++) {
temp = rnd.nextInt(212);
IntEvent evt = new IntEvent (temp);
ep01.insert(evt);
sentCount++;
}
Thread.sleep (0x1);
}
double duration = (System.nanoTime() - startTS)/1000000.0;
System.out.println(LIMIT +" events generated in "+ duration +"
milliseconds");
System.out.println("Last temperature submitted: "+ temp);
for (int i = 0; i < 5; i++) {
System.out.println ("Sec "+ i +": "+ hiwaesdk.currentTemp
+",
"+ hiwaesdk.readingCount);
Thread.sleep (1000);
}
System.exit(0);
}
}
### basic.drl ###
package drools53fusioneval
declare IntEvent
@role ( event )
end
rule "number rule"
when
$e : IntEvent () from entry-point ep01
$s : Server (hostname == "hiwaesdk")
then
$s.currentTemp = $e.data;
$s.readingCount++;
end
On Thu, May 16, 2013 at 4:25 AM, Wolfgang Laun <wolfgang.laun(a)gmail.com>wrote:
What you describe is definitely not "expected behaviour" -
it's more
like that race condition (or another one) already being in 5.3. Posting
your code might be a good idea ;-)
-W
On 16/05/2013, Jason Barto <jason.p.barto(a)gmail.com> wrote:
> I've been working to load test Drools 5.3 to determine if its a fit for a
> project. As part of the test I programmatically insert events as rapidly
> as possible; an example, my earlier test inserted 10k events in about
> 300ms. There is currently a single rule which reads the event and stores
> the event's value into the only fact in Drools. I'm very happy to report,
> and I'm sure it will be no surprise to anyone, that the engine processes
> all the events in roughly 1 sec. However I have noticed that any large
> number of events (~>1000) usually sees that a small number of events
don't
> get processed. I think after 10k events as many as 7 appear to have gone
> unprocessed. If 100 events are inserted, rather than 10k, no events get
> disregarded. Being new to Drools I can easily accept that my java code
or
> DRL code is off or unoptimized in some way. However not knowing how
Drools
> does its magic I'm currently inclined to think that Drools gets swamped
> (10k in 300ms is a lot) and a few events get dropped so Drools can keep
> operating. Is this a known or expected behavior from Drools? If not I am
> happy to post my code, it is similar to the other code sets I've posted
in
> the last few days. I'm still new to Drools and trying to learn its
behavior
> so any insight or enlightenment is greatly appreciated.
>
> Sincerely,
> Jason
>
_______________________________________________
rules-users mailing list
rules-users(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/rules-users
1:41 a.m.
Last week I found and fixed a very bad bug that seems to fit your case
https://issues.jboss.org/browse/DROOLS-131
This code should work in 5.6.0-SNAPSHOT and later versions.
I have seen DROOLS-138, but won't be able to work on it before the weekend
Davide
On 05/15/2013 11:12 PM, Jason Barto wrote:
Wolfgang,
the code to reproduce is below. I'm hoping to process between 20k and
50k events through Drools per second thus the extreme high-throughput
testing. I could settle for a single Drools node handling only say 5K
per second IFF I could cluster Drools but I've not yet found a way to
distribute workload across an active-active Drools cluster (seems
there is no such thing?).
Since you're recommendation I've shifted to using Drools 5.3 just FYI:
### Average.java ###
package drools53fusioneval;
import java.io.IOException;
import java.util.Random;
import org.drools.KnowledgeBase;
import org.drools.KnowledgeBaseConfiguration;
import org.drools.KnowledgeBaseFactory;
import org.drools.builder.KnowledgeBuilder;
import org.drools.builder.KnowledgeBuilderFactory;
import org.drools.builder.ResourceType;
import org.drools.conf.EventProcessingOption;
import org.drools.io.ResourceFactory;
import org.drools.runtime.StatefulKnowledgeSession;
import org.drools.runtime.rule.WorkingMemoryEntryPoint;
class AvgDFEChannel implements org.drools.runtime.Channel {
@Override
public void send(Object o) {
System.err.println("Recieved channel message: " + o);
}
}
public class Average {
public static void main(String[] args) throws
InterruptedException, IOException {
KnowledgeBaseConfiguration kbconfig =
KnowledgeBaseFactory.newKnowledgeBaseConfiguration();
kbconfig.setOption(EventProcessingOption.STREAM);
KnowledgeBase kbase =
KnowledgeBaseFactory.newKnowledgeBase(kbconfig);
KnowledgeBuilder kbuilder =
KnowledgeBuilderFactory.newKnowledgeBuilder();
kbuilder.add(ResourceFactory.newClassPathResource("drools53fusioneval/basic.drl"),
ResourceType.DRL);
if (kbuilder.hasErrors()) {
System.err.println(kbuilder.getErrors().toString());
}
kbase.addKnowledgePackages(kbuilder.getKnowledgePackages());
final StatefulKnowledgeSession session =
kbase.newStatefulKnowledgeSession();
session.registerChannel("heartbeat", new AvgDFEChannel());
WorkingMemoryEntryPoint ep01 =
session.getWorkingMemoryEntryPoint("ep01");
new Thread() {
public void run() {
session.fireUntilHalt();
}
}.start();
Thread.sleep(5000); // give the engine time to get setup
Server hiwaesdk = new Server("hiwaesdk");
session.insert(hiwaesdk);
long LIMIT = 10000;
long sentCount = 0;
int batchSize = 10000;
Random rnd = new Random(System.nanoTime());
int temp = 0;
long startTS = System.nanoTime();
while (sentCount < LIMIT) {
for (int i = 0; i < batchSize; i++) {
temp = rnd.nextInt(212);
IntEvent evt = new IntEvent (temp);
ep01.insert(evt);
sentCount++;
}
Thread.sleep (0x1);
}
double duration = (System.nanoTime() - startTS)/1000000.0;
System.out.println(LIMIT +" events generated in "+ duration +"
milliseconds");
System.out.println("Last temperature submitted: "+ temp);
for (int i = 0; i < 5; i++) {
System.out.println ("Sec "+ i +": "+
hiwaesdk.currentTemp
+", "+ hiwaesdk.readingCount);
Thread.sleep (1000);
}
System.exit(0);
}
}
### basic.drl ###
package drools53fusioneval
declare IntEvent
@role ( event )
end
rule "number rule"
when
$e : IntEvent () from entry-point ep01
$s : Server (hostname == "hiwaesdk")
then
$s.currentTemp = $e.data;
$s.readingCount++;
end
On Thu, May 16, 2013 at 4:25 AM, Wolfgang Laun
<wolfgang.laun(a)gmail.com <mailto:wolfgang.laun@gmail.com>> wrote:
What you describe is definitely not "expected behaviour" - it's more
like that race condition (or another one) already being in 5.3.
Posting
your code might be a good idea ;-)
-W
On 16/05/2013, Jason Barto <jason.p.barto(a)gmail.com
<mailto:jason.p.barto@gmail.com>> wrote:
> I've been working to load test Drools 5.3 to determine if its a
fit for a
> project. As part of the test I programmatically insert events
as rapidly
> as possible; an example, my earlier test inserted 10k events in
about
> 300ms. There is currently a single rule which reads the event
and stores
> the event's value into the only fact in Drools. I'm very happy
to report,
> and I'm sure it will be no surprise to anyone, that the engine
processes
> all the events in roughly 1 sec. However I have noticed that any
large
> number of events (~>1000) usually sees that a small number of
events don't
> get processed. I think after 10k events as many as 7 appear to
have gone
> unprocessed. If 100 events are inserted, rather than 10k, no
events get
> disregarded. Being new to Drools I can easily accept that my
java code or
> DRL code is off or unoptimized in some way. However not knowing
how Drools
> does its magic I'm currently inclined to think that Drools gets
swamped
> (10k in 300ms is a lot) and a few events get dropped so Drools
can keep
> operating. Is this a known or expected behavior from Drools? If
not I am
> happy to post my code, it is similar to the other code sets I've
posted in
> the last few days. I'm still new to Drools and trying to learn
its behavior
> so any insight or enlightenment is greatly appreciated.
>
> Sincerely,
> Jason
>
_______________________________________________
rules-users mailing list
rules-users(a)lists.jboss.org <mailto:rules-users@lists.jboss.org>
https://lists.jboss.org/mailman/listinfo/rules-users
_______________________________________________
rules-users mailing list
rules-users(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/rules-users
4242
days inactive
4243
days old
3 comments
3 participants
participants (3)
-
Davide Sottara -
Jason Barto -
Wolfgang Laun