[rules-users] Drools Fusion Dropping Events?

[Jason Barto]

Wolfgang,
the code to reproduce is below.  I'm hoping to process between 20k and 50k
events through Drools per second thus the extreme high-throughput testing.
I could settle for a single Drools node handling only say 5K per second IFF
I could cluster Drools but I've not yet found a way to distribute workload
across an active-active Drools cluster (seems there is no such thing?).

Since you're recommendation I've shifted to using Drools 5.3 just FYI:


### Average.java ###
package drools53fusioneval;

import java.io.IOException;
import java.util.Random;
import org.drools.KnowledgeBase;
import org.drools.KnowledgeBaseConfiguration;
import org.drools.KnowledgeBaseFactory;
import org.drools.builder.KnowledgeBuilder;
import org.drools.builder.KnowledgeBuilderFactory;
import org.drools.builder.ResourceType;
import org.drools.conf.EventProcessingOption;
import org.drools.io.ResourceFactory;
import org.drools.runtime.StatefulKnowledgeSession;
import org.drools.runtime.rule.WorkingMemoryEntryPoint;

class AvgDFEChannel implements org.drools.runtime.Channel {

    @Override
    public void send(Object o) {
        System.err.println("Recieved channel message: " + o);
    }
}

public class Average {

    public static void main(String[] args) throws InterruptedException,
IOException {
        KnowledgeBaseConfiguration kbconfig =
KnowledgeBaseFactory.newKnowledgeBaseConfiguration();
        kbconfig.setOption(EventProcessingOption.STREAM);

        KnowledgeBase kbase =
KnowledgeBaseFactory.newKnowledgeBase(kbconfig);

        KnowledgeBuilder kbuilder =
KnowledgeBuilderFactory.newKnowledgeBuilder();

kbuilder.add(ResourceFactory.newClassPathResource("drools53fusioneval/basic.drl"),
ResourceType.DRL);
        if (kbuilder.hasErrors()) {
            System.err.println(kbuilder.getErrors().toString());
        }
        kbase.addKnowledgePackages(kbuilder.getKnowledgePackages());

        final StatefulKnowledgeSession session =
kbase.newStatefulKnowledgeSession();
        session.registerChannel("heartbeat", new AvgDFEChannel());
        WorkingMemoryEntryPoint ep01 =
session.getWorkingMemoryEntryPoint("ep01");

        new Thread() {
            public void run() {
                session.fireUntilHalt();
            }
        }.start();

        Thread.sleep(5000); // give the engine time to get setup

        Server hiwaesdk = new Server("hiwaesdk");
        session.insert(hiwaesdk);

        long LIMIT = 10000;
        long sentCount = 0;
        int batchSize = 10000;
        Random rnd = new Random(System.nanoTime());
        int temp = 0;
        long startTS = System.nanoTime();
        while (sentCount < LIMIT) {
            for (int i = 0; i < batchSize; i++) {
                temp = rnd.nextInt(212);
                IntEvent evt = new IntEvent (temp);
                ep01.insert(evt);
                sentCount++;
            }
            Thread.sleep (0x1);
        }
        double duration = (System.nanoTime() - startTS)/1000000.0;
        System.out.println(LIMIT +" events generated in "+ duration +"
milliseconds");
        System.out.println("Last temperature submitted: "+ temp);
        for (int i = 0; i < 5; i++) {
            System.out.println ("Sec "+ i +": "+ hiwaesdk.currentTemp +",
"+ hiwaesdk.readingCount);
            Thread.sleep (1000);
        }
        System.exit(0);
    }
}


### basic.drl ###
package drools53fusioneval

declare IntEvent
    @role ( event )
end

rule "number rule"
when
    $e : IntEvent () from entry-point ep01
    $s : Server (hostname == "hiwaesdk")
then
    $s.currentTemp = $e.data;
    $s.readingCount++;
end




On Thu, May 16, 2013 at 4:25 AM, Wolfgang Laun <wolfgang.laun at gmail.com>wrote:

> What you describe is definitely not "expected behaviour" - it's more
> like that race condition (or another one) already being in 5.3. Posting
> your code might be a good idea ;-)
> -W
>
> On 16/05/2013, Jason Barto <jason.p.barto at gmail.com> wrote:
> > I've been working to load test Drools 5.3 to determine if its a fit for a
> > project.  As part of the test I programmatically insert events as rapidly
> > as possible; an example, my earlier test inserted 10k events in about
> > 300ms.  There is currently a single rule which reads the event and stores
> > the event's value into the only fact in Drools. I'm very happy to report,
> > and I'm sure it will be no surprise to anyone, that the engine processes
> > all the events in roughly 1 sec. However I have noticed that any large
> > number of events (~>1000) usually sees that a small number of events
> don't
> > get processed. I think after 10k events as many as 7 appear to have gone
> > unprocessed. If 100 events are inserted, rather than 10k, no events get
> > disregarded.  Being new to Drools I can easily accept that my java code
> or
> > DRL code is off or unoptimized in some way. However not knowing how
> Drools
> > does its magic I'm currently inclined to think that Drools gets swamped
> > (10k in 300ms is a lot) and a few events get dropped so Drools can keep
> > operating. Is this a known or expected behavior from Drools? If not I am
> > happy to post my code, it is similar to the other code sets I've posted
> in
> > the last few days. I'm still new to Drools and trying to learn its
> behavior
> > so any insight or enlightenment is greatly appreciated.
> >
> > Sincerely,
> > Jason
> >
> _______________________________________________
> rules-users mailing list
> rules-users at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/rules-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/rules-users/attachments/20130516/3b9b4d2a/attachment.html