3:09 p.m.
I am trying to run a partitioned rule base as we have a need to evaluate a
large number of facts. We have around 160 rules which are coordinated using
jBPM rule-flow groups(intermingled with some lightweight business logic).
The business logic in the process executed quickly(80ms for 10,000 facts)
but the rules take sometime to evaluate(thus why I thought a partitioned
rule base would work). However when I try to run it I get the stack trace
below. Do I need to do something to the session as well?
java.lang.IllegalStateException: Error launching test
com.nasdaq.backoffice.billing.process.PhlxCoreProcessTest public void
com.nasdaq.backoffice.billing.process.PhlxCoreProcessTest.testPhlxCoreProcess()
at
org.jboss.arquillian.protocol.servlet.ServletMethodExecutor.invoke(ServletMethodExecutor.java:122)
at
org.jboss.arquillian.container.test.impl.execution.RemoteTestExecuter.execute(RemoteTestExecuter.java:120)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:90)
at
org.jboss.arquillian.core.impl.EventContextImpl.invokeObservers(EventContextImpl.java:99)
at
org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:81)
at org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:134)
at org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:114)
at org.jboss.arquillian.core.impl.EventImpl.fire(EventImpl.java:67)
at
org.jboss.arquillian.container.test.impl.execution.ClientTestExecuter.execute(ClientTestExecuter.java:57)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:90)
at
org.jboss.arquillian.core.impl.EventContextImpl.invokeObservers(EventContextImpl.java:99)
at
org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:81)
at
org.jboss.arquillian.container.test.impl.client.ContainerEventController.createContext(ContainerEventController.java:142)
at
org.jboss.arquillian.container.test.impl.client.ContainerEventController.createTestContext(ContainerEventController.java:129)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:90)
at
org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:88)
at
org.jboss.arquillian.test.impl.TestContextHandler.createTestContext(TestContextHandler.java:89)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:90)
at
org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:88)
at
org.jboss.arquillian.test.impl.TestContextHandler.createClassContext(TestContextHandler.java:75)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:90)
at
org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:88)
at
org.jboss.arquillian.test.impl.TestContextHandler.createSuiteContext(TestContextHandler.java:60)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:90)
at
org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:88)
at org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:134)
at
org.jboss.arquillian.test.impl.EventTestRunnerAdaptor.test(EventTestRunnerAdaptor.java:111)
at org.jboss.arquillian.junit.Arquillian$6.evaluate(Arquillian.java:263)
at org.jboss.arquillian.junit.Arquillian$4.evaluate(Arquillian.java:226)
at org.jboss.arquillian.junit.Arquillian.multiExecute(Arquillian.java:314)
at org.jboss.arquillian.junit.Arquillian.access$100(Arquillian.java:46)
at org.jboss.arquillian.junit.Arquillian$5.evaluate(Arquillian.java:240)
at
org.junit.runners.BlockJUnit4ClassRunner.runNotIgnored(BlockJUnit4ClassRunner.java:79)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:71)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:49)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184)
at org.jboss.arquillian.junit.Arquillian$2.evaluate(Arquillian.java:185)
at org.jboss.arquillian.junit.Arquillian.multiExecute(Arquillian.java:314)
at org.jboss.arquillian.junit.Arquillian.access$100(Arquillian.java:46)
at org.jboss.arquillian.junit.Arquillian$3.evaluate(Arquillian.java:199)
at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
at org.jboss.arquillian.junit.Arquillian.run(Arquillian.java:147)
at
org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
at
org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
Caused by: java.io.WriteAbortedException: writing aborted;
java.io.NotSerializableException:
org.drools.impl.StatefulKnowledgeSessionImpl
at java.io.ObjectInputStream.readObject0(Unknown Source)
at java.io.ObjectInputStream.readObject(Unknown Source)
at
org.jboss.arquillian.test.spi.ExceptionProxy.readExternal(ExceptionProxy.java:177)
at java.io.ObjectInputStream.readExternalData(Unknown Source)
at java.io.ObjectInputStream.readOrdinaryObject(Unknown Source)
at java.io.ObjectInputStream.readObject0(Unknown Source)
at java.io.ObjectInputStream.defaultReadFields(Unknown Source)
at java.io.ObjectInputStream.readSerialData(Unknown Source)
at java.io.ObjectInputStream.readOrdinaryObject(Unknown Source)
at java.io.ObjectInputStream.readObject0(Unknown Source)
at java.io.ObjectInputStream.readObject(Unknown Source)
at
org.jboss.arquillian.protocol.servlet.ServletMethodExecutor.execute(ServletMethodExecutor.java:214)
at
org.jboss.arquillian.protocol.servlet.ServletMethodExecutor.executeWithRetry(ServletMethodExecutor.java:140)
at
org.jboss.arquillian.protocol.servlet.ServletMethodExecutor.invoke(ServletMethodExecutor.java:118)
... 75 more
Caused by: java.io.NotSerializableException:
org.drools.impl.StatefulKnowledgeSessionImpl
at java.io.ObjectOutputStream.writeObject0(Unknown Source)
at java.io.ObjectOutputStream.defaultWriteFields(Unknown Source)
at java.io.ObjectOutputStream.writeSerialData(Unknown Source)
at java.io.ObjectOutputStream.writeOrdinaryObject(Unknown Source)
at java.io.ObjectOutputStream.writeObject0(Unknown Source)
at java.io.ObjectOutputStream.writeObject(Unknown Source)
at
org.jboss.arquillian.test.spi.ExceptionProxy.writeExternal(ExceptionProxy.java:227)
at java.io.ObjectOutputStream.writeExternalData(Unknown Source)
at java.io.ObjectOutputStream.writeOrdinaryObject(Unknown Source)
at java.io.ObjectOutputStream.writeObject0(Unknown Source)
at java.io.ObjectOutputStream.defaultWriteFields(Unknown Source)
at java.io.ObjectOutputStream.writeSerialData(Unknown Source)
at java.io.ObjectOutputStream.writeOrdinaryObject(Unknown Source)
at java.io.ObjectOutputStream.writeObject0(Unknown Source)
at java.io.ObjectOutputStream.writeObject(Unknown Source)
at
org.jboss.arquillian.protocol.servlet.runner.ServletTestRunner.writeObject(ServletTestRunner.java:229)
at
org.jboss.arquillian.protocol.servlet.runner.ServletTestRunner.executeTest(ServletTestRunner.java:163)
at
org.jboss.arquillian.protocol.servlet.runner.ServletTestRunner.execute(ServletTestRunner.java:126)
at
org.jboss.arquillian.protocol.servlet.runner.ServletTestRunner.doGet(ServletTestRunner.java:90)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:734)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:847)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:329)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248)
at
org.jboss.weld.servlet.ConversationPropagationFilter.doFilter(ConversationPropagationFilter.java:62)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:275)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:161)
at
org.jboss.as.jpa.interceptor.WebNonTxEmCloserValve.invoke(WebNonTxEmCloserValve.java:50)
at
org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:154)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:155)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:368)
at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:877)
at
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:671)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:930)
at java.lang.Thread.run(Unknown Source)
--
View this message in context:
http://drools.46999.n3.nabble.com/Drools-5-3-partitioned-rule-base-tp3793...
Sent from the Drools: User forum mailing list archive at Nabble.com.
4:33 p.m.
After investagation I found the following:
10:29:45,421 ERROR [stderr] (pool-38-thread-2) at
org.drools.reteoo.SingleObjectSinkAdapter.propagateAssertObject(SingleObjectSinkAdapter.java:59)
10:29:45,423 ERROR [stderr] (pool-38-thread-2) at
org.drools.reteoo.AlphaNode.assertObject(AlphaNode.java:134)
10:29:45,424 ERROR [stderr] (pool-38-thread-2) at
org.drools.reteoo.PartitionTaskManager$FactAssertAction.execute(PartitionTaskManager.java:272)
10:29:45,424 ERROR [stderr] (pool-38-thread-2) at
org.drools.reteoo.PartitionTaskManager$PartitionTask.run(PartitionTaskManager.java:116)
10:29:45,425 ERROR [stderr] (pool-38-thread-2) at
org.drools.concurrent.ExternalExecutorService$ObservableRunnable.run(ExternalExecutorService.java:443)
10:29:45,425 ERROR [stderr] (pool-38-thread-2) at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
10:29:45,423 INFO [stdout] (pool-38-thread-1) SURCHARGES created
10:29:45,425 ERROR [stderr] (pool-38-thread-2) at
java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
10:29:45,426 ERROR [stderr] (pool-38-thread-2) at
java.lang.Thread.run(Unknown Source)
Any thoughts?
--
View this message in context:
http://drools.46999.n3.nabble.com/Drools-5-3-partitioned-rule-base-tp3793...
Sent from the Drools: User forum mailing list archive at Nabble.com.
4:54 p.m.
Why are you trying with that partitioning approach?
By the why I notice that most of the time when people uses ruleflow-groups
they didn't realize that all the evaluations are done at insertion time,
not when the process reaches the rule flow group, is that you case?
Doing consulting around these topics most of the clients doesn't need to
use rule flow groups and they just need some control facts to speed up the
processing times.
Cheers
On Fri, Mar 2, 2012 at 3:33 PM, gboro54 <gboro54(a)gmail.com> wrote:
After investagation I found the following:
10:29:45,421 ERROR [stderr] (pool-38-thread-2) at
org.drools.reteoo.SingleObjectSinkAdapter.propagateAssertObject(SingleObjectSinkAdapter.java:59)
10:29:45,423 ERROR [stderr] (pool-38-thread-2) at
org.drools.reteoo.AlphaNode.assertObject(AlphaNode.java:134)
10:29:45,424 ERROR [stderr] (pool-38-thread-2) at
org.drools.reteoo.PartitionTaskManager$FactAssertAction.execute(PartitionTaskManager.java:272)
10:29:45,424 ERROR [stderr] (pool-38-thread-2) at
org.drools.reteoo.PartitionTaskManager$PartitionTask.run(PartitionTaskManager.java:116)
10:29:45,425 ERROR [stderr] (pool-38-thread-2) at
org.drools.concurrent.ExternalExecutorService$ObservableRunnable.run(ExternalExecutorService.java:443)
10:29:45,425 ERROR [stderr] (pool-38-thread-2) at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
10:29:45,423 INFO [stdout] (pool-38-thread-1) SURCHARGES created
10:29:45,425 ERROR [stderr] (pool-38-thread-2) at
java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
10:29:45,426 ERROR [stderr] (pool-38-thread-2) at
java.lang.Thread.run(Unknown Source)
Any thoughts?
--
View this message in context:
http://drools.46999.n3.nabble.com/Drools-5-3-partitioned-rule-base-tp3793...
Sent from the Drools: User forum mailing list archive at Nabble.com.
_______________________________________________
rules-users mailing list
rules-users(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/rules-users
--
- MyJourney @ http://salaboy.wordpress.com
- Co-Founder @ http://www.jugargentina.org
- Co-Founder @ http://www.jbug.com.ar
- Salatino "Salaboy" Mauricio -
4:58 p.m.
I understand that all the evaluations are happening at insertion time. The
main reason for the groups is to intermix logic needed to execute outside of
rules(i.e I don't want rules invoking DAO's to retrieve data). I am trying
to partition because I need to be able to get a high level of messages
through these rules(a min of 1000/sec).
--
View this message in context:
http://drools.46999.n3.nabble.com/Drools-5-3-partitioned-rule-base-tp3793...
Sent from the Drools: User forum mailing list archive at Nabble.com.
5:02 p.m.
So the question is.. for processing those messages, you can set up multiple
sessions and evaluate each message separately?
or do you need to have all the messages in the same session?
Cheers
On Fri, Mar 2, 2012 at 3:58 PM, gboro54 <gboro54(a)gmail.com> wrote:
I understand that all the evaluations are happening at insertion
time. The
main reason for the groups is to intermix logic needed to execute outside
of
rules(i.e I don't want rules invoking DAO's to retrieve data). I am trying
to partition because I need to be able to get a high level of messages
through these rules(a min of 1000/sec).
--
View this message in context:
http://drools.46999.n3.nabble.com/Drools-5-3-partitioned-rule-base-tp3793...
Sent from the Drools: User forum mailing list archive at Nabble.com.
_______________________________________________
rules-users mailing list
rules-users(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/rules-users
--
- MyJourney @ http://salaboy.wordpress.com
- Co-Founder @ http://www.jugargentina.org
- Co-Founder @ http://www.jbug.com.ar
- Salatino "Salaboy" Mauricio -
5:12 p.m.
Executions need to happen in an order based on a data and a message id(at
least at one part of the evaluation). Because of this my plan was to use
dynamic salience and insert as many of the facts as I can at a time
--
View this message in context:
http://drools.46999.n3.nabble.com/Drools-5-3-partitioned-rule-base-tp3793...
Sent from the Drools: User forum mailing list archive at Nabble.com.
5:15 p.m.
But are the facts related with each other? do you need to analyze them all
together inside the same session?
Can you share a little bit more about your use cases? What do you mean with
"Execution" execution of what? What do you mean with "evaluation"?
Cheers
On Fri, Mar 2, 2012 at 4:12 PM, gboro54 <gboro54(a)gmail.com> wrote:
Executions need to happen in an order based on a data and a message
id(at
least at one part of the evaluation). Because of this my plan was to use
dynamic salience and insert as many of the facts as I can at a time
--
View this message in context:
http://drools.46999.n3.nabble.com/Drools-5-3-partitioned-rule-base-tp3793...
Sent from the Drools: User forum mailing list archive at Nabble.com.
_______________________________________________
rules-users mailing list
rules-users(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/rules-users
--
- MyJourney @ http://salaboy.wordpress.com
- Co-Founder @ http://www.jugargentina.org
- Co-Founder @ http://www.jbug.com.ar
- Salatino "Salaboy" Mauricio -
5:23 p.m.
We are writing a billing system using Drools to evaluate orders placed during
the day(this is a month to date process which will run nightly and we will
be bringing a real-time solution online later this year after we rewrite the
existing). Base fees of these orders can happen in parallel and in no way
affect one another, however we have price caps which do depend on the order
in which the cap is applied to the order(for certain conditions on an order
a surcharge may be created if a cap is applied). My past experience will
Drools has been it is quicker to do as much evaluation up front rather then
loop of a list of Orders and fire one at a time. However my experience in
running Drools with this load is limited(by the end of the month we will
have to process 15 million orders). I am open to suggestions on the best
way to do this. Additionally orders are reprocessed each night as orders
from a current day may affect the pricing of an order from previous days(i.e
a tier rate may apply, etc.)
On a side note I found that the issue is a NPE exception one of my rules
which only occurs if I partition the rule base(I am not sure why this would
make a difference).
--
View this message in context:
http://drools.46999.n3.nabble.com/Drools-5-3-partitioned-rule-base-tp3793...
Sent from the Drools: User forum mailing list archive at Nabble.com.
5:25 p.m.
Parition yourself ahead of time. Choose a key that is a true partition
of your data, chose the number of partitions, create a kbase per
partition and then hash on your inserted data to get the target kbase.
Mark
On 02/03/2012 16:23, gboro54 wrote:
We are writing a billing system using Drools to evaluate orders
placed during
the day(this is a month to date process which will run nightly and we will
be bringing a real-time solution online later this year after we rewrite the
existing). Base fees of these orders can happen in parallel and in no way
affect one another, however we have price caps which do depend on the order
in which the cap is applied to the order(for certain conditions on an order
a surcharge may be created if a cap is applied). My past experience will
Drools has been it is quicker to do as much evaluation up front rather then
loop of a list of Orders and fire one at a time. However my experience in
running Drools with this load is limited(by the end of the month we will
have to process 15 million orders). I am open to suggestions on the best
way to do this. Additionally orders are reprocessed each night as orders
from a current day may affect the pricing of an order from previous days(i.e
a tier rate may apply, etc.)
On a side note I found that the issue is a NPE exception one of my rules
which only occurs if I partition the rule base(I am not sure why this would
make a difference).
--
View this message in context:
http://drools.46999.n3.nabble.com/Drools-5-3-partitioned-rule-base-tp3793...
Sent from the Drools: User forum mailing list archive at Nabble.com.
_______________________________________________
rules-users mailing list
rules-users(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/rules-users
5:36 p.m.
Could you elaborate on that a little more Mark?
Mark Proctor wrote
Parition yourself ahead of time. Choose a key that is a true partition
of your data, chose the number of partitions, create a kbase per
partition and then hash on your inserted data to get the target kbase.
Mark
On 02/03/2012 16:23, gboro54 wrote:
> We are writing a billing system using Drools to evaluate orders placed
> during
> the day(this is a month to date process which will run nightly and we
> will
> be bringing a real-time solution online later this year after we rewrite
> the
> existing). Base fees of these orders can happen in parallel and in no way
> affect one another, however we have price caps which do depend on the
> order
> in which the cap is applied to the order(for certain conditions on an
> order
> a surcharge may be created if a cap is applied). My past experience will
> Drools has been it is quicker to do as much evaluation up front rather
> then
> loop of a list of Orders and fire one at a time. However my experience in
> running Drools with this load is limited(by the end of the month we will
> have to process 15 million orders). I am open to suggestions on the best
> way to do this. Additionally orders are reprocessed each night as orders
> from a current day may affect the pricing of an order from previous
> days(i.e
> a tier rate may apply, etc.)
>
>
> On a side note I found that the issue is a NPE exception one of my rules
> which only occurs if I partition the rule base(I am not sure why this
> would
> make a difference).
>
> --
> View this message in context:
>
http://drools.46999.n3.nabble.com/Drools-5-3-partitioned-rule-base-tp3793...
> Sent from the Drools: User forum mailing list archive at Nabble.com.
> _______________________________________________
> rules-users mailing list
> rules-users@.jboss
> https://lists.jboss.org/mailman/listinfo/rules-users
_______________________________________________
rules-users mailing list
rules-users@.jboss
https://lists.jboss.org/mailman/listinfo/rules-users
--
View this message in context:
http://drools.46999.n3.nabble.com/Drools-5-3-partitioned-rule-base-tp3793...
Sent from the Drools: User forum mailing list archive at Nabble.com.
6:33 p.m.
Could someone explain a little more about how Drools does a partition. I
believe I understand Mark's idea, I just need to flush it out some more
--
View this message in context:
http://drools.46999.n3.nabble.com/Drools-5-3-partitioned-rule-base-tp3793...
Sent from the Drools: User forum mailing list archive at Nabble.com.
7:01 p.m.
Drools doesn't partition. You will need to build an external framework
that partitions your data, and then assign a ksession per parition.
Whether it's a ksession or kbase per partition depends on whether you
can use the same rules per partition or not.
Mark
On 02/03/2012 17:33, gboro54 wrote:
Could someone explain a little more about how Drools does a
partition. I
believe I understand Mark's idea, I just need to flush it out some more
--
View this message in context:
http://drools.46999.n3.nabble.com/Drools-5-3-partitioned-rule-base-tp3793...
Sent from the Drools: User forum mailing list archive at Nabble.com.
_______________________________________________
rules-users mailing list
rules-users(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/rules-users
7:14 p.m.
How does MultithreadEvaluationOption work on a kBase? It is not explained
really clearly anywhere. I know if I insert single objects only this works
but as soon as two more of an object goes in I begin getting NPE...
Mark Proctor wrote
Drools doesn't partition. You will need to build an external framework
that partitions your data, and then assign a ksession per parition.
Whether it's a ksession or kbase per partition depends on whether you
can use the same rules per partition or not.
Mark
On 02/03/2012 17:33, gboro54 wrote:
> Could someone explain a little more about how Drools does a partition. I
> believe I understand Mark's idea, I just need to flush it out some more
>
> --
> View this message in context:
>
http://drools.46999.n3.nabble.com/Drools-5-3-partitioned-rule-base-tp3793...
> Sent from the Drools: User forum mailing list archive at Nabble.com.
> _______________________________________________
> rules-users mailing list
> rules-users@.jboss
> https://lists.jboss.org/mailman/listinfo/rules-users
_______________________________________________
rules-users mailing list
rules-users@.jboss
https://lists.jboss.org/mailman/listinfo/rules-users
--
View this message in context:
http://drools.46999.n3.nabble.com/Drools-5-3-partitioned-rule-base-tp3793...
Sent from the Drools: User forum mailing list archive at Nabble.com.
7:19 p.m.
Additionally when does partitioning by data cause performance to be
compromised? It has always been in my experience and to my knowledge that
building a complete session with facts is cheaper then iterating a list of
facts, creating a session, and firing. Perhaps I am mistaken? Additionally
is there documentation anywhere talking about utilizing drools in a high
volume environment?
--
View this message in context:
http://drools.46999.n3.nabble.com/Drools-5-3-partitioned-rule-base-tp3793...
Sent from the Drools: User forum mailing list archive at Nabble.com.
11:50 a.m.
On 02/03/2012 18:14, gboro54 wrote:
How does MultithreadEvaluationOption work on a kBase? It is not
explained
really clearly anywhere. I know if I insert single objects only this works
but as soon as two more of an object goes in I begin getting NPE...
The feature is
experimental and currently has problems, it's not
recommended you use it.
Mark
Mark Proctor wrote
> Drools doesn't partition. You will need to build an external framework
> that partitions your data, and then assign a ksession per parition.
> Whether it's a ksession or kbase per partition depends on whether you
> can use the same rules per partition or not.
>
> Mark
> On 02/03/2012 17:33, gboro54 wrote:
>> Could someone explain a little more about how Drools does a partition. I
>> believe I understand Mark's idea, I just need to flush it out some more
>>
>> --
>> View this message in context:
>>
http://drools.46999.n3.nabble.com/Drools-5-3-partitioned-rule-base-tp3793...
>> Sent from the Drools: User forum mailing list archive at Nabble.com.
>> _______________________________________________
>> rules-users mailing list
>> rules-users@.jboss
>> https://lists.jboss.org/mailman/listinfo/rules-users
> _______________________________________________
> rules-users mailing list
> rules-users@.jboss
> https://lists.jboss.org/mailman/listinfo/rules-users
>
--
View this message in context:
http://drools.46999.n3.nabble.com/Drools-5-3-partitioned-rule-base-tp3793...
Sent from the Drools: User forum mailing list archive at Nabble.com.
_______________________________________________
rules-users mailing list
rules-users(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/rules-users
6:52 p.m.
Would you partition the KnowledgeBase or the Sessions? Additional would each
Order being evaluated create a new session?
gboro54 wrote
Could you elaborate on that a little more Mark?
Mark Proctor wrote
>
> Parition yourself ahead of time. Choose a key that is a true partition
> of your data, chose the number of partitions, create a kbase per
> partition and then hash on your inserted data to get the target kbase.
>
> Mark
> On 02/03/2012 16:23, gboro54 wrote:
>> We are writing a billing system using Drools to evaluate orders placed
>> during
>> the day(this is a month to date process which will run nightly and we
>> will
>> be bringing a real-time solution online later this year after we rewrite
>> the
>> existing). Base fees of these orders can happen in parallel and in no
>> way
>> affect one another, however we have price caps which do depend on the
>> order
>> in which the cap is applied to the order(for certain conditions on an
>> order
>> a surcharge may be created if a cap is applied). My past experience will
>> Drools has been it is quicker to do as much evaluation up front rather
>> then
>> loop of a list of Orders and fire one at a time. However my experience
>> in
>> running Drools with this load is limited(by the end of the month we will
>> have to process 15 million orders). I am open to suggestions on the
>> best
>> way to do this. Additionally orders are reprocessed each night as orders
>> from a current day may affect the pricing of an order from previous
>> days(i.e
>> a tier rate may apply, etc.)
>>
>>
>> On a side note I found that the issue is a NPE exception one of my rules
>> which only occurs if I partition the rule base(I am not sure why this
>> would
>> make a difference).
>>
>> --
>> View this message in context:
>>
http://drools.46999.n3.nabble.com/Drools-5-3-partitioned-rule-base-tp3793...
>> Sent from the Drools: User forum mailing list archive at Nabble.com.
>> _______________________________________________
>> rules-users mailing list
>> rules-users@.jboss
>> https://lists.jboss.org/mailman/listinfo/rules-users
>
> _______________________________________________
> rules-users mailing list
> rules-users@.jboss
> https://lists.jboss.org/mailman/listinfo/rules-users
>
--
View this message in context:
http://drools.46999.n3.nabble.com/Drools-5-3-partitioned-rule-base-tp3793...
Sent from the Drools: User forum mailing list archive at Nabble.com.
11:51 a.m.
On 02/03/2012 17:52, gboro54 wrote:
Would you partition the KnowledgeBase or the Sessions? Additional
would each
Order being evaluated create a new session?
It depends on your problem and your
data.
Mark
gboro54 wrote
> Could you elaborate on that a little more Mark?
>
>
> Mark Proctor wrote
>> Parition yourself ahead of time. Choose a key that is a true partition
>> of your data, chose the number of partitions, create a kbase per
>> partition and then hash on your inserted data to get the target kbase.
>>
>> Mark
>> On 02/03/2012 16:23, gboro54 wrote:
>>> We are writing a billing system using Drools to evaluate orders placed
>>> during
>>> the day(this is a month to date process which will run nightly and we
>>> will
>>> be bringing a real-time solution online later this year after we rewrite
>>> the
>>> existing). Base fees of these orders can happen in parallel and in no
>>> way
>>> affect one another, however we have price caps which do depend on the
>>> order
>>> in which the cap is applied to the order(for certain conditions on an
>>> order
>>> a surcharge may be created if a cap is applied). My past experience will
>>> Drools has been it is quicker to do as much evaluation up front rather
>>> then
>>> loop of a list of Orders and fire one at a time. However my experience
>>> in
>>> running Drools with this load is limited(by the end of the month we will
>>> have to process 15 million orders). I am open to suggestions on the
>>> best
>>> way to do this. Additionally orders are reprocessed each night as orders
>>> from a current day may affect the pricing of an order from previous
>>> days(i.e
>>> a tier rate may apply, etc.)
>>>
>>>
>>> On a side note I found that the issue is a NPE exception one of my rules
>>> which only occurs if I partition the rule base(I am not sure why this
>>> would
>>> make a difference).
>>>
>>> --
>>> View this message in context:
>>>
http://drools.46999.n3.nabble.com/Drools-5-3-partitioned-rule-base-tp3793...
>>> Sent from the Drools: User forum mailing list archive at Nabble.com.
>>> _______________________________________________
>>> rules-users mailing list
>>> rules-users@.jboss
>>> https://lists.jboss.org/mailman/listinfo/rules-users
>> _______________________________________________
>> rules-users mailing list
>> rules-users@.jboss
>> https://lists.jboss.org/mailman/listinfo/rules-users
>>
--
View this message in context:
http://drools.46999.n3.nabble.com/Drools-5-3-partitioned-rule-base-tp3793...
Sent from the Drools: User forum mailing list archive at Nabble.com.
_______________________________________________
rules-users mailing list
rules-users(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/rules-users
2:11 p.m.
Mark,
I really appreciate the help and think I have come to a solution. Let me
know if this sounds reasonable.
1. We will continue to use jBPM to coordinate the rules and business logic
that need to occur in calculating charges for orders. However we will work
the process to only work on a per order level.
2. All orders are associated with accounts. We will keep one knowledgebase
as the rule sets are the same and we will partition sessions by accounts.
The flow will go as follows:
a. If the session exists insert the order, start a new process instance and
fire all rules
b. If the session has not been created: create the session, insert all
reference data that will be used by all orders in executions of the rule
set, insert the order, start a process, and fire all rules. This session is
then cached(via a hashmap more then likely)
c. This process will be invoked asyn from the main thread, allowing us to
control the multithreading
3. Orders will be treated as an event and will be set to expire in a certain
amount of time, allowing us to mange the memory footprint of each session.
Does this sound reasonable based on what you know of our usecase?
Additionally with expiring Orders does this cause a reevaluation of the
rules when the expiration occurs? The only other question I have is does the
expiration clock start when no more activation's get created for the given
event?
Thanks again.
--
View this message in context:
http://drools.46999.n3.nabble.com/Drools-5-3-partitioned-rule-base-tp3793...
Sent from the Drools: User forum mailing list archive at Nabble.com.
8:35 p.m.
Starting a new thread (if that's what you mean by process) for reusing
an existing session to process another order is likely to create more
overhead
and it'll just make threads compete for this resources.
Multiple threads each dedicated to a single session object might be a
better way to go.
I don't see any benefit to make simple order facts into events just for the
sake
of making them expire automatically. There ought to be a well defined
state (or states) when processed orders are retracted by some rule.
-W
On 3 March 2012 14:11, gboro54 <gboro54(a)gmail.com> wrote:
Mark,
I really appreciate the help and think I have come to a solution. Let me
know if this sounds reasonable.
1. We will continue to use jBPM to coordinate the rules and business logic
that need to occur in calculating charges for orders. However we will work
the process to only work on a per order level.
2. All orders are associated with accounts. We will keep one knowledgebase
as the rule sets are the same and we will partition sessions by accounts.
The flow will go as follows:
a. If the session exists insert the order, start a new process instance and
fire all rules
b. If the session has not been created: create the session, insert all
reference data that will be used by all orders in executions of the rule
set, insert the order, start a process, and fire all rules. This session is
then cached(via a hashmap more then likely)
c. This process will be invoked asyn from the main thread, allowing us to
control the multithreading
3. Orders will be treated as an event and will be set to expire in a
certain
amount of time, allowing us to mange the memory footprint of each session.
Does this sound reasonable based on what you know of our usecase?
Additionally with expiring Orders does this cause a reevaluation of the
rules when the expiration occurs? The only other question I have is does
the
expiration clock start when no more activation's get created for the given
event?
Thanks again.
--
View this message in context:
http://drools.46999.n3.nabble.com/Drools-5-3-partitioned-rule-base-tp3793...
Sent from the Drools: User forum mailing list archive at Nabble.com.
_______________________________________________
rules-users mailing list
rules-users(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/rules-users
8:45 p.m.
Agreed that I can have a rule to retract the order....i was just thinking if
the overhead of a reevaluation but can live with that...As far as the
threading...this logic is going to be deployed on a jboss instance using ejb
so I was thinking about, using the jboss thread pool...so when I say new
thread it may not, be new put, pulled from a pool....does that sound
reasonable?
laune wrote
Starting a new thread (if that's what you mean by process) for reusing
an existing session to process another order is likely to create more
overhead
and it'll just make threads compete for this resources.
Multiple threads each dedicated to a single session object might be a
better way to go.
I don't see any benefit to make simple order facts into events just for
the
sake
of making them expire automatically. There ought to be a well defined
state (or states) when processed orders are retracted by some rule.
-W
On 3 March 2012 14:11, gboro54 <gboro54@> wrote:
> Mark,
>
> I really appreciate the help and think I have come to a solution. Let me
> know if this sounds reasonable.
> 1. We will continue to use jBPM to coordinate the rules and business
> logic
> that need to occur in calculating charges for orders. However we will
> work
> the process to only work on a per order level.
> 2. All orders are associated with accounts. We will keep one
> knowledgebase
> as the rule sets are the same and we will partition sessions by accounts.
> The flow will go as follows:
> a. If the session exists insert the order, start a new process instance
> and
> fire all rules
> b. If the session has not been created: create the session, insert all
> reference data that will be used by all orders in executions of the rule
> set, insert the order, start a process, and fire all rules. This session
> is
> then cached(via a hashmap more then likely)
> c. This process will be invoked asyn from the main thread, allowing us to
> control the multithreading
> 3. Orders will be treated as an event and will be set to expire in a
> certain
> amount of time, allowing us to mange the memory footprint of each
> session.
>
> Does this sound reasonable based on what you know of our usecase?
> Additionally with expiring Orders does this cause a reevaluation of the
> rules when the expiration occurs? The only other question I have is does
> the
> expiration clock start when no more activation's get created for the
> given
> event?
>
> Thanks again.
>
> --
> View this message in context:
>
http://drools.46999.n3.nabble.com/Drools-5-3-partitioned-rule-base-tp3793...
> Sent from the Drools: User forum mailing list archive at Nabble.com.
> _______________________________________________
> rules-users mailing list
> rules-users@.jboss
> https://lists.jboss.org/mailman/listinfo/rules-users
>
_______________________________________________
rules-users mailing list
rules-users@.jboss
https://lists.jboss.org/mailman/listinfo/rules-users
--
View this message in context:
http://drools.46999.n3.nabble.com/Drools-5-3-partitioned-rule-base-tp3793...
Sent from the Drools: User forum mailing list archive at Nabble.com.
9:30 p.m.
My previous statement was poorly written(that is what I get for trying to
respond via my cell phone ). I agree that the Order fact is simple enough to
have a rule retract it once processing is completed. As far as the
threading, this is an ejb application being deployed on JBoss 7.1. Because
of this I was hoping to use @Asyn and allow the container to really manage
the threads. This method would simply request the cached session insert the
Order fact, and start the jBPM process to process the Order. Perhaps this
part of the design is still flawed as I have only recently been thinking
about this and I would welcome all suggestions.
Additionally is there any documentation on utilizing drools in a high
throughput environment(i.e white papers,etc.) This billing project is the
first of many potential projects we are looking to utilize drools for and I
would love to share some more documentation on this(as well as read on
myself).
gboro54 wrote
Agreed that I can have a rule to retract the order....i was just thinking
if the overhead of a reevaluation but can live with that...As far as the
threading...this logic is going to be deployed on a jboss instance using
ejb so I was thinking about, using the jboss thread pool...so when I say
new thread it may not, be new put, pulled from a pool....does that sound
reasonable?
laune wrote
>
> Starting a new thread (if that's what you mean by process) for reusing
> an existing session to process another order is likely to create more
> overhead
> and it'll just make threads compete for this resources.
>
> Multiple threads each dedicated to a single session object might be a
> better way to go.
>
> I don't see any benefit to make simple order facts into events just for
> the
> sake
> of making them expire automatically. There ought to be a well defined
> state (or states) when processed orders are retracted by some rule.
>
> -W
>
>
> On 3 March 2012 14:11, gboro54 <gboro54@> wrote:
>
>> Mark,
>>
>> I really appreciate the help and think I have come to a solution. Let
>> me
>> know if this sounds reasonable.
>> 1. We will continue to use jBPM to coordinate the rules and business
>> logic
>> that need to occur in calculating charges for orders. However we will
>> work
>> the process to only work on a per order level.
>> 2. All orders are associated with accounts. We will keep one
>> knowledgebase
>> as the rule sets are the same and we will partition sessions by
>> accounts.
>> The flow will go as follows:
>> a. If the session exists insert the order, start a new process instance
>> and
>> fire all rules
>> b. If the session has not been created: create the session, insert all
>> reference data that will be used by all orders in executions of the rule
>> set, insert the order, start a process, and fire all rules. This session
>> is
>> then cached(via a hashmap more then likely)
>> c. This process will be invoked asyn from the main thread, allowing us
>> to
>> control the multithreading
>> 3. Orders will be treated as an event and will be set to expire in a
>> certain
>> amount of time, allowing us to mange the memory footprint of each
>> session.
>>
>> Does this sound reasonable based on what you know of our usecase?
>> Additionally with expiring Orders does this cause a reevaluation of the
>> rules when the expiration occurs? The only other question I have is does
>> the
>> expiration clock start when no more activation's get created for the
>> given
>> event?
>>
>> Thanks again.
>>
>> --
>> View this message in context:
>>
http://drools.46999.n3.nabble.com/Drools-5-3-partitioned-rule-base-tp3793...
>> Sent from the Drools: User forum mailing list archive at Nabble.com.
>> _______________________________________________
>> rules-users mailing list
>> rules-users@.jboss
>> https://lists.jboss.org/mailman/listinfo/rules-users
>>
>
> _______________________________________________
> rules-users mailing list
> rules-users@.jboss
> https://lists.jboss.org/mailman/listinfo/rules-users
>
--
View this message in context:
http://drools.46999.n3.nabble.com/Drools-5-3-partitioned-rule-base-tp3793...
Sent from the Drools: User forum mailing list archive at Nabble.com.
7:31 a.m.
A few random observations.
A recent experiment has shown that insert takes 0.82 seconds for
100000 facts with one very simple rule with a CPU G620 @ 2.60GHz and:
java version "1.7.0_02"
Java(TM) SE Runtime Environment (build 1.7.0_02-b13)
Java HotSpot(TM) 64-Bit Server VM (build 22.0-b10, mixed mode)
There is, of course, no upper limit for insert as this depends on the
number and complexity of the rules. And if you apply the jBPM Process
paradigm, I'm not prepared to make any guesses. (It stands to reason
that this will incur more overhead than straightforward processing in
the Expert engine.)
Estimating 1K per order, I'd say that i/o time for 150M orders is not
to be neglected.
I'd investigate batch (but not bulk) insertion of quantities, so that
inserts and fire calls remain in good proportion. Also, balancing i/o
threads and processing threads in relation to available CPUs shouldn't
be different for an application using Drools.
I've been re-reading some of this thread, and the requirement of
ordered processing is going to be very difficult if the order this to
be maintained over 150M facts. I mentioned dynamic property, but many
different salience levels for a large number of activations is not
efficient, since this requires a sequential search.
-W
On 3 March 2012 21:30, gboro54 <gboro54(a)gmail.com> wrote:
My previous statement was poorly written(that is what I get for trying to
respond via my cell phone ). I agree that the Order fact is simple enough to
have a rule retract it once processing is completed. As far as the
threading, this is an ejb application being deployed on JBoss 7.1. Because
of this I was hoping to use @Asyn and allow the container to really manage
the threads. This method would simply request the cached session insert the
Order fact, and start the jBPM process to process the Order. Perhaps this
part of the design is still flawed as I have only recently been thinking
about this and I would welcome all suggestions.
Additionally is there any documentation on utilizing drools in a high
throughput environment(i.e white papers,etc.) This billing project is the
first of many potential projects we are looking to utilize drools for and I
would love to share some more documentation on this(as well as read on
myself).
gboro54 wrote
>
> Agreed that I can have a rule to retract the order....i was just thinking
> if the overhead of a reevaluation but can live with that...As far as the
> threading...this logic is going to be deployed on a jboss instance using
> ejb so I was thinking about, using the jboss thread pool...so when I say
> new thread it may not, be new put, pulled from a pool....does that sound
> reasonable?
>
>
>
> laune wrote
>>
>> Starting a new thread (if that's what you mean by process) for reusing
>> an existing session to process another order is likely to create more
>> overhead
>> and it'll just make threads compete for this resources.
>>
>> Multiple threads each dedicated to a single session object might be a
>> better way to go.
>>
>> I don't see any benefit to make simple order facts into events just for
>> the
>> sake
>> of making them expire automatically. There ought to be a well defined
>> state (or states) when processed orders are retracted by some rule.
>>
>> -W
>>
>>
>> On 3 March 2012 14:11, gboro54 <gboro54@> wrote:
>>
>>> Mark,
>>>
>>> I really appreciate the help and think I have come to a solution. Let
>>> me
>>> know if this sounds reasonable.
>>> 1. We will continue to use jBPM to coordinate the rules and business
>>> logic
>>> that need to occur in calculating charges for orders. However we will
>>> work
>>> the process to only work on a per order level.
>>> 2. All orders are associated with accounts. We will keep one
>>> knowledgebase
>>> as the rule sets are the same and we will partition sessions by
>>> accounts.
>>> The flow will go as follows:
>>> a. If the session exists insert the order, start a new process instance
>>> and
>>> fire all rules
>>> b. If the session has not been created: create the session, insert all
>>> reference data that will be used by all orders in executions of the rule
>>> set, insert the order, start a process, and fire all rules. This session
>>> is
>>> then cached(via a hashmap more then likely)
>>> c. This process will be invoked asyn from the main thread, allowing us
>>> to
>>> control the multithreading
>>> 3. Orders will be treated as an event and will be set to expire in a
>>> certain
>>> amount of time, allowing us to mange the memory footprint of each
>>> session.
>>>
>>> Does this sound reasonable based on what you know of our usecase?
>>> Additionally with expiring Orders does this cause a reevaluation of the
>>> rules when the expiration occurs? The only other question I have is does
>>> the
>>> expiration clock start when no more activation's get created for the
>>> given
>>> event?
>>>
>>> Thanks again.
>>>
>>> --
>>> View this message in context:
>>>
http://drools.46999.n3.nabble.com/Drools-5-3-partitioned-rule-base-tp3793...
>>> Sent from the Drools: User forum mailing list archive at Nabble.com.
>>> _______________________________________________
>>> rules-users mailing list
>>> rules-users@.jboss
>>> https://lists.jboss.org/mailman/listinfo/rules-users
>>>
>>
>> _______________________________________________
>> rules-users mailing list
>> rules-users@.jboss
>> https://lists.jboss.org/mailman/listinfo/rules-users
>>
>
--
View this message in context:
http://drools.46999.n3.nabble.com/Drools-5-3-partitioned-rule-base-tp3793...
Sent from the Drools: User forum mailing list archive at Nabble.com.
_______________________________________________
rules-users mailing list
rules-users(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/rules-users
1:57 p.m.
I have been reevaluating our use cases and have decided to make a couple
modifications:
1.Remove jBPM and Ruleflow groups. While it certainly fit a need I have
derived a new way to do the same thing.
2. After reviewing our current ruleset I decided to modify what the rules
actually do. At the end of the day the rules are written to infer certain
things about the order. With this in mind I have decided to let the rules
generate the billing meta-data about the Order(i.e I am may be eligible for
charge A and for cap B) and have post processing which will use this
information accordingly. The post processing is where the order of the
Orders will matter and I am currently thinking of a couple ways to handle
this.
3. Partitioning the data by account will allow us to multithread easily when
running rules. However rather then have 1 session per account I was thinking
about having sessions that represent a group of accounts.
4. With 3 in mind I am now trying to determine the best way to do this. My
thinking atm is to create x sessions(for this example let's say 5), each
associated with a thread via fireUntilHalt(). Since orders are streamed in
by another thread reading from the DB my thinking is this is the most
efficient way. However these leads me to a couple questions. How would I
determine that an order has been processes through the ruleset and is ready
for post processing(with fireAllRules I know that once the rules have fired
I can retract the Order and move it to post processing and perhaps this is
the answer that will be provided). With fireUntilHalt I thought about
raising an event which I would use CDI to listen for and then retract the
Order and pass it along for post processing. Is this a good approach? I am
open to any suggestion here.
5. After watching a video posted by salaboy(http://vimeo.com/27209589) I am
looking into some of the technologies he used. Particularly Kryo. The idea
would be that if an Order was eligible for certain post processing it would
be serialized to disc with a ranking(i.e key). Another part of the process
would then read one at a time and process them in order. Any thoughts around
this(I know this is not a drools question but I figured I would put it out
there anyway).
Thanks Again
-Mike
laune wrote
A few random observations.
A recent experiment has shown that insert takes 0.82 seconds for
100000 facts with one very simple rule with a CPU G620 @ 2.60GHz and:
java version "1.7.0_02"
Java(TM) SE Runtime Environment (build 1.7.0_02-b13)
Java HotSpot(TM) 64-Bit Server VM (build 22.0-b10, mixed mode)
There is, of course, no upper limit for insert as this depends on the
number and complexity of the rules. And if you apply the jBPM Process
paradigm, I'm not prepared to make any guesses. (It stands to reason
that this will incur more overhead than straightforward processing in
the Expert engine.)
Estimating 1K per order, I'd say that i/o time for 150M orders is not
to be neglected.
I'd investigate batch (but not bulk) insertion of quantities, so that
inserts and fire calls remain in good proportion. Also, balancing i/o
threads and processing threads in relation to available CPUs shouldn't
be different for an application using Drools.
I've been re-reading some of this thread, and the requirement of
ordered processing is going to be very difficult if the order this to
be maintained over 150M facts. I mentioned dynamic property, but many
different salience levels for a large number of activations is not
efficient, since this requires a sequential search.
-W
On 3 March 2012 21:30, gboro54 <gboro54@> wrote:
>
> My previous statement was poorly written(that is what I get for trying to
> respond via my cell phone ). I agree that the Order fact is simple enough
> to
> have a rule retract it once processing is completed. As far as the
> threading, this is an ejb application being deployed on JBoss 7.1.
> Because
> of this I was hoping to use @Asyn and allow the container to really
> manage
> the threads. This method would simply request the cached session insert
> the
> Order fact, and start the jBPM process to process the Order. Perhaps this
> part of the design is still flawed as I have only recently been thinking
> about this and I would welcome all suggestions.
>
> Additionally is there any documentation on utilizing drools in a high
> throughput environment(i.e white papers,etc.) This billing project is the
> first of many potential projects we are looking to utilize drools for and
> I
> would love to share some more documentation on this(as well as read on
> myself).
>
>
>
> gboro54 wrote
> >
> > Agreed that I can have a rule to retract the order....i was just
> thinking
> > if the overhead of a reevaluation but can live with that...As far as
> the
> > threading...this logic is going to be deployed on a jboss instance
> using
> > ejb so I was thinking about, using the jboss thread pool...so when I
> say
> > new thread it may not, be new put, pulled from a pool....does that
> sound
> > reasonable?
> >
> >
> >
> > laune wrote
> >>
> >> Starting a new thread (if that's what you mean by process) for reusing
> >> an existing session to process another order is likely to create more
> >> overhead
> >> and it'll just make threads compete for this resources.
> >>
> >> Multiple threads each dedicated to a single session object might be a
> >> better way to go.
> >>
> >> I don't see any benefit to make simple order facts into events just
> for
> >> the
> >> sake
> >> of making them expire automatically. There ought to be a well defined
> >> state (or states) when processed orders are retracted by some rule.
> >>
> >> -W
> >>
> >>
> >> On 3 March 2012 14:11, gboro54 <gboro54@> wrote:
> >>
> >>> Mark,
> >>>
> >>> I really appreciate the help and think I have come to a solution.
> Let
> >>> me
> >>> know if this sounds reasonable.
> >>> 1. We will continue to use jBPM to coordinate the rules and business
> >>> logic
> >>> that need to occur in calculating charges for orders. However we will
> >>> work
> >>> the process to only work on a per order level.
> >>> 2. All orders are associated with accounts. We will keep one
> >>> knowledgebase
> >>> as the rule sets are the same and we will partition sessions by
> >>> accounts.
> >>> The flow will go as follows:
> >>> a. If the session exists insert the order, start a new process
> instance
> >>> and
> >>> fire all rules
> >>> b. If the session has not been created: create the session, insert
> all
> >>> reference data that will be used by all orders in executions of the
> rule
> >>> set, insert the order, start a process, and fire all rules. This
> session
> >>> is
> >>> then cached(via a hashmap more then likely)
> >>> c. This process will be invoked asyn from the main thread, allowing
> us
> >>> to
> >>> control the multithreading
> >>> 3. Orders will be treated as an event and will be set to expire in a
> >>> certain
> >>> amount of time, allowing us to mange the memory footprint of each
> >>> session.
> >>>
> >>> Does this sound reasonable based on what you know of our usecase?
> >>> Additionally with expiring Orders does this cause a reevaluation of
> the
> >>> rules when the expiration occurs? The only other question I have is
> does
> >>> the
> >>> expiration clock start when no more activation's get created for
the
> >>> given
> >>> event?
> >>>
> >>> Thanks again.
> >>>
> >>> --
> >>> View this message in context:
> >>>
>
http://drools.46999.n3.nabble.com/Drools-5-3-partitioned-rule-base-tp3793...
> >>> Sent from the Drools: User forum mailing list archive at Nabble.com.
> >>> _______________________________________________
> >>> rules-users mailing list
> >>> rules-users@.jboss
> >>> https://lists.jboss.org/mailman/listinfo/rules-users
> >>>
> >>
> >> _______________________________________________
> >> rules-users mailing list
> >> rules-users@.jboss
> >> https://lists.jboss.org/mailman/listinfo/rules-users
> >>
> >
>
>
> --
> View this message in context:
>
http://drools.46999.n3.nabble.com/Drools-5-3-partitioned-rule-base-tp3793...
> Sent from the Drools: User forum mailing list archive at Nabble.com.
> _______________________________________________
> rules-users mailing list
> rules-users@.jboss
> https://lists.jboss.org/mailman/listinfo/rules-users
_______________________________________________
rules-users mailing list
rules-users@.jboss
https://lists.jboss.org/mailman/listinfo/rules-users
--
View this message in context:
http://drools.46999.n3.nabble.com/Drools-5-3-partitioned-rule-base-tp3793...
Sent from the Drools: User forum mailing list archive at Nabble.com.
3:35 p.m.
On 07/03/2012, gboro54 <gboro54(a)gmail.com> wrote:
I have been reevaluating our use cases and have decided to make a
couple
modifications:
4. With 3 in mind I am now trying to determine the best way to do
this. My
thinking atm is to create x sessions(for this example let's say 5), each
associated with a thread via fireUntilHalt(). Since orders are streamed in
by another thread reading from the DB my thinking is this is the most
efficient way. However these leads me to a couple questions. How would I
determine that an order has been processes through the ruleset and is ready
for post processing(with fireAllRules I know that once the rules have fired
I can retract the Order and move it to post processing and perhaps this is
the answer that will be provided). With fireUntilHalt I thought about
raising an event which I would use CDI to listen for and then retract the
Order and pass it along for post processing. Is this a good approach? I am
open to any suggestion here.
A low-salience rule ought to detect orders for which no other rules
fire any more - that's the moment when it has to be kicked into post
processing.
5. After watching a video posted by
salaboy(http://vimeo.com/27209589) I am
looking into some of the technologies he used. Particularly Kryo. The idea
would be that if an Order was eligible for certain post processing it would
be serialized to disc with a ranking(i.e key). Another part of the process
would then read one at a time and process them in order. Any thoughts around
this(I know this is not a drools question but I figured I would put it out
there anyway).
For the session, provide an abstract service via a global that accepts
the retracted order fact. Make postprocessing as independent as
possible, could even RMI to another system if cycles are a bottleneck.
Go to disk only if there's no other way, eg. when it is too slow and
you need to spill to disk.
-W
Thanks Again
-Mike
laune wrote
>
> A few random observations.
>
> A recent experiment has shown that insert takes 0.82 seconds for
> 100000 facts with one very simple rule with a CPU G620 @ 2.60GHz and:
> java version "1.7.0_02"
> Java(TM) SE Runtime Environment (build 1.7.0_02-b13)
> Java HotSpot(TM) 64-Bit Server VM (build 22.0-b10, mixed mode)
>
> There is, of course, no upper limit for insert as this depends on the
> number and complexity of the rules. And if you apply the jBPM Process
> paradigm, I'm not prepared to make any guesses. (It stands to reason
> that this will incur more overhead than straightforward processing in
> the Expert engine.)
>
> Estimating 1K per order, I'd say that i/o time for 150M orders is not
> to be neglected.
>
> I'd investigate batch (but not bulk) insertion of quantities, so that
> inserts and fire calls remain in good proportion. Also, balancing i/o
> threads and processing threads in relation to available CPUs shouldn't
> be different for an application using Drools.
>
> I've been re-reading some of this thread, and the requirement of
> ordered processing is going to be very difficult if the order this to
> be maintained over 150M facts. I mentioned dynamic property, but many
> different salience levels for a large number of activations is not
> efficient, since this requires a sequential search.
>
> -W
>
>
> On 3 March 2012 21:30, gboro54 <gboro54@> wrote:
>>
>> My previous statement was poorly written(that is what I get for trying to
>> respond via my cell phone ). I agree that the Order fact is simple enough
>> to
>> have a rule retract it once processing is completed. As far as the
>> threading, this is an ejb application being deployed on JBoss 7.1.
>> Because
>> of this I was hoping to use @Asyn and allow the container to really
>> manage
>> the threads. This method would simply request the cached session insert
>> the
>> Order fact, and start the jBPM process to process the Order. Perhaps this
>> part of the design is still flawed as I have only recently been thinking
>> about this and I would welcome all suggestions.
>>
>> Additionally is there any documentation on utilizing drools in a high
>> throughput environment(i.e white papers,etc.) This billing project is the
>> first of many potential projects we are looking to utilize drools for and
>> I
>> would love to share some more documentation on this(as well as read on
>> myself).
>>
>>
>>
>> gboro54 wrote
>> >
>> > Agreed that I can have a rule to retract the order....i was just
>> thinking
>> > if the overhead of a reevaluation but can live with that...As far as
>> the
>> > threading...this logic is going to be deployed on a jboss instance
>> using
>> > ejb so I was thinking about, using the jboss thread pool...so when I
>> say
>> > new thread it may not, be new put, pulled from a pool....does that
>> sound
>> > reasonable?
>> >
>> >
>> >
>> > laune wrote
>> >>
>> >> Starting a new thread (if that's what you mean by process) for
reusing
>> >> an existing session to process another order is likely to create more
>> >> overhead
>> >> and it'll just make threads compete for this resources.
>> >>
>> >> Multiple threads each dedicated to a single session object might be a
>> >> better way to go.
>> >>
>> >> I don't see any benefit to make simple order facts into events just
>> for
>> >> the
>> >> sake
>> >> of making them expire automatically. There ought to be a well defined
>> >> state (or states) when processed orders are retracted by some rule.
>> >>
>> >> -W
>> >>
>> >>
>> >> On 3 March 2012 14:11, gboro54 <gboro54@> wrote:
>> >>
>> >>> Mark,
>> >>>
>> >>> I really appreciate the help and think I have come to a solution.
>> Let
>> >>> me
>> >>> know if this sounds reasonable.
>> >>> 1. We will continue to use jBPM to coordinate the rules and
business
>> >>> logic
>> >>> that need to occur in calculating charges for orders. However we
will
>> >>> work
>> >>> the process to only work on a per order level.
>> >>> 2. All orders are associated with accounts. We will keep one
>> >>> knowledgebase
>> >>> as the rule sets are the same and we will partition sessions by
>> >>> accounts.
>> >>> The flow will go as follows:
>> >>> a. If the session exists insert the order, start a new process
>> instance
>> >>> and
>> >>> fire all rules
>> >>> b. If the session has not been created: create the session, insert
>> all
>> >>> reference data that will be used by all orders in executions of the
>> rule
>> >>> set, insert the order, start a process, and fire all rules. This
>> session
>> >>> is
>> >>> then cached(via a hashmap more then likely)
>> >>> c. This process will be invoked asyn from the main thread, allowing
>> us
>> >>> to
>> >>> control the multithreading
>> >>> 3. Orders will be treated as an event and will be set to expire in
a
>> >>> certain
>> >>> amount of time, allowing us to mange the memory footprint of each
>> >>> session.
>> >>>
>> >>> Does this sound reasonable based on what you know of our usecase?
>> >>> Additionally with expiring Orders does this cause a reevaluation of
>> the
>> >>> rules when the expiration occurs? The only other question I have is
>> does
>> >>> the
>> >>> expiration clock start when no more activation's get created for
the
>> >>> given
>> >>> event?
>> >>>
>> >>> Thanks again.
>> >>>
>> >>> --
>> >>> View this message in context:
>> >>>
>>
http://drools.46999.n3.nabble.com/Drools-5-3-partitioned-rule-base-tp3793...
>> >>> Sent from the Drools: User forum mailing list archive at
Nabble.com.
>> >>> _______________________________________________
>> >>> rules-users mailing list
>> >>> rules-users@.jboss
>> >>> https://lists.jboss.org/mailman/listinfo/rules-users
>> >>>
>> >>
>> >> _______________________________________________
>> >> rules-users mailing list
>> >> rules-users@.jboss
>> >> https://lists.jboss.org/mailman/listinfo/rules-users
>> >>
>> >
>>
>>
>> --
>> View this message in context:
>>
http://drools.46999.n3.nabble.com/Drools-5-3-partitioned-rule-base-tp3793...
>> Sent from the Drools: User forum mailing list archive at Nabble.com.
>> _______________________________________________
>> rules-users mailing list
>> rules-users@.jboss
>> https://lists.jboss.org/mailman/listinfo/rules-users
>
> _______________________________________________
> rules-users mailing list
> rules-users@.jboss
> https://lists.jboss.org/mailman/listinfo/rules-users
>
--
View this message in context:
http://drools.46999.n3.nabble.com/Drools-5-3-partitioned-rule-base-tp3793...
Sent from the Drools: User forum mailing list archive at Nabble.com.
_______________________________________________
rules-users mailing list
rules-users(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/rules-users
8:07 p.m.
I was thinking we could do separate sessions then queue up the results in
order and process them....I also like Mark's idea of partitioning
--
View this message in context:
http://drools.46999.n3.nabble.com/Drools-5-3-partitioned-rule-base-tp3793...
Sent from the Drools: User forum mailing list archive at Nabble.com.
4:58 p.m.
I understand that all the evaluations are happening at insertion time. The
main reason for the groups is to intermix logic needed to execute outside of
rules(i.e I don't want rules invoking DAO's to retrieve data). I am trying
to partition because I need to be able to get a high level of messages
through these rules(a min of 1000/sec).
--
View this message in context:
http://drools.46999.n3.nabble.com/Drools-5-3-partitioned-rule-base-tp3793...
Sent from the Drools: User forum mailing list archive at Nabble.com.
9 p.m.
Many of the systems use large amounts of data or large numbers of rules by
applying indices inside rules or by processing transactions that could be
batched - which really increases the number of rules or that that are seen.
One of my customers used indices in rules creating rulesets of ~ 200,000
rules. With analysis the number of rules was reduced to ~ 10 + some data
structures. If account processing transactions can be grouped together then
the number of transactions may be dramatically reduced.
The approach of using a thinner client with seperate sessions on seperate
processes using partitioned data is a good idea. If you don't have an
intuititve key then you may want processes to processes chunks of queued
data as processes are ready.
--
View this message in context:
http://drools.46999.n3.nabble.com/Drools-5-3-partitioned-rule-base-tp3793...
Sent from the Drools: User forum mailing list archive at Nabble.com.
4441
days inactive
4446
days old
26 comments
5 participants
participants (5)
-
drdaveg -
gboro54 -
Mark Proctor -
Mauricio Salatino -
Wolfgang Laun