[JBoss JIRA] (DROOLS-67) deadlock when invoking rules concrrently
by Mario Fusco (JIRA)
[ https://issues.jboss.org/browse/DROOLS-67?page=com.atlassian.jira.plugin.... ]
Mario Fusco commented on DROOLS-67:
-----------------------------------
It is really impossible for me to guess what's going on just looking at that stack trace.
Could you send a reproducer or at least give me some more information to help me to reproduce it?
> deadlock when invoking rules concrrently
> ----------------------------------------
>
> Key: DROOLS-67
> URL: https://issues.jboss.org/browse/DROOLS-67
> Project: Drools
> Issue Type: Bug
> Security Level: Public(Everyone can see)
> Affects Versions: 5.5
> Reporter: smita pande
> Assignee: Mark Proctor
>
> Owns Monitor Lock on org/drools/rule/JavaDialectRuntimeData$PackageClassLoader@0E99F9B8/0E99F9C4
> Waiting for Monitor Lock on org/drools/rule/CompositeClassLoader@0DB50DC0/0DB50DCC
> at java/lang/ClassLoader.loadClass(ClassLoader.java:606(Compiled Code))
> at java/lang/Class.forNameImpl(Native Method)
> at java/lang/Class.forName(Class.java:163(Compiled Code))
> at org/drools/rule/JavaDialectRuntimeData$PackageClassLoader.loadClass(JavaDialectRuntimeData.java:458(Compiled Code))
> at java/lang/ClassLoader.loadClass(ClassLoader.java:606(Compiled Code))
> at /Rule_AutoUW_Rule_BR12Constants_153_0ConsequenceInvoker.evaluate(Rule_AutoUW_Rule_BR12Constants_153_0ConsequenceInvoker.java:16)
> at org/drools/common/DefaultAgenda.fireActivation(DefaultAgenda.java:934(Compiled Code))
> at org/drools/common/DefaultAgenda.fireNextItem(DefaultAgenda.java:885(Compiled Code))
> at org/drools/common/DefaultAgenda.fireAllRules(DefaultAgenda.java:1086(Compiled Code))
> at org/drools/common/AbstractWorkingMemory.fireAllRules(AbstractWorkingMemory.java:661(Compiled Code))
> at org/drools/common/AbstractWorkingMemory.fireAllRules(AbstractWorkingMemory.java:637(Compiled Code))
> at org/drools/reteoo/ReteooStatelessSession.execute(ReteooStatelessSession.java:203(Compiled Code))
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
13 years, 4 months
[JBoss JIRA] (JGRP-1604) SEQUENCER problems and slowness
by Bela Ban (JIRA)
[ https://issues.jboss.org/browse/JGRP-1604?page=com.atlassian.jira.plugin.... ]
Bela Ban commented on JGRP-1604:
--------------------------------
We don't need the blocking on forward-table; memory consumption always stayed ok with 10 sender threads. Can be added later if necessary. However, the change where messages were not serialized and added to forward-table as byte[] buffers, but rather as messages, probably helped more, as we're now storing a message only once (the same msg is stored in forward-table and UNICAST(2,3).
> SEQUENCER problems and slowness
> -------------------------------
>
> Key: JGRP-1604
> URL: https://issues.jboss.org/browse/JGRP-1604
> Project: JGroups
> Issue Type: Enhancement
> Reporter: Bela Ban
> Assignee: Bela Ban
> Fix For: 3.3
>
>
> On high volumes of messages (e.g. MPerf), SEQUENCER runs out of memory.
> The culprit is the forward-table, which is unbounded, and if messages are added as a faster rate than the time it takes to forward the message to the coordinator (sequencer) and for it to be received as a multicast and subsequently be removed from the forward-table, memory increases.
> This also makes SEQUENCER slow.
> SOLUTION:
> * Look into bounding the forward-table, e.g. define the max number of bytes to be in the table at any given time
> * When a message is to be added, its length is checked (Message.getLength())
> ** If the length is 0 or length + accumulated bytes is less than max: add, else block
> * A received message (sent by self) causes the entry to be removed from the forward-table and its length to be subtracted from accumulated bytes (unblocking potentially blocked threads)
> We should also look at delivery table and see if we can make it simpler, e.g. by running a gathering round when a new sequencer is chosen, determining the highest seqnos of all members
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
13 years, 4 months
[JBoss JIRA] (DROOLS-67) deadlock when invoking rules concrrently
by Mark Proctor (JIRA)
[ https://issues.jboss.org/browse/DROOLS-67?page=com.atlassian.jira.plugin.... ]
Mark Proctor updated DROOLS-67:
-------------------------------
Priority: Major (was: Blocker)
> deadlock when invoking rules concrrently
> ----------------------------------------
>
> Key: DROOLS-67
> URL: https://issues.jboss.org/browse/DROOLS-67
> Project: Drools
> Issue Type: Bug
> Security Level: Public(Everyone can see)
> Affects Versions: 5.5
> Reporter: smita pande
> Assignee: Mark Proctor
>
> Owns Monitor Lock on org/drools/rule/JavaDialectRuntimeData$PackageClassLoader@0E99F9B8/0E99F9C4
> Waiting for Monitor Lock on org/drools/rule/CompositeClassLoader@0DB50DC0/0DB50DCC
> at java/lang/ClassLoader.loadClass(ClassLoader.java:606(Compiled Code))
> at java/lang/Class.forNameImpl(Native Method)
> at java/lang/Class.forName(Class.java:163(Compiled Code))
> at org/drools/rule/JavaDialectRuntimeData$PackageClassLoader.loadClass(JavaDialectRuntimeData.java:458(Compiled Code))
> at java/lang/ClassLoader.loadClass(ClassLoader.java:606(Compiled Code))
> at /Rule_AutoUW_Rule_BR12Constants_153_0ConsequenceInvoker.evaluate(Rule_AutoUW_Rule_BR12Constants_153_0ConsequenceInvoker.java:16)
> at org/drools/common/DefaultAgenda.fireActivation(DefaultAgenda.java:934(Compiled Code))
> at org/drools/common/DefaultAgenda.fireNextItem(DefaultAgenda.java:885(Compiled Code))
> at org/drools/common/DefaultAgenda.fireAllRules(DefaultAgenda.java:1086(Compiled Code))
> at org/drools/common/AbstractWorkingMemory.fireAllRules(AbstractWorkingMemory.java:661(Compiled Code))
> at org/drools/common/AbstractWorkingMemory.fireAllRules(AbstractWorkingMemory.java:637(Compiled Code))
> at org/drools/reteoo/ReteooStatelessSession.execute(ReteooStatelessSession.java:203(Compiled Code))
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
13 years, 4 months
[JBoss JIRA] (JGRP-1604) SEQUENCER problems and slowness
by Bela Ban (JIRA)
[ https://issues.jboss.org/browse/JGRP-1604?page=com.atlassian.jira.plugin.... ]
Bela Ban commented on JGRP-1604:
--------------------------------
I found out that having the coordinator (sequencer) broadcast its messages directly, versus other members having to forward their messages to the coord, was *starving* the messages of the other members: the coordinator always preferred sending its own messages.
I changed this, so that now the coord forwards messages to itself, to have a fairer scheduling and MPerf shows ca. 50MB/sec/node, compared to 20 before !
> SEQUENCER problems and slowness
> -------------------------------
>
> Key: JGRP-1604
> URL: https://issues.jboss.org/browse/JGRP-1604
> Project: JGroups
> Issue Type: Enhancement
> Reporter: Bela Ban
> Assignee: Bela Ban
> Fix For: 3.3
>
>
> On high volumes of messages (e.g. MPerf), SEQUENCER runs out of memory.
> The culprit is the forward-table, which is unbounded, and if messages are added as a faster rate than the time it takes to forward the message to the coordinator (sequencer) and for it to be received as a multicast and subsequently be removed from the forward-table, memory increases.
> This also makes SEQUENCER slow.
> SOLUTION:
> * Look into bounding the forward-table, e.g. define the max number of bytes to be in the table at any given time
> * When a message is to be added, its length is checked (Message.getLength())
> ** If the length is 0 or length + accumulated bytes is less than max: add, else block
> * A received message (sent by self) causes the entry to be removed from the forward-table and its length to be subtracted from accumulated bytes (unblocking potentially blocked threads)
> We should also look at delivery table and see if we can make it simpler, e.g. by running a gathering round when a new sequencer is chosen, determining the highest seqnos of all members
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
13 years, 4 months
[JBoss JIRA] (JGRP-1604) SEQUENCER problems and slowness
by Bela Ban (JIRA)
[ https://issues.jboss.org/browse/JGRP-1604?page=com.atlassian.jira.plugin.... ]
Bela Ban commented on JGRP-1604:
--------------------------------
The culprit is delivery_table and canDeliver(), which calls size() on the ConcurrentSkipListSet: size() has a cost linear to the number of elements in the set, and size() is called on *every* delivery.
Suggested SOLUTION:
# Make canDeliver() fast, e.g. use a linked hashmap with seqnos as keys for delivery_table
# (Indepenent from 1) Make addition to forward-table blocking, so we can't run out of memory when many threads are adding messages to the forward-table. The blocking should be based on the number of bytes in the forward-table (Message.getLength())
> SEQUENCER problems and slowness
> -------------------------------
>
> Key: JGRP-1604
> URL: https://issues.jboss.org/browse/JGRP-1604
> Project: JGroups
> Issue Type: Enhancement
> Reporter: Bela Ban
> Assignee: Bela Ban
> Fix For: 3.3
>
>
> On high volumes of messages (e.g. MPerf), SEQUENCER runs out of memory.
> The culprit is the forward-table, which is unbounded, and if messages are added as a faster rate than the time it takes to forward the message to the coordinator (sequencer) and for it to be received as a multicast and subsequently be removed from the forward-table, memory increases.
> This also makes SEQUENCER slow.
> SOLUTION:
> * Look into bounding the forward-table, e.g. define the max number of bytes to be in the table at any given time
> * When a message is to be added, its length is checked (Message.getLength())
> ** If the length is 0 or length + accumulated bytes is less than max: add, else block
> * A received message (sent by self) causes the entry to be removed from the forward-table and its length to be subtracted from accumulated bytes (unblocking potentially blocked threads)
> We should also look at delivery table and see if we can make it simpler, e.g. by running a gathering round when a new sequencer is chosen, determining the highest seqnos of all members
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
13 years, 4 months