5:30 a.m.
Deadlock between GroupRequest and FLUSH during concurrent startup.
------------------------------------------------------------------
Key: JGRP-750
URL: http://jira.jboss.com/jira/browse/JGRP-750
Project: JGroups
Issue Type: Bug
Affects Versions: 2.6.3
Environment: Debian etch (i386), Java HotSpot(TM) Server VM (build 1.6.0-b105,
mixed mode)
Reporter: Robert Newson
Assigned To: Bela Ban
We've been having more trouble with concurrent start up and now think
we've isolated a deadlock between FLUSH and GroupRequest during
concurrent startup.
We have four boxes that join a channel and use MessageDispatcher
immediately after connecting. This frequently blocks indefinitely.
GroupRequest.execute() obtains a lock, then a subsequent view change
comes in which does likewise. The upshot is that we can see all
Incoming threads are blocked for the lock and the only way it can be
released is for a stop_flush message to occur. With all incoming
threads blocked, that never happens.
In the attached unit test if you add this after the call to connect("A"), it
passes, implying a deadlock;
if (j ==0) {
Thread.sleep(500);
}
Additionally, and this is more speculative, it seems the wait/notify code in pbcast does
not account for the spurious wakeup case. I don't know under what circumstances they
happen, and I don't believe we're seeing spurious wakes at this time, but it
should be fixed at some stage.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
5:32 a.m.
New subject: [JBoss JIRA] Updated: (JGRP-750) Deadlock between GroupRequest and FLUSH during concurrent startup.
[ http://jira.jboss.com/jira/browse/JGRP-750?page=all ]
Robert Newson updated JGRP-750:
-------------------------------
Attachment: ConcurrentStartupWithGroupRequestTest.java
Test to demonstrate lockup with GroupRequest and FLUSH.
The test uses a countdown latch to verify that each of the four MessageDispatchers,
collectively, have sent 50 messages. This never happens unless you add a sleep() after
connect() to allow members to join (and flush) in isolation.
Deadlock between GroupRequest and FLUSH during concurrent startup.
------------------------------------------------------------------
Key: JGRP-750
URL: http://jira.jboss.com/jira/browse/JGRP-750
Project: JGroups
Issue Type: Bug
Affects Versions: 2.6.3
Environment: Debian etch (i386), Java HotSpot(TM) Server VM (build 1.6.0-b105,
mixed mode)
Reporter: Robert Newson
Assigned To: Bela Ban
Attachments: ConcurrentStartupWithGroupRequestTest.java
We've been having more trouble with concurrent start up and now think
we've isolated a deadlock between FLUSH and GroupRequest during
concurrent startup.
We have four boxes that join a channel and use MessageDispatcher
immediately after connecting. This frequently blocks indefinitely.
GroupRequest.execute() obtains a lock, then a subsequent view change
comes in which does likewise. The upshot is that we can see all
Incoming threads are blocked for the lock and the only way it can be
released is for a stop_flush message to occur. With all incoming
threads blocked, that never happens.
In the attached unit test if you add this after the call to connect("A"), it
passes, implying a deadlock;
if (j ==0) {
Thread.sleep(500);
}
Additionally, and this is more speculative, it seems the wait/notify code in pbcast does
not account for the spurious wakeup case. I don't know under what circumstances they
happen, and I don't believe we're seeing spurious wakes at this time, but it
should be fixed at some stage.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
5:34 a.m.
New subject: [JBoss JIRA] Updated: (JGRP-750) Deadlock between GroupRequest and FLUSH during concurrent startup.
[ http://jira.jboss.com/jira/browse/JGRP-750?page=all ]
Robert Newson updated JGRP-750:
-------------------------------
Attachment: stacktrace.txt
Complete run with stack trace at the point it wedges.
Deadlock between GroupRequest and FLUSH during concurrent startup.
------------------------------------------------------------------
Key: JGRP-750
URL: http://jira.jboss.com/jira/browse/JGRP-750
Project: JGroups
Issue Type: Bug
Affects Versions: 2.6.3
Environment: Debian etch (i386), Java HotSpot(TM) Server VM (build 1.6.0-b105,
mixed mode)
Reporter: Robert Newson
Assigned To: Bela Ban
Attachments: ConcurrentStartupWithGroupRequestTest.java, stacktrace.txt
We've been having more trouble with concurrent start up and now think
we've isolated a deadlock between FLUSH and GroupRequest during
concurrent startup.
We have four boxes that join a channel and use MessageDispatcher
immediately after connecting. This frequently blocks indefinitely.
GroupRequest.execute() obtains a lock, then a subsequent view change
comes in which does likewise. The upshot is that we can see all
Incoming threads are blocked for the lock and the only way it can be
released is for a stop_flush message to occur. With all incoming
threads blocked, that never happens.
In the attached unit test if you add this after the call to connect("A"), it
passes, implying a deadlock;
if (j ==0) {
Thread.sleep(500);
}
Additionally, and this is more speculative, it seems the wait/notify code in pbcast does
not account for the spurious wakeup case. I don't know under what circumstances they
happen, and I don't believe we're seeing spurious wakes at this time, but it
should be fixed at some stage.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
5:38 a.m.
New subject: [JBoss JIRA] Updated: (JGRP-750) Deadlock between GroupRequest and FLUSH during concurrent startup.
[ http://jira.jboss.com/jira/browse/JGRP-750?page=all ]
Robert Newson updated JGRP-750:
-------------------------------
Attachment: ConcurrentStartupWithGroupRequestTest.java
Updated version. I had the connect() inside the loop rather than outside. It still fails,
sadly.
There's a Thread.sleep line that is commented out. If you uncomment it, the test
passes.
Deadlock between GroupRequest and FLUSH during concurrent startup.
------------------------------------------------------------------
Key: JGRP-750
URL: http://jira.jboss.com/jira/browse/JGRP-750
Project: JGroups
Issue Type: Bug
Affects Versions: 2.6.3
Environment: Debian etch (i386), Java HotSpot(TM) Server VM (build 1.6.0-b105,
mixed mode)
Reporter: Robert Newson
Assigned To: Bela Ban
Attachments: ConcurrentStartupWithGroupRequestTest.java,
ConcurrentStartupWithGroupRequestTest.java, stacktrace.txt
We've been having more trouble with concurrent start up and now think
we've isolated a deadlock between FLUSH and GroupRequest during
concurrent startup.
We have four boxes that join a channel and use MessageDispatcher
immediately after connecting. This frequently blocks indefinitely.
GroupRequest.execute() obtains a lock, then a subsequent view change
comes in which does likewise. The upshot is that we can see all
Incoming threads are blocked for the lock and the only way it can be
released is for a stop_flush message to occur. With all incoming
threads blocked, that never happens.
In the attached unit test if you add this after the call to connect("A"), it
passes, implying a deadlock;
if (j ==0) {
Thread.sleep(500);
}
Additionally, and this is more speculative, it seems the wait/notify code in pbcast does
not account for the spurious wakeup case. I don't know under what circumstances they
happen, and I don't believe we're seeing spurious wakes at this time, but it
should be fixed at some stage.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
7:58 a.m.
New subject: [JBoss JIRA] Commented: (JGRP-750) Deadlock between GroupRequest and FLUSH during concurrent startup.
[ http://jira.jboss.com/jira/browse/JGRP-750?page=comments#action_12411465 ]
Michael Newcomb commented on JGRP-750:
--------------------------------------
Do you do channel.setOpt(Channel.BLOCK, true)?
Also, this uses multiplexer?
Deadlock between GroupRequest and FLUSH during concurrent startup.
------------------------------------------------------------------
Key: JGRP-750
URL: http://jira.jboss.com/jira/browse/JGRP-750
Project: JGroups
Issue Type: Bug
Affects Versions: 2.6.3
Environment: Debian etch (i386), Java HotSpot(TM) Server VM (build 1.6.0-b105,
mixed mode)
Reporter: Robert Newson
Assigned To: Bela Ban
Attachments: ConcurrentStartupWithGroupRequestTest.java,
ConcurrentStartupWithGroupRequestTest.java, stacktrace.txt
We've been having more trouble with concurrent start up and now think
we've isolated a deadlock between FLUSH and GroupRequest during
concurrent startup.
We have four boxes that join a channel and use MessageDispatcher
immediately after connecting. This frequently blocks indefinitely.
GroupRequest.execute() obtains a lock, then a subsequent view change
comes in which does likewise. The upshot is that we can see all
Incoming threads are blocked for the lock and the only way it can be
released is for a stop_flush message to occur. With all incoming
threads blocked, that never happens.
In the attached unit test if you add this after the call to connect("A"), it
passes, implying a deadlock;
if (j ==0) {
Thread.sleep(500);
}
Additionally, and this is more speculative, it seems the wait/notify code in pbcast does
not account for the spurious wakeup case. I don't know under what circumstances they
happen, and I don't believe we're seeing spurious wakes at this time, but it
should be fixed at some stage.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
9:34 a.m.
New subject: [JBoss JIRA] Assigned: (JGRP-750) Deadlock between GroupRequest and FLUSH during concurrent startup.
[ http://jira.jboss.com/jira/browse/JGRP-750?page=all ]
Vladimir Blagojevic reassigned JGRP-750:
----------------------------------------
Assignee: Vladimir Blagojevic (was: Bela Ban)
Deadlock between GroupRequest and FLUSH during concurrent startup.
------------------------------------------------------------------
Key: JGRP-750
URL: http://jira.jboss.com/jira/browse/JGRP-750
Project: JGroups
Issue Type: Bug
Affects Versions: 2.6.3
Environment: Debian etch (i386), Java HotSpot(TM) Server VM (build 1.6.0-b105,
mixed mode)
Reporter: Robert Newson
Assigned To: Vladimir Blagojevic
Attachments: ConcurrentStartupWithGroupRequestTest.java,
ConcurrentStartupWithGroupRequestTest.java, stacktrace.txt
We've been having more trouble with concurrent start up and now think
we've isolated a deadlock between FLUSH and GroupRequest during
concurrent startup.
We have four boxes that join a channel and use MessageDispatcher
immediately after connecting. This frequently blocks indefinitely.
GroupRequest.execute() obtains a lock, then a subsequent view change
comes in which does likewise. The upshot is that we can see all
Incoming threads are blocked for the lock and the only way it can be
released is for a stop_flush message to occur. With all incoming
threads blocked, that never happens.
In the attached unit test if you add this after the call to connect("A"), it
passes, implying a deadlock;
if (j ==0) {
Thread.sleep(500);
}
Additionally, and this is more speculative, it seems the wait/notify code in pbcast does
not account for the spurious wakeup case. I don't know under what circumstances they
happen, and I don't believe we're seeing spurious wakes at this time, but it
should be fixed at some stage.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
5:06 p.m.
New subject: [JBoss JIRA] Commented: (JGRP-750) Deadlock between GroupRequest and FLUSH during concurrent startup.
[ http://jira.jboss.com/jira/browse/JGRP-750?page=comments#action_12411508 ]
Robert Newson commented on JGRP-750:
------------------------------------
This is with the multiplexer. I do not set BlOCK to true. Setting it does not make the
test pass, but disabling the multiplexer does.
Deadlock between GroupRequest and FLUSH during concurrent startup.
------------------------------------------------------------------
Key: JGRP-750
URL: http://jira.jboss.com/jira/browse/JGRP-750
Project: JGroups
Issue Type: Bug
Affects Versions: 2.6.3
Environment: Debian etch (i386), Java HotSpot(TM) Server VM (build 1.6.0-b105,
mixed mode)
Reporter: Robert Newson
Assigned To: Vladimir Blagojevic
Attachments: ConcurrentStartupWithGroupRequestTest.java,
ConcurrentStartupWithGroupRequestTest.java, stacktrace.txt
We've been having more trouble with concurrent start up and now think
we've isolated a deadlock between FLUSH and GroupRequest during
concurrent startup.
We have four boxes that join a channel and use MessageDispatcher
immediately after connecting. This frequently blocks indefinitely.
GroupRequest.execute() obtains a lock, then a subsequent view change
comes in which does likewise. The upshot is that we can see all
Incoming threads are blocked for the lock and the only way it can be
released is for a stop_flush message to occur. With all incoming
threads blocked, that never happens.
In the attached unit test if you add this after the call to connect("A"), it
passes, implying a deadlock;
if (j ==0) {
Thread.sleep(500);
}
Additionally, and this is more speculative, it seems the wait/notify code in pbcast does
not account for the spurious wakeup case. I don't know under what circumstances they
happen, and I don't believe we're seeing spurious wakes at this time, but it
should be fixed at some stage.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
8:15 a.m.
New subject: [JBoss JIRA] Commented: (JGRP-750) Deadlock between GroupRequest and FLUSH during concurrent startup.
[ http://jira.jboss.com/jira/browse/JGRP-750?page=comments#action_12411583 ]
Vladimir Blagojevic commented on JGRP-750:
------------------------------------------
I found a cause for this problem. When application thread calls castMessage underlying
plumbing in MessageDispatcher sends a GroupRequest which in turns obtains a lock held
until response arrives. In MuxChannel#connect we send a sync call across the cluster to
update a service view. When this request comes to a node that has sent GroupRequest it
needs to obtain the same lock to update a view in GroupRequest#viewChange. However, it
cannot obtain a lock since it is held by an application thread which has been in turned
blocked in FLUSH.down. So request times out.
So you ask how come this happens in MuxChannel and not in JChannel. Well, in
MuxChannel#connect we do two flushes in case it is a first service connecting on top of a
unconnected "real" channel. The infringing call castMessage squeezes itself in
between these two flushes.
Deadlock between GroupRequest and FLUSH during concurrent startup.
------------------------------------------------------------------
Key: JGRP-750
URL: http://jira.jboss.com/jira/browse/JGRP-750
Project: JGroups
Issue Type: Bug
Affects Versions: 2.6.3
Environment: Debian etch (i386), Java HotSpot(TM) Server VM (build 1.6.0-b105,
mixed mode)
Reporter: Robert Newson
Assigned To: Vladimir Blagojevic
Attachments: ConcurrentStartupWithGroupRequestTest.java,
ConcurrentStartupWithGroupRequestTest.java, stacktrace.txt
We've been having more trouble with concurrent start up and now think
we've isolated a deadlock between FLUSH and GroupRequest during
concurrent startup.
We have four boxes that join a channel and use MessageDispatcher
immediately after connecting. This frequently blocks indefinitely.
GroupRequest.execute() obtains a lock, then a subsequent view change
comes in which does likewise. The upshot is that we can see all
Incoming threads are blocked for the lock and the only way it can be
released is for a stop_flush message to occur. With all incoming
threads blocked, that never happens.
In the attached unit test if you add this after the call to connect("A"), it
passes, implying a deadlock;
if (j ==0) {
Thread.sleep(500);
}
Additionally, and this is more speculative, it seems the wait/notify code in pbcast does
not account for the spurious wakeup case. I don't know under what circumstances they
happen, and I don't believe we're seeing spurious wakes at this time, but it
should be fixed at some stage.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
1:09 a.m.
New subject: [JBoss JIRA] Updated: (JGRP-750) Deadlock between GroupRequest and FLUSH during concurrent startup.
[ http://jira.jboss.com/jira/browse/JGRP-750?page=all ]
Bela Ban updated JGRP-750:
--------------------------
Fix Version/s: 2.6.3
Deadlock between GroupRequest and FLUSH during concurrent startup.
------------------------------------------------------------------
Key: JGRP-750
URL: http://jira.jboss.com/jira/browse/JGRP-750
Project: JGroups
Issue Type: Bug
Affects Versions: 2.6.3
Environment: Debian etch (i386), Java HotSpot(TM) Server VM (build 1.6.0-b105,
mixed mode)
Reporter: Robert Newson
Assigned To: Vladimir Blagojevic
Fix For: 2.7, 2.6.3
Attachments: ConcurrentStartupWithGroupRequestTest.java,
ConcurrentStartupWithGroupRequestTest.java, stacktrace.txt
We've been having more trouble with concurrent start up and now think
we've isolated a deadlock between FLUSH and GroupRequest during
concurrent startup.
We have four boxes that join a channel and use MessageDispatcher
immediately after connecting. This frequently blocks indefinitely.
GroupRequest.execute() obtains a lock, then a subsequent view change
comes in which does likewise. The upshot is that we can see all
Incoming threads are blocked for the lock and the only way it can be
released is for a stop_flush message to occur. With all incoming
threads blocked, that never happens.
In the attached unit test if you add this after the call to connect("A"), it
passes, implying a deadlock;
if (j ==0) {
Thread.sleep(500);
}
Additionally, and this is more speculative, it seems the wait/notify code in pbcast does
not account for the spurious wakeup case. I don't know under what circumstances they
happen, and I don't believe we're seeing spurious wakes at this time, but it
should be fixed at some stage.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
1:09 a.m.
New subject: [JBoss JIRA] Updated: (JGRP-750) Deadlock between GroupRequest and FLUSH during concurrent startup.
[ http://jira.jboss.com/jira/browse/JGRP-750?page=all ]
Bela Ban updated JGRP-750:
--------------------------
Fix Version/s: 2.7
Deadlock between GroupRequest and FLUSH during concurrent startup.
------------------------------------------------------------------
Key: JGRP-750
URL: http://jira.jboss.com/jira/browse/JGRP-750
Project: JGroups
Issue Type: Bug
Affects Versions: 2.6.3
Environment: Debian etch (i386), Java HotSpot(TM) Server VM (build 1.6.0-b105,
mixed mode)
Reporter: Robert Newson
Assigned To: Vladimir Blagojevic
Fix For: 2.7, 2.6.3
Attachments: ConcurrentStartupWithGroupRequestTest.java,
ConcurrentStartupWithGroupRequestTest.java, stacktrace.txt
We've been having more trouble with concurrent start up and now think
we've isolated a deadlock between FLUSH and GroupRequest during
concurrent startup.
We have four boxes that join a channel and use MessageDispatcher
immediately after connecting. This frequently blocks indefinitely.
GroupRequest.execute() obtains a lock, then a subsequent view change
comes in which does likewise. The upshot is that we can see all
Incoming threads are blocked for the lock and the only way it can be
released is for a stop_flush message to occur. With all incoming
threads blocked, that never happens.
In the attached unit test if you add this after the call to connect("A"), it
passes, implying a deadlock;
if (j ==0) {
Thread.sleep(500);
}
Additionally, and this is more speculative, it seems the wait/notify code in pbcast does
not account for the spurious wakeup case. I don't know under what circumstances they
happen, and I don't believe we're seeing spurious wakes at this time, but it
should be fixed at some stage.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
4:03 a.m.
New subject: [JBoss JIRA] Commented: (JGRP-750) Deadlock between GroupRequest and FLUSH during concurrent startup.
[ http://jira.jboss.com/jira/browse/JGRP-750?page=comments#action_12412563 ]
Vladimir Blagojevic commented on JGRP-750:
------------------------------------------
Don't think we can resolve this problem without major retrofitting of Multiplexer.
Workaround exists. Simply add sleep after each channel joins. Consider closing.
Deadlock between GroupRequest and FLUSH during concurrent startup.
------------------------------------------------------------------
Key: JGRP-750
URL: http://jira.jboss.com/jira/browse/JGRP-750
Project: JGroups
Issue Type: Bug
Affects Versions: 2.6.3
Environment: Debian etch (i386), Java HotSpot(TM) Server VM (build 1.6.0-b105,
mixed mode)
Reporter: Robert Newson
Assigned To: Vladimir Blagojevic
Fix For: 2.6.3, 2.7
Attachments: ConcurrentStartupWithGroupRequestTest.java,
ConcurrentStartupWithGroupRequestTest.java, stacktrace.txt
We've been having more trouble with concurrent start up and now think
we've isolated a deadlock between FLUSH and GroupRequest during
concurrent startup.
We have four boxes that join a channel and use MessageDispatcher
immediately after connecting. This frequently blocks indefinitely.
GroupRequest.execute() obtains a lock, then a subsequent view change
comes in which does likewise. The upshot is that we can see all
Incoming threads are blocked for the lock and the only way it can be
released is for a stop_flush message to occur. With all incoming
threads blocked, that never happens.
In the attached unit test if you add this after the call to connect("A"), it
passes, implying a deadlock;
if (j ==0) {
Thread.sleep(500);
}
Additionally, and this is more speculative, it seems the wait/notify code in pbcast does
not account for the spurious wakeup case. I don't know under what circumstances they
happen, and I don't believe we're seeing spurious wakes at this time, but it
should be fixed at some stage.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
4:05 a.m.
New subject: [JBoss JIRA] Updated: (JGRP-750) Deadlock between GroupRequest and FLUSH during concurrent startup.
[ http://jira.jboss.com/jira/browse/JGRP-750?page=all ]
Vladimir Blagojevic updated JGRP-750:
-------------------------------------
Workaround Description: Add sleep after connect and avoid concurrent channel startup
Deadlock between GroupRequest and FLUSH during concurrent startup.
------------------------------------------------------------------
Key: JGRP-750
URL: http://jira.jboss.com/jira/browse/JGRP-750
Project: JGroups
Issue Type: Bug
Affects Versions: 2.6.3
Environment: Debian etch (i386), Java HotSpot(TM) Server VM (build 1.6.0-b105,
mixed mode)
Reporter: Robert Newson
Assigned To: Vladimir Blagojevic
Fix For: 2.6.3, 2.7
Attachments: ConcurrentStartupWithGroupRequestTest.java,
ConcurrentStartupWithGroupRequestTest.java, stacktrace.txt
We've been having more trouble with concurrent start up and now think
we've isolated a deadlock between FLUSH and GroupRequest during
concurrent startup.
We have four boxes that join a channel and use MessageDispatcher
immediately after connecting. This frequently blocks indefinitely.
GroupRequest.execute() obtains a lock, then a subsequent view change
comes in which does likewise. The upshot is that we can see all
Incoming threads are blocked for the lock and the only way it can be
released is for a stop_flush message to occur. With all incoming
threads blocked, that never happens.
In the attached unit test if you add this after the call to connect("A"), it
passes, implying a deadlock;
if (j ==0) {
Thread.sleep(500);
}
Additionally, and this is more speculative, it seems the wait/notify code in pbcast does
not account for the spurious wakeup case. I don't know under what circumstances they
happen, and I don't believe we're seeing spurious wakes at this time, but it
should be fixed at some stage.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
12:16 p.m.
New subject: [JBoss JIRA] Assigned: (JGRP-750) Deadlock between GroupRequest and FLUSH during concurrent startup.
[ http://jira.jboss.com/jira/browse/JGRP-750?page=all ]
Vladimir Blagojevic reassigned JGRP-750:
----------------------------------------
Assignee: Bela Ban (was: Vladimir Blagojevic)
This is the problem inherit to two flushes we do in Multiplexer when new channel joins.
There is no fix without major reworking. Workaround exists. Consider closing.
Deadlock between GroupRequest and FLUSH during concurrent startup.
------------------------------------------------------------------
Key: JGRP-750
URL: http://jira.jboss.com/jira/browse/JGRP-750
Project: JGroups
Issue Type: Bug
Affects Versions: 2.6.3
Environment: Debian etch (i386), Java HotSpot(TM) Server VM (build 1.6.0-b105,
mixed mode)
Reporter: Robert Newson
Assigned To: Bela Ban
Fix For: 2.7, 2.6.3
Attachments: ConcurrentStartupWithGroupRequestTest.java,
ConcurrentStartupWithGroupRequestTest.java, stacktrace.txt
We've been having more trouble with concurrent start up and now think
we've isolated a deadlock between FLUSH and GroupRequest during
concurrent startup.
We have four boxes that join a channel and use MessageDispatcher
immediately after connecting. This frequently blocks indefinitely.
GroupRequest.execute() obtains a lock, then a subsequent view change
comes in which does likewise. The upshot is that we can see all
Incoming threads are blocked for the lock and the only way it can be
released is for a stop_flush message to occur. With all incoming
threads blocked, that never happens.
In the attached unit test if you add this after the call to connect("A"), it
passes, implying a deadlock;
if (j ==0) {
Thread.sleep(500);
}
Additionally, and this is more speculative, it seems the wait/notify code in pbcast does
not account for the spurious wakeup case. I don't know under what circumstances they
happen, and I don't believe we're seeing spurious wakes at this time, but it
should be fixed at some stage.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
9:03 a.m.
New subject: [JBoss JIRA] Closed: (JGRP-750) Deadlock between GroupRequest and FLUSH during concurrent startup.
[ http://jira.jboss.com/jira/browse/JGRP-750?page=all ]
Vladimir Blagojevic closed JGRP-750.
------------------------------------
Resolution: Done
This is hard to fix and we are not going to support Multiplexer anyway! Closing this
issue
Deadlock between GroupRequest and FLUSH during concurrent startup.
------------------------------------------------------------------
Key: JGRP-750
URL: http://jira.jboss.com/jira/browse/JGRP-750
Project: JGroups
Issue Type: Bug
Affects Versions: 2.6.3
Environment: Debian etch (i386), Java HotSpot(TM) Server VM (build 1.6.0-b105,
mixed mode)
Reporter: Robert Newson
Assigned To: Bela Ban
Fix For: 2.6.3, 2.7
Attachments: ConcurrentStartupWithGroupRequestTest.java,
ConcurrentStartupWithGroupRequestTest.java, stacktrace.txt
We've been having more trouble with concurrent start up and now think
we've isolated a deadlock between FLUSH and GroupRequest during
concurrent startup.
We have four boxes that join a channel and use MessageDispatcher
immediately after connecting. This frequently blocks indefinitely.
GroupRequest.execute() obtains a lock, then a subsequent view change
comes in which does likewise. The upshot is that we can see all
Incoming threads are blocked for the lock and the only way it can be
released is for a stop_flush message to occur. With all incoming
threads blocked, that never happens.
In the attached unit test if you add this after the call to connect("A"), it
passes, implying a deadlock;
if (j ==0) {
Thread.sleep(500);
}
Additionally, and this is more speculative, it seems the wait/notify code in pbcast does
not account for the spurious wakeup case. I don't know under what circumstances they
happen, and I don't believe we're seeing spurious wakes at this time, but it
should be fixed at some stage.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
6633
days inactive
6654
days old
13 comments
4 participants
participants (4)
-
Bela Ban (JIRA) -
Michael Newcomb (JIRA)
-
Robert Newson (JIRA)
-
Vladimir Blagojevic (JIRA)