[ http://jira.jboss.com/jira/browse/JGRP-532?page=all ] Bela Ban updated JGRP-532: -------------------------- Attachment: config.txt -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://jira.jboss.com/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ http://jira.jboss.com/jira/browse/JGRP-532?page=comments#action_12365326 ] Bela Ban commented on JGRP-532: ------------------------------- Seems to happen only when TCP.use_send_queues is enabled -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://jira.jboss.com/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ http://jira.jboss.com/jira/browse/JGRP-532?page=comments#action_12365487 ] Bela Ban commented on JGRP-532: ------------------------------- Not true, also happens when send queues are disabled. What happens is that Connection._send() can block on the TCP write(), as the receiver node set the sliding window to 0. Example: - A and B - A (NAKACK) receives a message from B, acquires a lock on B's NakReceiverWindow and passes the message up. - In A.STABLE, we determine that we've seen enough bytes, and send down a STABLE message - The STABLE message blocks on write() (B has temporarily set its sliding window to 0) - Now all messages from B will not get processed as the lock for B's NRW is held by the blocked write() thread - Therefore A will send a sliding window size of 0 to B, which will cause B's write() to block ==> deadlock -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://jira.jboss.com/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ http://jira.jboss.com/jira/browse/JGRP-532?page=comments#action_12365509 ] Bela Ban commented on JGRP-532: ------------------------------- Set the rejection policy to "discard" for all TCP based stacks, also in 2.4 -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://jira.jboss.com/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ http://jira.jboss.com/jira/browse/JGRP-532?page=comments#action_12365524 ] Bela Ban commented on JGRP-532: ------------------------------- We need to revert the rejection level to "run", otherwise performance is worse. The stack trace below shows that STABLE needs to release the lock on NakReceiverWindow before passing down the message, which would prevent the deadlock. This is related to 535. "Incoming Thread,perf,192.168.0.5:7801" daemon prio=10 tid=0x00002aaacba7ac00 nid=0x24b6 waiting on condition [0x0000000040aa9000..0x0000000040aa9d10] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00002aaab4046dd8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1889) at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:254) at org.jgroups.blocks.BasicConnectionTable$Connection.send(BasicConnectionTable.java:433) at org.jgroups.blocks.BasicConnectionTable.send(BasicConnectionTable.java:243) at org.jgroups.protocols.TCP.send(TCP.java:54) at org.jgroups.protocols.BasicTCP.sendToSingleMember(BasicTCP.java:194) at org.jgroups.protocols.BasicTCP.sendToAllMembers(BasicTCP.java:179) at org.jgroups.protocols.TP.doSend(TP.java:1187) at org.jgroups.protocols.TP.send(TP.java:1177) at org.jgroups.protocols.TP.down(TP.java:944) at org.jgroups.protocols.Discovery.down(Discovery.java:321) at org.jgroups.protocols.MERGE2.down(MERGE2.java:184) at org.jgroups.protocols.FD_SOCK.down(FD_SOCK.java:403) at org.jgroups.protocols.FD.down(FD.java:363) at org.jgroups.stack.Protocol.down(Protocol.java:252) at org.jgroups.protocols.BARRIER.down(BARRIER.java:97) at org.jgroups.protocols.pbcast.NAKACK.send(NAKACK.java:648) at org.jgroups.protocols.pbcast.NAKACK.down(NAKACK.java:438) at org.jgroups.protocols.UNICAST.down(UNICAST.java:430) at org.jgroups.protocols.pbcast.STABLE.sendStableMessage(STABLE.java:640) at org.jgroups.protocols.pbcast.STABLE.handleRegularMessage(STABLE.java:286) at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:233) at org.jgroups.protocols.UNICAST.up(UNICAST.java:263) at org.jgroups.protocols.pbcast.NAKACK.handleMessage(NAKACK.java:720) - locked <0x00002aaab404afc0> (a org.jgroups.stack.NakReceiverWindow) at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:546) at org.jgroups.protocols.BARRIER.up(BARRIER.java:122) at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:154) at org.jgroups.protocols.FD.up(FD.java:328) at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:298) at org.jgroups.protocols.MERGE2.up(MERGE2.java:145) at org.jgroups.protocols.Discovery.up(Discovery.java:220) at org.jgroups.protocols.MPING.up(MPING.java:151) at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1542) at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1491) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) at java.lang.Thread.run(Thread.java:619) -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://jira.jboss.com/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ http://jira.jboss.com/jira/browse/JGRP-532?page=comments#action_12365526 ] Bela Ban commented on JGRP-532: ------------------------------- So, the basic issue is this: - We have 1 thread that receives messages from each node (Connection.Receiver thread) - When all threads are used up in the pool, rejection policy of "run" make that receiver thread deliver the message - If that's the case and the receiver thread blocks on NAKACK's NakReceiverWindow, then it will not be able to pull further data off of the socket. After the socket's receive window fills up, the sender will block in write(). Alternatively, if use_send_queues is true, the receiver won't take messages off the queue, so put() will block (as the queue is bounded). - How can the receiver thread block ? If another receiver thread ahead of it acquires the lock for the NakReceiverWindow in NAKACK and calls down() on the same thread and happens to block on write() ! Example: - Threads 1,2 and 3 have submitted messages to the thread pool - Now the thread pool is exhausted, so "run" makes sure the Connection.Receiver calls up() directly (on its own thread) - The Connection.Receiver blocks until it can acquire the NRW lock, therefore it doesn't pull data off of the socket - Threads 1-3 send data, at one point, the TCP write() might block, say for example for Thread 3. - Thread-3 now blocks on write() and has the NRW lock - The Connection.Receiver thread waits for the NRW lock, but meanwhile doesn't cause the write() to unblock as it doesn't pull data off of the socket: ===> DEADLOCK ! SOLUTION: threads 1-3 need to unlock the NRW *before* they send down messages ! This is JIRA 535 -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://jira.jboss.com/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ http://jira.jboss.com/jira/browse/JGRP-532?page=all ] Bela Ban resolved JGRP-532. --------------------------- Resolution: Done The deadlock only happens when down() can block (e.g. in the transport). We cannot avoid a block on TCP write(), so we have to avoid blocking the receiver, and therefore run the sending of stable messages in a separate thread. Of course, we can always avoid the receiver from ever hanging by simply defining a rejection policy of "discard" rather than "run" -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://jira.jboss.com/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira