[JBoss JIRA] Created: (JGRP-1003) deadlock in TP.send - socket write hangs inside a Lock, preventing other threads to do TP.send()
4:58 a.m.
deadlock in TP.send - socket write hangs inside a Lock, preventing other threads to do
TP.send()
------------------------------------------------------------------------------------------------
Key: JGRP-1003
URL: https://jira.jboss.org/jira/browse/JGRP-1003
Project: JGroups
Issue Type: Bug
Affects Versions: 2.7
Environment: Linux (Debian), kernel 2.6.x
Reporter: Victor N
Assignee: Bela Ban
I am using JGroups 2.7.0 GA with the typical protocols stack based on TCP (taken from
tcp.xml).
Sometimes it occurs that socket write operation hangs inside TP.send() -- this is not so
untypical for blocking I/O approach! -- after that JGroups is not working at all because
most of threads are blocked inside TP.send() waiting for that first thread releasing
"out_stream_lock". Below is a fragment from my stack trace.
1) the thread that hangs in socket write until forever (maybe due to a broken socket or
something else); this thread owns "out_stream_lock in TP class":
"Timer-1,name,IP:port" daemon prio=10 tid=0x084e5c00 nid=0x15f0 runnable
[0x211c1000..0x211c2040]
java.lang.Thread.State: RUNNABLE
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
- locked <0x43427300> (a java.io.BufferedOutputStream)
at java.io.DataOutputStream.flush(DataOutputStream.java:106)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection.doSend(TCPConnectionMap.java:472)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection._send(TCPConnectionMap.java:453)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection.send(TCPConnectionMap.java:436)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection.access$100(TCPConnectionMap.java:341)
at org.jgroups.blocks.TCPConnectionMap.send(TCPConnectionMap.java:137)
at org.jgroups.protocols.TCP.send(TCP.java:53)
at org.jgroups.protocols.BasicTCP.sendToSingleMember(BasicTCP.java:141)
at org.jgroups.protocols.TP.doSend(TP.java:1105)
at org.jgroups.protocols.TP.send(TP.java:1088)
at org.jgroups.protocols.TP.down(TP.java:907)
at org.jgroups.protocols.Discovery.down(Discovery.java:363)
at org.jgroups.protocols.MERGE2.down(MERGE2.java:169)
at org.jgroups.protocols.FD_SOCK.down(FD_SOCK.java:333)
at org.jgroups.protocols.FD.down(FD.java:327)
at org.jgroups.protocols.VERIFY_SUSPECT.down(VERIFY_SUSPECT.java:72)
at org.jgroups.protocols.BARRIER.down(BARRIER.java:96)
at org.jgroups.protocols.pbcast.NAKACK.retransmit(NAKACK.java:1530)
at org.jgroups.protocols.pbcast.NAKACK.retransmit(NAKACK.java:1476)
at org.jgroups.stack.Retransmitter$Task.run(Retransmitter.java:207)
at org.jgroups.util.TimeScheduler$TaskWrapper.run(TimeScheduler.java:218)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:207)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Locked ownable synchronizers:
- <0x2b5522e0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
- <0x2b5550f8> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
- <0x434273f0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
2) and there are many threads like this blocked "on out_stream_lock.lock()" call
inside TP class:
"Connection.Receiver [ip:port - ip:port],name,ip:port" prio=10 tid=0x020dc000
nid=0x1fab waiting on condition [0x1365f000..0x1365ffc0]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x2b5550f8> (a
java.util.concurrent.locks.ReentrantLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:778)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1114)
at
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186)
at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262)
at org.jgroups.protocols.TP.send(TP.java:1082)
at org.jgroups.protocols.TP.down(TP.java:907)
at org.jgroups.protocols.Discovery.up(Discovery.java:274)
at org.jgroups.protocols.TP.passMessageUp(TP.java:995)
at org.jgroups.protocols.TP.access$100(TP.java:52)
at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1451)
at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1427)
at
java.util.concurrent.ThreadPoolExecutor$CallerRunsPolicy.rejectedExecution(ThreadPoolExecutor.java:1738)
at
org.jgroups.util.ShutdownRejectedExecutionHandler.rejectedExecution(ShutdownRejectedExecutionHandler.java:34)
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
at org.jgroups.protocols.TP.dispatchToThreadPool(TP.java:1061)
at org.jgroups.protocols.TP.receive(TP.java:1038)
at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:180)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection$ConnectionPeerReceiver.run(TCPConnectionMap.java:553)
at java.lang.Thread.run(Thread.java:619)
Locked ownable synchronizers:
- None
(Since all threads from JGroups' ThreadPool are locked, you can see
"rejectedExecution" in the last stack trace).
Maybe we could implement some solution for TCP (blocking I/O)? Maybe it is possible to
release the lock before socket write?
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
5 a.m.
New subject: [JBoss JIRA] Updated: (JGRP-1003) deadlock in TP.send - socket write hangs inside a Lock, preventing other threads to do TP.send()
[
https://jira.jboss.org/jira/browse/JGRP-1003?page=com.atlassian.jira.plug...
]
Bela Ban updated JGRP-1003:
---------------------------
Fix Version/s: 2.8
deadlock in TP.send - socket write hangs inside a Lock, preventing
other threads to do TP.send()
------------------------------------------------------------------------------------------------
Key: JGRP-1003
URL: https://jira.jboss.org/jira/browse/JGRP-1003
Project: JGroups
Issue Type: Bug
Affects Versions: 2.7
Environment: Linux (Debian), kernel 2.6.x
Reporter: Victor N
Assignee: Bela Ban
Fix For: 2.8
I am using JGroups 2.7.0 GA with the typical protocols stack based on TCP (taken from
tcp.xml).
Sometimes it occurs that socket write operation hangs inside TP.send() -- this is not so
untypical for blocking I/O approach! -- after that JGroups is not working at all because
most of threads are blocked inside TP.send() waiting for that first thread releasing
"out_stream_lock". Below is a fragment from my stack trace.
1) the thread that hangs in socket write until forever (maybe due to a broken socket or
something else); this thread owns "out_stream_lock in TP class":
"Timer-1,name,IP:port" daemon prio=10 tid=0x084e5c00 nid=0x15f0 runnable
[0x211c1000..0x211c2040]
java.lang.Thread.State: RUNNABLE
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
- locked <0x43427300> (a java.io.BufferedOutputStream)
at java.io.DataOutputStream.flush(DataOutputStream.java:106)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection.doSend(TCPConnectionMap.java:472)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection._send(TCPConnectionMap.java:453)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection.send(TCPConnectionMap.java:436)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection.access$100(TCPConnectionMap.java:341)
at org.jgroups.blocks.TCPConnectionMap.send(TCPConnectionMap.java:137)
at org.jgroups.protocols.TCP.send(TCP.java:53)
at org.jgroups.protocols.BasicTCP.sendToSingleMember(BasicTCP.java:141)
at org.jgroups.protocols.TP.doSend(TP.java:1105)
at org.jgroups.protocols.TP.send(TP.java:1088)
at org.jgroups.protocols.TP.down(TP.java:907)
at org.jgroups.protocols.Discovery.down(Discovery.java:363)
at org.jgroups.protocols.MERGE2.down(MERGE2.java:169)
at org.jgroups.protocols.FD_SOCK.down(FD_SOCK.java:333)
at org.jgroups.protocols.FD.down(FD.java:327)
at org.jgroups.protocols.VERIFY_SUSPECT.down(VERIFY_SUSPECT.java:72)
at org.jgroups.protocols.BARRIER.down(BARRIER.java:96)
at org.jgroups.protocols.pbcast.NAKACK.retransmit(NAKACK.java:1530)
at org.jgroups.protocols.pbcast.NAKACK.retransmit(NAKACK.java:1476)
at org.jgroups.stack.Retransmitter$Task.run(Retransmitter.java:207)
at org.jgroups.util.TimeScheduler$TaskWrapper.run(TimeScheduler.java:218)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:207)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Locked ownable synchronizers:
- <0x2b5522e0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
- <0x2b5550f8> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
- <0x434273f0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
2) and there are many threads like this blocked "on out_stream_lock.lock()"
call inside TP class:
"Connection.Receiver [ip:port - ip:port],name,ip:port" prio=10 tid=0x020dc000
nid=0x1fab waiting on condition [0x1365f000..0x1365ffc0]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x2b5550f8> (a
java.util.concurrent.locks.ReentrantLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:778)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1114)
at
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186)
at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262)
at org.jgroups.protocols.TP.send(TP.java:1082)
at org.jgroups.protocols.TP.down(TP.java:907)
at org.jgroups.protocols.Discovery.up(Discovery.java:274)
at org.jgroups.protocols.TP.passMessageUp(TP.java:995)
at org.jgroups.protocols.TP.access$100(TP.java:52)
at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1451)
at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1427)
at
java.util.concurrent.ThreadPoolExecutor$CallerRunsPolicy.rejectedExecution(ThreadPoolExecutor.java:1738)
at
org.jgroups.util.ShutdownRejectedExecutionHandler.rejectedExecution(ShutdownRejectedExecutionHandler.java:34)
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
at org.jgroups.protocols.TP.dispatchToThreadPool(TP.java:1061)
at org.jgroups.protocols.TP.receive(TP.java:1038)
at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:180)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection$ConnectionPeerReceiver.run(TCPConnectionMap.java:553)
at java.lang.Thread.run(Thread.java:619)
Locked ownable synchronizers:
- None
(Since all threads from JGroups' ThreadPool are locked, you can see
"rejectedExecution" in the last stack trace).
Maybe we could implement some solution for TCP (blocking I/O)? Maybe it is possible to
release the lock before socket write?
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
5:09 a.m.
New subject: [JBoss JIRA] Commented: (JGRP-1003) deadlock in TP.send - socket write hangs inside a Lock, preventing other threads to do TP.send()
[
https://jira.jboss.org/jira/browse/JGRP-1003?page=com.atlassian.jira.plug...
]
Bela Ban commented on JGRP-1003:
--------------------------------
Note that we don't support 2.7, I suggest switch to either 2.6.x or 2.8
deadlock in TP.send - socket write hangs inside a Lock, preventing
other threads to do TP.send()
------------------------------------------------------------------------------------------------
Key: JGRP-1003
URL: https://jira.jboss.org/jira/browse/JGRP-1003
Project: JGroups
Issue Type: Bug
Affects Versions: 2.7
Environment: Linux (Debian), kernel 2.6.x
Reporter: Victor N
Assignee: Bela Ban
Fix For: 2.8
I am using JGroups 2.7.0 GA with the typical protocols stack based on TCP (taken from
tcp.xml).
Sometimes it occurs that socket write operation hangs inside TP.send() -- this is not so
untypical for blocking I/O approach! -- after that JGroups is not working at all because
most of threads are blocked inside TP.send() waiting for that first thread releasing
"out_stream_lock". Below is a fragment from my stack trace.
1) the thread that hangs in socket write until forever (maybe due to a broken socket or
something else); this thread owns "out_stream_lock in TP class":
"Timer-1,name,IP:port" daemon prio=10 tid=0x084e5c00 nid=0x15f0 runnable
[0x211c1000..0x211c2040]
java.lang.Thread.State: RUNNABLE
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
- locked <0x43427300> (a java.io.BufferedOutputStream)
at java.io.DataOutputStream.flush(DataOutputStream.java:106)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection.doSend(TCPConnectionMap.java:472)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection._send(TCPConnectionMap.java:453)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection.send(TCPConnectionMap.java:436)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection.access$100(TCPConnectionMap.java:341)
at org.jgroups.blocks.TCPConnectionMap.send(TCPConnectionMap.java:137)
at org.jgroups.protocols.TCP.send(TCP.java:53)
at org.jgroups.protocols.BasicTCP.sendToSingleMember(BasicTCP.java:141)
at org.jgroups.protocols.TP.doSend(TP.java:1105)
at org.jgroups.protocols.TP.send(TP.java:1088)
at org.jgroups.protocols.TP.down(TP.java:907)
at org.jgroups.protocols.Discovery.down(Discovery.java:363)
at org.jgroups.protocols.MERGE2.down(MERGE2.java:169)
at org.jgroups.protocols.FD_SOCK.down(FD_SOCK.java:333)
at org.jgroups.protocols.FD.down(FD.java:327)
at org.jgroups.protocols.VERIFY_SUSPECT.down(VERIFY_SUSPECT.java:72)
at org.jgroups.protocols.BARRIER.down(BARRIER.java:96)
at org.jgroups.protocols.pbcast.NAKACK.retransmit(NAKACK.java:1530)
at org.jgroups.protocols.pbcast.NAKACK.retransmit(NAKACK.java:1476)
at org.jgroups.stack.Retransmitter$Task.run(Retransmitter.java:207)
at org.jgroups.util.TimeScheduler$TaskWrapper.run(TimeScheduler.java:218)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:207)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Locked ownable synchronizers:
- <0x2b5522e0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
- <0x2b5550f8> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
- <0x434273f0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
2) and there are many threads like this blocked "on out_stream_lock.lock()"
call inside TP class:
"Connection.Receiver [ip:port - ip:port],name,ip:port" prio=10 tid=0x020dc000
nid=0x1fab waiting on condition [0x1365f000..0x1365ffc0]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x2b5550f8> (a
java.util.concurrent.locks.ReentrantLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:778)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1114)
at
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186)
at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262)
at org.jgroups.protocols.TP.send(TP.java:1082)
at org.jgroups.protocols.TP.down(TP.java:907)
at org.jgroups.protocols.Discovery.up(Discovery.java:274)
at org.jgroups.protocols.TP.passMessageUp(TP.java:995)
at org.jgroups.protocols.TP.access$100(TP.java:52)
at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1451)
at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1427)
at
java.util.concurrent.ThreadPoolExecutor$CallerRunsPolicy.rejectedExecution(ThreadPoolExecutor.java:1738)
at
org.jgroups.util.ShutdownRejectedExecutionHandler.rejectedExecution(ShutdownRejectedExecutionHandler.java:34)
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
at org.jgroups.protocols.TP.dispatchToThreadPool(TP.java:1061)
at org.jgroups.protocols.TP.receive(TP.java:1038)
at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:180)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection$ConnectionPeerReceiver.run(TCPConnectionMap.java:553)
at java.lang.Thread.run(Thread.java:619)
Locked ownable synchronizers:
- None
(Since all threads from JGroups' ThreadPool are locked, you can see
"rejectedExecution" in the last stack trace).
Maybe we could implement some solution for TCP (blocking I/O)? Maybe it is possible to
release the lock before socket write?
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
5:09 a.m.
New subject: [JBoss JIRA] Commented: (JGRP-1003) deadlock in TP.send - socket write hangs inside a Lock, preventing other threads to do TP.send()
[
https://jira.jboss.org/jira/browse/JGRP-1003?page=com.atlassian.jira.plug...
]
Bela Ban commented on JGRP-1003:
--------------------------------
The question is why does it hang in write(). The reason is that the peer doesn't call
Socket.receive(), but we need to find out why.
A few things to try out:
- Use bundling: enable_bundling="true"
- Increase the socket buffer size: recv_buf_size / send_buf_size. The former is more
important
- Use send queues in TCP: use_send_queues="true"
- Change the transport's thread pool config, a good one would be
thread_pool.enabled="true"
thread_pool.min_threads="2"
thread_pool.max_threads="8"
thread_pool.keep_alive_time="5000"
thread_pool.queue_enabled="true"
thread_pool.queue_max_size="10000"
thread_pool.rejection_policy="discard"
deadlock in TP.send - socket write hangs inside a Lock, preventing
other threads to do TP.send()
------------------------------------------------------------------------------------------------
Key: JGRP-1003
URL: https://jira.jboss.org/jira/browse/JGRP-1003
Project: JGroups
Issue Type: Bug
Affects Versions: 2.7
Environment: Linux (Debian), kernel 2.6.x
Reporter: Victor N
Assignee: Bela Ban
Fix For: 2.8
I am using JGroups 2.7.0 GA with the typical protocols stack based on TCP (taken from
tcp.xml).
Sometimes it occurs that socket write operation hangs inside TP.send() -- this is not so
untypical for blocking I/O approach! -- after that JGroups is not working at all because
most of threads are blocked inside TP.send() waiting for that first thread releasing
"out_stream_lock". Below is a fragment from my stack trace.
1) the thread that hangs in socket write until forever (maybe due to a broken socket or
something else); this thread owns "out_stream_lock in TP class":
"Timer-1,name,IP:port" daemon prio=10 tid=0x084e5c00 nid=0x15f0 runnable
[0x211c1000..0x211c2040]
java.lang.Thread.State: RUNNABLE
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
- locked <0x43427300> (a java.io.BufferedOutputStream)
at java.io.DataOutputStream.flush(DataOutputStream.java:106)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection.doSend(TCPConnectionMap.java:472)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection._send(TCPConnectionMap.java:453)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection.send(TCPConnectionMap.java:436)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection.access$100(TCPConnectionMap.java:341)
at org.jgroups.blocks.TCPConnectionMap.send(TCPConnectionMap.java:137)
at org.jgroups.protocols.TCP.send(TCP.java:53)
at org.jgroups.protocols.BasicTCP.sendToSingleMember(BasicTCP.java:141)
at org.jgroups.protocols.TP.doSend(TP.java:1105)
at org.jgroups.protocols.TP.send(TP.java:1088)
at org.jgroups.protocols.TP.down(TP.java:907)
at org.jgroups.protocols.Discovery.down(Discovery.java:363)
at org.jgroups.protocols.MERGE2.down(MERGE2.java:169)
at org.jgroups.protocols.FD_SOCK.down(FD_SOCK.java:333)
at org.jgroups.protocols.FD.down(FD.java:327)
at org.jgroups.protocols.VERIFY_SUSPECT.down(VERIFY_SUSPECT.java:72)
at org.jgroups.protocols.BARRIER.down(BARRIER.java:96)
at org.jgroups.protocols.pbcast.NAKACK.retransmit(NAKACK.java:1530)
at org.jgroups.protocols.pbcast.NAKACK.retransmit(NAKACK.java:1476)
at org.jgroups.stack.Retransmitter$Task.run(Retransmitter.java:207)
at org.jgroups.util.TimeScheduler$TaskWrapper.run(TimeScheduler.java:218)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:207)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Locked ownable synchronizers:
- <0x2b5522e0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
- <0x2b5550f8> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
- <0x434273f0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
2) and there are many threads like this blocked "on out_stream_lock.lock()"
call inside TP class:
"Connection.Receiver [ip:port - ip:port],name,ip:port" prio=10 tid=0x020dc000
nid=0x1fab waiting on condition [0x1365f000..0x1365ffc0]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x2b5550f8> (a
java.util.concurrent.locks.ReentrantLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:778)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1114)
at
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186)
at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262)
at org.jgroups.protocols.TP.send(TP.java:1082)
at org.jgroups.protocols.TP.down(TP.java:907)
at org.jgroups.protocols.Discovery.up(Discovery.java:274)
at org.jgroups.protocols.TP.passMessageUp(TP.java:995)
at org.jgroups.protocols.TP.access$100(TP.java:52)
at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1451)
at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1427)
at
java.util.concurrent.ThreadPoolExecutor$CallerRunsPolicy.rejectedExecution(ThreadPoolExecutor.java:1738)
at
org.jgroups.util.ShutdownRejectedExecutionHandler.rejectedExecution(ShutdownRejectedExecutionHandler.java:34)
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
at org.jgroups.protocols.TP.dispatchToThreadPool(TP.java:1061)
at org.jgroups.protocols.TP.receive(TP.java:1038)
at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:180)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection$ConnectionPeerReceiver.run(TCPConnectionMap.java:553)
at java.lang.Thread.run(Thread.java:619)
Locked ownable synchronizers:
- None
(Since all threads from JGroups' ThreadPool are locked, you can see
"rejectedExecution" in the last stack trace).
Maybe we could implement some solution for TCP (blocking I/O)? Maybe it is possible to
release the lock before socket write?
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
5:39 a.m.
New subject: [JBoss JIRA] Commented: (JGRP-1003) deadlock in TP.send - socket write hangs inside a Lock, preventing other threads to do TP.send()
[
https://jira.jboss.org/jira/browse/JGRP-1003?page=com.atlassian.jira.plug...
]
Victor N commented on JGRP-1003:
--------------------------------
Yes, Bela,I will try with 2.8 and I am going to use TCP_NIO, this should be more safe.
As I know, there is no way to prevent socket write from hanging in blocking I/O approach
in Java, because most of operating system does not allow to set a socket write timeout.
There is SO_TIMEOUT which means socket read timeout, but there is not timeout for writing.
So I suppose that in blocking I/O it is not safe to do socket write inside a lock.
I do not think that socket buffer sizes can help. Also, thread pool settings may be
different, but if some thread owns an exclusive lock forever, this will not help.
I will look into 2.8 code to see how enable_bundling and use_send_queues changes
behavior.
deadlock in TP.send - socket write hangs inside a Lock, preventing
other threads to do TP.send()
------------------------------------------------------------------------------------------------
Key: JGRP-1003
URL: https://jira.jboss.org/jira/browse/JGRP-1003
Project: JGroups
Issue Type: Bug
Affects Versions: 2.7
Environment: Linux (Debian), kernel 2.6.x
Reporter: Victor N
Assignee: Bela Ban
Fix For: 2.8
I am using JGroups 2.7.0 GA with the typical protocols stack based on TCP (taken from
tcp.xml).
Sometimes it occurs that socket write operation hangs inside TP.send() -- this is not so
untypical for blocking I/O approach! -- after that JGroups is not working at all because
most of threads are blocked inside TP.send() waiting for that first thread releasing
"out_stream_lock". Below is a fragment from my stack trace.
1) the thread that hangs in socket write until forever (maybe due to a broken socket or
something else); this thread owns "out_stream_lock in TP class":
"Timer-1,name,IP:port" daemon prio=10 tid=0x084e5c00 nid=0x15f0 runnable
[0x211c1000..0x211c2040]
java.lang.Thread.State: RUNNABLE
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
- locked <0x43427300> (a java.io.BufferedOutputStream)
at java.io.DataOutputStream.flush(DataOutputStream.java:106)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection.doSend(TCPConnectionMap.java:472)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection._send(TCPConnectionMap.java:453)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection.send(TCPConnectionMap.java:436)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection.access$100(TCPConnectionMap.java:341)
at org.jgroups.blocks.TCPConnectionMap.send(TCPConnectionMap.java:137)
at org.jgroups.protocols.TCP.send(TCP.java:53)
at org.jgroups.protocols.BasicTCP.sendToSingleMember(BasicTCP.java:141)
at org.jgroups.protocols.TP.doSend(TP.java:1105)
at org.jgroups.protocols.TP.send(TP.java:1088)
at org.jgroups.protocols.TP.down(TP.java:907)
at org.jgroups.protocols.Discovery.down(Discovery.java:363)
at org.jgroups.protocols.MERGE2.down(MERGE2.java:169)
at org.jgroups.protocols.FD_SOCK.down(FD_SOCK.java:333)
at org.jgroups.protocols.FD.down(FD.java:327)
at org.jgroups.protocols.VERIFY_SUSPECT.down(VERIFY_SUSPECT.java:72)
at org.jgroups.protocols.BARRIER.down(BARRIER.java:96)
at org.jgroups.protocols.pbcast.NAKACK.retransmit(NAKACK.java:1530)
at org.jgroups.protocols.pbcast.NAKACK.retransmit(NAKACK.java:1476)
at org.jgroups.stack.Retransmitter$Task.run(Retransmitter.java:207)
at org.jgroups.util.TimeScheduler$TaskWrapper.run(TimeScheduler.java:218)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:207)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Locked ownable synchronizers:
- <0x2b5522e0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
- <0x2b5550f8> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
- <0x434273f0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
2) and there are many threads like this blocked "on out_stream_lock.lock()"
call inside TP class:
"Connection.Receiver [ip:port - ip:port],name,ip:port" prio=10 tid=0x020dc000
nid=0x1fab waiting on condition [0x1365f000..0x1365ffc0]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x2b5550f8> (a
java.util.concurrent.locks.ReentrantLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:778)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1114)
at
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186)
at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262)
at org.jgroups.protocols.TP.send(TP.java:1082)
at org.jgroups.protocols.TP.down(TP.java:907)
at org.jgroups.protocols.Discovery.up(Discovery.java:274)
at org.jgroups.protocols.TP.passMessageUp(TP.java:995)
at org.jgroups.protocols.TP.access$100(TP.java:52)
at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1451)
at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1427)
at
java.util.concurrent.ThreadPoolExecutor$CallerRunsPolicy.rejectedExecution(ThreadPoolExecutor.java:1738)
at
org.jgroups.util.ShutdownRejectedExecutionHandler.rejectedExecution(ShutdownRejectedExecutionHandler.java:34)
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
at org.jgroups.protocols.TP.dispatchToThreadPool(TP.java:1061)
at org.jgroups.protocols.TP.receive(TP.java:1038)
at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:180)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection$ConnectionPeerReceiver.run(TCPConnectionMap.java:553)
at java.lang.Thread.run(Thread.java:619)
Locked ownable synchronizers:
- None
(Since all threads from JGroups' ThreadPool are locked, you can see
"rejectedExecution" in the last stack trace).
Maybe we could implement some solution for TCP (blocking I/O)? Maybe it is possible to
release the lock before socket write?
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
5:42 a.m.
New subject: [JBoss JIRA] Commented: (JGRP-1003) deadlock in TP.send - socket write hangs inside a Lock, preventing other threads to do TP.send()
[
https://jira.jboss.org/jira/browse/JGRP-1003?page=com.atlassian.jira.plug...
]
Victor N commented on JGRP-1003:
--------------------------------
I can see that 2.8 uses the same code in TP.send:
out_stream_lock.lock();
try {
out_stream.reset();
dos.reset();
writeMessage(msg, dos, multicast);
Buffer buf=new Buffer(out_stream.getRawBuffer(), 0, out_stream.size());
doSend(buf, dest, multicast);
}
finally {
out_stream_lock.unlock();
}
So socket write is done inside a lock, and the behavior does not depend on enable_bundling
and use_send_queues/
deadlock in TP.send - socket write hangs inside a Lock, preventing
other threads to do TP.send()
------------------------------------------------------------------------------------------------
Key: JGRP-1003
URL: https://jira.jboss.org/jira/browse/JGRP-1003
Project: JGroups
Issue Type: Bug
Affects Versions: 2.7
Environment: Linux (Debian), kernel 2.6.x
Reporter: Victor N
Assignee: Bela Ban
Fix For: 2.8
I am using JGroups 2.7.0 GA with the typical protocols stack based on TCP (taken from
tcp.xml).
Sometimes it occurs that socket write operation hangs inside TP.send() -- this is not so
untypical for blocking I/O approach! -- after that JGroups is not working at all because
most of threads are blocked inside TP.send() waiting for that first thread releasing
"out_stream_lock". Below is a fragment from my stack trace.
1) the thread that hangs in socket write until forever (maybe due to a broken socket or
something else); this thread owns "out_stream_lock in TP class":
"Timer-1,name,IP:port" daemon prio=10 tid=0x084e5c00 nid=0x15f0 runnable
[0x211c1000..0x211c2040]
java.lang.Thread.State: RUNNABLE
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
- locked <0x43427300> (a java.io.BufferedOutputStream)
at java.io.DataOutputStream.flush(DataOutputStream.java:106)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection.doSend(TCPConnectionMap.java:472)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection._send(TCPConnectionMap.java:453)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection.send(TCPConnectionMap.java:436)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection.access$100(TCPConnectionMap.java:341)
at org.jgroups.blocks.TCPConnectionMap.send(TCPConnectionMap.java:137)
at org.jgroups.protocols.TCP.send(TCP.java:53)
at org.jgroups.protocols.BasicTCP.sendToSingleMember(BasicTCP.java:141)
at org.jgroups.protocols.TP.doSend(TP.java:1105)
at org.jgroups.protocols.TP.send(TP.java:1088)
at org.jgroups.protocols.TP.down(TP.java:907)
at org.jgroups.protocols.Discovery.down(Discovery.java:363)
at org.jgroups.protocols.MERGE2.down(MERGE2.java:169)
at org.jgroups.protocols.FD_SOCK.down(FD_SOCK.java:333)
at org.jgroups.protocols.FD.down(FD.java:327)
at org.jgroups.protocols.VERIFY_SUSPECT.down(VERIFY_SUSPECT.java:72)
at org.jgroups.protocols.BARRIER.down(BARRIER.java:96)
at org.jgroups.protocols.pbcast.NAKACK.retransmit(NAKACK.java:1530)
at org.jgroups.protocols.pbcast.NAKACK.retransmit(NAKACK.java:1476)
at org.jgroups.stack.Retransmitter$Task.run(Retransmitter.java:207)
at org.jgroups.util.TimeScheduler$TaskWrapper.run(TimeScheduler.java:218)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:207)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Locked ownable synchronizers:
- <0x2b5522e0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
- <0x2b5550f8> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
- <0x434273f0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
2) and there are many threads like this blocked "on out_stream_lock.lock()"
call inside TP class:
"Connection.Receiver [ip:port - ip:port],name,ip:port" prio=10 tid=0x020dc000
nid=0x1fab waiting on condition [0x1365f000..0x1365ffc0]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x2b5550f8> (a
java.util.concurrent.locks.ReentrantLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:778)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1114)
at
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186)
at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262)
at org.jgroups.protocols.TP.send(TP.java:1082)
at org.jgroups.protocols.TP.down(TP.java:907)
at org.jgroups.protocols.Discovery.up(Discovery.java:274)
at org.jgroups.protocols.TP.passMessageUp(TP.java:995)
at org.jgroups.protocols.TP.access$100(TP.java:52)
at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1451)
at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1427)
at
java.util.concurrent.ThreadPoolExecutor$CallerRunsPolicy.rejectedExecution(ThreadPoolExecutor.java:1738)
at
org.jgroups.util.ShutdownRejectedExecutionHandler.rejectedExecution(ShutdownRejectedExecutionHandler.java:34)
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
at org.jgroups.protocols.TP.dispatchToThreadPool(TP.java:1061)
at org.jgroups.protocols.TP.receive(TP.java:1038)
at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:180)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection$ConnectionPeerReceiver.run(TCPConnectionMap.java:553)
at java.lang.Thread.run(Thread.java:619)
Locked ownable synchronizers:
- None
(Since all threads from JGroups' ThreadPool are locked, you can see
"rejectedExecution" in the last stack trace).
Maybe we could implement some solution for TCP (blocking I/O)? Maybe it is possible to
release the lock before socket write?
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
6:51 a.m.
New subject: [JBoss JIRA] Commented: (JGRP-1003) deadlock in TP.send - socket write hangs inside a Lock, preventing other threads to do TP.send()
[
https://jira.jboss.org/jira/browse/JGRP-1003?page=com.atlassian.jira.plug...
]
Bela Ban commented on JGRP-1003:
--------------------------------
use_send_queues will *copy* the buffer and add it to a queue, then return immediately and
release the lock. A separate thread will then send (and possibly block). This should
help.
deadlock in TP.send - socket write hangs inside a Lock, preventing
other threads to do TP.send()
------------------------------------------------------------------------------------------------
Key: JGRP-1003
URL: https://jira.jboss.org/jira/browse/JGRP-1003
Project: JGroups
Issue Type: Bug
Affects Versions: 2.7
Environment: Linux (Debian), kernel 2.6.x
Reporter: Victor N
Assignee: Bela Ban
Fix For: 2.8
I am using JGroups 2.7.0 GA with the typical protocols stack based on TCP (taken from
tcp.xml).
Sometimes it occurs that socket write operation hangs inside TP.send() -- this is not so
untypical for blocking I/O approach! -- after that JGroups is not working at all because
most of threads are blocked inside TP.send() waiting for that first thread releasing
"out_stream_lock". Below is a fragment from my stack trace.
1) the thread that hangs in socket write until forever (maybe due to a broken socket or
something else); this thread owns "out_stream_lock in TP class":
"Timer-1,name,IP:port" daemon prio=10 tid=0x084e5c00 nid=0x15f0 runnable
[0x211c1000..0x211c2040]
java.lang.Thread.State: RUNNABLE
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
- locked <0x43427300> (a java.io.BufferedOutputStream)
at java.io.DataOutputStream.flush(DataOutputStream.java:106)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection.doSend(TCPConnectionMap.java:472)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection._send(TCPConnectionMap.java:453)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection.send(TCPConnectionMap.java:436)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection.access$100(TCPConnectionMap.java:341)
at org.jgroups.blocks.TCPConnectionMap.send(TCPConnectionMap.java:137)
at org.jgroups.protocols.TCP.send(TCP.java:53)
at org.jgroups.protocols.BasicTCP.sendToSingleMember(BasicTCP.java:141)
at org.jgroups.protocols.TP.doSend(TP.java:1105)
at org.jgroups.protocols.TP.send(TP.java:1088)
at org.jgroups.protocols.TP.down(TP.java:907)
at org.jgroups.protocols.Discovery.down(Discovery.java:363)
at org.jgroups.protocols.MERGE2.down(MERGE2.java:169)
at org.jgroups.protocols.FD_SOCK.down(FD_SOCK.java:333)
at org.jgroups.protocols.FD.down(FD.java:327)
at org.jgroups.protocols.VERIFY_SUSPECT.down(VERIFY_SUSPECT.java:72)
at org.jgroups.protocols.BARRIER.down(BARRIER.java:96)
at org.jgroups.protocols.pbcast.NAKACK.retransmit(NAKACK.java:1530)
at org.jgroups.protocols.pbcast.NAKACK.retransmit(NAKACK.java:1476)
at org.jgroups.stack.Retransmitter$Task.run(Retransmitter.java:207)
at org.jgroups.util.TimeScheduler$TaskWrapper.run(TimeScheduler.java:218)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:207)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Locked ownable synchronizers:
- <0x2b5522e0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
- <0x2b5550f8> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
- <0x434273f0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
2) and there are many threads like this blocked "on out_stream_lock.lock()"
call inside TP class:
"Connection.Receiver [ip:port - ip:port],name,ip:port" prio=10 tid=0x020dc000
nid=0x1fab waiting on condition [0x1365f000..0x1365ffc0]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x2b5550f8> (a
java.util.concurrent.locks.ReentrantLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:778)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1114)
at
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186)
at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262)
at org.jgroups.protocols.TP.send(TP.java:1082)
at org.jgroups.protocols.TP.down(TP.java:907)
at org.jgroups.protocols.Discovery.up(Discovery.java:274)
at org.jgroups.protocols.TP.passMessageUp(TP.java:995)
at org.jgroups.protocols.TP.access$100(TP.java:52)
at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1451)
at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1427)
at
java.util.concurrent.ThreadPoolExecutor$CallerRunsPolicy.rejectedExecution(ThreadPoolExecutor.java:1738)
at
org.jgroups.util.ShutdownRejectedExecutionHandler.rejectedExecution(ShutdownRejectedExecutionHandler.java:34)
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
at org.jgroups.protocols.TP.dispatchToThreadPool(TP.java:1061)
at org.jgroups.protocols.TP.receive(TP.java:1038)
at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:180)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection$ConnectionPeerReceiver.run(TCPConnectionMap.java:553)
at java.lang.Thread.run(Thread.java:619)
Locked ownable synchronizers:
- None
(Since all threads from JGroups' ThreadPool are locked, you can see
"rejectedExecution" in the last stack trace).
Maybe we could implement some solution for TCP (blocking I/O)? Maybe it is possible to
release the lock before socket write?
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
7:27 a.m.
New subject: [JBoss JIRA] Commented: (JGRP-1003) deadlock in TP.send - socket write hangs inside a Lock, preventing other threads to do TP.send()
[
https://jira.jboss.org/jira/browse/JGRP-1003?page=com.atlassian.jira.plug...
]
Victor N commented on JGRP-1003:
--------------------------------
Thanks, I see this in the code now!
use_send_queues should help.
deadlock in TP.send - socket write hangs inside a Lock, preventing
other threads to do TP.send()
------------------------------------------------------------------------------------------------
Key: JGRP-1003
URL: https://jira.jboss.org/jira/browse/JGRP-1003
Project: JGroups
Issue Type: Bug
Affects Versions: 2.7
Environment: Linux (Debian), kernel 2.6.x
Reporter: Victor N
Assignee: Bela Ban
Fix For: 2.8
I am using JGroups 2.7.0 GA with the typical protocols stack based on TCP (taken from
tcp.xml).
Sometimes it occurs that socket write operation hangs inside TP.send() -- this is not so
untypical for blocking I/O approach! -- after that JGroups is not working at all because
most of threads are blocked inside TP.send() waiting for that first thread releasing
"out_stream_lock". Below is a fragment from my stack trace.
1) the thread that hangs in socket write until forever (maybe due to a broken socket or
something else); this thread owns "out_stream_lock in TP class":
"Timer-1,name,IP:port" daemon prio=10 tid=0x084e5c00 nid=0x15f0 runnable
[0x211c1000..0x211c2040]
java.lang.Thread.State: RUNNABLE
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
- locked <0x43427300> (a java.io.BufferedOutputStream)
at java.io.DataOutputStream.flush(DataOutputStream.java:106)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection.doSend(TCPConnectionMap.java:472)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection._send(TCPConnectionMap.java:453)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection.send(TCPConnectionMap.java:436)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection.access$100(TCPConnectionMap.java:341)
at org.jgroups.blocks.TCPConnectionMap.send(TCPConnectionMap.java:137)
at org.jgroups.protocols.TCP.send(TCP.java:53)
at org.jgroups.protocols.BasicTCP.sendToSingleMember(BasicTCP.java:141)
at org.jgroups.protocols.TP.doSend(TP.java:1105)
at org.jgroups.protocols.TP.send(TP.java:1088)
at org.jgroups.protocols.TP.down(TP.java:907)
at org.jgroups.protocols.Discovery.down(Discovery.java:363)
at org.jgroups.protocols.MERGE2.down(MERGE2.java:169)
at org.jgroups.protocols.FD_SOCK.down(FD_SOCK.java:333)
at org.jgroups.protocols.FD.down(FD.java:327)
at org.jgroups.protocols.VERIFY_SUSPECT.down(VERIFY_SUSPECT.java:72)
at org.jgroups.protocols.BARRIER.down(BARRIER.java:96)
at org.jgroups.protocols.pbcast.NAKACK.retransmit(NAKACK.java:1530)
at org.jgroups.protocols.pbcast.NAKACK.retransmit(NAKACK.java:1476)
at org.jgroups.stack.Retransmitter$Task.run(Retransmitter.java:207)
at org.jgroups.util.TimeScheduler$TaskWrapper.run(TimeScheduler.java:218)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:207)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Locked ownable synchronizers:
- <0x2b5522e0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
- <0x2b5550f8> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
- <0x434273f0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
2) and there are many threads like this blocked "on out_stream_lock.lock()"
call inside TP class:
"Connection.Receiver [ip:port - ip:port],name,ip:port" prio=10 tid=0x020dc000
nid=0x1fab waiting on condition [0x1365f000..0x1365ffc0]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x2b5550f8> (a
java.util.concurrent.locks.ReentrantLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:778)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1114)
at
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186)
at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262)
at org.jgroups.protocols.TP.send(TP.java:1082)
at org.jgroups.protocols.TP.down(TP.java:907)
at org.jgroups.protocols.Discovery.up(Discovery.java:274)
at org.jgroups.protocols.TP.passMessageUp(TP.java:995)
at org.jgroups.protocols.TP.access$100(TP.java:52)
at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1451)
at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1427)
at
java.util.concurrent.ThreadPoolExecutor$CallerRunsPolicy.rejectedExecution(ThreadPoolExecutor.java:1738)
at
org.jgroups.util.ShutdownRejectedExecutionHandler.rejectedExecution(ShutdownRejectedExecutionHandler.java:34)
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
at org.jgroups.protocols.TP.dispatchToThreadPool(TP.java:1061)
at org.jgroups.protocols.TP.receive(TP.java:1038)
at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:180)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection$ConnectionPeerReceiver.run(TCPConnectionMap.java:553)
at java.lang.Thread.run(Thread.java:619)
Locked ownable synchronizers:
- None
(Since all threads from JGroups' ThreadPool are locked, you can see
"rejectedExecution" in the last stack trace).
Maybe we could implement some solution for TCP (blocking I/O)? Maybe it is possible to
release the lock before socket write?
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
10:39 a.m.
New subject: [JBoss JIRA] Commented: (JGRP-1003) deadlock in TP.send - socket write hangs inside a Lock, preventing other threads to do TP.send()
[
https://jira.jboss.org/jira/browse/JGRP-1003?page=com.atlassian.jira.plug...
]
Bela Ban commented on JGRP-1003:
--------------------------------
Let me know if this fixes the issue, so I can close it. I plan on rewriting a TCP
transport once we're on JDK 1.7 anyway, to run on NIO, so I don't want to make too
many changes to TCP as it stands for now
deadlock in TP.send - socket write hangs inside a Lock, preventing
other threads to do TP.send()
------------------------------------------------------------------------------------------------
Key: JGRP-1003
URL: https://jira.jboss.org/jira/browse/JGRP-1003
Project: JGroups
Issue Type: Bug
Affects Versions: 2.7
Environment: Linux (Debian), kernel 2.6.x
Reporter: Victor N
Assignee: Bela Ban
Fix For: 2.8
I am using JGroups 2.7.0 GA with the typical protocols stack based on TCP (taken from
tcp.xml).
Sometimes it occurs that socket write operation hangs inside TP.send() -- this is not so
untypical for blocking I/O approach! -- after that JGroups is not working at all because
most of threads are blocked inside TP.send() waiting for that first thread releasing
"out_stream_lock". Below is a fragment from my stack trace.
1) the thread that hangs in socket write until forever (maybe due to a broken socket or
something else); this thread owns "out_stream_lock in TP class":
"Timer-1,name,IP:port" daemon prio=10 tid=0x084e5c00 nid=0x15f0 runnable
[0x211c1000..0x211c2040]
java.lang.Thread.State: RUNNABLE
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
- locked <0x43427300> (a java.io.BufferedOutputStream)
at java.io.DataOutputStream.flush(DataOutputStream.java:106)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection.doSend(TCPConnectionMap.java:472)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection._send(TCPConnectionMap.java:453)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection.send(TCPConnectionMap.java:436)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection.access$100(TCPConnectionMap.java:341)
at org.jgroups.blocks.TCPConnectionMap.send(TCPConnectionMap.java:137)
at org.jgroups.protocols.TCP.send(TCP.java:53)
at org.jgroups.protocols.BasicTCP.sendToSingleMember(BasicTCP.java:141)
at org.jgroups.protocols.TP.doSend(TP.java:1105)
at org.jgroups.protocols.TP.send(TP.java:1088)
at org.jgroups.protocols.TP.down(TP.java:907)
at org.jgroups.protocols.Discovery.down(Discovery.java:363)
at org.jgroups.protocols.MERGE2.down(MERGE2.java:169)
at org.jgroups.protocols.FD_SOCK.down(FD_SOCK.java:333)
at org.jgroups.protocols.FD.down(FD.java:327)
at org.jgroups.protocols.VERIFY_SUSPECT.down(VERIFY_SUSPECT.java:72)
at org.jgroups.protocols.BARRIER.down(BARRIER.java:96)
at org.jgroups.protocols.pbcast.NAKACK.retransmit(NAKACK.java:1530)
at org.jgroups.protocols.pbcast.NAKACK.retransmit(NAKACK.java:1476)
at org.jgroups.stack.Retransmitter$Task.run(Retransmitter.java:207)
at org.jgroups.util.TimeScheduler$TaskWrapper.run(TimeScheduler.java:218)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:207)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Locked ownable synchronizers:
- <0x2b5522e0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
- <0x2b5550f8> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
- <0x434273f0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
2) and there are many threads like this blocked "on out_stream_lock.lock()"
call inside TP class:
"Connection.Receiver [ip:port - ip:port],name,ip:port" prio=10 tid=0x020dc000
nid=0x1fab waiting on condition [0x1365f000..0x1365ffc0]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x2b5550f8> (a
java.util.concurrent.locks.ReentrantLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:778)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1114)
at
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186)
at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262)
at org.jgroups.protocols.TP.send(TP.java:1082)
at org.jgroups.protocols.TP.down(TP.java:907)
at org.jgroups.protocols.Discovery.up(Discovery.java:274)
at org.jgroups.protocols.TP.passMessageUp(TP.java:995)
at org.jgroups.protocols.TP.access$100(TP.java:52)
at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1451)
at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1427)
at
java.util.concurrent.ThreadPoolExecutor$CallerRunsPolicy.rejectedExecution(ThreadPoolExecutor.java:1738)
at
org.jgroups.util.ShutdownRejectedExecutionHandler.rejectedExecution(ShutdownRejectedExecutionHandler.java:34)
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
at org.jgroups.protocols.TP.dispatchToThreadPool(TP.java:1061)
at org.jgroups.protocols.TP.receive(TP.java:1038)
at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:180)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection$ConnectionPeerReceiver.run(TCPConnectionMap.java:553)
at java.lang.Thread.run(Thread.java:619)
Locked ownable synchronizers:
- None
(Since all threads from JGroups' ThreadPool are locked, you can see
"rejectedExecution" in the last stack trace).
Maybe we could implement some solution for TCP (blocking I/O)? Maybe it is possible to
release the lock before socket write?
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
1:56 a.m.
New subject: [JBoss JIRA] Closed: (JGRP-1003) deadlock in TP.send - socket write hangs inside a Lock, preventing other threads to do TP.send()
[
https://jira.jboss.org/jira/browse/JGRP-1003?page=com.atlassian.jira.plug...
]
Bela Ban closed JGRP-1003.
--------------------------
Resolution: Won't Fix
This should work with use_send_queues="true". Please reopen if not. Note that
the entire TCP / TCP_NIO impl will get rewritten once we switch to JDK 1.7 as baseline
(NIO2)
deadlock in TP.send - socket write hangs inside a Lock, preventing
other threads to do TP.send()
------------------------------------------------------------------------------------------------
Key: JGRP-1003
URL: https://jira.jboss.org/jira/browse/JGRP-1003
Project: JGroups
Issue Type: Bug
Affects Versions: 2.7
Environment: Linux (Debian), kernel 2.6.x
Reporter: Victor N
Assignee: Bela Ban
Fix For: 2.8
I am using JGroups 2.7.0 GA with the typical protocols stack based on TCP (taken from
tcp.xml).
Sometimes it occurs that socket write operation hangs inside TP.send() -- this is not so
untypical for blocking I/O approach! -- after that JGroups is not working at all because
most of threads are blocked inside TP.send() waiting for that first thread releasing
"out_stream_lock". Below is a fragment from my stack trace.
1) the thread that hangs in socket write until forever (maybe due to a broken socket or
something else); this thread owns "out_stream_lock in TP class":
"Timer-1,name,IP:port" daemon prio=10 tid=0x084e5c00 nid=0x15f0 runnable
[0x211c1000..0x211c2040]
java.lang.Thread.State: RUNNABLE
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
- locked <0x43427300> (a java.io.BufferedOutputStream)
at java.io.DataOutputStream.flush(DataOutputStream.java:106)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection.doSend(TCPConnectionMap.java:472)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection._send(TCPConnectionMap.java:453)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection.send(TCPConnectionMap.java:436)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection.access$100(TCPConnectionMap.java:341)
at org.jgroups.blocks.TCPConnectionMap.send(TCPConnectionMap.java:137)
at org.jgroups.protocols.TCP.send(TCP.java:53)
at org.jgroups.protocols.BasicTCP.sendToSingleMember(BasicTCP.java:141)
at org.jgroups.protocols.TP.doSend(TP.java:1105)
at org.jgroups.protocols.TP.send(TP.java:1088)
at org.jgroups.protocols.TP.down(TP.java:907)
at org.jgroups.protocols.Discovery.down(Discovery.java:363)
at org.jgroups.protocols.MERGE2.down(MERGE2.java:169)
at org.jgroups.protocols.FD_SOCK.down(FD_SOCK.java:333)
at org.jgroups.protocols.FD.down(FD.java:327)
at org.jgroups.protocols.VERIFY_SUSPECT.down(VERIFY_SUSPECT.java:72)
at org.jgroups.protocols.BARRIER.down(BARRIER.java:96)
at org.jgroups.protocols.pbcast.NAKACK.retransmit(NAKACK.java:1530)
at org.jgroups.protocols.pbcast.NAKACK.retransmit(NAKACK.java:1476)
at org.jgroups.stack.Retransmitter$Task.run(Retransmitter.java:207)
at org.jgroups.util.TimeScheduler$TaskWrapper.run(TimeScheduler.java:218)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:207)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Locked ownable synchronizers:
- <0x2b5522e0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
- <0x2b5550f8> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
- <0x434273f0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
2) and there are many threads like this blocked "on out_stream_lock.lock()"
call inside TP class:
"Connection.Receiver [ip:port - ip:port],name,ip:port" prio=10 tid=0x020dc000
nid=0x1fab waiting on condition [0x1365f000..0x1365ffc0]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x2b5550f8> (a
java.util.concurrent.locks.ReentrantLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:778)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1114)
at
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186)
at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262)
at org.jgroups.protocols.TP.send(TP.java:1082)
at org.jgroups.protocols.TP.down(TP.java:907)
at org.jgroups.protocols.Discovery.up(Discovery.java:274)
at org.jgroups.protocols.TP.passMessageUp(TP.java:995)
at org.jgroups.protocols.TP.access$100(TP.java:52)
at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1451)
at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1427)
at
java.util.concurrent.ThreadPoolExecutor$CallerRunsPolicy.rejectedExecution(ThreadPoolExecutor.java:1738)
at
org.jgroups.util.ShutdownRejectedExecutionHandler.rejectedExecution(ShutdownRejectedExecutionHandler.java:34)
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
at org.jgroups.protocols.TP.dispatchToThreadPool(TP.java:1061)
at org.jgroups.protocols.TP.receive(TP.java:1038)
at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:180)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection$ConnectionPeerReceiver.run(TCPConnectionMap.java:553)
at java.lang.Thread.run(Thread.java:619)
Locked ownable synchronizers:
- None
(Since all threads from JGroups' ThreadPool are locked, you can see
"rejectedExecution" in the last stack trace).
Maybe we could implement some solution for TCP (blocking I/O)? Maybe it is possible to
release the lock before socket write?
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
7:14 a.m.
New subject: [JBoss JIRA] Commented: (JGRP-1003) deadlock in TP.send - socket write hangs inside a Lock, preventing other threads to do TP.send()
[
https://jira.jboss.org/jira/browse/JGRP-1003?page=com.atlassian.jira.plug...
]
Victor N commented on JGRP-1003:
--------------------------------
I will check how it works now.
By the way, I would recommend to set use_send_queues="true" in the default
tcp.xml in JGroups sources/examples!
deadlock in TP.send - socket write hangs inside a Lock, preventing
other threads to do TP.send()
------------------------------------------------------------------------------------------------
Key: JGRP-1003
URL: https://jira.jboss.org/jira/browse/JGRP-1003
Project: JGroups
Issue Type: Bug
Affects Versions: 2.7
Environment: Linux (Debian), kernel 2.6.x
Reporter: Victor N
Assignee: Bela Ban
Fix For: 2.8
I am using JGroups 2.7.0 GA with the typical protocols stack based on TCP (taken from
tcp.xml).
Sometimes it occurs that socket write operation hangs inside TP.send() -- this is not so
untypical for blocking I/O approach! -- after that JGroups is not working at all because
most of threads are blocked inside TP.send() waiting for that first thread releasing
"out_stream_lock". Below is a fragment from my stack trace.
1) the thread that hangs in socket write until forever (maybe due to a broken socket or
something else); this thread owns "out_stream_lock in TP class":
"Timer-1,name,IP:port" daemon prio=10 tid=0x084e5c00 nid=0x15f0 runnable
[0x211c1000..0x211c2040]
java.lang.Thread.State: RUNNABLE
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
- locked <0x43427300> (a java.io.BufferedOutputStream)
at java.io.DataOutputStream.flush(DataOutputStream.java:106)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection.doSend(TCPConnectionMap.java:472)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection._send(TCPConnectionMap.java:453)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection.send(TCPConnectionMap.java:436)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection.access$100(TCPConnectionMap.java:341)
at org.jgroups.blocks.TCPConnectionMap.send(TCPConnectionMap.java:137)
at org.jgroups.protocols.TCP.send(TCP.java:53)
at org.jgroups.protocols.BasicTCP.sendToSingleMember(BasicTCP.java:141)
at org.jgroups.protocols.TP.doSend(TP.java:1105)
at org.jgroups.protocols.TP.send(TP.java:1088)
at org.jgroups.protocols.TP.down(TP.java:907)
at org.jgroups.protocols.Discovery.down(Discovery.java:363)
at org.jgroups.protocols.MERGE2.down(MERGE2.java:169)
at org.jgroups.protocols.FD_SOCK.down(FD_SOCK.java:333)
at org.jgroups.protocols.FD.down(FD.java:327)
at org.jgroups.protocols.VERIFY_SUSPECT.down(VERIFY_SUSPECT.java:72)
at org.jgroups.protocols.BARRIER.down(BARRIER.java:96)
at org.jgroups.protocols.pbcast.NAKACK.retransmit(NAKACK.java:1530)
at org.jgroups.protocols.pbcast.NAKACK.retransmit(NAKACK.java:1476)
at org.jgroups.stack.Retransmitter$Task.run(Retransmitter.java:207)
at org.jgroups.util.TimeScheduler$TaskWrapper.run(TimeScheduler.java:218)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:207)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Locked ownable synchronizers:
- <0x2b5522e0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
- <0x2b5550f8> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
- <0x434273f0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
2) and there are many threads like this blocked "on out_stream_lock.lock()"
call inside TP class:
"Connection.Receiver [ip:port - ip:port],name,ip:port" prio=10 tid=0x020dc000
nid=0x1fab waiting on condition [0x1365f000..0x1365ffc0]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x2b5550f8> (a
java.util.concurrent.locks.ReentrantLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:778)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1114)
at
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186)
at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262)
at org.jgroups.protocols.TP.send(TP.java:1082)
at org.jgroups.protocols.TP.down(TP.java:907)
at org.jgroups.protocols.Discovery.up(Discovery.java:274)
at org.jgroups.protocols.TP.passMessageUp(TP.java:995)
at org.jgroups.protocols.TP.access$100(TP.java:52)
at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1451)
at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1427)
at
java.util.concurrent.ThreadPoolExecutor$CallerRunsPolicy.rejectedExecution(ThreadPoolExecutor.java:1738)
at
org.jgroups.util.ShutdownRejectedExecutionHandler.rejectedExecution(ShutdownRejectedExecutionHandler.java:34)
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
at org.jgroups.protocols.TP.dispatchToThreadPool(TP.java:1061)
at org.jgroups.protocols.TP.receive(TP.java:1038)
at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:180)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection$ConnectionPeerReceiver.run(TCPConnectionMap.java:553)
at java.lang.Thread.run(Thread.java:619)
Locked ownable synchronizers:
- None
(Since all threads from JGroups' ThreadPool are locked, you can see
"rejectedExecution" in the last stack trace).
Maybe we could implement some solution for TCP (blocking I/O)? Maybe it is possible to
release the lock before socket write?
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
7:22 a.m.
New subject: [JBoss JIRA] Commented: (JGRP-1003) deadlock in TP.send - socket write hangs inside a Lock, preventing other threads to do TP.send()
[
https://jira.jboss.org/jira/browse/JGRP-1003?page=com.atlassian.jira.plug...
]
Bela Ban commented on JGRP-1003:
--------------------------------
set use_send_queue="true" in all TCP configs
deadlock in TP.send - socket write hangs inside a Lock, preventing
other threads to do TP.send()
------------------------------------------------------------------------------------------------
Key: JGRP-1003
URL: https://jira.jboss.org/jira/browse/JGRP-1003
Project: JGroups
Issue Type: Bug
Affects Versions: 2.7
Environment: Linux (Debian), kernel 2.6.x
Reporter: Victor N
Assignee: Bela Ban
Fix For: 2.8
I am using JGroups 2.7.0 GA with the typical protocols stack based on TCP (taken from
tcp.xml).
Sometimes it occurs that socket write operation hangs inside TP.send() -- this is not so
untypical for blocking I/O approach! -- after that JGroups is not working at all because
most of threads are blocked inside TP.send() waiting for that first thread releasing
"out_stream_lock". Below is a fragment from my stack trace.
1) the thread that hangs in socket write until forever (maybe due to a broken socket or
something else); this thread owns "out_stream_lock in TP class":
"Timer-1,name,IP:port" daemon prio=10 tid=0x084e5c00 nid=0x15f0 runnable
[0x211c1000..0x211c2040]
java.lang.Thread.State: RUNNABLE
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
- locked <0x43427300> (a java.io.BufferedOutputStream)
at java.io.DataOutputStream.flush(DataOutputStream.java:106)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection.doSend(TCPConnectionMap.java:472)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection._send(TCPConnectionMap.java:453)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection.send(TCPConnectionMap.java:436)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection.access$100(TCPConnectionMap.java:341)
at org.jgroups.blocks.TCPConnectionMap.send(TCPConnectionMap.java:137)
at org.jgroups.protocols.TCP.send(TCP.java:53)
at org.jgroups.protocols.BasicTCP.sendToSingleMember(BasicTCP.java:141)
at org.jgroups.protocols.TP.doSend(TP.java:1105)
at org.jgroups.protocols.TP.send(TP.java:1088)
at org.jgroups.protocols.TP.down(TP.java:907)
at org.jgroups.protocols.Discovery.down(Discovery.java:363)
at org.jgroups.protocols.MERGE2.down(MERGE2.java:169)
at org.jgroups.protocols.FD_SOCK.down(FD_SOCK.java:333)
at org.jgroups.protocols.FD.down(FD.java:327)
at org.jgroups.protocols.VERIFY_SUSPECT.down(VERIFY_SUSPECT.java:72)
at org.jgroups.protocols.BARRIER.down(BARRIER.java:96)
at org.jgroups.protocols.pbcast.NAKACK.retransmit(NAKACK.java:1530)
at org.jgroups.protocols.pbcast.NAKACK.retransmit(NAKACK.java:1476)
at org.jgroups.stack.Retransmitter$Task.run(Retransmitter.java:207)
at org.jgroups.util.TimeScheduler$TaskWrapper.run(TimeScheduler.java:218)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:207)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Locked ownable synchronizers:
- <0x2b5522e0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
- <0x2b5550f8> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
- <0x434273f0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
2) and there are many threads like this blocked "on out_stream_lock.lock()"
call inside TP class:
"Connection.Receiver [ip:port - ip:port],name,ip:port" prio=10 tid=0x020dc000
nid=0x1fab waiting on condition [0x1365f000..0x1365ffc0]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x2b5550f8> (a
java.util.concurrent.locks.ReentrantLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:778)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1114)
at
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186)
at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262)
at org.jgroups.protocols.TP.send(TP.java:1082)
at org.jgroups.protocols.TP.down(TP.java:907)
at org.jgroups.protocols.Discovery.up(Discovery.java:274)
at org.jgroups.protocols.TP.passMessageUp(TP.java:995)
at org.jgroups.protocols.TP.access$100(TP.java:52)
at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1451)
at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1427)
at
java.util.concurrent.ThreadPoolExecutor$CallerRunsPolicy.rejectedExecution(ThreadPoolExecutor.java:1738)
at
org.jgroups.util.ShutdownRejectedExecutionHandler.rejectedExecution(ShutdownRejectedExecutionHandler.java:34)
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
at org.jgroups.protocols.TP.dispatchToThreadPool(TP.java:1061)
at org.jgroups.protocols.TP.receive(TP.java:1038)
at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:180)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection$ConnectionPeerReceiver.run(TCPConnectionMap.java:553)
at java.lang.Thread.run(Thread.java:619)
Locked ownable synchronizers:
- None
(Since all threads from JGroups' ThreadPool are locked, you can see
"rejectedExecution" in the last stack trace).
Maybe we could implement some solution for TCP (blocking I/O)? Maybe it is possible to
release the lock before socket write?
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
1:18 p.m.
New subject: [JBoss JIRA] Reopened: (JGRP-1003) deadlock in TP.send - socket write hangs inside a Lock, preventing other threads to do TP.send()
[
https://jira.jboss.org/jira/browse/JGRP-1003?page=com.atlassian.jira.plug...
]
Victor N reopened JGRP-1003:
----------------------------
Hi Bela,
another day, another bug :)
So... I have configured use_send_queues="true" for TCP. After some time I have
got another strange situation - one of my servers was excluded from group and never
rejoined. I looked into stack traces - I am sending you "jgroups_s1.txt" file by
email, please look.
You will see one thread:
"Connection.Sender [server1:40001 - server2:33324],GateCluster,server1:40001"
prio=10 tid=0x08f9d400 nid=0x147e waiting on condition [0x187d6000..0x187d7140]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x830c5d28> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection$Sender.run(TCPConnectionMap.java:594)
at java.lang.Thread.run(Thread.java:619)
This means that the thread is waiting until S ender's queue becomes non-empty
(LinkedBlockingQueue.java:358):
while (count.get() == 0)
notEmpty.await();
But at the same time there are many threads like this (trying to send something to
network):
"pool-2-thread-2" prio=10 tid=0x0891a000 nid=0x1f9d waiting on condition
[0x20afe000..0x20aff0c0]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x63ac53e8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:254)
at org.jgroups.blocks.TCPConnectionMap$TCPConnection.send(TCPConnectionMap.java:429)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection.access$100(TCPConnectionMap.java:341)
at org.jgroups.blocks.TCPConnectionMap.send(TCPConnectionMap.java:137)
at org.jgroups.protocols.TCP.send(TCP.java:53)
at org.jgroups.protocols.BasicTCP.sendToSingleMember(BasicTCP.java:141)
at org.jgroups.protocols.BasicTCP.sendToAllMembers(BasicTCP.java:126)
at org.jgroups.protocols.TP.doSend(TP.java:1102)
at org.jgroups.protocols.TP.access$1600(TP.java:52)
at org.jgroups.protocols.TP$Bundler.sendBundledMessages(TP.java:1548)
at org.jgroups.protocols.TP$Bundler.send(TP.java:1484)
at org.jgroups.protocols.TP$Bundler.access$200(TP.java:1461)
at org.jgroups.protocols.TP.send(TP.java:1077)
at org.jgroups.protocols.TP.down(TP.java:907)
at org.jgroups.protocols.Discovery.down(Discovery.java:363)
at org.jgroups.protocols.MERGE2.down(MERGE2.java:169)
at org.jgroups.protocols.FD_SOCK.down(FD_SOCK.java:333)
at org.jgroups.protocols.FD.down(FD.java:327)
at org.jgroups.protocols.VERIFY_SUSPECT.down(VERIFY_SUSPECT.java:72)
at org.jgroups.protocols.BARRIER.down(BARRIER.java:96)
at org.jgroups.protocols.pbcast.NAKACK.send(NAKACK.java:771)
at org.jgroups.protocols.pbcast.NAKACK.down(NAKACK.java:572)
at org.jgroups.protocols.UNICAST.down(UNICAST.java:468)
at org.jgroups.protocols.pbcast.STABLE.down(STABLE.java:322)
at org.jgroups.protocols.VIEW_SYNC.down(VIEW_SYNC.java:196)
at org.jgroups.protocols.pbcast.GMS.down(GMS.java:939)
at org.jgroups.protocols.FC.handleDownMessage(FC.java:617)
at org.jgroups.protocols.FC.down(FC.java:434)
at org.jgroups.protocols.FRAG2.down(FRAG2.java:158)
at
org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER.down(STREAMING_STATE_TRANSFER.java:275)
at org.jgroups.stack.ProtocolStack.down(ProtocolStack.java:838)
at org.jgroups.JChannel.down(JChannel.java:1504)
at org.jgroups.JChannel.send(JChannel.java:670)
at org.jgroups.JChannel.send(JChannel.java:686)
at
mytest.cluster.JGroupsClusterManagerImpl.doSendMessageToAllGates(JGroupsClusterManagerImpl.java:93)
at
mytest.cluster.JGroupsClusterManagerImpl.access$000(JGroupsClusterManagerImpl.java:24)
at mytest.cluster.JGroupsClusterManagerImpl$1.run(JGroupsClusterManagerImpl.java:81)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
This means that this 2nd thread is waiting until Sender's queue becomes non-full
(LinkedBlockingQueue.java:254):
while (count.get() == capacity)
notFull.await();
How can this be at the same time - the same queue is empty and full?
Maybe there are 2 different instances of sender queue and 2 different senders? Is a new
sender created in some situations (shunning, reconnect, etc.)?
deadlock in TP.send - socket write hangs inside a Lock, preventing
other threads to do TP.send()
------------------------------------------------------------------------------------------------
Key: JGRP-1003
URL: https://jira.jboss.org/jira/browse/JGRP-1003
Project: JGroups
Issue Type: Bug
Affects Versions: 2.7
Environment: Linux (Debian), kernel 2.6.x
Reporter: Victor N
Assignee: Bela Ban
Fix For: 2.8
I am using JGroups 2.7.0 GA with the typical protocols stack based on TCP (taken from
tcp.xml).
Sometimes it occurs that socket write operation hangs inside TP.send() -- this is not so
untypical for blocking I/O approach! -- after that JGroups is not working at all because
most of threads are blocked inside TP.send() waiting for that first thread releasing
"out_stream_lock". Below is a fragment from my stack trace.
1) the thread that hangs in socket write until forever (maybe due to a broken socket or
something else); this thread owns "out_stream_lock in TP class":
"Timer-1,name,IP:port" daemon prio=10 tid=0x084e5c00 nid=0x15f0 runnable
[0x211c1000..0x211c2040]
java.lang.Thread.State: RUNNABLE
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
- locked <0x43427300> (a java.io.BufferedOutputStream)
at java.io.DataOutputStream.flush(DataOutputStream.java:106)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection.doSend(TCPConnectionMap.java:472)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection._send(TCPConnectionMap.java:453)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection.send(TCPConnectionMap.java:436)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection.access$100(TCPConnectionMap.java:341)
at org.jgroups.blocks.TCPConnectionMap.send(TCPConnectionMap.java:137)
at org.jgroups.protocols.TCP.send(TCP.java:53)
at org.jgroups.protocols.BasicTCP.sendToSingleMember(BasicTCP.java:141)
at org.jgroups.protocols.TP.doSend(TP.java:1105)
at org.jgroups.protocols.TP.send(TP.java:1088)
at org.jgroups.protocols.TP.down(TP.java:907)
at org.jgroups.protocols.Discovery.down(Discovery.java:363)
at org.jgroups.protocols.MERGE2.down(MERGE2.java:169)
at org.jgroups.protocols.FD_SOCK.down(FD_SOCK.java:333)
at org.jgroups.protocols.FD.down(FD.java:327)
at org.jgroups.protocols.VERIFY_SUSPECT.down(VERIFY_SUSPECT.java:72)
at org.jgroups.protocols.BARRIER.down(BARRIER.java:96)
at org.jgroups.protocols.pbcast.NAKACK.retransmit(NAKACK.java:1530)
at org.jgroups.protocols.pbcast.NAKACK.retransmit(NAKACK.java:1476)
at org.jgroups.stack.Retransmitter$Task.run(Retransmitter.java:207)
at org.jgroups.util.TimeScheduler$TaskWrapper.run(TimeScheduler.java:218)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:207)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Locked ownable synchronizers:
- <0x2b5522e0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
- <0x2b5550f8> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
- <0x434273f0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
2) and there are many threads like this blocked "on out_stream_lock.lock()"
call inside TP class:
"Connection.Receiver [ip:port - ip:port],name,ip:port" prio=10 tid=0x020dc000
nid=0x1fab waiting on condition [0x1365f000..0x1365ffc0]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x2b5550f8> (a
java.util.concurrent.locks.ReentrantLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:778)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1114)
at
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186)
at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262)
at org.jgroups.protocols.TP.send(TP.java:1082)
at org.jgroups.protocols.TP.down(TP.java:907)
at org.jgroups.protocols.Discovery.up(Discovery.java:274)
at org.jgroups.protocols.TP.passMessageUp(TP.java:995)
at org.jgroups.protocols.TP.access$100(TP.java:52)
at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1451)
at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1427)
at
java.util.concurrent.ThreadPoolExecutor$CallerRunsPolicy.rejectedExecution(ThreadPoolExecutor.java:1738)
at
org.jgroups.util.ShutdownRejectedExecutionHandler.rejectedExecution(ShutdownRejectedExecutionHandler.java:34)
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
at org.jgroups.protocols.TP.dispatchToThreadPool(TP.java:1061)
at org.jgroups.protocols.TP.receive(TP.java:1038)
at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:180)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection$ConnectionPeerReceiver.run(TCPConnectionMap.java:553)
at java.lang.Thread.run(Thread.java:619)
Locked ownable synchronizers:
- None
(Since all threads from JGroups' ThreadPool are locked, you can see
"rejectedExecution" in the last stack trace).
Maybe we could implement some solution for TCP (blocking I/O)? Maybe it is possible to
release the lock before socket write?
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
1:24 p.m.
New subject: [JBoss JIRA] Commented: (JGRP-1003) deadlock in TP.send - socket write hangs inside a Lock, preventing other threads to do TP.send()
[
https://jira.jboss.org/jira/browse/JGRP-1003?page=com.atlassian.jira.plug...
]
Victor N commented on JGRP-1003:
--------------------------------
Definitely, I am going to test with TCP_NIO ! I hope it does not need send queues and does
not block in "send" call.
deadlock in TP.send - socket write hangs inside a Lock, preventing
other threads to do TP.send()
------------------------------------------------------------------------------------------------
Key: JGRP-1003
URL: https://jira.jboss.org/jira/browse/JGRP-1003
Project: JGroups
Issue Type: Bug
Affects Versions: 2.7
Environment: Linux (Debian), kernel 2.6.x
Reporter: Victor N
Assignee: Bela Ban
Fix For: 2.8
I am using JGroups 2.7.0 GA with the typical protocols stack based on TCP (taken from
tcp.xml).
Sometimes it occurs that socket write operation hangs inside TP.send() -- this is not so
untypical for blocking I/O approach! -- after that JGroups is not working at all because
most of threads are blocked inside TP.send() waiting for that first thread releasing
"out_stream_lock". Below is a fragment from my stack trace.
1) the thread that hangs in socket write until forever (maybe due to a broken socket or
something else); this thread owns "out_stream_lock in TP class":
"Timer-1,name,IP:port" daemon prio=10 tid=0x084e5c00 nid=0x15f0 runnable
[0x211c1000..0x211c2040]
java.lang.Thread.State: RUNNABLE
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
- locked <0x43427300> (a java.io.BufferedOutputStream)
at java.io.DataOutputStream.flush(DataOutputStream.java:106)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection.doSend(TCPConnectionMap.java:472)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection._send(TCPConnectionMap.java:453)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection.send(TCPConnectionMap.java:436)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection.access$100(TCPConnectionMap.java:341)
at org.jgroups.blocks.TCPConnectionMap.send(TCPConnectionMap.java:137)
at org.jgroups.protocols.TCP.send(TCP.java:53)
at org.jgroups.protocols.BasicTCP.sendToSingleMember(BasicTCP.java:141)
at org.jgroups.protocols.TP.doSend(TP.java:1105)
at org.jgroups.protocols.TP.send(TP.java:1088)
at org.jgroups.protocols.TP.down(TP.java:907)
at org.jgroups.protocols.Discovery.down(Discovery.java:363)
at org.jgroups.protocols.MERGE2.down(MERGE2.java:169)
at org.jgroups.protocols.FD_SOCK.down(FD_SOCK.java:333)
at org.jgroups.protocols.FD.down(FD.java:327)
at org.jgroups.protocols.VERIFY_SUSPECT.down(VERIFY_SUSPECT.java:72)
at org.jgroups.protocols.BARRIER.down(BARRIER.java:96)
at org.jgroups.protocols.pbcast.NAKACK.retransmit(NAKACK.java:1530)
at org.jgroups.protocols.pbcast.NAKACK.retransmit(NAKACK.java:1476)
at org.jgroups.stack.Retransmitter$Task.run(Retransmitter.java:207)
at org.jgroups.util.TimeScheduler$TaskWrapper.run(TimeScheduler.java:218)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:207)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Locked ownable synchronizers:
- <0x2b5522e0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
- <0x2b5550f8> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
- <0x434273f0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
2) and there are many threads like this blocked "on out_stream_lock.lock()"
call inside TP class:
"Connection.Receiver [ip:port - ip:port],name,ip:port" prio=10 tid=0x020dc000
nid=0x1fab waiting on condition [0x1365f000..0x1365ffc0]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x2b5550f8> (a
java.util.concurrent.locks.ReentrantLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:778)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1114)
at
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186)
at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262)
at org.jgroups.protocols.TP.send(TP.java:1082)
at org.jgroups.protocols.TP.down(TP.java:907)
at org.jgroups.protocols.Discovery.up(Discovery.java:274)
at org.jgroups.protocols.TP.passMessageUp(TP.java:995)
at org.jgroups.protocols.TP.access$100(TP.java:52)
at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1451)
at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1427)
at
java.util.concurrent.ThreadPoolExecutor$CallerRunsPolicy.rejectedExecution(ThreadPoolExecutor.java:1738)
at
org.jgroups.util.ShutdownRejectedExecutionHandler.rejectedExecution(ShutdownRejectedExecutionHandler.java:34)
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
at org.jgroups.protocols.TP.dispatchToThreadPool(TP.java:1061)
at org.jgroups.protocols.TP.receive(TP.java:1038)
at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:180)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection$ConnectionPeerReceiver.run(TCPConnectionMap.java:553)
at java.lang.Thread.run(Thread.java:619)
Locked ownable synchronizers:
- None
(Since all threads from JGroups' ThreadPool are locked, you can see
"rejectedExecution" in the last stack trace).
Maybe we could implement some solution for TCP (blocking I/O)? Maybe it is possible to
release the lock before socket write?
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
1:01 a.m.
New subject: [JBoss JIRA] Updated: (JGRP-1003) deadlock in TP.send - socket write hangs inside a Lock, preventing other threads to do TP.send()
[
https://jira.jboss.org/jira/browse/JGRP-1003?page=com.atlassian.jira.plug...
]
Bela Ban updated JGRP-1003:
---------------------------
Attachment: stacktrace.txt
deadlock in TP.send - socket write hangs inside a Lock, preventing
other threads to do TP.send()
------------------------------------------------------------------------------------------------
Key: JGRP-1003
URL: https://jira.jboss.org/jira/browse/JGRP-1003
Project: JGroups
Issue Type: Bug
Affects Versions: 2.7
Environment: Linux (Debian), kernel 2.6.x
Reporter: Victor N
Assignee: Bela Ban
Fix For: 2.8
Attachments: stacktrace.txt
I am using JGroups 2.7.0 GA with the typical protocols stack based on TCP (taken from
tcp.xml).
Sometimes it occurs that socket write operation hangs inside TP.send() -- this is not so
untypical for blocking I/O approach! -- after that JGroups is not working at all because
most of threads are blocked inside TP.send() waiting for that first thread releasing
"out_stream_lock". Below is a fragment from my stack trace.
1) the thread that hangs in socket write until forever (maybe due to a broken socket or
something else); this thread owns "out_stream_lock in TP class":
"Timer-1,name,IP:port" daemon prio=10 tid=0x084e5c00 nid=0x15f0 runnable
[0x211c1000..0x211c2040]
java.lang.Thread.State: RUNNABLE
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
- locked <0x43427300> (a java.io.BufferedOutputStream)
at java.io.DataOutputStream.flush(DataOutputStream.java:106)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection.doSend(TCPConnectionMap.java:472)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection._send(TCPConnectionMap.java:453)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection.send(TCPConnectionMap.java:436)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection.access$100(TCPConnectionMap.java:341)
at org.jgroups.blocks.TCPConnectionMap.send(TCPConnectionMap.java:137)
at org.jgroups.protocols.TCP.send(TCP.java:53)
at org.jgroups.protocols.BasicTCP.sendToSingleMember(BasicTCP.java:141)
at org.jgroups.protocols.TP.doSend(TP.java:1105)
at org.jgroups.protocols.TP.send(TP.java:1088)
at org.jgroups.protocols.TP.down(TP.java:907)
at org.jgroups.protocols.Discovery.down(Discovery.java:363)
at org.jgroups.protocols.MERGE2.down(MERGE2.java:169)
at org.jgroups.protocols.FD_SOCK.down(FD_SOCK.java:333)
at org.jgroups.protocols.FD.down(FD.java:327)
at org.jgroups.protocols.VERIFY_SUSPECT.down(VERIFY_SUSPECT.java:72)
at org.jgroups.protocols.BARRIER.down(BARRIER.java:96)
at org.jgroups.protocols.pbcast.NAKACK.retransmit(NAKACK.java:1530)
at org.jgroups.protocols.pbcast.NAKACK.retransmit(NAKACK.java:1476)
at org.jgroups.stack.Retransmitter$Task.run(Retransmitter.java:207)
at org.jgroups.util.TimeScheduler$TaskWrapper.run(TimeScheduler.java:218)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:207)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Locked ownable synchronizers:
- <0x2b5522e0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
- <0x2b5550f8> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
- <0x434273f0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
2) and there are many threads like this blocked "on out_stream_lock.lock()"
call inside TP class:
"Connection.Receiver [ip:port - ip:port],name,ip:port" prio=10 tid=0x020dc000
nid=0x1fab waiting on condition [0x1365f000..0x1365ffc0]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x2b5550f8> (a
java.util.concurrent.locks.ReentrantLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:778)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1114)
at
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186)
at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262)
at org.jgroups.protocols.TP.send(TP.java:1082)
at org.jgroups.protocols.TP.down(TP.java:907)
at org.jgroups.protocols.Discovery.up(Discovery.java:274)
at org.jgroups.protocols.TP.passMessageUp(TP.java:995)
at org.jgroups.protocols.TP.access$100(TP.java:52)
at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1451)
at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1427)
at
java.util.concurrent.ThreadPoolExecutor$CallerRunsPolicy.rejectedExecution(ThreadPoolExecutor.java:1738)
at
org.jgroups.util.ShutdownRejectedExecutionHandler.rejectedExecution(ShutdownRejectedExecutionHandler.java:34)
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
at org.jgroups.protocols.TP.dispatchToThreadPool(TP.java:1061)
at org.jgroups.protocols.TP.receive(TP.java:1038)
at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:180)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection$ConnectionPeerReceiver.run(TCPConnectionMap.java:553)
at java.lang.Thread.run(Thread.java:619)
Locked ownable synchronizers:
- None
(Since all threads from JGroups' ThreadPool are locked, you can see
"rejectedExecution" in the last stack trace).
Maybe we could implement some solution for TCP (blocking I/O)? Maybe it is possible to
release the lock before socket write?
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
3:54 a.m.
New subject: [JBoss JIRA] Commented: (JGRP-1003) deadlock in TP.send - socket write hangs inside a Lock, preventing other threads to do TP.send()
[
https://jira.jboss.org/jira/browse/JGRP-1003?page=com.atlassian.jira.plug...
]
Victor N commented on JGRP-1003:
--------------------------------
Bela, I reproduced the problem again with the same symptoms - please look into the
attached jgroups_s2.txt.
deadlock in TP.send - socket write hangs inside a Lock, preventing
other threads to do TP.send()
------------------------------------------------------------------------------------------------
Key: JGRP-1003
URL: https://jira.jboss.org/jira/browse/JGRP-1003
Project: JGroups
Issue Type: Bug
Affects Versions: 2.7
Environment: Linux (Debian), kernel 2.6.x
Reporter: Victor N
Assignee: Bela Ban
Fix For: 2.8
Attachments: jgroups_s2.txt, stacktrace.txt
I am using JGroups 2.7.0 GA with the typical protocols stack based on TCP (taken from
tcp.xml).
Sometimes it occurs that socket write operation hangs inside TP.send() -- this is not so
untypical for blocking I/O approach! -- after that JGroups is not working at all because
most of threads are blocked inside TP.send() waiting for that first thread releasing
"out_stream_lock". Below is a fragment from my stack trace.
1) the thread that hangs in socket write until forever (maybe due to a broken socket or
something else); this thread owns "out_stream_lock in TP class":
"Timer-1,name,IP:port" daemon prio=10 tid=0x084e5c00 nid=0x15f0 runnable
[0x211c1000..0x211c2040]
java.lang.Thread.State: RUNNABLE
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
- locked <0x43427300> (a java.io.BufferedOutputStream)
at java.io.DataOutputStream.flush(DataOutputStream.java:106)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection.doSend(TCPConnectionMap.java:472)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection._send(TCPConnectionMap.java:453)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection.send(TCPConnectionMap.java:436)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection.access$100(TCPConnectionMap.java:341)
at org.jgroups.blocks.TCPConnectionMap.send(TCPConnectionMap.java:137)
at org.jgroups.protocols.TCP.send(TCP.java:53)
at org.jgroups.protocols.BasicTCP.sendToSingleMember(BasicTCP.java:141)
at org.jgroups.protocols.TP.doSend(TP.java:1105)
at org.jgroups.protocols.TP.send(TP.java:1088)
at org.jgroups.protocols.TP.down(TP.java:907)
at org.jgroups.protocols.Discovery.down(Discovery.java:363)
at org.jgroups.protocols.MERGE2.down(MERGE2.java:169)
at org.jgroups.protocols.FD_SOCK.down(FD_SOCK.java:333)
at org.jgroups.protocols.FD.down(FD.java:327)
at org.jgroups.protocols.VERIFY_SUSPECT.down(VERIFY_SUSPECT.java:72)
at org.jgroups.protocols.BARRIER.down(BARRIER.java:96)
at org.jgroups.protocols.pbcast.NAKACK.retransmit(NAKACK.java:1530)
at org.jgroups.protocols.pbcast.NAKACK.retransmit(NAKACK.java:1476)
at org.jgroups.stack.Retransmitter$Task.run(Retransmitter.java:207)
at org.jgroups.util.TimeScheduler$TaskWrapper.run(TimeScheduler.java:218)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:207)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Locked ownable synchronizers:
- <0x2b5522e0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
- <0x2b5550f8> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
- <0x434273f0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
2) and there are many threads like this blocked "on out_stream_lock.lock()"
call inside TP class:
"Connection.Receiver [ip:port - ip:port],name,ip:port" prio=10 tid=0x020dc000
nid=0x1fab waiting on condition [0x1365f000..0x1365ffc0]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x2b5550f8> (a
java.util.concurrent.locks.ReentrantLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:778)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1114)
at
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186)
at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262)
at org.jgroups.protocols.TP.send(TP.java:1082)
at org.jgroups.protocols.TP.down(TP.java:907)
at org.jgroups.protocols.Discovery.up(Discovery.java:274)
at org.jgroups.protocols.TP.passMessageUp(TP.java:995)
at org.jgroups.protocols.TP.access$100(TP.java:52)
at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1451)
at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1427)
at
java.util.concurrent.ThreadPoolExecutor$CallerRunsPolicy.rejectedExecution(ThreadPoolExecutor.java:1738)
at
org.jgroups.util.ShutdownRejectedExecutionHandler.rejectedExecution(ShutdownRejectedExecutionHandler.java:34)
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
at org.jgroups.protocols.TP.dispatchToThreadPool(TP.java:1061)
at org.jgroups.protocols.TP.receive(TP.java:1038)
at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:180)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection$ConnectionPeerReceiver.run(TCPConnectionMap.java:553)
at java.lang.Thread.run(Thread.java:619)
Locked ownable synchronizers:
- None
(Since all threads from JGroups' ThreadPool are locked, you can see
"rejectedExecution" in the last stack trace).
Maybe we could implement some solution for TCP (blocking I/O)? Maybe it is possible to
release the lock before socket write?
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
3:54 a.m.
New subject: [JBoss JIRA] Updated: (JGRP-1003) deadlock in TP.send - socket write hangs inside a Lock, preventing other threads to do TP.send()
[
https://jira.jboss.org/jira/browse/JGRP-1003?page=com.atlassian.jira.plug...
]
Victor N updated JGRP-1003:
---------------------------
Attachment: jgroups_s2.txt
deadlock in TP.send - socket write hangs inside a Lock, preventing
other threads to do TP.send()
------------------------------------------------------------------------------------------------
Key: JGRP-1003
URL: https://jira.jboss.org/jira/browse/JGRP-1003
Project: JGroups
Issue Type: Bug
Affects Versions: 2.7
Environment: Linux (Debian), kernel 2.6.x
Reporter: Victor N
Assignee: Bela Ban
Fix For: 2.8
Attachments: jgroups_s2.txt, stacktrace.txt
I am using JGroups 2.7.0 GA with the typical protocols stack based on TCP (taken from
tcp.xml).
Sometimes it occurs that socket write operation hangs inside TP.send() -- this is not so
untypical for blocking I/O approach! -- after that JGroups is not working at all because
most of threads are blocked inside TP.send() waiting for that first thread releasing
"out_stream_lock". Below is a fragment from my stack trace.
1) the thread that hangs in socket write until forever (maybe due to a broken socket or
something else); this thread owns "out_stream_lock in TP class":
"Timer-1,name,IP:port" daemon prio=10 tid=0x084e5c00 nid=0x15f0 runnable
[0x211c1000..0x211c2040]
java.lang.Thread.State: RUNNABLE
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
- locked <0x43427300> (a java.io.BufferedOutputStream)
at java.io.DataOutputStream.flush(DataOutputStream.java:106)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection.doSend(TCPConnectionMap.java:472)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection._send(TCPConnectionMap.java:453)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection.send(TCPConnectionMap.java:436)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection.access$100(TCPConnectionMap.java:341)
at org.jgroups.blocks.TCPConnectionMap.send(TCPConnectionMap.java:137)
at org.jgroups.protocols.TCP.send(TCP.java:53)
at org.jgroups.protocols.BasicTCP.sendToSingleMember(BasicTCP.java:141)
at org.jgroups.protocols.TP.doSend(TP.java:1105)
at org.jgroups.protocols.TP.send(TP.java:1088)
at org.jgroups.protocols.TP.down(TP.java:907)
at org.jgroups.protocols.Discovery.down(Discovery.java:363)
at org.jgroups.protocols.MERGE2.down(MERGE2.java:169)
at org.jgroups.protocols.FD_SOCK.down(FD_SOCK.java:333)
at org.jgroups.protocols.FD.down(FD.java:327)
at org.jgroups.protocols.VERIFY_SUSPECT.down(VERIFY_SUSPECT.java:72)
at org.jgroups.protocols.BARRIER.down(BARRIER.java:96)
at org.jgroups.protocols.pbcast.NAKACK.retransmit(NAKACK.java:1530)
at org.jgroups.protocols.pbcast.NAKACK.retransmit(NAKACK.java:1476)
at org.jgroups.stack.Retransmitter$Task.run(Retransmitter.java:207)
at org.jgroups.util.TimeScheduler$TaskWrapper.run(TimeScheduler.java:218)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:207)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Locked ownable synchronizers:
- <0x2b5522e0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
- <0x2b5550f8> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
- <0x434273f0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
2) and there are many threads like this blocked "on out_stream_lock.lock()"
call inside TP class:
"Connection.Receiver [ip:port - ip:port],name,ip:port" prio=10 tid=0x020dc000
nid=0x1fab waiting on condition [0x1365f000..0x1365ffc0]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x2b5550f8> (a
java.util.concurrent.locks.ReentrantLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:778)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1114)
at
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186)
at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262)
at org.jgroups.protocols.TP.send(TP.java:1082)
at org.jgroups.protocols.TP.down(TP.java:907)
at org.jgroups.protocols.Discovery.up(Discovery.java:274)
at org.jgroups.protocols.TP.passMessageUp(TP.java:995)
at org.jgroups.protocols.TP.access$100(TP.java:52)
at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1451)
at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1427)
at
java.util.concurrent.ThreadPoolExecutor$CallerRunsPolicy.rejectedExecution(ThreadPoolExecutor.java:1738)
at
org.jgroups.util.ShutdownRejectedExecutionHandler.rejectedExecution(ShutdownRejectedExecutionHandler.java:34)
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
at org.jgroups.protocols.TP.dispatchToThreadPool(TP.java:1061)
at org.jgroups.protocols.TP.receive(TP.java:1038)
at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:180)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection$ConnectionPeerReceiver.run(TCPConnectionMap.java:553)
at java.lang.Thread.run(Thread.java:619)
Locked ownable synchronizers:
- None
(Since all threads from JGroups' ThreadPool are locked, you can see
"rejectedExecution" in the last stack trace).
Maybe we could implement some solution for TCP (blocking I/O)? Maybe it is possible to
release the lock before socket write?
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
10:19 a.m.
New subject: [JBoss JIRA] Commented: (JGRP-1003) deadlock in TP.send - socket write hangs inside a Lock, preventing other threads to do TP.send()
[
https://jira.jboss.org/jira/browse/JGRP-1003?page=com.atlassian.jira.plug...
]
Victor N commented on JGRP-1003:
--------------------------------
Forgot so say - I took stack traces 3 time with some intervals, it's interesting that
one of threads (I marked it with ****) is always reading something from network:
"Connection.Sender [server1:40001 - server2:51740],TestCluster,server1:40001"
prio=10 tid=0x08da7c00 nid=0x2a0e waiting on condition [0x1926d000..0x1926de40]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x77389808> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
***** at
org.jgroups.blocks.TCPConnectionMap$TCPConnection$Sender.run(TCPConnectionMap.java:594)
at java.lang.Thread.run(Thread.java:619)
Maybe, this thread is blocked in socket read forever? Is SO_TIMEOUT set here?
deadlock in TP.send - socket write hangs inside a Lock, preventing
other threads to do TP.send()
------------------------------------------------------------------------------------------------
Key: JGRP-1003
URL: https://jira.jboss.org/jira/browse/JGRP-1003
Project: JGroups
Issue Type: Bug
Affects Versions: 2.7
Environment: Linux (Debian), kernel 2.6.x
Reporter: Victor N
Assignee: Bela Ban
Fix For: 2.8
Attachments: jgroups_s2.txt, stacktrace.txt
I am using JGroups 2.7.0 GA with the typical protocols stack based on TCP (taken from
tcp.xml).
Sometimes it occurs that socket write operation hangs inside TP.send() -- this is not so
untypical for blocking I/O approach! -- after that JGroups is not working at all because
most of threads are blocked inside TP.send() waiting for that first thread releasing
"out_stream_lock". Below is a fragment from my stack trace.
1) the thread that hangs in socket write until forever (maybe due to a broken socket or
something else); this thread owns "out_stream_lock in TP class":
"Timer-1,name,IP:port" daemon prio=10 tid=0x084e5c00 nid=0x15f0 runnable
[0x211c1000..0x211c2040]
java.lang.Thread.State: RUNNABLE
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
- locked <0x43427300> (a java.io.BufferedOutputStream)
at java.io.DataOutputStream.flush(DataOutputStream.java:106)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection.doSend(TCPConnectionMap.java:472)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection._send(TCPConnectionMap.java:453)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection.send(TCPConnectionMap.java:436)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection.access$100(TCPConnectionMap.java:341)
at org.jgroups.blocks.TCPConnectionMap.send(TCPConnectionMap.java:137)
at org.jgroups.protocols.TCP.send(TCP.java:53)
at org.jgroups.protocols.BasicTCP.sendToSingleMember(BasicTCP.java:141)
at org.jgroups.protocols.TP.doSend(TP.java:1105)
at org.jgroups.protocols.TP.send(TP.java:1088)
at org.jgroups.protocols.TP.down(TP.java:907)
at org.jgroups.protocols.Discovery.down(Discovery.java:363)
at org.jgroups.protocols.MERGE2.down(MERGE2.java:169)
at org.jgroups.protocols.FD_SOCK.down(FD_SOCK.java:333)
at org.jgroups.protocols.FD.down(FD.java:327)
at org.jgroups.protocols.VERIFY_SUSPECT.down(VERIFY_SUSPECT.java:72)
at org.jgroups.protocols.BARRIER.down(BARRIER.java:96)
at org.jgroups.protocols.pbcast.NAKACK.retransmit(NAKACK.java:1530)
at org.jgroups.protocols.pbcast.NAKACK.retransmit(NAKACK.java:1476)
at org.jgroups.stack.Retransmitter$Task.run(Retransmitter.java:207)
at org.jgroups.util.TimeScheduler$TaskWrapper.run(TimeScheduler.java:218)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:207)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Locked ownable synchronizers:
- <0x2b5522e0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
- <0x2b5550f8> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
- <0x434273f0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
2) and there are many threads like this blocked "on out_stream_lock.lock()"
call inside TP class:
"Connection.Receiver [ip:port - ip:port],name,ip:port" prio=10 tid=0x020dc000
nid=0x1fab waiting on condition [0x1365f000..0x1365ffc0]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x2b5550f8> (a
java.util.concurrent.locks.ReentrantLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:778)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1114)
at
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186)
at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262)
at org.jgroups.protocols.TP.send(TP.java:1082)
at org.jgroups.protocols.TP.down(TP.java:907)
at org.jgroups.protocols.Discovery.up(Discovery.java:274)
at org.jgroups.protocols.TP.passMessageUp(TP.java:995)
at org.jgroups.protocols.TP.access$100(TP.java:52)
at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1451)
at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1427)
at
java.util.concurrent.ThreadPoolExecutor$CallerRunsPolicy.rejectedExecution(ThreadPoolExecutor.java:1738)
at
org.jgroups.util.ShutdownRejectedExecutionHandler.rejectedExecution(ShutdownRejectedExecutionHandler.java:34)
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
at org.jgroups.protocols.TP.dispatchToThreadPool(TP.java:1061)
at org.jgroups.protocols.TP.receive(TP.java:1038)
at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:180)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection$ConnectionPeerReceiver.run(TCPConnectionMap.java:553)
at java.lang.Thread.run(Thread.java:619)
Locked ownable synchronizers:
- None
(Since all threads from JGroups' ThreadPool are locked, you can see
"rejectedExecution" in the last stack trace).
Maybe we could implement some solution for TCP (blocking I/O)? Maybe it is possible to
release the lock before socket write?
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
3:15 a.m.
New subject: [JBoss JIRA] Commented: (JGRP-1003) deadlock in TP.send - socket write hangs inside a Lock, preventing other threads to do TP.send()
[
https://jira.jboss.org/jira/browse/JGRP-1003?page=com.atlassian.jira.plug...
]
Bela Ban commented on JGRP-1003:
--------------------------------
Search for LinkedBlockingQueue:
There are 2 different send queues we're talking about: 2 threads (pool-2-thread-4 and
Timer-3,TestCluster,server1:40001) are blocked on lock <0x93c25718> associated with
one queue, trying to put data into the queue (which is full).
1 thread (Connection.Sender [server1:40001 - server2:51740],TestCluster,server1:40001) is
blocked taking elements from the queue which is locked by <0x77389808> and is
empty.
Case #2 is normal as there is no data for this queue. However, case #1 is weird, as there
should be a sender thread which removes elements from the queue and sends them, so that
the queue becomes not full and the threads which put elements onto it get unblocked.
I changed one catch(Exception) clause to catch(Throwable), maybe the sender thread
terminated through an exception and was therefore unable to remove elements from the send
queue.
I committed the change to CVS head, can you try it out ?
Any stack trace must have a sender thread !
deadlock in TP.send - socket write hangs inside a Lock, preventing
other threads to do TP.send()
------------------------------------------------------------------------------------------------
Key: JGRP-1003
URL: https://jira.jboss.org/jira/browse/JGRP-1003
Project: JGroups
Issue Type: Bug
Affects Versions: 2.7
Environment: Linux (Debian), kernel 2.6.x
Reporter: Victor N
Assignee: Bela Ban
Fix For: 2.8
Attachments: jgroups_s2.txt, stacktrace.txt
I am using JGroups 2.7.0 GA with the typical protocols stack based on TCP (taken from
tcp.xml).
Sometimes it occurs that socket write operation hangs inside TP.send() -- this is not so
untypical for blocking I/O approach! -- after that JGroups is not working at all because
most of threads are blocked inside TP.send() waiting for that first thread releasing
"out_stream_lock". Below is a fragment from my stack trace.
1) the thread that hangs in socket write until forever (maybe due to a broken socket or
something else); this thread owns "out_stream_lock in TP class":
"Timer-1,name,IP:port" daemon prio=10 tid=0x084e5c00 nid=0x15f0 runnable
[0x211c1000..0x211c2040]
java.lang.Thread.State: RUNNABLE
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
- locked <0x43427300> (a java.io.BufferedOutputStream)
at java.io.DataOutputStream.flush(DataOutputStream.java:106)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection.doSend(TCPConnectionMap.java:472)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection._send(TCPConnectionMap.java:453)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection.send(TCPConnectionMap.java:436)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection.access$100(TCPConnectionMap.java:341)
at org.jgroups.blocks.TCPConnectionMap.send(TCPConnectionMap.java:137)
at org.jgroups.protocols.TCP.send(TCP.java:53)
at org.jgroups.protocols.BasicTCP.sendToSingleMember(BasicTCP.java:141)
at org.jgroups.protocols.TP.doSend(TP.java:1105)
at org.jgroups.protocols.TP.send(TP.java:1088)
at org.jgroups.protocols.TP.down(TP.java:907)
at org.jgroups.protocols.Discovery.down(Discovery.java:363)
at org.jgroups.protocols.MERGE2.down(MERGE2.java:169)
at org.jgroups.protocols.FD_SOCK.down(FD_SOCK.java:333)
at org.jgroups.protocols.FD.down(FD.java:327)
at org.jgroups.protocols.VERIFY_SUSPECT.down(VERIFY_SUSPECT.java:72)
at org.jgroups.protocols.BARRIER.down(BARRIER.java:96)
at org.jgroups.protocols.pbcast.NAKACK.retransmit(NAKACK.java:1530)
at org.jgroups.protocols.pbcast.NAKACK.retransmit(NAKACK.java:1476)
at org.jgroups.stack.Retransmitter$Task.run(Retransmitter.java:207)
at org.jgroups.util.TimeScheduler$TaskWrapper.run(TimeScheduler.java:218)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:207)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Locked ownable synchronizers:
- <0x2b5522e0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
- <0x2b5550f8> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
- <0x434273f0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
2) and there are many threads like this blocked "on out_stream_lock.lock()"
call inside TP class:
"Connection.Receiver [ip:port - ip:port],name,ip:port" prio=10 tid=0x020dc000
nid=0x1fab waiting on condition [0x1365f000..0x1365ffc0]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x2b5550f8> (a
java.util.concurrent.locks.ReentrantLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:778)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1114)
at
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186)
at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262)
at org.jgroups.protocols.TP.send(TP.java:1082)
at org.jgroups.protocols.TP.down(TP.java:907)
at org.jgroups.protocols.Discovery.up(Discovery.java:274)
at org.jgroups.protocols.TP.passMessageUp(TP.java:995)
at org.jgroups.protocols.TP.access$100(TP.java:52)
at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1451)
at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1427)
at
java.util.concurrent.ThreadPoolExecutor$CallerRunsPolicy.rejectedExecution(ThreadPoolExecutor.java:1738)
at
org.jgroups.util.ShutdownRejectedExecutionHandler.rejectedExecution(ShutdownRejectedExecutionHandler.java:34)
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
at org.jgroups.protocols.TP.dispatchToThreadPool(TP.java:1061)
at org.jgroups.protocols.TP.receive(TP.java:1038)
at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:180)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection$ConnectionPeerReceiver.run(TCPConnectionMap.java:553)
at java.lang.Thread.run(Thread.java:619)
Locked ownable synchronizers:
- None
(Since all threads from JGroups' ThreadPool are locked, you can see
"rejectedExecution" in the last stack trace).
Maybe we could implement some solution for TCP (blocking I/O)? Maybe it is possible to
release the lock before socket write?
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
1:34 a.m.
New subject: [JBoss JIRA] Resolved: (JGRP-1003) deadlock in TP.send - socket write hangs inside a Lock, preventing other threads to do TP.send()
[
https://jira.jboss.org/jira/browse/JGRP-1003?page=com.atlassian.jira.plug...
]
Bela Ban resolved JGRP-1003.
----------------------------
Resolution: Done
Haven't heard from you regarding the fix I commmitted - does it work ?
Anyway, I'm resolving this issue. If the bug re-occurs, please re-open the issue
(under 2.9 though, as we want to release 2.8.GA soon)
deadlock in TP.send - socket write hangs inside a Lock, preventing
other threads to do TP.send()
------------------------------------------------------------------------------------------------
Key: JGRP-1003
URL: https://jira.jboss.org/jira/browse/JGRP-1003
Project: JGroups
Issue Type: Bug
Affects Versions: 2.7
Environment: Linux (Debian), kernel 2.6.x
Reporter: Victor N
Assignee: Bela Ban
Fix For: 2.8
Attachments: jgroups_s2.txt, stacktrace.txt
I am using JGroups 2.7.0 GA with the typical protocols stack based on TCP (taken from
tcp.xml).
Sometimes it occurs that socket write operation hangs inside TP.send() -- this is not so
untypical for blocking I/O approach! -- after that JGroups is not working at all because
most of threads are blocked inside TP.send() waiting for that first thread releasing
"out_stream_lock". Below is a fragment from my stack trace.
1) the thread that hangs in socket write until forever (maybe due to a broken socket or
something else); this thread owns "out_stream_lock in TP class":
"Timer-1,name,IP:port" daemon prio=10 tid=0x084e5c00 nid=0x15f0 runnable
[0x211c1000..0x211c2040]
java.lang.Thread.State: RUNNABLE
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
- locked <0x43427300> (a java.io.BufferedOutputStream)
at java.io.DataOutputStream.flush(DataOutputStream.java:106)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection.doSend(TCPConnectionMap.java:472)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection._send(TCPConnectionMap.java:453)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection.send(TCPConnectionMap.java:436)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection.access$100(TCPConnectionMap.java:341)
at org.jgroups.blocks.TCPConnectionMap.send(TCPConnectionMap.java:137)
at org.jgroups.protocols.TCP.send(TCP.java:53)
at org.jgroups.protocols.BasicTCP.sendToSingleMember(BasicTCP.java:141)
at org.jgroups.protocols.TP.doSend(TP.java:1105)
at org.jgroups.protocols.TP.send(TP.java:1088)
at org.jgroups.protocols.TP.down(TP.java:907)
at org.jgroups.protocols.Discovery.down(Discovery.java:363)
at org.jgroups.protocols.MERGE2.down(MERGE2.java:169)
at org.jgroups.protocols.FD_SOCK.down(FD_SOCK.java:333)
at org.jgroups.protocols.FD.down(FD.java:327)
at org.jgroups.protocols.VERIFY_SUSPECT.down(VERIFY_SUSPECT.java:72)
at org.jgroups.protocols.BARRIER.down(BARRIER.java:96)
at org.jgroups.protocols.pbcast.NAKACK.retransmit(NAKACK.java:1530)
at org.jgroups.protocols.pbcast.NAKACK.retransmit(NAKACK.java:1476)
at org.jgroups.stack.Retransmitter$Task.run(Retransmitter.java:207)
at org.jgroups.util.TimeScheduler$TaskWrapper.run(TimeScheduler.java:218)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:207)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Locked ownable synchronizers:
- <0x2b5522e0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
- <0x2b5550f8> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
- <0x434273f0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
2) and there are many threads like this blocked "on out_stream_lock.lock()"
call inside TP class:
"Connection.Receiver [ip:port - ip:port],name,ip:port" prio=10 tid=0x020dc000
nid=0x1fab waiting on condition [0x1365f000..0x1365ffc0]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x2b5550f8> (a
java.util.concurrent.locks.ReentrantLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:778)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1114)
at
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186)
at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262)
at org.jgroups.protocols.TP.send(TP.java:1082)
at org.jgroups.protocols.TP.down(TP.java:907)
at org.jgroups.protocols.Discovery.up(Discovery.java:274)
at org.jgroups.protocols.TP.passMessageUp(TP.java:995)
at org.jgroups.protocols.TP.access$100(TP.java:52)
at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1451)
at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1427)
at
java.util.concurrent.ThreadPoolExecutor$CallerRunsPolicy.rejectedExecution(ThreadPoolExecutor.java:1738)
at
org.jgroups.util.ShutdownRejectedExecutionHandler.rejectedExecution(ShutdownRejectedExecutionHandler.java:34)
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
at org.jgroups.protocols.TP.dispatchToThreadPool(TP.java:1061)
at org.jgroups.protocols.TP.receive(TP.java:1038)
at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:180)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection$ConnectionPeerReceiver.run(TCPConnectionMap.java:553)
at java.lang.Thread.run(Thread.java:619)
Locked ownable synchronizers:
- None
(Since all threads from JGroups' ThreadPool are locked, you can see
"rejectedExecution" in the last stack trace).
Maybe we could implement some solution for TCP (blocking I/O)? Maybe it is possible to
release the lock before socket write?
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
3:41 a.m.
New subject: [JBoss JIRA] Commented: (JGRP-1003) deadlock in TP.send - socket write hangs inside a Lock, preventing other threads to do TP.send()
[
https://jira.jboss.org/jira/browse/JGRP-1003?page=com.atlassian.jira.plug...
]
Victor N commented on JGRP-1003:
--------------------------------
Bela, I think I will test it in 2.8.GA - I am going to play with both TCP & TCP_NIO.
deadlock in TP.send - socket write hangs inside a Lock, preventing
other threads to do TP.send()
------------------------------------------------------------------------------------------------
Key: JGRP-1003
URL: https://jira.jboss.org/jira/browse/JGRP-1003
Project: JGroups
Issue Type: Bug
Affects Versions: 2.7
Environment: Linux (Debian), kernel 2.6.x
Reporter: Victor N
Assignee: Bela Ban
Fix For: 2.8
Attachments: jgroups_s2.txt, stacktrace.txt
I am using JGroups 2.7.0 GA with the typical protocols stack based on TCP (taken from
tcp.xml).
Sometimes it occurs that socket write operation hangs inside TP.send() -- this is not so
untypical for blocking I/O approach! -- after that JGroups is not working at all because
most of threads are blocked inside TP.send() waiting for that first thread releasing
"out_stream_lock". Below is a fragment from my stack trace.
1) the thread that hangs in socket write until forever (maybe due to a broken socket or
something else); this thread owns "out_stream_lock in TP class":
"Timer-1,name,IP:port" daemon prio=10 tid=0x084e5c00 nid=0x15f0 runnable
[0x211c1000..0x211c2040]
java.lang.Thread.State: RUNNABLE
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
- locked <0x43427300> (a java.io.BufferedOutputStream)
at java.io.DataOutputStream.flush(DataOutputStream.java:106)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection.doSend(TCPConnectionMap.java:472)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection._send(TCPConnectionMap.java:453)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection.send(TCPConnectionMap.java:436)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection.access$100(TCPConnectionMap.java:341)
at org.jgroups.blocks.TCPConnectionMap.send(TCPConnectionMap.java:137)
at org.jgroups.protocols.TCP.send(TCP.java:53)
at org.jgroups.protocols.BasicTCP.sendToSingleMember(BasicTCP.java:141)
at org.jgroups.protocols.TP.doSend(TP.java:1105)
at org.jgroups.protocols.TP.send(TP.java:1088)
at org.jgroups.protocols.TP.down(TP.java:907)
at org.jgroups.protocols.Discovery.down(Discovery.java:363)
at org.jgroups.protocols.MERGE2.down(MERGE2.java:169)
at org.jgroups.protocols.FD_SOCK.down(FD_SOCK.java:333)
at org.jgroups.protocols.FD.down(FD.java:327)
at org.jgroups.protocols.VERIFY_SUSPECT.down(VERIFY_SUSPECT.java:72)
at org.jgroups.protocols.BARRIER.down(BARRIER.java:96)
at org.jgroups.protocols.pbcast.NAKACK.retransmit(NAKACK.java:1530)
at org.jgroups.protocols.pbcast.NAKACK.retransmit(NAKACK.java:1476)
at org.jgroups.stack.Retransmitter$Task.run(Retransmitter.java:207)
at org.jgroups.util.TimeScheduler$TaskWrapper.run(TimeScheduler.java:218)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:207)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Locked ownable synchronizers:
- <0x2b5522e0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
- <0x2b5550f8> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
- <0x434273f0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
2) and there are many threads like this blocked "on out_stream_lock.lock()"
call inside TP class:
"Connection.Receiver [ip:port - ip:port],name,ip:port" prio=10 tid=0x020dc000
nid=0x1fab waiting on condition [0x1365f000..0x1365ffc0]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x2b5550f8> (a
java.util.concurrent.locks.ReentrantLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:778)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1114)
at
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186)
at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262)
at org.jgroups.protocols.TP.send(TP.java:1082)
at org.jgroups.protocols.TP.down(TP.java:907)
at org.jgroups.protocols.Discovery.up(Discovery.java:274)
at org.jgroups.protocols.TP.passMessageUp(TP.java:995)
at org.jgroups.protocols.TP.access$100(TP.java:52)
at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1451)
at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1427)
at
java.util.concurrent.ThreadPoolExecutor$CallerRunsPolicy.rejectedExecution(ThreadPoolExecutor.java:1738)
at
org.jgroups.util.ShutdownRejectedExecutionHandler.rejectedExecution(ShutdownRejectedExecutionHandler.java:34)
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
at org.jgroups.protocols.TP.dispatchToThreadPool(TP.java:1061)
at org.jgroups.protocols.TP.receive(TP.java:1038)
at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:180)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection$ConnectionPeerReceiver.run(TCPConnectionMap.java:553)
at java.lang.Thread.run(Thread.java:619)
Locked ownable synchronizers:
- None
(Since all threads from JGroups' ThreadPool are locked, you can see
"rejectedExecution" in the last stack trace).
Maybe we could implement some solution for TCP (blocking I/O)? Maybe it is possible to
release the lock before socket write?
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
5:44 a.m.
New subject: [JBoss JIRA] Commented: (JGRP-1003) deadlock in TP.send - socket write hangs inside a Lock, preventing other threads to do TP.send()
[
https://jira.jboss.org/jira/browse/JGRP-1003?page=com.atlassian.jira.plug...
]
Victor N commented on JGRP-1003:
--------------------------------
Bela, I can confirm that "Connection.Sender" thread disappears when the bug is
reproduced.
Sorry, I did not test with CVS head nor 2.8 yet.
deadlock in TP.send - socket write hangs inside a Lock, preventing
other threads to do TP.send()
------------------------------------------------------------------------------------------------
Key: JGRP-1003
URL: https://jira.jboss.org/jira/browse/JGRP-1003
Project: JGroups
Issue Type: Bug
Affects Versions: 2.7
Environment: Linux (Debian), kernel 2.6.x
Reporter: Victor N
Assignee: Bela Ban
Fix For: 2.8
Attachments: jgroups_s2.txt, stacktrace.txt
I am using JGroups 2.7.0 GA with the typical protocols stack based on TCP (taken from
tcp.xml).
Sometimes it occurs that socket write operation hangs inside TP.send() -- this is not so
untypical for blocking I/O approach! -- after that JGroups is not working at all because
most of threads are blocked inside TP.send() waiting for that first thread releasing
"out_stream_lock". Below is a fragment from my stack trace.
1) the thread that hangs in socket write until forever (maybe due to a broken socket or
something else); this thread owns "out_stream_lock in TP class":
"Timer-1,name,IP:port" daemon prio=10 tid=0x084e5c00 nid=0x15f0 runnable
[0x211c1000..0x211c2040]
java.lang.Thread.State: RUNNABLE
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
- locked <0x43427300> (a java.io.BufferedOutputStream)
at java.io.DataOutputStream.flush(DataOutputStream.java:106)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection.doSend(TCPConnectionMap.java:472)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection._send(TCPConnectionMap.java:453)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection.send(TCPConnectionMap.java:436)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection.access$100(TCPConnectionMap.java:341)
at org.jgroups.blocks.TCPConnectionMap.send(TCPConnectionMap.java:137)
at org.jgroups.protocols.TCP.send(TCP.java:53)
at org.jgroups.protocols.BasicTCP.sendToSingleMember(BasicTCP.java:141)
at org.jgroups.protocols.TP.doSend(TP.java:1105)
at org.jgroups.protocols.TP.send(TP.java:1088)
at org.jgroups.protocols.TP.down(TP.java:907)
at org.jgroups.protocols.Discovery.down(Discovery.java:363)
at org.jgroups.protocols.MERGE2.down(MERGE2.java:169)
at org.jgroups.protocols.FD_SOCK.down(FD_SOCK.java:333)
at org.jgroups.protocols.FD.down(FD.java:327)
at org.jgroups.protocols.VERIFY_SUSPECT.down(VERIFY_SUSPECT.java:72)
at org.jgroups.protocols.BARRIER.down(BARRIER.java:96)
at org.jgroups.protocols.pbcast.NAKACK.retransmit(NAKACK.java:1530)
at org.jgroups.protocols.pbcast.NAKACK.retransmit(NAKACK.java:1476)
at org.jgroups.stack.Retransmitter$Task.run(Retransmitter.java:207)
at org.jgroups.util.TimeScheduler$TaskWrapper.run(TimeScheduler.java:218)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:207)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Locked ownable synchronizers:
- <0x2b5522e0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
- <0x2b5550f8> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
- <0x434273f0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
2) and there are many threads like this blocked "on out_stream_lock.lock()"
call inside TP class:
"Connection.Receiver [ip:port - ip:port],name,ip:port" prio=10 tid=0x020dc000
nid=0x1fab waiting on condition [0x1365f000..0x1365ffc0]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x2b5550f8> (a
java.util.concurrent.locks.ReentrantLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:778)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1114)
at
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186)
at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262)
at org.jgroups.protocols.TP.send(TP.java:1082)
at org.jgroups.protocols.TP.down(TP.java:907)
at org.jgroups.protocols.Discovery.up(Discovery.java:274)
at org.jgroups.protocols.TP.passMessageUp(TP.java:995)
at org.jgroups.protocols.TP.access$100(TP.java:52)
at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1451)
at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1427)
at
java.util.concurrent.ThreadPoolExecutor$CallerRunsPolicy.rejectedExecution(ThreadPoolExecutor.java:1738)
at
org.jgroups.util.ShutdownRejectedExecutionHandler.rejectedExecution(ShutdownRejectedExecutionHandler.java:34)
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
at org.jgroups.protocols.TP.dispatchToThreadPool(TP.java:1061)
at org.jgroups.protocols.TP.receive(TP.java:1038)
at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:180)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection$ConnectionPeerReceiver.run(TCPConnectionMap.java:553)
at java.lang.Thread.run(Thread.java:619)
Locked ownable synchronizers:
- None
(Since all threads from JGroups' ThreadPool are locked, you can see
"rejectedExecution" in the last stack trace).
Maybe we could implement some solution for TCP (blocking I/O)? Maybe it is possible to
release the lock before socket write?
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
5:06 a.m.
New subject: [JBoss JIRA] Reopened: (JGRP-1003) deadlock in TP.send - socket write hangs inside a Lock, preventing other threads to do TP.send()
[
https://jira.jboss.org/jira/browse/JGRP-1003?page=com.atlassian.jira.plug...
]
Victor N reopened JGRP-1003:
----------------------------
Bela, the problem is reproduced in 2.8.0 CR6.
A similar environment as it was before: 8 nodes (named node5 ... node 12).
After some time (~2 days) something went wrong on node5 and I have seen that all my
threads were blocked in JGroups code and can not send my messages.
My config is based on tcp.xml:
-----------------------------------------------------------------
<config xmlns="urn:org:jgroups"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="urn:org:jgroups file:schema/JGroups-2.8.xsd">
<TCP bind_addr="${bindAddress}"
bind_port="40002"
loopback="true"
recv_buf_size="20000000"
send_buf_size="640000"
discard_incompatible_packets="true"
max_bundle_size="64000"
max_bundle_timeout="30"
enable_bundling="true"
use_send_queues="true"
sock_conn_timeout="300"
skip_suspected_members="true"
thread_pool.enabled="true"
thread_pool.min_threads="1"
thread_pool.max_threads="25"
thread_pool.keep_alive_time="5000"
thread_pool.queue_enabled="false"
thread_pool.queue_max_size="10000"
thread_pool.rejection_policy="run"
oob_thread_pool.enabled="true"
oob_thread_pool.min_threads="1"
oob_thread_pool.max_threads="8"
oob_thread_pool.keep_alive_time="5000"
oob_thread_pool.queue_enabled="false"
oob_thread_pool.queue_max_size="1000"
oob_thread_pool.rejection_policy="run"/>
<TCPPING timeout="3000"
initial_hosts="${nodes}"
port_range="1"
num_initial_members="3"/>
<MERGE2 min_interval="10000"
max_interval="30000"/>
<FD_SOCK/>
<FD timeout="10000" max_tries="5" />
<VERIFY_SUSPECT timeout="1500" />
<BARRIER />
<pbcast.NAKACK
use_mcast_xmit="false" gc_lag="0"
retransmit_timeout="300,600,1200,2400,4800"
discard_delivered_msgs="true"/>
<UNICAST timeout="300,600,1200" />
<pbcast.STABLE stability_delay="1000"
desired_avg_gossip="50000"
max_bytes="400000"/>
<pbcast.GMS print_local_addr="true" join_timeout="3000"
view_bundling="true"/>
<FC max_credits="2000000"
min_threshold="0.10"/>
<FRAG2 frag_size="60000" />
<!-- <pbcast.STREAMING_STATE_TRANSFER/> -->
<!-- <pbcast.STATE_TRANSFER/> -->
</config>
-----------------------------------------------------------------
Stack-traces from node5 are attached in j1__.txt file ; I changed IP addresses and names a
little bit (node5 - 192.168.1.5, node6 - 192.168.1.6 ...).
As you can see there are 7 Connection.Sender and 7 Connection.Receiver threads, this is ok
for 8 nodes.
So, about my locked threads... I have 10 threads named "pool-1-thread-XXX", all
of them are locked trying to acquire lock in Bundler:
at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262)
at org.jgroups.protocols.TP$Bundler.send(TP.java:1538)
The only thread which took this lock is "Timer-2,TestCluster,node5" locked
here:
at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:254)
at org.jgroups.blocks.TCPConnectionMap$TCPConnection.send(TCPConnectionMap.java:420)
This thread is locked here because Sender's queue is full.
Since this is the only place where the lock is owned, seems that all my threads are
waiting for the SAME lock instance, and this means they are working on the same Sender.
If you look at all Connection.Sender's, they are all trying to take from queue:
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection$Sender.run(TCPConnectionMap.java:603)
And this stack-trace says that all Senders have empty queues. It is strange: one thread is
waiting until the queue is not full but I do not see any full queue!
The only idea I have is that there are different lock instances, and so, different Sender
instances - maybe Sender thread terminated somehow and was restarted, but all my threads
are working on the old Sender instance. But In do not see any warning about this in my
logs at INFO/WARN/ERROR levels.
Bela, do you have any ideas?
deadlock in TP.send - socket write hangs inside a Lock, preventing
other threads to do TP.send()
------------------------------------------------------------------------------------------------
Key: JGRP-1003
URL: https://jira.jboss.org/jira/browse/JGRP-1003
Project: JGroups
Issue Type: Bug
Affects Versions: 2.7
Environment: Linux (Debian), kernel 2.6.x
Reporter: Victor N
Assignee: Bela Ban
Fix For: 2.8
Attachments: jgroups_s2.txt, stacktrace.txt
I am using JGroups 2.7.0 GA with the typical protocols stack based on TCP (taken from
tcp.xml).
Sometimes it occurs that socket write operation hangs inside TP.send() -- this is not so
untypical for blocking I/O approach! -- after that JGroups is not working at all because
most of threads are blocked inside TP.send() waiting for that first thread releasing
"out_stream_lock". Below is a fragment from my stack trace.
1) the thread that hangs in socket write until forever (maybe due to a broken socket or
something else); this thread owns "out_stream_lock in TP class":
"Timer-1,name,IP:port" daemon prio=10 tid=0x084e5c00 nid=0x15f0 runnable
[0x211c1000..0x211c2040]
java.lang.Thread.State: RUNNABLE
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
- locked <0x43427300> (a java.io.BufferedOutputStream)
at java.io.DataOutputStream.flush(DataOutputStream.java:106)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection.doSend(TCPConnectionMap.java:472)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection._send(TCPConnectionMap.java:453)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection.send(TCPConnectionMap.java:436)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection.access$100(TCPConnectionMap.java:341)
at org.jgroups.blocks.TCPConnectionMap.send(TCPConnectionMap.java:137)
at org.jgroups.protocols.TCP.send(TCP.java:53)
at org.jgroups.protocols.BasicTCP.sendToSingleMember(BasicTCP.java:141)
at org.jgroups.protocols.TP.doSend(TP.java:1105)
at org.jgroups.protocols.TP.send(TP.java:1088)
at org.jgroups.protocols.TP.down(TP.java:907)
at org.jgroups.protocols.Discovery.down(Discovery.java:363)
at org.jgroups.protocols.MERGE2.down(MERGE2.java:169)
at org.jgroups.protocols.FD_SOCK.down(FD_SOCK.java:333)
at org.jgroups.protocols.FD.down(FD.java:327)
at org.jgroups.protocols.VERIFY_SUSPECT.down(VERIFY_SUSPECT.java:72)
at org.jgroups.protocols.BARRIER.down(BARRIER.java:96)
at org.jgroups.protocols.pbcast.NAKACK.retransmit(NAKACK.java:1530)
at org.jgroups.protocols.pbcast.NAKACK.retransmit(NAKACK.java:1476)
at org.jgroups.stack.Retransmitter$Task.run(Retransmitter.java:207)
at org.jgroups.util.TimeScheduler$TaskWrapper.run(TimeScheduler.java:218)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:207)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Locked ownable synchronizers:
- <0x2b5522e0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
- <0x2b5550f8> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
- <0x434273f0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
2) and there are many threads like this blocked "on out_stream_lock.lock()"
call inside TP class:
"Connection.Receiver [ip:port - ip:port],name,ip:port" prio=10 tid=0x020dc000
nid=0x1fab waiting on condition [0x1365f000..0x1365ffc0]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x2b5550f8> (a
java.util.concurrent.locks.ReentrantLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:778)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1114)
at
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186)
at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262)
at org.jgroups.protocols.TP.send(TP.java:1082)
at org.jgroups.protocols.TP.down(TP.java:907)
at org.jgroups.protocols.Discovery.up(Discovery.java:274)
at org.jgroups.protocols.TP.passMessageUp(TP.java:995)
at org.jgroups.protocols.TP.access$100(TP.java:52)
at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1451)
at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1427)
at
java.util.concurrent.ThreadPoolExecutor$CallerRunsPolicy.rejectedExecution(ThreadPoolExecutor.java:1738)
at
org.jgroups.util.ShutdownRejectedExecutionHandler.rejectedExecution(ShutdownRejectedExecutionHandler.java:34)
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
at org.jgroups.protocols.TP.dispatchToThreadPool(TP.java:1061)
at org.jgroups.protocols.TP.receive(TP.java:1038)
at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:180)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection$ConnectionPeerReceiver.run(TCPConnectionMap.java:553)
at java.lang.Thread.run(Thread.java:619)
Locked ownable synchronizers:
- None
(Since all threads from JGroups' ThreadPool are locked, you can see
"rejectedExecution" in the last stack trace).
Maybe we could implement some solution for TCP (blocking I/O)? Maybe it is possible to
release the lock before socket write?
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
5:08 a.m.
New subject: [JBoss JIRA] Updated: (JGRP-1003) deadlock in TP.send - socket write hangs inside a Lock, preventing other threads to do TP.send()
[
https://jira.jboss.org/jira/browse/JGRP-1003?page=com.atlassian.jira.plug...
]
Victor N updated JGRP-1003:
---------------------------
Attachment: j1__.txt
Attaching stack-traces
deadlock in TP.send - socket write hangs inside a Lock, preventing
other threads to do TP.send()
------------------------------------------------------------------------------------------------
Key: JGRP-1003
URL: https://jira.jboss.org/jira/browse/JGRP-1003
Project: JGroups
Issue Type: Bug
Affects Versions: 2.7
Environment: Linux (Debian), kernel 2.6.x
Reporter: Victor N
Assignee: Bela Ban
Fix For: 2.8
Attachments: j1__.txt, jgroups_s2.txt, stacktrace.txt
I am using JGroups 2.7.0 GA with the typical protocols stack based on TCP (taken from
tcp.xml).
Sometimes it occurs that socket write operation hangs inside TP.send() -- this is not so
untypical for blocking I/O approach! -- after that JGroups is not working at all because
most of threads are blocked inside TP.send() waiting for that first thread releasing
"out_stream_lock". Below is a fragment from my stack trace.
1) the thread that hangs in socket write until forever (maybe due to a broken socket or
something else); this thread owns "out_stream_lock in TP class":
"Timer-1,name,IP:port" daemon prio=10 tid=0x084e5c00 nid=0x15f0 runnable
[0x211c1000..0x211c2040]
java.lang.Thread.State: RUNNABLE
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
- locked <0x43427300> (a java.io.BufferedOutputStream)
at java.io.DataOutputStream.flush(DataOutputStream.java:106)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection.doSend(TCPConnectionMap.java:472)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection._send(TCPConnectionMap.java:453)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection.send(TCPConnectionMap.java:436)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection.access$100(TCPConnectionMap.java:341)
at org.jgroups.blocks.TCPConnectionMap.send(TCPConnectionMap.java:137)
at org.jgroups.protocols.TCP.send(TCP.java:53)
at org.jgroups.protocols.BasicTCP.sendToSingleMember(BasicTCP.java:141)
at org.jgroups.protocols.TP.doSend(TP.java:1105)
at org.jgroups.protocols.TP.send(TP.java:1088)
at org.jgroups.protocols.TP.down(TP.java:907)
at org.jgroups.protocols.Discovery.down(Discovery.java:363)
at org.jgroups.protocols.MERGE2.down(MERGE2.java:169)
at org.jgroups.protocols.FD_SOCK.down(FD_SOCK.java:333)
at org.jgroups.protocols.FD.down(FD.java:327)
at org.jgroups.protocols.VERIFY_SUSPECT.down(VERIFY_SUSPECT.java:72)
at org.jgroups.protocols.BARRIER.down(BARRIER.java:96)
at org.jgroups.protocols.pbcast.NAKACK.retransmit(NAKACK.java:1530)
at org.jgroups.protocols.pbcast.NAKACK.retransmit(NAKACK.java:1476)
at org.jgroups.stack.Retransmitter$Task.run(Retransmitter.java:207)
at org.jgroups.util.TimeScheduler$TaskWrapper.run(TimeScheduler.java:218)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:207)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Locked ownable synchronizers:
- <0x2b5522e0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
- <0x2b5550f8> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
- <0x434273f0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
2) and there are many threads like this blocked "on out_stream_lock.lock()"
call inside TP class:
"Connection.Receiver [ip:port - ip:port],name,ip:port" prio=10 tid=0x020dc000
nid=0x1fab waiting on condition [0x1365f000..0x1365ffc0]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x2b5550f8> (a
java.util.concurrent.locks.ReentrantLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:778)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1114)
at
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186)
at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262)
at org.jgroups.protocols.TP.send(TP.java:1082)
at org.jgroups.protocols.TP.down(TP.java:907)
at org.jgroups.protocols.Discovery.up(Discovery.java:274)
at org.jgroups.protocols.TP.passMessageUp(TP.java:995)
at org.jgroups.protocols.TP.access$100(TP.java:52)
at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1451)
at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1427)
at
java.util.concurrent.ThreadPoolExecutor$CallerRunsPolicy.rejectedExecution(ThreadPoolExecutor.java:1738)
at
org.jgroups.util.ShutdownRejectedExecutionHandler.rejectedExecution(ShutdownRejectedExecutionHandler.java:34)
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
at org.jgroups.protocols.TP.dispatchToThreadPool(TP.java:1061)
at org.jgroups.protocols.TP.receive(TP.java:1038)
at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:180)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection$ConnectionPeerReceiver.run(TCPConnectionMap.java:553)
at java.lang.Thread.run(Thread.java:619)
Locked ownable synchronizers:
- None
(Since all threads from JGroups' ThreadPool are locked, you can see
"rejectedExecution" in the last stack trace).
Maybe we could implement some solution for TCP (blocking I/O)? Maybe it is possible to
release the lock before socket write?
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
9:52 a.m.
New subject: [JBoss JIRA] Commented: (JGRP-1003) deadlock in TP.send - socket write hangs inside a Lock, preventing other threads to do TP.send()
[
https://jira.jboss.org/jira/browse/JGRP-1003?page=com.atlassian.jira.plug...
]
Bela Ban commented on JGRP-1003:
--------------------------------
You still have a rejection_policy of "run" As I mentioned earlier, this is not
good and can lead to deadlocks like yours. Please replace it with "discard" in
both pools !
deadlock in TP.send - socket write hangs inside a Lock, preventing
other threads to do TP.send()
------------------------------------------------------------------------------------------------
Key: JGRP-1003
URL: https://jira.jboss.org/jira/browse/JGRP-1003
Project: JGroups
Issue Type: Bug
Affects Versions: 2.7
Environment: Linux (Debian), kernel 2.6.x
Reporter: Victor N
Assignee: Bela Ban
Fix For: 2.8
Attachments: j1__.txt, jgroups_s2.txt, stacktrace.txt
I am using JGroups 2.7.0 GA with the typical protocols stack based on TCP (taken from
tcp.xml).
Sometimes it occurs that socket write operation hangs inside TP.send() -- this is not so
untypical for blocking I/O approach! -- after that JGroups is not working at all because
most of threads are blocked inside TP.send() waiting for that first thread releasing
"out_stream_lock". Below is a fragment from my stack trace.
1) the thread that hangs in socket write until forever (maybe due to a broken socket or
something else); this thread owns "out_stream_lock in TP class":
"Timer-1,name,IP:port" daemon prio=10 tid=0x084e5c00 nid=0x15f0 runnable
[0x211c1000..0x211c2040]
java.lang.Thread.State: RUNNABLE
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
- locked <0x43427300> (a java.io.BufferedOutputStream)
at java.io.DataOutputStream.flush(DataOutputStream.java:106)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection.doSend(TCPConnectionMap.java:472)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection._send(TCPConnectionMap.java:453)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection.send(TCPConnectionMap.java:436)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection.access$100(TCPConnectionMap.java:341)
at org.jgroups.blocks.TCPConnectionMap.send(TCPConnectionMap.java:137)
at org.jgroups.protocols.TCP.send(TCP.java:53)
at org.jgroups.protocols.BasicTCP.sendToSingleMember(BasicTCP.java:141)
at org.jgroups.protocols.TP.doSend(TP.java:1105)
at org.jgroups.protocols.TP.send(TP.java:1088)
at org.jgroups.protocols.TP.down(TP.java:907)
at org.jgroups.protocols.Discovery.down(Discovery.java:363)
at org.jgroups.protocols.MERGE2.down(MERGE2.java:169)
at org.jgroups.protocols.FD_SOCK.down(FD_SOCK.java:333)
at org.jgroups.protocols.FD.down(FD.java:327)
at org.jgroups.protocols.VERIFY_SUSPECT.down(VERIFY_SUSPECT.java:72)
at org.jgroups.protocols.BARRIER.down(BARRIER.java:96)
at org.jgroups.protocols.pbcast.NAKACK.retransmit(NAKACK.java:1530)
at org.jgroups.protocols.pbcast.NAKACK.retransmit(NAKACK.java:1476)
at org.jgroups.stack.Retransmitter$Task.run(Retransmitter.java:207)
at org.jgroups.util.TimeScheduler$TaskWrapper.run(TimeScheduler.java:218)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:207)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Locked ownable synchronizers:
- <0x2b5522e0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
- <0x2b5550f8> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
- <0x434273f0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
2) and there are many threads like this blocked "on out_stream_lock.lock()"
call inside TP class:
"Connection.Receiver [ip:port - ip:port],name,ip:port" prio=10 tid=0x020dc000
nid=0x1fab waiting on condition [0x1365f000..0x1365ffc0]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x2b5550f8> (a
java.util.concurrent.locks.ReentrantLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:778)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1114)
at
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186)
at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262)
at org.jgroups.protocols.TP.send(TP.java:1082)
at org.jgroups.protocols.TP.down(TP.java:907)
at org.jgroups.protocols.Discovery.up(Discovery.java:274)
at org.jgroups.protocols.TP.passMessageUp(TP.java:995)
at org.jgroups.protocols.TP.access$100(TP.java:52)
at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1451)
at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1427)
at
java.util.concurrent.ThreadPoolExecutor$CallerRunsPolicy.rejectedExecution(ThreadPoolExecutor.java:1738)
at
org.jgroups.util.ShutdownRejectedExecutionHandler.rejectedExecution(ShutdownRejectedExecutionHandler.java:34)
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
at org.jgroups.protocols.TP.dispatchToThreadPool(TP.java:1061)
at org.jgroups.protocols.TP.receive(TP.java:1038)
at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:180)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection$ConnectionPeerReceiver.run(TCPConnectionMap.java:553)
at java.lang.Thread.run(Thread.java:619)
Locked ownable synchronizers:
- None
(Since all threads from JGroups' ThreadPool are locked, you can see
"rejectedExecution" in the last stack trace).
Maybe we could implement some solution for TCP (blocking I/O)? Maybe it is possible to
release the lock before socket write?
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
10:28 a.m.
New subject: [JBoss JIRA] Commented: (JGRP-1003) deadlock in TP.send - socket write hangs inside a Lock, preventing other threads to do TP.send()
[
https://jira.jboss.org/jira/browse/JGRP-1003?page=com.atlassian.jira.plug...
]
Victor N commented on JGRP-1003:
--------------------------------
OK, I will try with "discard" tomorrow and test how it works.
Definitely, "threading in JGroups" theme needs a separate book or at least a
good detailed paragraph ;)
Seem that I tried different configs; for now only one, maybe, will work (at least for TCP)
- with use_send_queues=true, queue_enabled=false and rejection_policy=discard.
deadlock in TP.send - socket write hangs inside a Lock, preventing
other threads to do TP.send()
------------------------------------------------------------------------------------------------
Key: JGRP-1003
URL: https://jira.jboss.org/jira/browse/JGRP-1003
Project: JGroups
Issue Type: Bug
Affects Versions: 2.7
Environment: Linux (Debian), kernel 2.6.x
Reporter: Victor N
Assignee: Bela Ban
Fix For: 2.8
Attachments: j1__.txt, jgroups_s2.txt, stacktrace.txt
I am using JGroups 2.7.0 GA with the typical protocols stack based on TCP (taken from
tcp.xml).
Sometimes it occurs that socket write operation hangs inside TP.send() -- this is not so
untypical for blocking I/O approach! -- after that JGroups is not working at all because
most of threads are blocked inside TP.send() waiting for that first thread releasing
"out_stream_lock". Below is a fragment from my stack trace.
1) the thread that hangs in socket write until forever (maybe due to a broken socket or
something else); this thread owns "out_stream_lock in TP class":
"Timer-1,name,IP:port" daemon prio=10 tid=0x084e5c00 nid=0x15f0 runnable
[0x211c1000..0x211c2040]
java.lang.Thread.State: RUNNABLE
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
- locked <0x43427300> (a java.io.BufferedOutputStream)
at java.io.DataOutputStream.flush(DataOutputStream.java:106)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection.doSend(TCPConnectionMap.java:472)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection._send(TCPConnectionMap.java:453)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection.send(TCPConnectionMap.java:436)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection.access$100(TCPConnectionMap.java:341)
at org.jgroups.blocks.TCPConnectionMap.send(TCPConnectionMap.java:137)
at org.jgroups.protocols.TCP.send(TCP.java:53)
at org.jgroups.protocols.BasicTCP.sendToSingleMember(BasicTCP.java:141)
at org.jgroups.protocols.TP.doSend(TP.java:1105)
at org.jgroups.protocols.TP.send(TP.java:1088)
at org.jgroups.protocols.TP.down(TP.java:907)
at org.jgroups.protocols.Discovery.down(Discovery.java:363)
at org.jgroups.protocols.MERGE2.down(MERGE2.java:169)
at org.jgroups.protocols.FD_SOCK.down(FD_SOCK.java:333)
at org.jgroups.protocols.FD.down(FD.java:327)
at org.jgroups.protocols.VERIFY_SUSPECT.down(VERIFY_SUSPECT.java:72)
at org.jgroups.protocols.BARRIER.down(BARRIER.java:96)
at org.jgroups.protocols.pbcast.NAKACK.retransmit(NAKACK.java:1530)
at org.jgroups.protocols.pbcast.NAKACK.retransmit(NAKACK.java:1476)
at org.jgroups.stack.Retransmitter$Task.run(Retransmitter.java:207)
at org.jgroups.util.TimeScheduler$TaskWrapper.run(TimeScheduler.java:218)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:207)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Locked ownable synchronizers:
- <0x2b5522e0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
- <0x2b5550f8> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
- <0x434273f0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
2) and there are many threads like this blocked "on out_stream_lock.lock()"
call inside TP class:
"Connection.Receiver [ip:port - ip:port],name,ip:port" prio=10 tid=0x020dc000
nid=0x1fab waiting on condition [0x1365f000..0x1365ffc0]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x2b5550f8> (a
java.util.concurrent.locks.ReentrantLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:778)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1114)
at
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186)
at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262)
at org.jgroups.protocols.TP.send(TP.java:1082)
at org.jgroups.protocols.TP.down(TP.java:907)
at org.jgroups.protocols.Discovery.up(Discovery.java:274)
at org.jgroups.protocols.TP.passMessageUp(TP.java:995)
at org.jgroups.protocols.TP.access$100(TP.java:52)
at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1451)
at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1427)
at
java.util.concurrent.ThreadPoolExecutor$CallerRunsPolicy.rejectedExecution(ThreadPoolExecutor.java:1738)
at
org.jgroups.util.ShutdownRejectedExecutionHandler.rejectedExecution(ShutdownRejectedExecutionHandler.java:34)
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
at org.jgroups.protocols.TP.dispatchToThreadPool(TP.java:1061)
at org.jgroups.protocols.TP.receive(TP.java:1038)
at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:180)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection$ConnectionPeerReceiver.run(TCPConnectionMap.java:553)
at java.lang.Thread.run(Thread.java:619)
Locked ownable synchronizers:
- None
(Since all threads from JGroups' ThreadPool are locked, you can see
"rejectedExecution" in the last stack trace).
Maybe we could implement some solution for TCP (blocking I/O)? Maybe it is possible to
release the lock before socket write?
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
9:42 a.m.
New subject: [JBoss JIRA] Closed: (JGRP-1003) deadlock in TP.send - socket write hangs inside a Lock, preventing other threads to do TP.send()
[
https://jira.jboss.org/jira/browse/JGRP-1003?page=com.atlassian.jira.plug...
]
Bela Ban closed JGRP-1003.
--------------------------
Resolution: Done
deadlock in TP.send - socket write hangs inside a Lock, preventing
other threads to do TP.send()
------------------------------------------------------------------------------------------------
Key: JGRP-1003
URL: https://jira.jboss.org/jira/browse/JGRP-1003
Project: JGroups
Issue Type: Bug
Affects Versions: 2.7
Environment: Linux (Debian), kernel 2.6.x
Reporter: Victor N
Assignee: Bela Ban
Fix For: 2.8
Attachments: j1__.txt, jgroups_s2.txt, stacktrace.txt
I am using JGroups 2.7.0 GA with the typical protocols stack based on TCP (taken from
tcp.xml).
Sometimes it occurs that socket write operation hangs inside TP.send() -- this is not so
untypical for blocking I/O approach! -- after that JGroups is not working at all because
most of threads are blocked inside TP.send() waiting for that first thread releasing
"out_stream_lock". Below is a fragment from my stack trace.
1) the thread that hangs in socket write until forever (maybe due to a broken socket or
something else); this thread owns "out_stream_lock in TP class":
"Timer-1,name,IP:port" daemon prio=10 tid=0x084e5c00 nid=0x15f0 runnable
[0x211c1000..0x211c2040]
java.lang.Thread.State: RUNNABLE
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
- locked <0x43427300> (a java.io.BufferedOutputStream)
at java.io.DataOutputStream.flush(DataOutputStream.java:106)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection.doSend(TCPConnectionMap.java:472)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection._send(TCPConnectionMap.java:453)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection.send(TCPConnectionMap.java:436)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection.access$100(TCPConnectionMap.java:341)
at org.jgroups.blocks.TCPConnectionMap.send(TCPConnectionMap.java:137)
at org.jgroups.protocols.TCP.send(TCP.java:53)
at org.jgroups.protocols.BasicTCP.sendToSingleMember(BasicTCP.java:141)
at org.jgroups.protocols.TP.doSend(TP.java:1105)
at org.jgroups.protocols.TP.send(TP.java:1088)
at org.jgroups.protocols.TP.down(TP.java:907)
at org.jgroups.protocols.Discovery.down(Discovery.java:363)
at org.jgroups.protocols.MERGE2.down(MERGE2.java:169)
at org.jgroups.protocols.FD_SOCK.down(FD_SOCK.java:333)
at org.jgroups.protocols.FD.down(FD.java:327)
at org.jgroups.protocols.VERIFY_SUSPECT.down(VERIFY_SUSPECT.java:72)
at org.jgroups.protocols.BARRIER.down(BARRIER.java:96)
at org.jgroups.protocols.pbcast.NAKACK.retransmit(NAKACK.java:1530)
at org.jgroups.protocols.pbcast.NAKACK.retransmit(NAKACK.java:1476)
at org.jgroups.stack.Retransmitter$Task.run(Retransmitter.java:207)
at org.jgroups.util.TimeScheduler$TaskWrapper.run(TimeScheduler.java:218)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:207)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Locked ownable synchronizers:
- <0x2b5522e0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
- <0x2b5550f8> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
- <0x434273f0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
2) and there are many threads like this blocked "on out_stream_lock.lock()"
call inside TP class:
"Connection.Receiver [ip:port - ip:port],name,ip:port" prio=10 tid=0x020dc000
nid=0x1fab waiting on condition [0x1365f000..0x1365ffc0]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x2b5550f8> (a
java.util.concurrent.locks.ReentrantLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:778)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1114)
at
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186)
at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262)
at org.jgroups.protocols.TP.send(TP.java:1082)
at org.jgroups.protocols.TP.down(TP.java:907)
at org.jgroups.protocols.Discovery.up(Discovery.java:274)
at org.jgroups.protocols.TP.passMessageUp(TP.java:995)
at org.jgroups.protocols.TP.access$100(TP.java:52)
at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1451)
at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1427)
at
java.util.concurrent.ThreadPoolExecutor$CallerRunsPolicy.rejectedExecution(ThreadPoolExecutor.java:1738)
at
org.jgroups.util.ShutdownRejectedExecutionHandler.rejectedExecution(ShutdownRejectedExecutionHandler.java:34)
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
at org.jgroups.protocols.TP.dispatchToThreadPool(TP.java:1061)
at org.jgroups.protocols.TP.receive(TP.java:1038)
at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:180)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection$ConnectionPeerReceiver.run(TCPConnectionMap.java:553)
at java.lang.Thread.run(Thread.java:619)
Locked ownable synchronizers:
- None
(Since all threads from JGroups' ThreadPool are locked, you can see
"rejectedExecution" in the last stack trace).
Maybe we could implement some solution for TCP (blocking I/O)? Maybe it is possible to
release the lock before socket write?
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
4:37 a.m.
New subject: [JBoss JIRA] Commented: (JGRP-1003) deadlock in TP.send - socket write hangs inside a Lock, preventing other threads to do TP.send()
[
https://jira.jboss.org/jira/browse/JGRP-1003?page=com.atlassian.jira.plug...
]
Victor N commented on JGRP-1003:
--------------------------------
I have changed to "discard" - it seems to work during ~ 1 week!
deadlock in TP.send - socket write hangs inside a Lock, preventing
other threads to do TP.send()
------------------------------------------------------------------------------------------------
Key: JGRP-1003
URL: https://jira.jboss.org/jira/browse/JGRP-1003
Project: JGroups
Issue Type: Bug
Affects Versions: 2.7
Environment: Linux (Debian), kernel 2.6.x
Reporter: Victor N
Assignee: Bela Ban
Fix For: 2.8
Attachments: j1__.txt, jgroups_s2.txt, stacktrace.txt
I am using JGroups 2.7.0 GA with the typical protocols stack based on TCP (taken from
tcp.xml).
Sometimes it occurs that socket write operation hangs inside TP.send() -- this is not so
untypical for blocking I/O approach! -- after that JGroups is not working at all because
most of threads are blocked inside TP.send() waiting for that first thread releasing
"out_stream_lock". Below is a fragment from my stack trace.
1) the thread that hangs in socket write until forever (maybe due to a broken socket or
something else); this thread owns "out_stream_lock in TP class":
"Timer-1,name,IP:port" daemon prio=10 tid=0x084e5c00 nid=0x15f0 runnable
[0x211c1000..0x211c2040]
java.lang.Thread.State: RUNNABLE
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
- locked <0x43427300> (a java.io.BufferedOutputStream)
at java.io.DataOutputStream.flush(DataOutputStream.java:106)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection.doSend(TCPConnectionMap.java:472)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection._send(TCPConnectionMap.java:453)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection.send(TCPConnectionMap.java:436)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection.access$100(TCPConnectionMap.java:341)
at org.jgroups.blocks.TCPConnectionMap.send(TCPConnectionMap.java:137)
at org.jgroups.protocols.TCP.send(TCP.java:53)
at org.jgroups.protocols.BasicTCP.sendToSingleMember(BasicTCP.java:141)
at org.jgroups.protocols.TP.doSend(TP.java:1105)
at org.jgroups.protocols.TP.send(TP.java:1088)
at org.jgroups.protocols.TP.down(TP.java:907)
at org.jgroups.protocols.Discovery.down(Discovery.java:363)
at org.jgroups.protocols.MERGE2.down(MERGE2.java:169)
at org.jgroups.protocols.FD_SOCK.down(FD_SOCK.java:333)
at org.jgroups.protocols.FD.down(FD.java:327)
at org.jgroups.protocols.VERIFY_SUSPECT.down(VERIFY_SUSPECT.java:72)
at org.jgroups.protocols.BARRIER.down(BARRIER.java:96)
at org.jgroups.protocols.pbcast.NAKACK.retransmit(NAKACK.java:1530)
at org.jgroups.protocols.pbcast.NAKACK.retransmit(NAKACK.java:1476)
at org.jgroups.stack.Retransmitter$Task.run(Retransmitter.java:207)
at org.jgroups.util.TimeScheduler$TaskWrapper.run(TimeScheduler.java:218)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:207)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Locked ownable synchronizers:
- <0x2b5522e0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
- <0x2b5550f8> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
- <0x434273f0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
2) and there are many threads like this blocked "on out_stream_lock.lock()"
call inside TP class:
"Connection.Receiver [ip:port - ip:port],name,ip:port" prio=10 tid=0x020dc000
nid=0x1fab waiting on condition [0x1365f000..0x1365ffc0]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x2b5550f8> (a
java.util.concurrent.locks.ReentrantLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:778)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1114)
at
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186)
at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262)
at org.jgroups.protocols.TP.send(TP.java:1082)
at org.jgroups.protocols.TP.down(TP.java:907)
at org.jgroups.protocols.Discovery.up(Discovery.java:274)
at org.jgroups.protocols.TP.passMessageUp(TP.java:995)
at org.jgroups.protocols.TP.access$100(TP.java:52)
at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1451)
at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1427)
at
java.util.concurrent.ThreadPoolExecutor$CallerRunsPolicy.rejectedExecution(ThreadPoolExecutor.java:1738)
at
org.jgroups.util.ShutdownRejectedExecutionHandler.rejectedExecution(ShutdownRejectedExecutionHandler.java:34)
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
at org.jgroups.protocols.TP.dispatchToThreadPool(TP.java:1061)
at org.jgroups.protocols.TP.receive(TP.java:1038)
at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:180)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection$ConnectionPeerReceiver.run(TCPConnectionMap.java:553)
at java.lang.Thread.run(Thread.java:619)
Locked ownable synchronizers:
- None
(Since all threads from JGroups' ThreadPool are locked, you can see
"rejectedExecution" in the last stack trace).
Maybe we could implement some solution for TCP (blocking I/O)? Maybe it is possible to
release the lock before socket write?
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
3:46 a.m.
New subject: [JBoss JIRA] Reopened: (JGRP-1003) deadlock in TP.send - socket write hangs inside a Lock, preventing other threads to do TP.send()
[
https://jira.jboss.org/jira/browse/JGRP-1003?page=com.atlassian.jira.plug...
]
Victor N reopened JGRP-1003:
----------------------------
Bela, the problem was reproduced with 2.8.0.CR6 and the following config:
<config xmlns="urn:org:jgroups"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="urn:org:jgroups file:schema/JGroups-2.8.xsd">
<TCP bind_addr="${bindAddress}"
bind_port="40001"
loopback="true"
recv_buf_size="20000000"
send_buf_size="640000"
discard_incompatible_packets="true"
max_bundle_size="64000"
max_bundle_timeout="30"
enable_bundling="true"
use_send_queues="true"
sock_conn_timeout="300"
skip_suspected_members="true"
thread_pool.enabled="true"
thread_pool.min_threads="1"
thread_pool.max_threads="25"
thread_pool.keep_alive_time="5000"
thread_pool.queue_enabled="false"
thread_pool.queue_max_size="10000"
thread_pool.rejection_policy="discard"
oob_thread_pool.enabled="true"
oob_thread_pool.min_threads="1"
oob_thread_pool.max_threads="8"
oob_thread_pool.keep_alive_time="5000"
oob_thread_pool.queue_enabled="false"
oob_thread_pool.queue_max_size="1000"
oob_thread_pool.rejection_policy="discard"/>
<TCPPING timeout="3000"
initial_hosts="${nodes}"
port_range="1"
num_initial_members="3"/>
<MERGE2 min_interval="10000"
max_interval="30000"/>
<FD_SOCK/>
<FD timeout="10000" max_tries="5" />
<VERIFY_SUSPECT timeout="1500" />
<BARRIER />
<pbcast.NAKACK
use_mcast_xmit="false" gc_lag="0"
retransmit_timeout="300,600,1200,2400,4800"
discard_delivered_msgs="true"/>
<UNICAST timeout="300,600,1200" />
<pbcast.STABLE stability_delay="1000"
desired_avg_gossip="50000"
max_bytes="400000"/>
<pbcast.GMS print_local_addr="true" join_timeout="3000"
view_bundling="true"/>
<FC max_credits="2000000"
min_threshold="0.10"/>
<FRAG2 frag_size="60000" />
</config>
I have 10 nodes in cluster: server3 ... server12
The problem is still the same - please look at new stack traces
(jgroups_stacktrace_09feb2010.txt). Threads are blocked on a full queue, so jgroups can
not send messages (incl. MERGE). This stacktrace was taken from "server8" - all
other nodes were working perfectly excepting server8.
From log file at server8:
08/02/2010 20:23:54| ERROR | org.jgroups.protocols.TP.down(): failed sending message to
server10 (182 bytes): java.net.SocketTimeoutException: connect timed out
08/02/2010 20:23:56| WARN | org.jgroups.protocols.TP.sendToSingleMember(): server8: no
physical address for server10, dropping message
08/02/2010 20:24:06| WARN | org.jgroups.protocols.pbcast.GMS.installView(): server8: not
member of view [server3|460] [server3, server6, server5, server10, server11, server9,
server12, server7, server4]; discarding it
08/02/2010 20:41:03| WARN | org.jgroups.protocols.pbcast.GMS.installView(): server8: not
member of view [server3|462] [server3, server6, server5, server10, server11, server9,
server12, server7, server4]; discarding it
08/02/2010 20:55:08| WARN | org.jgroups.protocols.pbcast.GMS.installView(): server8: not
member of view [server3|464] [server3, server6, server5, server10, server11, server9,
server12, server7, server4]; discarding it
08/02/2010 20:55:22| WARN | org.jgroups.protocols.pbcast.GMS.castViewChangeWithDest():
server8: failed to collect all ACKs (expected=10) for view MergeView::[server3|465]
[server3, server6, server5, server8, server10, server11, server9, server12, server7,
server4], subgroups=[[server3|464] [server3, server6, server5, server10, server11,
server9, server12, server7, server4], [server3|463] [server8]] after 2000ms, missing ACKs
from [server10, server9]
08/02/2010 21:10:44| WARN | org.jgroups.protocols.pbcast.GMS.castViewChangeWithDest():
server8: failed to collect all ACKs (expected=10) for view MergeView::[server3|467]
[server3, server6, server5, server8, server10, server11, server9, server12, server7,
server4], subgroups=[[server3|466] [server3, server6, server5, server10, server11,
server9, server12, server7, server4], [server3|465] [server8]] after 2000ms, missing ACKs
from [server3, server6, server5, server10, server11, server9, server12, server7, server4]
08/02/2010 21:26:30| WARN | org.jgroups.protocols.pbcast.Merger.handleMergeRequest():
server8: merge is already in progress
08/02/2010 21:26:55| WARN | org.jgroups.protocols.pbcast.Merger.handleMergeRequest():
server8: merge is already in progress
...(again and again - merge is already in progress)...
at the same time at server3 (it was coordinator) - system clock here differs a little bit,
I checked:
08/02/2010 20:55:38| WARN | org.jgroups.protocols.pbcast.NAKACK.handleMessage(): server3:
dropped message from server8 (not in retransmission table), members are [server11,
server10, server7, server12, server3, server5, server9, server6, server4],
view=[server3|464] [server3, server6, server5, server10, server11, server9, server12,
server7, server4]
08/02/2010 21:11:01| WARN | org.jgroups.protocols.pbcast.NAKACK.handleMessage(): server3:
dropped message from server8 (not in retransmission table), members are [server11,
server10, server7, server12, server3, server5, server9, server6, server4],
view=[server3|466] [server3, server6, server5, server10, server11, server9, server12,
server7, server4]
08/02/2010 21:26:23| WARN | org.jgroups.protocols.pbcast.NAKACK.handleMessage(): server3:
dropped message from server8 (not in retransmission table), members are [server11,
server10, server7, server12, server3, server5, server9, server6, server4],
view=[server3|468] [server3, server6, server5, server10, server11, server9, server12,
server7, server4]
08/02/2010 21:26:28| WARN | org.jgroups.protocols.pbcast.Merger$MergeTask.run(): server3:
merge leader did not get data from all partition coordinators [server8, server3], merge is
cancelled
08/02/2010 21:27:15| ERROR | org.jgroups.protocols.pbcast.Merger.handleMergeResponse():
server3: this.merge_id (server3::21) is different from merge_id (server3::19)
08/02/2010 21:27:19| WARN | org.jgroups.protocols.pbcast.Merger$MergeTask.run(): server3:
merge leader did not get data from all partition coordinators [server8, server3], merge is
cancelled
08/02/2010 21:27:45| WARN | org.jgroups.protocols.pbcast.Merger$MergeTask.run(): server3:
merge leader did not get data from all partition coordinators [server8, server3], merge is
cancelled
08/02/2010 21:28:31| ERROR | org.jgroups.protocols.pbcast.Merger.handleMergeResponse():
server3: this.merge_id (server3::24) is different from merge_id (server3::20)
and at server5 (for comparison):
08/02/2010 20:55:16| WARN | org.jgroups.protocols.pbcast.NAKACK.handleMessage(): server5:
dropped message from server8 (not in retransmission table), members are [server11,
server10, server7, server12, server3, server5, server9, server6, server4],
view=[server3|464] [server3, server6, server5, server10, server11, server9, server12,
server7, server4]
08/02/2010 21:10:38| WARN | org.jgroups.protocols.pbcast.NAKACK.handleMessage(): server5:
dropped message from server8 (not in retransmission table), members are [server11,
server10, server7, server12, server3, server5, server9, server6, server4],
view=[server3|466] [server3, server6, server5, server10, server11, server9, server12,
server7, server4]
08/02/2010 21:26:01| WARN | org.jgroups.protocols.pbcast.NAKACK.handleMessage(): server5:
dropped message from server8 (not in retransmission table), members are [server11,
server10, server7, server12, server3, server5, server9, server6, server4],
view=[server3|468] [server3, server6, server5, server10, server11, server9, server12,
server7, server4]
deadlock in TP.send - socket write hangs inside a Lock, preventing
other threads to do TP.send()
------------------------------------------------------------------------------------------------
Key: JGRP-1003
URL: https://jira.jboss.org/jira/browse/JGRP-1003
Project: JGroups
Issue Type: Bug
Affects Versions: 2.7
Environment: Linux (Debian), kernel 2.6.x
Reporter: Victor N
Assignee: Bela Ban
Fix For: 2.8
Attachments: j1__.txt, jgroups_s2.txt, jgroups_stacktrace_09feb2010.txt,
stacktrace.txt
I am using JGroups 2.7.0 GA with the typical protocols stack based on TCP (taken from
tcp.xml).
Sometimes it occurs that socket write operation hangs inside TP.send() -- this is not so
untypical for blocking I/O approach! -- after that JGroups is not working at all because
most of threads are blocked inside TP.send() waiting for that first thread releasing
"out_stream_lock". Below is a fragment from my stack trace.
1) the thread that hangs in socket write until forever (maybe due to a broken socket or
something else); this thread owns "out_stream_lock in TP class":
"Timer-1,name,IP:port" daemon prio=10 tid=0x084e5c00 nid=0x15f0 runnable
[0x211c1000..0x211c2040]
java.lang.Thread.State: RUNNABLE
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
- locked <0x43427300> (a java.io.BufferedOutputStream)
at java.io.DataOutputStream.flush(DataOutputStream.java:106)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection.doSend(TCPConnectionMap.java:472)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection._send(TCPConnectionMap.java:453)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection.send(TCPConnectionMap.java:436)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection.access$100(TCPConnectionMap.java:341)
at org.jgroups.blocks.TCPConnectionMap.send(TCPConnectionMap.java:137)
at org.jgroups.protocols.TCP.send(TCP.java:53)
at org.jgroups.protocols.BasicTCP.sendToSingleMember(BasicTCP.java:141)
at org.jgroups.protocols.TP.doSend(TP.java:1105)
at org.jgroups.protocols.TP.send(TP.java:1088)
at org.jgroups.protocols.TP.down(TP.java:907)
at org.jgroups.protocols.Discovery.down(Discovery.java:363)
at org.jgroups.protocols.MERGE2.down(MERGE2.java:169)
at org.jgroups.protocols.FD_SOCK.down(FD_SOCK.java:333)
at org.jgroups.protocols.FD.down(FD.java:327)
at org.jgroups.protocols.VERIFY_SUSPECT.down(VERIFY_SUSPECT.java:72)
at org.jgroups.protocols.BARRIER.down(BARRIER.java:96)
at org.jgroups.protocols.pbcast.NAKACK.retransmit(NAKACK.java:1530)
at org.jgroups.protocols.pbcast.NAKACK.retransmit(NAKACK.java:1476)
at org.jgroups.stack.Retransmitter$Task.run(Retransmitter.java:207)
at org.jgroups.util.TimeScheduler$TaskWrapper.run(TimeScheduler.java:218)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:207)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Locked ownable synchronizers:
- <0x2b5522e0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
- <0x2b5550f8> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
- <0x434273f0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
2) and there are many threads like this blocked "on out_stream_lock.lock()"
call inside TP class:
"Connection.Receiver [ip:port - ip:port],name,ip:port" prio=10 tid=0x020dc000
nid=0x1fab waiting on condition [0x1365f000..0x1365ffc0]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x2b5550f8> (a
java.util.concurrent.locks.ReentrantLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:778)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1114)
at
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186)
at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262)
at org.jgroups.protocols.TP.send(TP.java:1082)
at org.jgroups.protocols.TP.down(TP.java:907)
at org.jgroups.protocols.Discovery.up(Discovery.java:274)
at org.jgroups.protocols.TP.passMessageUp(TP.java:995)
at org.jgroups.protocols.TP.access$100(TP.java:52)
at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1451)
at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1427)
at
java.util.concurrent.ThreadPoolExecutor$CallerRunsPolicy.rejectedExecution(ThreadPoolExecutor.java:1738)
at
org.jgroups.util.ShutdownRejectedExecutionHandler.rejectedExecution(ShutdownRejectedExecutionHandler.java:34)
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
at org.jgroups.protocols.TP.dispatchToThreadPool(TP.java:1061)
at org.jgroups.protocols.TP.receive(TP.java:1038)
at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:180)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection$ConnectionPeerReceiver.run(TCPConnectionMap.java:553)
at java.lang.Thread.run(Thread.java:619)
Locked ownable synchronizers:
- None
(Since all threads from JGroups' ThreadPool are locked, you can see
"rejectedExecution" in the last stack trace).
Maybe we could implement some solution for TCP (blocking I/O)? Maybe it is possible to
release the lock before socket write?
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
3:46 a.m.
New subject: [JBoss JIRA] Updated: (JGRP-1003) deadlock in TP.send - socket write hangs inside a Lock, preventing other threads to do TP.send()
[
https://jira.jboss.org/jira/browse/JGRP-1003?page=com.atlassian.jira.plug...
]
Victor N updated JGRP-1003:
---------------------------
Attachment: jgroups_stacktrace_09feb2010.txt
deadlock in TP.send - socket write hangs inside a Lock, preventing
other threads to do TP.send()
------------------------------------------------------------------------------------------------
Key: JGRP-1003
URL: https://jira.jboss.org/jira/browse/JGRP-1003
Project: JGroups
Issue Type: Bug
Affects Versions: 2.7
Environment: Linux (Debian), kernel 2.6.x
Reporter: Victor N
Assignee: Bela Ban
Fix For: 2.8
Attachments: j1__.txt, jgroups_s2.txt, jgroups_stacktrace_09feb2010.txt,
stacktrace.txt
I am using JGroups 2.7.0 GA with the typical protocols stack based on TCP (taken from
tcp.xml).
Sometimes it occurs that socket write operation hangs inside TP.send() -- this is not so
untypical for blocking I/O approach! -- after that JGroups is not working at all because
most of threads are blocked inside TP.send() waiting for that first thread releasing
"out_stream_lock". Below is a fragment from my stack trace.
1) the thread that hangs in socket write until forever (maybe due to a broken socket or
something else); this thread owns "out_stream_lock in TP class":
"Timer-1,name,IP:port" daemon prio=10 tid=0x084e5c00 nid=0x15f0 runnable
[0x211c1000..0x211c2040]
java.lang.Thread.State: RUNNABLE
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
- locked <0x43427300> (a java.io.BufferedOutputStream)
at java.io.DataOutputStream.flush(DataOutputStream.java:106)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection.doSend(TCPConnectionMap.java:472)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection._send(TCPConnectionMap.java:453)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection.send(TCPConnectionMap.java:436)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection.access$100(TCPConnectionMap.java:341)
at org.jgroups.blocks.TCPConnectionMap.send(TCPConnectionMap.java:137)
at org.jgroups.protocols.TCP.send(TCP.java:53)
at org.jgroups.protocols.BasicTCP.sendToSingleMember(BasicTCP.java:141)
at org.jgroups.protocols.TP.doSend(TP.java:1105)
at org.jgroups.protocols.TP.send(TP.java:1088)
at org.jgroups.protocols.TP.down(TP.java:907)
at org.jgroups.protocols.Discovery.down(Discovery.java:363)
at org.jgroups.protocols.MERGE2.down(MERGE2.java:169)
at org.jgroups.protocols.FD_SOCK.down(FD_SOCK.java:333)
at org.jgroups.protocols.FD.down(FD.java:327)
at org.jgroups.protocols.VERIFY_SUSPECT.down(VERIFY_SUSPECT.java:72)
at org.jgroups.protocols.BARRIER.down(BARRIER.java:96)
at org.jgroups.protocols.pbcast.NAKACK.retransmit(NAKACK.java:1530)
at org.jgroups.protocols.pbcast.NAKACK.retransmit(NAKACK.java:1476)
at org.jgroups.stack.Retransmitter$Task.run(Retransmitter.java:207)
at org.jgroups.util.TimeScheduler$TaskWrapper.run(TimeScheduler.java:218)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:207)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Locked ownable synchronizers:
- <0x2b5522e0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
- <0x2b5550f8> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
- <0x434273f0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
2) and there are many threads like this blocked "on out_stream_lock.lock()"
call inside TP class:
"Connection.Receiver [ip:port - ip:port],name,ip:port" prio=10 tid=0x020dc000
nid=0x1fab waiting on condition [0x1365f000..0x1365ffc0]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x2b5550f8> (a
java.util.concurrent.locks.ReentrantLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:778)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1114)
at
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186)
at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262)
at org.jgroups.protocols.TP.send(TP.java:1082)
at org.jgroups.protocols.TP.down(TP.java:907)
at org.jgroups.protocols.Discovery.up(Discovery.java:274)
at org.jgroups.protocols.TP.passMessageUp(TP.java:995)
at org.jgroups.protocols.TP.access$100(TP.java:52)
at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1451)
at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1427)
at
java.util.concurrent.ThreadPoolExecutor$CallerRunsPolicy.rejectedExecution(ThreadPoolExecutor.java:1738)
at
org.jgroups.util.ShutdownRejectedExecutionHandler.rejectedExecution(ShutdownRejectedExecutionHandler.java:34)
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
at org.jgroups.protocols.TP.dispatchToThreadPool(TP.java:1061)
at org.jgroups.protocols.TP.receive(TP.java:1038)
at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:180)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection$ConnectionPeerReceiver.run(TCPConnectionMap.java:553)
at java.lang.Thread.run(Thread.java:619)
Locked ownable synchronizers:
- None
(Since all threads from JGroups' ThreadPool are locked, you can see
"rejectedExecution" in the last stack trace).
Maybe we could implement some solution for TCP (blocking I/O)? Maybe it is possible to
release the lock before socket write?
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
3:51 a.m.
New subject: [JBoss JIRA] Updated: (JGRP-1003) deadlock in TP.send - socket write hangs inside a Lock, preventing other threads to do TP.send()
[
https://jira.jboss.org/jira/browse/JGRP-1003?page=com.atlassian.jira.plug...
]
Bela Ban updated JGRP-1003:
---------------------------
Fix Version/s: 2.10
(was: 2.8)
deadlock in TP.send - socket write hangs inside a Lock, preventing
other threads to do TP.send()
------------------------------------------------------------------------------------------------
Key: JGRP-1003
URL: https://jira.jboss.org/jira/browse/JGRP-1003
Project: JGroups
Issue Type: Bug
Affects Versions: 2.7
Environment: Linux (Debian), kernel 2.6.x
Reporter: Victor N
Assignee: Bela Ban
Fix For: 2.10
Attachments: j1__.txt, jgroups_s2.txt, jgroups_stacktrace_09feb2010.txt,
stacktrace.txt
I am using JGroups 2.7.0 GA with the typical protocols stack based on TCP (taken from
tcp.xml).
Sometimes it occurs that socket write operation hangs inside TP.send() -- this is not so
untypical for blocking I/O approach! -- after that JGroups is not working at all because
most of threads are blocked inside TP.send() waiting for that first thread releasing
"out_stream_lock". Below is a fragment from my stack trace.
1) the thread that hangs in socket write until forever (maybe due to a broken socket or
something else); this thread owns "out_stream_lock in TP class":
"Timer-1,name,IP:port" daemon prio=10 tid=0x084e5c00 nid=0x15f0 runnable
[0x211c1000..0x211c2040]
java.lang.Thread.State: RUNNABLE
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
- locked <0x43427300> (a java.io.BufferedOutputStream)
at java.io.DataOutputStream.flush(DataOutputStream.java:106)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection.doSend(TCPConnectionMap.java:472)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection._send(TCPConnectionMap.java:453)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection.send(TCPConnectionMap.java:436)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection.access$100(TCPConnectionMap.java:341)
at org.jgroups.blocks.TCPConnectionMap.send(TCPConnectionMap.java:137)
at org.jgroups.protocols.TCP.send(TCP.java:53)
at org.jgroups.protocols.BasicTCP.sendToSingleMember(BasicTCP.java:141)
at org.jgroups.protocols.TP.doSend(TP.java:1105)
at org.jgroups.protocols.TP.send(TP.java:1088)
at org.jgroups.protocols.TP.down(TP.java:907)
at org.jgroups.protocols.Discovery.down(Discovery.java:363)
at org.jgroups.protocols.MERGE2.down(MERGE2.java:169)
at org.jgroups.protocols.FD_SOCK.down(FD_SOCK.java:333)
at org.jgroups.protocols.FD.down(FD.java:327)
at org.jgroups.protocols.VERIFY_SUSPECT.down(VERIFY_SUSPECT.java:72)
at org.jgroups.protocols.BARRIER.down(BARRIER.java:96)
at org.jgroups.protocols.pbcast.NAKACK.retransmit(NAKACK.java:1530)
at org.jgroups.protocols.pbcast.NAKACK.retransmit(NAKACK.java:1476)
at org.jgroups.stack.Retransmitter$Task.run(Retransmitter.java:207)
at org.jgroups.util.TimeScheduler$TaskWrapper.run(TimeScheduler.java:218)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:207)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Locked ownable synchronizers:
- <0x2b5522e0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
- <0x2b5550f8> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
- <0x434273f0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
2) and there are many threads like this blocked "on out_stream_lock.lock()"
call inside TP class:
"Connection.Receiver [ip:port - ip:port],name,ip:port" prio=10 tid=0x020dc000
nid=0x1fab waiting on condition [0x1365f000..0x1365ffc0]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x2b5550f8> (a
java.util.concurrent.locks.ReentrantLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:778)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1114)
at
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186)
at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262)
at org.jgroups.protocols.TP.send(TP.java:1082)
at org.jgroups.protocols.TP.down(TP.java:907)
at org.jgroups.protocols.Discovery.up(Discovery.java:274)
at org.jgroups.protocols.TP.passMessageUp(TP.java:995)
at org.jgroups.protocols.TP.access$100(TP.java:52)
at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1451)
at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1427)
at
java.util.concurrent.ThreadPoolExecutor$CallerRunsPolicy.rejectedExecution(ThreadPoolExecutor.java:1738)
at
org.jgroups.util.ShutdownRejectedExecutionHandler.rejectedExecution(ShutdownRejectedExecutionHandler.java:34)
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
at org.jgroups.protocols.TP.dispatchToThreadPool(TP.java:1061)
at org.jgroups.protocols.TP.receive(TP.java:1038)
at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:180)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection$ConnectionPeerReceiver.run(TCPConnectionMap.java:553)
at java.lang.Thread.run(Thread.java:619)
Locked ownable synchronizers:
- None
(Since all threads from JGroups' ThreadPool are locked, you can see
"rejectedExecution" in the last stack trace).
Maybe we could implement some solution for TCP (blocking I/O)? Maybe it is possible to
release the lock before socket write?
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
3:54 a.m.
New subject: [JBoss JIRA] Commented: (JGRP-1003) deadlock in TP.send - socket write hangs inside a Lock, preventing other threads to do TP.send()
[
https://jira.jboss.org/jira/browse/JGRP-1003?page=com.atlassian.jira.plug...
]
Victor N commented on JGRP-1003:
--------------------------------
Seems the cluster splits into 2 partitions:
1) with server3 as coordinator - its view contains 9 members (all excepting server8)
2) with server8 as coordinator - its view contains 6 members (I don't know why - maybe
there were 6 members visible before everything stalled in JGroups)
and these 2 partitions do not merge. But where is the root cause of the problem? Is this
because of blocked threads (one queue is full, another is empty) in serve8 or something
else? Maybe some timeouts in my config are too small?
Can we monitor/diagnose somehow why that queues are full/empty? If I should turn on some
logging in JGroups, please let me know.
deadlock in TP.send - socket write hangs inside a Lock, preventing
other threads to do TP.send()
------------------------------------------------------------------------------------------------
Key: JGRP-1003
URL: https://jira.jboss.org/jira/browse/JGRP-1003
Project: JGroups
Issue Type: Bug
Affects Versions: 2.7
Environment: Linux (Debian), kernel 2.6.x
Reporter: Victor N
Assignee: Bela Ban
Fix For: 2.10
Attachments: j1__.txt, jgroups_s2.txt, jgroups_stacktrace_09feb2010.txt,
stacktrace.txt
I am using JGroups 2.7.0 GA with the typical protocols stack based on TCP (taken from
tcp.xml).
Sometimes it occurs that socket write operation hangs inside TP.send() -- this is not so
untypical for blocking I/O approach! -- after that JGroups is not working at all because
most of threads are blocked inside TP.send() waiting for that first thread releasing
"out_stream_lock". Below is a fragment from my stack trace.
1) the thread that hangs in socket write until forever (maybe due to a broken socket or
something else); this thread owns "out_stream_lock in TP class":
"Timer-1,name,IP:port" daemon prio=10 tid=0x084e5c00 nid=0x15f0 runnable
[0x211c1000..0x211c2040]
java.lang.Thread.State: RUNNABLE
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
- locked <0x43427300> (a java.io.BufferedOutputStream)
at java.io.DataOutputStream.flush(DataOutputStream.java:106)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection.doSend(TCPConnectionMap.java:472)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection._send(TCPConnectionMap.java:453)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection.send(TCPConnectionMap.java:436)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection.access$100(TCPConnectionMap.java:341)
at org.jgroups.blocks.TCPConnectionMap.send(TCPConnectionMap.java:137)
at org.jgroups.protocols.TCP.send(TCP.java:53)
at org.jgroups.protocols.BasicTCP.sendToSingleMember(BasicTCP.java:141)
at org.jgroups.protocols.TP.doSend(TP.java:1105)
at org.jgroups.protocols.TP.send(TP.java:1088)
at org.jgroups.protocols.TP.down(TP.java:907)
at org.jgroups.protocols.Discovery.down(Discovery.java:363)
at org.jgroups.protocols.MERGE2.down(MERGE2.java:169)
at org.jgroups.protocols.FD_SOCK.down(FD_SOCK.java:333)
at org.jgroups.protocols.FD.down(FD.java:327)
at org.jgroups.protocols.VERIFY_SUSPECT.down(VERIFY_SUSPECT.java:72)
at org.jgroups.protocols.BARRIER.down(BARRIER.java:96)
at org.jgroups.protocols.pbcast.NAKACK.retransmit(NAKACK.java:1530)
at org.jgroups.protocols.pbcast.NAKACK.retransmit(NAKACK.java:1476)
at org.jgroups.stack.Retransmitter$Task.run(Retransmitter.java:207)
at org.jgroups.util.TimeScheduler$TaskWrapper.run(TimeScheduler.java:218)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:207)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Locked ownable synchronizers:
- <0x2b5522e0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
- <0x2b5550f8> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
- <0x434273f0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
2) and there are many threads like this blocked "on out_stream_lock.lock()"
call inside TP class:
"Connection.Receiver [ip:port - ip:port],name,ip:port" prio=10 tid=0x020dc000
nid=0x1fab waiting on condition [0x1365f000..0x1365ffc0]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x2b5550f8> (a
java.util.concurrent.locks.ReentrantLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:778)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1114)
at
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186)
at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262)
at org.jgroups.protocols.TP.send(TP.java:1082)
at org.jgroups.protocols.TP.down(TP.java:907)
at org.jgroups.protocols.Discovery.up(Discovery.java:274)
at org.jgroups.protocols.TP.passMessageUp(TP.java:995)
at org.jgroups.protocols.TP.access$100(TP.java:52)
at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1451)
at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1427)
at
java.util.concurrent.ThreadPoolExecutor$CallerRunsPolicy.rejectedExecution(ThreadPoolExecutor.java:1738)
at
org.jgroups.util.ShutdownRejectedExecutionHandler.rejectedExecution(ShutdownRejectedExecutionHandler.java:34)
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
at org.jgroups.protocols.TP.dispatchToThreadPool(TP.java:1061)
at org.jgroups.protocols.TP.receive(TP.java:1038)
at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:180)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection$ConnectionPeerReceiver.run(TCPConnectionMap.java:553)
at java.lang.Thread.run(Thread.java:619)
Locked ownable synchronizers:
- None
(Since all threads from JGroups' ThreadPool are locked, you can see
"rejectedExecution" in the last stack trace).
Maybe we could implement some solution for TCP (blocking I/O)? Maybe it is possible to
release the lock before socket write?
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
4:20 p.m.
New subject: [JBoss JIRA] Commented: (JGRP-1003) deadlock in TP.send - socket write hangs inside a Lock, preventing other threads to do TP.send()
[
https://jira.jboss.org/jira/browse/JGRP-1003?page=com.atlassian.jira.plug...
]
Victor N commented on JGRP-1003:
--------------------------------
Bela, I do not know how to reproduce it in tests. Seems that the only thing we can do is
to add more logging for such strange situations and to analyze them and the source code.
If send queue is full, we should understand what a sender's thread is doing. Maybe it
is blocked in some socket read or write operation...
deadlock in TP.send - socket write hangs inside a Lock, preventing
other threads to do TP.send()
------------------------------------------------------------------------------------------------
Key: JGRP-1003
URL: https://jira.jboss.org/jira/browse/JGRP-1003
Project: JGroups
Issue Type: Bug
Affects Versions: 2.7
Environment: Linux (Debian), kernel 2.6.x
Reporter: Victor N
Assignee: Bela Ban
Fix For: 2.10
Attachments: j1__.txt, jgroups_s2.txt, jgroups_stacktrace_09feb2010.txt,
stacktrace.txt
I am using JGroups 2.7.0 GA with the typical protocols stack based on TCP (taken from
tcp.xml).
Sometimes it occurs that socket write operation hangs inside TP.send() -- this is not so
untypical for blocking I/O approach! -- after that JGroups is not working at all because
most of threads are blocked inside TP.send() waiting for that first thread releasing
"out_stream_lock". Below is a fragment from my stack trace.
1) the thread that hangs in socket write until forever (maybe due to a broken socket or
something else); this thread owns "out_stream_lock in TP class":
"Timer-1,name,IP:port" daemon prio=10 tid=0x084e5c00 nid=0x15f0 runnable
[0x211c1000..0x211c2040]
java.lang.Thread.State: RUNNABLE
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
- locked <0x43427300> (a java.io.BufferedOutputStream)
at java.io.DataOutputStream.flush(DataOutputStream.java:106)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection.doSend(TCPConnectionMap.java:472)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection._send(TCPConnectionMap.java:453)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection.send(TCPConnectionMap.java:436)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection.access$100(TCPConnectionMap.java:341)
at org.jgroups.blocks.TCPConnectionMap.send(TCPConnectionMap.java:137)
at org.jgroups.protocols.TCP.send(TCP.java:53)
at org.jgroups.protocols.BasicTCP.sendToSingleMember(BasicTCP.java:141)
at org.jgroups.protocols.TP.doSend(TP.java:1105)
at org.jgroups.protocols.TP.send(TP.java:1088)
at org.jgroups.protocols.TP.down(TP.java:907)
at org.jgroups.protocols.Discovery.down(Discovery.java:363)
at org.jgroups.protocols.MERGE2.down(MERGE2.java:169)
at org.jgroups.protocols.FD_SOCK.down(FD_SOCK.java:333)
at org.jgroups.protocols.FD.down(FD.java:327)
at org.jgroups.protocols.VERIFY_SUSPECT.down(VERIFY_SUSPECT.java:72)
at org.jgroups.protocols.BARRIER.down(BARRIER.java:96)
at org.jgroups.protocols.pbcast.NAKACK.retransmit(NAKACK.java:1530)
at org.jgroups.protocols.pbcast.NAKACK.retransmit(NAKACK.java:1476)
at org.jgroups.stack.Retransmitter$Task.run(Retransmitter.java:207)
at org.jgroups.util.TimeScheduler$TaskWrapper.run(TimeScheduler.java:218)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:207)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Locked ownable synchronizers:
- <0x2b5522e0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
- <0x2b5550f8> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
- <0x434273f0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
2) and there are many threads like this blocked "on out_stream_lock.lock()"
call inside TP class:
"Connection.Receiver [ip:port - ip:port],name,ip:port" prio=10 tid=0x020dc000
nid=0x1fab waiting on condition [0x1365f000..0x1365ffc0]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x2b5550f8> (a
java.util.concurrent.locks.ReentrantLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:778)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1114)
at
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186)
at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262)
at org.jgroups.protocols.TP.send(TP.java:1082)
at org.jgroups.protocols.TP.down(TP.java:907)
at org.jgroups.protocols.Discovery.up(Discovery.java:274)
at org.jgroups.protocols.TP.passMessageUp(TP.java:995)
at org.jgroups.protocols.TP.access$100(TP.java:52)
at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1451)
at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1427)
at
java.util.concurrent.ThreadPoolExecutor$CallerRunsPolicy.rejectedExecution(ThreadPoolExecutor.java:1738)
at
org.jgroups.util.ShutdownRejectedExecutionHandler.rejectedExecution(ShutdownRejectedExecutionHandler.java:34)
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
at org.jgroups.protocols.TP.dispatchToThreadPool(TP.java:1061)
at org.jgroups.protocols.TP.receive(TP.java:1038)
at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:180)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection$ConnectionPeerReceiver.run(TCPConnectionMap.java:553)
at java.lang.Thread.run(Thread.java:619)
Locked ownable synchronizers:
- None
(Since all threads from JGroups' ThreadPool are locked, you can see
"rejectedExecution" in the last stack trace).
Maybe we could implement some solution for TCP (blocking I/O)? Maybe it is possible to
release the lock before socket write?
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
3:35 a.m.
New subject: [JBoss JIRA] Commented: (JGRP-1003) deadlock in TP.send - socket write hangs inside a Lock, preventing other threads to do TP.send()
[
https://jira.jboss.org/jira/browse/JGRP-1003?page=com.atlassian.jira.plug...
]
Victor N commented on JGRP-1003:
--------------------------------
The problem is reproduced again at one of our servers. I do not see anything specific in
log files - I just see that all my threads are blocked trying to send via JGroups. And
JGroups itself can not process its join/merge procedure (because of the queue problem
mentioned earlier). No exceptions in my log file around the time the problem has appeared.
It just hangs waiting on the queue.
Bela, do you have any ideas? Maybe we could add some try/catch(Throwable) blocks and log
these events in all critical places? Maybe some threads are not working?
deadlock in TP.send - socket write hangs inside a Lock, preventing
other threads to do TP.send()
------------------------------------------------------------------------------------------------
Key: JGRP-1003
URL: https://jira.jboss.org/jira/browse/JGRP-1003
Project: JGroups
Issue Type: Bug
Affects Versions: 2.7
Environment: Linux (Debian), kernel 2.6.x
Reporter: Victor N
Assignee: Bela Ban
Fix For: 2.10
Attachments: j1__.txt, jgroups_s2.txt, jgroups_stacktrace_09feb2010.txt,
stacktrace.txt
I am using JGroups 2.7.0 GA with the typical protocols stack based on TCP (taken from
tcp.xml).
Sometimes it occurs that socket write operation hangs inside TP.send() -- this is not so
untypical for blocking I/O approach! -- after that JGroups is not working at all because
most of threads are blocked inside TP.send() waiting for that first thread releasing
"out_stream_lock". Below is a fragment from my stack trace.
1) the thread that hangs in socket write until forever (maybe due to a broken socket or
something else); this thread owns "out_stream_lock in TP class":
"Timer-1,name,IP:port" daemon prio=10 tid=0x084e5c00 nid=0x15f0 runnable
[0x211c1000..0x211c2040]
java.lang.Thread.State: RUNNABLE
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
- locked <0x43427300> (a java.io.BufferedOutputStream)
at java.io.DataOutputStream.flush(DataOutputStream.java:106)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection.doSend(TCPConnectionMap.java:472)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection._send(TCPConnectionMap.java:453)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection.send(TCPConnectionMap.java:436)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection.access$100(TCPConnectionMap.java:341)
at org.jgroups.blocks.TCPConnectionMap.send(TCPConnectionMap.java:137)
at org.jgroups.protocols.TCP.send(TCP.java:53)
at org.jgroups.protocols.BasicTCP.sendToSingleMember(BasicTCP.java:141)
at org.jgroups.protocols.TP.doSend(TP.java:1105)
at org.jgroups.protocols.TP.send(TP.java:1088)
at org.jgroups.protocols.TP.down(TP.java:907)
at org.jgroups.protocols.Discovery.down(Discovery.java:363)
at org.jgroups.protocols.MERGE2.down(MERGE2.java:169)
at org.jgroups.protocols.FD_SOCK.down(FD_SOCK.java:333)
at org.jgroups.protocols.FD.down(FD.java:327)
at org.jgroups.protocols.VERIFY_SUSPECT.down(VERIFY_SUSPECT.java:72)
at org.jgroups.protocols.BARRIER.down(BARRIER.java:96)
at org.jgroups.protocols.pbcast.NAKACK.retransmit(NAKACK.java:1530)
at org.jgroups.protocols.pbcast.NAKACK.retransmit(NAKACK.java:1476)
at org.jgroups.stack.Retransmitter$Task.run(Retransmitter.java:207)
at org.jgroups.util.TimeScheduler$TaskWrapper.run(TimeScheduler.java:218)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:207)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Locked ownable synchronizers:
- <0x2b5522e0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
- <0x2b5550f8> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
- <0x434273f0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
2) and there are many threads like this blocked "on out_stream_lock.lock()"
call inside TP class:
"Connection.Receiver [ip:port - ip:port],name,ip:port" prio=10 tid=0x020dc000
nid=0x1fab waiting on condition [0x1365f000..0x1365ffc0]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x2b5550f8> (a
java.util.concurrent.locks.ReentrantLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:778)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1114)
at
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186)
at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262)
at org.jgroups.protocols.TP.send(TP.java:1082)
at org.jgroups.protocols.TP.down(TP.java:907)
at org.jgroups.protocols.Discovery.up(Discovery.java:274)
at org.jgroups.protocols.TP.passMessageUp(TP.java:995)
at org.jgroups.protocols.TP.access$100(TP.java:52)
at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1451)
at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1427)
at
java.util.concurrent.ThreadPoolExecutor$CallerRunsPolicy.rejectedExecution(ThreadPoolExecutor.java:1738)
at
org.jgroups.util.ShutdownRejectedExecutionHandler.rejectedExecution(ShutdownRejectedExecutionHandler.java:34)
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
at org.jgroups.protocols.TP.dispatchToThreadPool(TP.java:1061)
at org.jgroups.protocols.TP.receive(TP.java:1038)
at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:180)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection$ConnectionPeerReceiver.run(TCPConnectionMap.java:553)
at java.lang.Thread.run(Thread.java:619)
Locked ownable synchronizers:
- None
(Since all threads from JGroups' ThreadPool are locked, you can see
"rejectedExecution" in the last stack trace).
Maybe we could implement some solution for TCP (blocking I/O)? Maybe it is possible to
release the lock before socket write?
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
10:56 a.m.
New subject: [JBoss JIRA] Commented: (JGRP-1003) deadlock in TP.send - socket write hangs inside a Lock, preventing other threads to do TP.send()
[
https://jira.jboss.org/jira/browse/JGRP-1003?page=com.atlassian.jira.plug...
]
Victor N commented on JGRP-1003:
--------------------------------
I can not find the reason of this bug, but seems there is a workaround:
if I call JChannel.close(), all blocked threads are alive again and of course I can create
a new JChannel after that.
So, seems there is some condition in JGroups code that brings the situation when one queue
is empty, but another is full. But this condition goes away when the channel is closed -
after that the queues go to their normal state again.
deadlock in TP.send - socket write hangs inside a Lock, preventing
other threads to do TP.send()
------------------------------------------------------------------------------------------------
Key: JGRP-1003
URL: https://jira.jboss.org/jira/browse/JGRP-1003
Project: JGroups
Issue Type: Bug
Affects Versions: 2.7
Environment: Linux (Debian), kernel 2.6.x
Reporter: Victor N
Assignee: Bela Ban
Fix For: 2.10
Attachments: j1__.txt, jgroups_s2.txt, jgroups_stacktrace_09feb2010.txt,
stacktrace.txt
I am using JGroups 2.7.0 GA with the typical protocols stack based on TCP (taken from
tcp.xml).
Sometimes it occurs that socket write operation hangs inside TP.send() -- this is not so
untypical for blocking I/O approach! -- after that JGroups is not working at all because
most of threads are blocked inside TP.send() waiting for that first thread releasing
"out_stream_lock". Below is a fragment from my stack trace.
1) the thread that hangs in socket write until forever (maybe due to a broken socket or
something else); this thread owns "out_stream_lock in TP class":
"Timer-1,name,IP:port" daemon prio=10 tid=0x084e5c00 nid=0x15f0 runnable
[0x211c1000..0x211c2040]
java.lang.Thread.State: RUNNABLE
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
- locked <0x43427300> (a java.io.BufferedOutputStream)
at java.io.DataOutputStream.flush(DataOutputStream.java:106)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection.doSend(TCPConnectionMap.java:472)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection._send(TCPConnectionMap.java:453)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection.send(TCPConnectionMap.java:436)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection.access$100(TCPConnectionMap.java:341)
at org.jgroups.blocks.TCPConnectionMap.send(TCPConnectionMap.java:137)
at org.jgroups.protocols.TCP.send(TCP.java:53)
at org.jgroups.protocols.BasicTCP.sendToSingleMember(BasicTCP.java:141)
at org.jgroups.protocols.TP.doSend(TP.java:1105)
at org.jgroups.protocols.TP.send(TP.java:1088)
at org.jgroups.protocols.TP.down(TP.java:907)
at org.jgroups.protocols.Discovery.down(Discovery.java:363)
at org.jgroups.protocols.MERGE2.down(MERGE2.java:169)
at org.jgroups.protocols.FD_SOCK.down(FD_SOCK.java:333)
at org.jgroups.protocols.FD.down(FD.java:327)
at org.jgroups.protocols.VERIFY_SUSPECT.down(VERIFY_SUSPECT.java:72)
at org.jgroups.protocols.BARRIER.down(BARRIER.java:96)
at org.jgroups.protocols.pbcast.NAKACK.retransmit(NAKACK.java:1530)
at org.jgroups.protocols.pbcast.NAKACK.retransmit(NAKACK.java:1476)
at org.jgroups.stack.Retransmitter$Task.run(Retransmitter.java:207)
at org.jgroups.util.TimeScheduler$TaskWrapper.run(TimeScheduler.java:218)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:207)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Locked ownable synchronizers:
- <0x2b5522e0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
- <0x2b5550f8> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
- <0x434273f0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
2) and there are many threads like this blocked "on out_stream_lock.lock()"
call inside TP class:
"Connection.Receiver [ip:port - ip:port],name,ip:port" prio=10 tid=0x020dc000
nid=0x1fab waiting on condition [0x1365f000..0x1365ffc0]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x2b5550f8> (a
java.util.concurrent.locks.ReentrantLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:778)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1114)
at
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186)
at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262)
at org.jgroups.protocols.TP.send(TP.java:1082)
at org.jgroups.protocols.TP.down(TP.java:907)
at org.jgroups.protocols.Discovery.up(Discovery.java:274)
at org.jgroups.protocols.TP.passMessageUp(TP.java:995)
at org.jgroups.protocols.TP.access$100(TP.java:52)
at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1451)
at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1427)
at
java.util.concurrent.ThreadPoolExecutor$CallerRunsPolicy.rejectedExecution(ThreadPoolExecutor.java:1738)
at
org.jgroups.util.ShutdownRejectedExecutionHandler.rejectedExecution(ShutdownRejectedExecutionHandler.java:34)
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
at org.jgroups.protocols.TP.dispatchToThreadPool(TP.java:1061)
at org.jgroups.protocols.TP.receive(TP.java:1038)
at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:180)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection$ConnectionPeerReceiver.run(TCPConnectionMap.java:553)
at java.lang.Thread.run(Thread.java:619)
Locked ownable synchronizers:
- None
(Since all threads from JGroups' ThreadPool are locked, you can see
"rejectedExecution" in the last stack trace).
Maybe we could implement some solution for TCP (blocking I/O)? Maybe it is possible to
release the lock before socket write?
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
10:58 a.m.
New subject: [JBoss JIRA] Commented: (JGRP-1003) deadlock in TP.send - socket write hangs inside a Lock, preventing other threads to do TP.send()
[
https://jira.jboss.org/jira/browse/JGRP-1003?page=com.atlassian.jira.plug...
]
Victor N commented on JGRP-1003:
--------------------------------
Forgot to say, the problem can be reproduced only under high load, when some nodes can be
unresponsive for many seconds (and even minutes sometimes).
I have never seen it under normal load. Maybe, there are some race condition in the code.
deadlock in TP.send - socket write hangs inside a Lock, preventing
other threads to do TP.send()
------------------------------------------------------------------------------------------------
Key: JGRP-1003
URL: https://jira.jboss.org/jira/browse/JGRP-1003
Project: JGroups
Issue Type: Bug
Affects Versions: 2.7
Environment: Linux (Debian), kernel 2.6.x
Reporter: Victor N
Assignee: Bela Ban
Fix For: 2.10
Attachments: j1__.txt, jgroups_s2.txt, jgroups_stacktrace_09feb2010.txt,
stacktrace.txt
I am using JGroups 2.7.0 GA with the typical protocols stack based on TCP (taken from
tcp.xml).
Sometimes it occurs that socket write operation hangs inside TP.send() -- this is not so
untypical for blocking I/O approach! -- after that JGroups is not working at all because
most of threads are blocked inside TP.send() waiting for that first thread releasing
"out_stream_lock". Below is a fragment from my stack trace.
1) the thread that hangs in socket write until forever (maybe due to a broken socket or
something else); this thread owns "out_stream_lock in TP class":
"Timer-1,name,IP:port" daemon prio=10 tid=0x084e5c00 nid=0x15f0 runnable
[0x211c1000..0x211c2040]
java.lang.Thread.State: RUNNABLE
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
- locked <0x43427300> (a java.io.BufferedOutputStream)
at java.io.DataOutputStream.flush(DataOutputStream.java:106)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection.doSend(TCPConnectionMap.java:472)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection._send(TCPConnectionMap.java:453)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection.send(TCPConnectionMap.java:436)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection.access$100(TCPConnectionMap.java:341)
at org.jgroups.blocks.TCPConnectionMap.send(TCPConnectionMap.java:137)
at org.jgroups.protocols.TCP.send(TCP.java:53)
at org.jgroups.protocols.BasicTCP.sendToSingleMember(BasicTCP.java:141)
at org.jgroups.protocols.TP.doSend(TP.java:1105)
at org.jgroups.protocols.TP.send(TP.java:1088)
at org.jgroups.protocols.TP.down(TP.java:907)
at org.jgroups.protocols.Discovery.down(Discovery.java:363)
at org.jgroups.protocols.MERGE2.down(MERGE2.java:169)
at org.jgroups.protocols.FD_SOCK.down(FD_SOCK.java:333)
at org.jgroups.protocols.FD.down(FD.java:327)
at org.jgroups.protocols.VERIFY_SUSPECT.down(VERIFY_SUSPECT.java:72)
at org.jgroups.protocols.BARRIER.down(BARRIER.java:96)
at org.jgroups.protocols.pbcast.NAKACK.retransmit(NAKACK.java:1530)
at org.jgroups.protocols.pbcast.NAKACK.retransmit(NAKACK.java:1476)
at org.jgroups.stack.Retransmitter$Task.run(Retransmitter.java:207)
at org.jgroups.util.TimeScheduler$TaskWrapper.run(TimeScheduler.java:218)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:207)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Locked ownable synchronizers:
- <0x2b5522e0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
- <0x2b5550f8> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
- <0x434273f0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
2) and there are many threads like this blocked "on out_stream_lock.lock()"
call inside TP class:
"Connection.Receiver [ip:port - ip:port],name,ip:port" prio=10 tid=0x020dc000
nid=0x1fab waiting on condition [0x1365f000..0x1365ffc0]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x2b5550f8> (a
java.util.concurrent.locks.ReentrantLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:778)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1114)
at
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186)
at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262)
at org.jgroups.protocols.TP.send(TP.java:1082)
at org.jgroups.protocols.TP.down(TP.java:907)
at org.jgroups.protocols.Discovery.up(Discovery.java:274)
at org.jgroups.protocols.TP.passMessageUp(TP.java:995)
at org.jgroups.protocols.TP.access$100(TP.java:52)
at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1451)
at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1427)
at
java.util.concurrent.ThreadPoolExecutor$CallerRunsPolicy.rejectedExecution(ThreadPoolExecutor.java:1738)
at
org.jgroups.util.ShutdownRejectedExecutionHandler.rejectedExecution(ShutdownRejectedExecutionHandler.java:34)
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
at org.jgroups.protocols.TP.dispatchToThreadPool(TP.java:1061)
at org.jgroups.protocols.TP.receive(TP.java:1038)
at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:180)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection$ConnectionPeerReceiver.run(TCPConnectionMap.java:553)
at java.lang.Thread.run(Thread.java:619)
Locked ownable synchronizers:
- None
(Since all threads from JGroups' ThreadPool are locked, you can see
"rejectedExecution" in the last stack trace).
Maybe we could implement some solution for TCP (blocking I/O)? Maybe it is possible to
release the lock before socket write?
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
1 a.m.
New subject: [JBoss JIRA] Commented: (JGRP-1003) deadlock in TP.send - socket write hangs inside a Lock, preventing other threads to do TP.send()
[
https://jira.jboss.org/jira/browse/JGRP-1003?page=com.atlassian.jira.plug...
]
Victor N commented on JGRP-1003:
--------------------------------
Bela,
finally, I've got a TRACE log when the problem is reproduced. It is very large, I can
archive it and send you directly. What is strange there:
1) no GMS/merge messages at all (for ~10 min); this node was NOT coordinator (if it is
important)
2) "internal" communication works partially :
JGroups receives messages from other nodes (I see TP.passMessageUp() and
NAKACK.handleMessage() from other servers)
and sends messages (I see BasicTCP.sendUnicast(), UNICAST.retransmit(), TP.down());
discovery via TCP_PING is working too (I see GET_MBRS_REQ and GET_MBRS_RSP)
3) views are exactly the same at all nodes - from this point of view, the communication
seems to work, but all my sending threads are blocked in JGroups in TP.send(), so I can
not send messages. Maybe, if I create new threads, they will work, I don't know.
My config is still the same. JGroups version is 2.8.0 GA.
Should I collect more info to understand the problem?
deadlock in TP.send - socket write hangs inside a Lock, preventing
other threads to do TP.send()
------------------------------------------------------------------------------------------------
Key: JGRP-1003
URL: https://jira.jboss.org/jira/browse/JGRP-1003
Project: JGroups
Issue Type: Bug
Affects Versions: 2.7
Environment: Linux (Debian), kernel 2.6.x
Reporter: Victor N
Assignee: Bela Ban
Fix For: 2.10
Attachments: j1__.txt, jgroups_s2.txt, jgroups_stacktrace_09feb2010.txt,
stacktrace.txt
I am using JGroups 2.7.0 GA with the typical protocols stack based on TCP (taken from
tcp.xml).
Sometimes it occurs that socket write operation hangs inside TP.send() -- this is not so
untypical for blocking I/O approach! -- after that JGroups is not working at all because
most of threads are blocked inside TP.send() waiting for that first thread releasing
"out_stream_lock". Below is a fragment from my stack trace.
1) the thread that hangs in socket write until forever (maybe due to a broken socket or
something else); this thread owns "out_stream_lock in TP class":
"Timer-1,name,IP:port" daemon prio=10 tid=0x084e5c00 nid=0x15f0 runnable
[0x211c1000..0x211c2040]
java.lang.Thread.State: RUNNABLE
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
- locked <0x43427300> (a java.io.BufferedOutputStream)
at java.io.DataOutputStream.flush(DataOutputStream.java:106)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection.doSend(TCPConnectionMap.java:472)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection._send(TCPConnectionMap.java:453)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection.send(TCPConnectionMap.java:436)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection.access$100(TCPConnectionMap.java:341)
at org.jgroups.blocks.TCPConnectionMap.send(TCPConnectionMap.java:137)
at org.jgroups.protocols.TCP.send(TCP.java:53)
at org.jgroups.protocols.BasicTCP.sendToSingleMember(BasicTCP.java:141)
at org.jgroups.protocols.TP.doSend(TP.java:1105)
at org.jgroups.protocols.TP.send(TP.java:1088)
at org.jgroups.protocols.TP.down(TP.java:907)
at org.jgroups.protocols.Discovery.down(Discovery.java:363)
at org.jgroups.protocols.MERGE2.down(MERGE2.java:169)
at org.jgroups.protocols.FD_SOCK.down(FD_SOCK.java:333)
at org.jgroups.protocols.FD.down(FD.java:327)
at org.jgroups.protocols.VERIFY_SUSPECT.down(VERIFY_SUSPECT.java:72)
at org.jgroups.protocols.BARRIER.down(BARRIER.java:96)
at org.jgroups.protocols.pbcast.NAKACK.retransmit(NAKACK.java:1530)
at org.jgroups.protocols.pbcast.NAKACK.retransmit(NAKACK.java:1476)
at org.jgroups.stack.Retransmitter$Task.run(Retransmitter.java:207)
at org.jgroups.util.TimeScheduler$TaskWrapper.run(TimeScheduler.java:218)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:207)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Locked ownable synchronizers:
- <0x2b5522e0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
- <0x2b5550f8> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
- <0x434273f0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
2) and there are many threads like this blocked "on out_stream_lock.lock()"
call inside TP class:
"Connection.Receiver [ip:port - ip:port],name,ip:port" prio=10 tid=0x020dc000
nid=0x1fab waiting on condition [0x1365f000..0x1365ffc0]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x2b5550f8> (a
java.util.concurrent.locks.ReentrantLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:778)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1114)
at
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186)
at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262)
at org.jgroups.protocols.TP.send(TP.java:1082)
at org.jgroups.protocols.TP.down(TP.java:907)
at org.jgroups.protocols.Discovery.up(Discovery.java:274)
at org.jgroups.protocols.TP.passMessageUp(TP.java:995)
at org.jgroups.protocols.TP.access$100(TP.java:52)
at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1451)
at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1427)
at
java.util.concurrent.ThreadPoolExecutor$CallerRunsPolicy.rejectedExecution(ThreadPoolExecutor.java:1738)
at
org.jgroups.util.ShutdownRejectedExecutionHandler.rejectedExecution(ShutdownRejectedExecutionHandler.java:34)
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
at org.jgroups.protocols.TP.dispatchToThreadPool(TP.java:1061)
at org.jgroups.protocols.TP.receive(TP.java:1038)
at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:180)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection$ConnectionPeerReceiver.run(TCPConnectionMap.java:553)
at java.lang.Thread.run(Thread.java:619)
Locked ownable synchronizers:
- None
(Since all threads from JGroups' ThreadPool are locked, you can see
"rejectedExecution" in the last stack trace).
Maybe we could implement some solution for TCP (blocking I/O)? Maybe it is possible to
release the lock before socket write?
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
1:34 a.m.
New subject: [JBoss JIRA] Commented: (JGRP-1003) deadlock in TP.send - socket write hangs inside a Lock, preventing other threads to do TP.send()
[
https://jira.jboss.org/jira/browse/JGRP-1003?page=com.atlassian.jira.plug...
]
Victor N commented on JGRP-1003:
--------------------------------
I compared logs from this problematic node with logs from coordinator and logs from the
neighbor sending heartbeats. What I see:
4) the "problematic" log is much bigger in size (270 Mb vs 20 Mb!)
There are many many UNICAST.retransmit() calls in it! Almost endlessly.
Logs from the neighbor confirms this (frequent XMIT_REQ from the problematic node).
6) (maybe this is interesting) from the coordinator's log I can see (we have 11 nodes
in our cluster, gate3/node3 is the coordinator):
14/04/2010 06:04:14 UTC| TRACE |
org.jgroups.protocols.MERGE2$FindSubgroupsTask.findAndNotify(): Discovery results:
[gate9.mydomain]: MergeView::[gate3.mydomain|6831] [gate3.mydomain, gate5.mydomain,
gate4.mydomain, gate11.mydomain, gate2.mydomain, gate10.mydomain, gate7.mydomain,
gate8.mydomain, gate6.mydomain, gate9.mydomain, gate12.mydomain],
subgroups=[[gate3.mydomain|6829] [gate9.mydomain], [gate3.mydomain|6830] [gate3.mydomain,
gate5.mydomain, gate4.mydomain, gate11.mydomain, gate2.mydomain, gate10.mydomain,
gate7.mydomain, gate8.mydomain, gate6.mydomain, gate12.mydomain]]
[gate6.mydomain]: MergeView::[gate3.mydomain|6831] [gate3.mydomain, gate5.mydomain,
gate4.mydomain, gate11.mydomain, gate2.mydomain, gate10.mydomain, gate7.mydomain,
gate8.mydomain, gate6.mydomain, gate9.mydomain, gate12.mydomain],
subgroups=[[gate3.mydomain|6829] [gate9.mydomain], [gate3.mydomain|6830] [gate3.mydomain,
gate5.mydomain, gate4.mydomain, gate11.mydomain, gate2.mydomain, gate10.mydomain,
gate7.mydomain, gate8.mydomain, gate6.mydomain, gate12.mydomain]]
[gate7.mydomain]: MergeView::[gate3.mydomain|6831] [gate3.mydomain, gate5.mydomain,
gate4.mydomain, gate11.mydomain, gate2.mydomain, gate10.mydomain, gate7.mydomain,
gate8.mydomain, gate6.mydomain, gate9.mydomain, gate12.mydomain],
subgroups=[[gate3.mydomain|6829] [gate9.mydomain], [gate3.mydomain|6830] [gate3.mydomain,
gate5.mydomain, gate4.mydomain, gate11.mydomain, gate2.mydomain, gate10.mydomain,
gate7.mydomain, gate8.mydomain, gate6.mydomain, gate12.mydomain]]
[gate5.mydomain]: MergeView::[gate3.mydomain|6831] [gate3.mydomain, gate5.mydomain,
gate4.mydomain, gate11.mydomain, gate2.mydomain, gate10.mydomain, gate7.mydomain,
gate8.mydomain, gate6.mydomain, gate9.mydomain, gate12.mydomain],
subgroups=[[gate3.mydomain|6829] [gate9.mydomain], [gate3.mydomain|6830] [gate3.mydomain,
gate5.mydomain, gate4.mydomain, gate11.mydomain, gate2.mydomain, gate10.mydomain,
gate7.mydomain, gate8.mydomain, gate6.mydomain, gate12.mydomain]]
[gate8.mydomain]: MergeView::[gate3.mydomain|6831] [gate3.mydomain, gate5.mydomain,
gate4.mydomain, gate11.mydomain, gate2.mydomain, gate10.mydomain, gate7.mydomain,
gate8.mydomain, gate6.mydomain, gate9.mydomain, gate12.mydomain],
subgroups=[[gate3.mydomain|6829] [gate9.mydomain], [gate3.mydomain|6830] [gate3.mydomain,
gate5.mydomain, gate4.mydomain, gate11.mydomain, gate2.mydomain, gate10.mydomain,
gate7.mydomain, gate8.mydomain, gate6.mydomain, gate12.mydomain]]
[gate11.mydomain]: MergeView::[gate3.mydomain|6831] [gate3.mydomain, gate5.mydomain,
gate4.mydomain, gate11.mydomain, gate2.mydomain, gate10.mydomain, gate7.mydomain,
gate8.mydomain, gate6.mydomain, gate9.mydomain, gate12.mydomain],
subgroups=[[gate3.mydomain|6829] [gate9.mydomain], [gate3.mydomain|6830] [gate3.mydomain,
gate5.mydomain, gate4.mydomain, gate11.mydomain, gate2.mydomain, gate10.mydomain,
gate7.mydomain, gate8.mydomain, gate6.mydomain, gate12.mydomain]]
[gate2.mydomain]: MergeView::[gate3.mydomain|6831] [gate3.mydomain, gate5.mydomain,
gate4.mydomain, gate11.mydomain, gate2.mydomain, gate10.mydomain, gate7.mydomain,
gate8.mydomain, gate6.mydomain, gate9.mydomain, gate12.mydomain],
subgroups=[[gate3.mydomain|6829] [gate9.mydomain], [gate3.mydomain|6830] [gate3.mydomain,
gate5.mydomain, gate4.mydomain, gate11.mydomain, gate2.mydomain, gate10.mydomain,
gate7.mydomain, gate8.mydomain, gate6.mydomain, gate12.mydomain]]
[gate4.mydomain]: MergeView::[gate3.mydomain|6831] [gate3.mydomain, gate5.mydomain,
gate4.mydomain, gate11.mydomain, gate2.mydomain, gate10.mydomain, gate7.mydomain,
gate8.mydomain, gate6.mydomain, gate9.mydomain, gate12.mydomain],
subgroups=[[gate3.mydomain|6829] [gate9.mydomain], [gate3.mydomain|6830] [gate3.mydomain,
gate5.mydomain, gate4.mydomain, gate11.mydomain, gate2.mydomain, gate10.mydomain,
gate7.mydomain, gate8.mydomain, gate6.mydomain, gate12.mydomain]]
[gate12.mydomain]: MergeView::[gate3.mydomain|6831] [gate3.mydomain, gate5.mydomain,
gate4.mydomain, gate11.mydomain, gate2.mydomain, gate10.mydomain, gate7.mydomain,
gate8.mydomain, gate6.mydomain, gate9.mydomain, gate12.mydomain],
subgroups=[[gate3.mydomain|6829] [gate9.mydomain], [gate3.mydomain|6830] [gate3.mydomain,
gate5.mydomain, gate4.mydomain, gate11.mydomain, gate2.mydomain, gate10.mydomain,
gate7.mydomain, gate8.mydomain, gate6.mydomain, gate12.mydomain]]
[gate10.mydomain]: MergeView::[gate3.mydomain|6831] [gate3.mydomain, gate5.mydomain,
gate4.mydomain, gate11.mydomain, gate2.mydomain, gate10.mydomain, gate7.mydomain,
gate8.mydomain, gate6.mydomain, gate9.mydomain, gate12.mydomain],
subgroups=[[gate3.mydomain|6829] [gate9.mydomain], [gate3.mydomain|6830] [gate3.mydomain,
gate5.mydomain, gate4.mydomain, gate11.mydomain, gate2.mydomain, gate10.mydomain,
gate7.mydomain, gate8.mydomain, gate6.mydomain, gate12.mydomain]]
[gate3.mydomain]: MergeView::[gate3.mydomain|6831] [gate3.mydomain, gate5.mydomain,
gate4.mydomain, gate11.mydomain, gate2.mydomain, gate10.mydomain, gate7.mydomain,
gate8.mydomain, gate6.mydomain, gate9.mydomain, gate12.mydomain],
subgroups=[[gate3.mydomain|6829] [gate9.mydomain], [gate3.mydomain|6830] [gate3.mydomain,
gate5.mydomain, gate4.mydomain, gate11.mydomain, gate2.mydomain, gate10.mydomain,
gate7.mydomain, gate8.mydomain, gate6.mydomain, gate12.mydomain]]
deadlock in TP.send - socket write hangs inside a Lock, preventing
other threads to do TP.send()
------------------------------------------------------------------------------------------------
Key: JGRP-1003
URL: https://jira.jboss.org/jira/browse/JGRP-1003
Project: JGroups
Issue Type: Bug
Affects Versions: 2.7
Environment: Linux (Debian), kernel 2.6.x
Reporter: Victor N
Assignee: Bela Ban
Fix For: 2.10
Attachments: j1__.txt, jgroups_s2.txt, jgroups_stacktrace_09feb2010.txt,
stacktrace.txt
I am using JGroups 2.7.0 GA with the typical protocols stack based on TCP (taken from
tcp.xml).
Sometimes it occurs that socket write operation hangs inside TP.send() -- this is not so
untypical for blocking I/O approach! -- after that JGroups is not working at all because
most of threads are blocked inside TP.send() waiting for that first thread releasing
"out_stream_lock". Below is a fragment from my stack trace.
1) the thread that hangs in socket write until forever (maybe due to a broken socket or
something else); this thread owns "out_stream_lock in TP class":
"Timer-1,name,IP:port" daemon prio=10 tid=0x084e5c00 nid=0x15f0 runnable
[0x211c1000..0x211c2040]
java.lang.Thread.State: RUNNABLE
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
- locked <0x43427300> (a java.io.BufferedOutputStream)
at java.io.DataOutputStream.flush(DataOutputStream.java:106)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection.doSend(TCPConnectionMap.java:472)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection._send(TCPConnectionMap.java:453)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection.send(TCPConnectionMap.java:436)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection.access$100(TCPConnectionMap.java:341)
at org.jgroups.blocks.TCPConnectionMap.send(TCPConnectionMap.java:137)
at org.jgroups.protocols.TCP.send(TCP.java:53)
at org.jgroups.protocols.BasicTCP.sendToSingleMember(BasicTCP.java:141)
at org.jgroups.protocols.TP.doSend(TP.java:1105)
at org.jgroups.protocols.TP.send(TP.java:1088)
at org.jgroups.protocols.TP.down(TP.java:907)
at org.jgroups.protocols.Discovery.down(Discovery.java:363)
at org.jgroups.protocols.MERGE2.down(MERGE2.java:169)
at org.jgroups.protocols.FD_SOCK.down(FD_SOCK.java:333)
at org.jgroups.protocols.FD.down(FD.java:327)
at org.jgroups.protocols.VERIFY_SUSPECT.down(VERIFY_SUSPECT.java:72)
at org.jgroups.protocols.BARRIER.down(BARRIER.java:96)
at org.jgroups.protocols.pbcast.NAKACK.retransmit(NAKACK.java:1530)
at org.jgroups.protocols.pbcast.NAKACK.retransmit(NAKACK.java:1476)
at org.jgroups.stack.Retransmitter$Task.run(Retransmitter.java:207)
at org.jgroups.util.TimeScheduler$TaskWrapper.run(TimeScheduler.java:218)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:207)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Locked ownable synchronizers:
- <0x2b5522e0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
- <0x2b5550f8> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
- <0x434273f0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
2) and there are many threads like this blocked "on out_stream_lock.lock()"
call inside TP class:
"Connection.Receiver [ip:port - ip:port],name,ip:port" prio=10 tid=0x020dc000
nid=0x1fab waiting on condition [0x1365f000..0x1365ffc0]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x2b5550f8> (a
java.util.concurrent.locks.ReentrantLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:778)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1114)
at
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186)
at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262)
at org.jgroups.protocols.TP.send(TP.java:1082)
at org.jgroups.protocols.TP.down(TP.java:907)
at org.jgroups.protocols.Discovery.up(Discovery.java:274)
at org.jgroups.protocols.TP.passMessageUp(TP.java:995)
at org.jgroups.protocols.TP.access$100(TP.java:52)
at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1451)
at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1427)
at
java.util.concurrent.ThreadPoolExecutor$CallerRunsPolicy.rejectedExecution(ThreadPoolExecutor.java:1738)
at
org.jgroups.util.ShutdownRejectedExecutionHandler.rejectedExecution(ShutdownRejectedExecutionHandler.java:34)
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
at org.jgroups.protocols.TP.dispatchToThreadPool(TP.java:1061)
at org.jgroups.protocols.TP.receive(TP.java:1038)
at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:180)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection$ConnectionPeerReceiver.run(TCPConnectionMap.java:553)
at java.lang.Thread.run(Thread.java:619)
Locked ownable synchronizers:
- None
(Since all threads from JGroups' ThreadPool are locked, you can see
"rejectedExecution" in the last stack trace).
Maybe we could implement some solution for TCP (blocking I/O)? Maybe it is possible to
release the lock before socket write?
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
6:44 a.m.
New subject: [JBoss JIRA] Commented: (JGRP-1003) deadlock in TP.send - socket write hangs inside a Lock, preventing other threads to do TP.send()
[
https://jira.jboss.org/jira/browse/JGRP-1003?page=com.atlassian.jira.plug...
]
Bela Ban commented on JGRP-1003:
--------------------------------
You've listed many issues here, ranging from TCP send blocking, to merge issues, to
unicast issues etc. The underlying issue might, however, still be the hang in the TCP
send.
I suggest try out JGroups 2.9 (or even better, 2.10 (Beta1)).
2.9 got rid of the send lock (you see in the first comment on this case).
2.10 also got rid of the lock when sending message bundles; it now uses a bounded transfer
queue.
I'm closing this case, I suggest you open a new one with (ideally) reproduceable steps
to re-create the original issue on 2.9 or 2.10...
deadlock in TP.send - socket write hangs inside a Lock, preventing
other threads to do TP.send()
------------------------------------------------------------------------------------------------
Key: JGRP-1003
URL: https://jira.jboss.org/jira/browse/JGRP-1003
Project: JGroups
Issue Type: Bug
Affects Versions: 2.7
Environment: Linux (Debian), kernel 2.6.x
Reporter: Victor N
Assignee: Bela Ban
Fix For: 2.10
Attachments: j1__.txt, jgroups_s2.txt, jgroups_stacktrace_09feb2010.txt,
stacktrace.txt
I am using JGroups 2.7.0 GA with the typical protocols stack based on TCP (taken from
tcp.xml).
Sometimes it occurs that socket write operation hangs inside TP.send() -- this is not so
untypical for blocking I/O approach! -- after that JGroups is not working at all because
most of threads are blocked inside TP.send() waiting for that first thread releasing
"out_stream_lock". Below is a fragment from my stack trace.
1) the thread that hangs in socket write until forever (maybe due to a broken socket or
something else); this thread owns "out_stream_lock in TP class":
"Timer-1,name,IP:port" daemon prio=10 tid=0x084e5c00 nid=0x15f0 runnable
[0x211c1000..0x211c2040]
java.lang.Thread.State: RUNNABLE
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
- locked <0x43427300> (a java.io.BufferedOutputStream)
at java.io.DataOutputStream.flush(DataOutputStream.java:106)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection.doSend(TCPConnectionMap.java:472)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection._send(TCPConnectionMap.java:453)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection.send(TCPConnectionMap.java:436)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection.access$100(TCPConnectionMap.java:341)
at org.jgroups.blocks.TCPConnectionMap.send(TCPConnectionMap.java:137)
at org.jgroups.protocols.TCP.send(TCP.java:53)
at org.jgroups.protocols.BasicTCP.sendToSingleMember(BasicTCP.java:141)
at org.jgroups.protocols.TP.doSend(TP.java:1105)
at org.jgroups.protocols.TP.send(TP.java:1088)
at org.jgroups.protocols.TP.down(TP.java:907)
at org.jgroups.protocols.Discovery.down(Discovery.java:363)
at org.jgroups.protocols.MERGE2.down(MERGE2.java:169)
at org.jgroups.protocols.FD_SOCK.down(FD_SOCK.java:333)
at org.jgroups.protocols.FD.down(FD.java:327)
at org.jgroups.protocols.VERIFY_SUSPECT.down(VERIFY_SUSPECT.java:72)
at org.jgroups.protocols.BARRIER.down(BARRIER.java:96)
at org.jgroups.protocols.pbcast.NAKACK.retransmit(NAKACK.java:1530)
at org.jgroups.protocols.pbcast.NAKACK.retransmit(NAKACK.java:1476)
at org.jgroups.stack.Retransmitter$Task.run(Retransmitter.java:207)
at org.jgroups.util.TimeScheduler$TaskWrapper.run(TimeScheduler.java:218)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:207)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Locked ownable synchronizers:
- <0x2b5522e0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
- <0x2b5550f8> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
- <0x434273f0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
2) and there are many threads like this blocked "on out_stream_lock.lock()"
call inside TP class:
"Connection.Receiver [ip:port - ip:port],name,ip:port" prio=10 tid=0x020dc000
nid=0x1fab waiting on condition [0x1365f000..0x1365ffc0]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x2b5550f8> (a
java.util.concurrent.locks.ReentrantLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:778)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1114)
at
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186)
at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262)
at org.jgroups.protocols.TP.send(TP.java:1082)
at org.jgroups.protocols.TP.down(TP.java:907)
at org.jgroups.protocols.Discovery.up(Discovery.java:274)
at org.jgroups.protocols.TP.passMessageUp(TP.java:995)
at org.jgroups.protocols.TP.access$100(TP.java:52)
at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1451)
at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1427)
at
java.util.concurrent.ThreadPoolExecutor$CallerRunsPolicy.rejectedExecution(ThreadPoolExecutor.java:1738)
at
org.jgroups.util.ShutdownRejectedExecutionHandler.rejectedExecution(ShutdownRejectedExecutionHandler.java:34)
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
at org.jgroups.protocols.TP.dispatchToThreadPool(TP.java:1061)
at org.jgroups.protocols.TP.receive(TP.java:1038)
at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:180)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection$ConnectionPeerReceiver.run(TCPConnectionMap.java:553)
at java.lang.Thread.run(Thread.java:619)
Locked ownable synchronizers:
- None
(Since all threads from JGroups' ThreadPool are locked, you can see
"rejectedExecution" in the last stack trace).
Maybe we could implement some solution for TCP (blocking I/O)? Maybe it is possible to
release the lock before socket write?
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
6:44 a.m.
New subject: [JBoss JIRA] Closed: (JGRP-1003) deadlock in TP.send - socket write hangs inside a Lock, preventing other threads to do TP.send()
[
https://jira.jboss.org/jira/browse/JGRP-1003?page=com.atlassian.jira.plug...
]
Bela Ban closed JGRP-1003.
--------------------------
Resolution: Deferred
deadlock in TP.send - socket write hangs inside a Lock, preventing
other threads to do TP.send()
------------------------------------------------------------------------------------------------
Key: JGRP-1003
URL: https://jira.jboss.org/jira/browse/JGRP-1003
Project: JGroups
Issue Type: Bug
Affects Versions: 2.7
Environment: Linux (Debian), kernel 2.6.x
Reporter: Victor N
Assignee: Bela Ban
Fix For: 2.10
Attachments: j1__.txt, jgroups_s2.txt, jgroups_stacktrace_09feb2010.txt,
stacktrace.txt
I am using JGroups 2.7.0 GA with the typical protocols stack based on TCP (taken from
tcp.xml).
Sometimes it occurs that socket write operation hangs inside TP.send() -- this is not so
untypical for blocking I/O approach! -- after that JGroups is not working at all because
most of threads are blocked inside TP.send() waiting for that first thread releasing
"out_stream_lock". Below is a fragment from my stack trace.
1) the thread that hangs in socket write until forever (maybe due to a broken socket or
something else); this thread owns "out_stream_lock in TP class":
"Timer-1,name,IP:port" daemon prio=10 tid=0x084e5c00 nid=0x15f0 runnable
[0x211c1000..0x211c2040]
java.lang.Thread.State: RUNNABLE
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
- locked <0x43427300> (a java.io.BufferedOutputStream)
at java.io.DataOutputStream.flush(DataOutputStream.java:106)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection.doSend(TCPConnectionMap.java:472)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection._send(TCPConnectionMap.java:453)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection.send(TCPConnectionMap.java:436)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection.access$100(TCPConnectionMap.java:341)
at org.jgroups.blocks.TCPConnectionMap.send(TCPConnectionMap.java:137)
at org.jgroups.protocols.TCP.send(TCP.java:53)
at org.jgroups.protocols.BasicTCP.sendToSingleMember(BasicTCP.java:141)
at org.jgroups.protocols.TP.doSend(TP.java:1105)
at org.jgroups.protocols.TP.send(TP.java:1088)
at org.jgroups.protocols.TP.down(TP.java:907)
at org.jgroups.protocols.Discovery.down(Discovery.java:363)
at org.jgroups.protocols.MERGE2.down(MERGE2.java:169)
at org.jgroups.protocols.FD_SOCK.down(FD_SOCK.java:333)
at org.jgroups.protocols.FD.down(FD.java:327)
at org.jgroups.protocols.VERIFY_SUSPECT.down(VERIFY_SUSPECT.java:72)
at org.jgroups.protocols.BARRIER.down(BARRIER.java:96)
at org.jgroups.protocols.pbcast.NAKACK.retransmit(NAKACK.java:1530)
at org.jgroups.protocols.pbcast.NAKACK.retransmit(NAKACK.java:1476)
at org.jgroups.stack.Retransmitter$Task.run(Retransmitter.java:207)
at org.jgroups.util.TimeScheduler$TaskWrapper.run(TimeScheduler.java:218)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:207)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Locked ownable synchronizers:
- <0x2b5522e0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
- <0x2b5550f8> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
- <0x434273f0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
2) and there are many threads like this blocked "on out_stream_lock.lock()"
call inside TP class:
"Connection.Receiver [ip:port - ip:port],name,ip:port" prio=10 tid=0x020dc000
nid=0x1fab waiting on condition [0x1365f000..0x1365ffc0]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x2b5550f8> (a
java.util.concurrent.locks.ReentrantLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:778)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1114)
at
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186)
at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262)
at org.jgroups.protocols.TP.send(TP.java:1082)
at org.jgroups.protocols.TP.down(TP.java:907)
at org.jgroups.protocols.Discovery.up(Discovery.java:274)
at org.jgroups.protocols.TP.passMessageUp(TP.java:995)
at org.jgroups.protocols.TP.access$100(TP.java:52)
at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1451)
at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1427)
at
java.util.concurrent.ThreadPoolExecutor$CallerRunsPolicy.rejectedExecution(ThreadPoolExecutor.java:1738)
at
org.jgroups.util.ShutdownRejectedExecutionHandler.rejectedExecution(ShutdownRejectedExecutionHandler.java:34)
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
at org.jgroups.protocols.TP.dispatchToThreadPool(TP.java:1061)
at org.jgroups.protocols.TP.receive(TP.java:1038)
at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:180)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection$ConnectionPeerReceiver.run(TCPConnectionMap.java:553)
at java.lang.Thread.run(Thread.java:619)
Locked ownable synchronizers:
- None
(Since all threads from JGroups' ThreadPool are locked, you can see
"rejectedExecution" in the last stack trace).
Maybe we could implement some solution for TCP (blocking I/O)? Maybe it is possible to
release the lock before socket write?
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
1:36 p.m.
New subject: [JBoss JIRA] Commented: (JGRP-1003) deadlock in TP.send - socket write hangs inside a Lock, preventing other threads to do TP.send()
[
https://jira.jboss.org/browse/JGRP-1003?page=com.atlassian.jira.plugin.sy...
]
Victor N commented on JGRP-1003:
--------------------------------
Can confirm - after using 2.10 for a long time in production, seems the problem went
away!
deadlock in TP.send - socket write hangs inside a Lock, preventing
other threads to do TP.send()
------------------------------------------------------------------------------------------------
Key: JGRP-1003
URL: https://jira.jboss.org/browse/JGRP-1003
Project: JGroups
Issue Type: Bug
Affects Versions: 2.7
Environment: Linux (Debian), kernel 2.6.x
Reporter: Victor N
Assignee: Bela Ban
Fix For: 2.10
Attachments: j1__.txt, jgroups_s2.txt, jgroups_stacktrace_09feb2010.txt,
stacktrace.txt
I am using JGroups 2.7.0 GA with the typical protocols stack based on TCP (taken from
tcp.xml).
Sometimes it occurs that socket write operation hangs inside TP.send() -- this is not so
untypical for blocking I/O approach! -- after that JGroups is not working at all because
most of threads are blocked inside TP.send() waiting for that first thread releasing
"out_stream_lock". Below is a fragment from my stack trace.
1) the thread that hangs in socket write until forever (maybe due to a broken socket or
something else); this thread owns "out_stream_lock in TP class":
"Timer-1,name,IP:port" daemon prio=10 tid=0x084e5c00 nid=0x15f0 runnable
[0x211c1000..0x211c2040]
java.lang.Thread.State: RUNNABLE
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
- locked <0x43427300> (a java.io.BufferedOutputStream)
at java.io.DataOutputStream.flush(DataOutputStream.java:106)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection.doSend(TCPConnectionMap.java:472)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection._send(TCPConnectionMap.java:453)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection.send(TCPConnectionMap.java:436)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection.access$100(TCPConnectionMap.java:341)
at org.jgroups.blocks.TCPConnectionMap.send(TCPConnectionMap.java:137)
at org.jgroups.protocols.TCP.send(TCP.java:53)
at org.jgroups.protocols.BasicTCP.sendToSingleMember(BasicTCP.java:141)
at org.jgroups.protocols.TP.doSend(TP.java:1105)
at org.jgroups.protocols.TP.send(TP.java:1088)
at org.jgroups.protocols.TP.down(TP.java:907)
at org.jgroups.protocols.Discovery.down(Discovery.java:363)
at org.jgroups.protocols.MERGE2.down(MERGE2.java:169)
at org.jgroups.protocols.FD_SOCK.down(FD_SOCK.java:333)
at org.jgroups.protocols.FD.down(FD.java:327)
at org.jgroups.protocols.VERIFY_SUSPECT.down(VERIFY_SUSPECT.java:72)
at org.jgroups.protocols.BARRIER.down(BARRIER.java:96)
at org.jgroups.protocols.pbcast.NAKACK.retransmit(NAKACK.java:1530)
at org.jgroups.protocols.pbcast.NAKACK.retransmit(NAKACK.java:1476)
at org.jgroups.stack.Retransmitter$Task.run(Retransmitter.java:207)
at org.jgroups.util.TimeScheduler$TaskWrapper.run(TimeScheduler.java:218)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:207)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Locked ownable synchronizers:
- <0x2b5522e0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
- <0x2b5550f8> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
- <0x434273f0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
2) and there are many threads like this blocked "on out_stream_lock.lock()"
call inside TP class:
"Connection.Receiver [ip:port - ip:port],name,ip:port" prio=10 tid=0x020dc000
nid=0x1fab waiting on condition [0x1365f000..0x1365ffc0]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x2b5550f8> (a
java.util.concurrent.locks.ReentrantLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:778)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1114)
at
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186)
at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262)
at org.jgroups.protocols.TP.send(TP.java:1082)
at org.jgroups.protocols.TP.down(TP.java:907)
at org.jgroups.protocols.Discovery.up(Discovery.java:274)
at org.jgroups.protocols.TP.passMessageUp(TP.java:995)
at org.jgroups.protocols.TP.access$100(TP.java:52)
at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1451)
at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1427)
at
java.util.concurrent.ThreadPoolExecutor$CallerRunsPolicy.rejectedExecution(ThreadPoolExecutor.java:1738)
at
org.jgroups.util.ShutdownRejectedExecutionHandler.rejectedExecution(ShutdownRejectedExecutionHandler.java:34)
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
at org.jgroups.protocols.TP.dispatchToThreadPool(TP.java:1061)
at org.jgroups.protocols.TP.receive(TP.java:1038)
at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:180)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection$ConnectionPeerReceiver.run(TCPConnectionMap.java:553)
at java.lang.Thread.run(Thread.java:619)
Locked ownable synchronizers:
- None
(Since all threads from JGroups' ThreadPool are locked, you can see
"rejectedExecution" in the last stack trace).
Maybe we could implement some solution for TCP (blocking I/O)? Maybe it is possible to
release the lock before socket write?
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
8:03 a.m.
New subject: [JBoss JIRA] Commented: (JGRP-1003) deadlock in TP.send - socket write hangs inside a Lock, preventing other threads to do TP.send()
[
https://issues.jboss.org/browse/JGRP-1003?page=com.atlassian.jira.plugin....
]
Victor N commented on JGRP-1003:
--------------------------------
I think it is a duplicate of JGRP-1157 and now fixed.
deadlock in TP.send - socket write hangs inside a Lock, preventing
other threads to do TP.send()
------------------------------------------------------------------------------------------------
Key: JGRP-1003
URL: https://issues.jboss.org/browse/JGRP-1003
Project: JGroups
Issue Type: Bug
Affects Versions: 2.7
Environment: Linux (Debian), kernel 2.6.x
Reporter: Victor N
Assignee: Bela Ban
Fix For: 2.10
Attachments: j1__.txt, jgroups_s2.txt, jgroups_stacktrace_09feb2010.txt,
stacktrace.txt
I am using JGroups 2.7.0 GA with the typical protocols stack based on TCP (taken from
tcp.xml).
Sometimes it occurs that socket write operation hangs inside TP.send() -- this is not so
untypical for blocking I/O approach! -- after that JGroups is not working at all because
most of threads are blocked inside TP.send() waiting for that first thread releasing
"out_stream_lock". Below is a fragment from my stack trace.
1) the thread that hangs in socket write until forever (maybe due to a broken socket or
something else); this thread owns "out_stream_lock in TP class":
"Timer-1,name,IP:port" daemon prio=10 tid=0x084e5c00 nid=0x15f0 runnable
[0x211c1000..0x211c2040]
java.lang.Thread.State: RUNNABLE
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
- locked <0x43427300> (a java.io.BufferedOutputStream)
at java.io.DataOutputStream.flush(DataOutputStream.java:106)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection.doSend(TCPConnectionMap.java:472)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection._send(TCPConnectionMap.java:453)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection.send(TCPConnectionMap.java:436)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection.access$100(TCPConnectionMap.java:341)
at org.jgroups.blocks.TCPConnectionMap.send(TCPConnectionMap.java:137)
at org.jgroups.protocols.TCP.send(TCP.java:53)
at org.jgroups.protocols.BasicTCP.sendToSingleMember(BasicTCP.java:141)
at org.jgroups.protocols.TP.doSend(TP.java:1105)
at org.jgroups.protocols.TP.send(TP.java:1088)
at org.jgroups.protocols.TP.down(TP.java:907)
at org.jgroups.protocols.Discovery.down(Discovery.java:363)
at org.jgroups.protocols.MERGE2.down(MERGE2.java:169)
at org.jgroups.protocols.FD_SOCK.down(FD_SOCK.java:333)
at org.jgroups.protocols.FD.down(FD.java:327)
at org.jgroups.protocols.VERIFY_SUSPECT.down(VERIFY_SUSPECT.java:72)
at org.jgroups.protocols.BARRIER.down(BARRIER.java:96)
at org.jgroups.protocols.pbcast.NAKACK.retransmit(NAKACK.java:1530)
at org.jgroups.protocols.pbcast.NAKACK.retransmit(NAKACK.java:1476)
at org.jgroups.stack.Retransmitter$Task.run(Retransmitter.java:207)
at org.jgroups.util.TimeScheduler$TaskWrapper.run(TimeScheduler.java:218)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:207)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Locked ownable synchronizers:
- <0x2b5522e0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
- <0x2b5550f8> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
- <0x434273f0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
2) and there are many threads like this blocked "on out_stream_lock.lock()"
call inside TP class:
"Connection.Receiver [ip:port - ip:port],name,ip:port" prio=10 tid=0x020dc000
nid=0x1fab waiting on condition [0x1365f000..0x1365ffc0]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x2b5550f8> (a
java.util.concurrent.locks.ReentrantLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:778)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1114)
at
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186)
at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262)
at org.jgroups.protocols.TP.send(TP.java:1082)
at org.jgroups.protocols.TP.down(TP.java:907)
at org.jgroups.protocols.Discovery.up(Discovery.java:274)
at org.jgroups.protocols.TP.passMessageUp(TP.java:995)
at org.jgroups.protocols.TP.access$100(TP.java:52)
at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1451)
at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1427)
at
java.util.concurrent.ThreadPoolExecutor$CallerRunsPolicy.rejectedExecution(ThreadPoolExecutor.java:1738)
at
org.jgroups.util.ShutdownRejectedExecutionHandler.rejectedExecution(ShutdownRejectedExecutionHandler.java:34)
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
at org.jgroups.protocols.TP.dispatchToThreadPool(TP.java:1061)
at org.jgroups.protocols.TP.receive(TP.java:1038)
at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:180)
at
org.jgroups.blocks.TCPConnectionMap$TCPConnection$ConnectionPeerReceiver.run(TCPConnectionMap.java:553)
at java.lang.Thread.run(Thread.java:619)
Locked ownable synchronizers:
- None
(Since all threads from JGroups' ThreadPool are locked, you can see
"rejectedExecution" in the last stack trace).
Maybe we could implement some solution for TCP (blocking I/O)? Maybe it is possible to
release the lock before socket write?
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
5707
days inactive
6258
days old
41 comments
2 participants
participants (2)
-
Bela Ban (JIRA) -
Victor N (JIRA)