[Design of Messaging on JBoss (Messaging/JBoss)] - PingStressTests hanging
by clebert.suconic@jboss.com
It' s not formally a deadlock...
But if you enable PingStressTest (getNumberOfIterations() > 10), you will see one thread waiting the executor to finish, and another thread waiting to lock:
| "LocalThread i = 10" prio=10 tid=0x00007fc50035c000 nid=0x6484 waiting on condition [0x0000000040849000..0x0000000040849c00]
| java.lang.Thread.State: TIMED_WAITING (parking)
| at sun.misc.Unsafe.park(Native Method)
| - parking to wait for <0x00007fc50a5eebe8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
| at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
| at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)
| at java.util.concurrent.ThreadPoolExecutor.awaitTermination(ThreadPoolExecutor.java:1244)
| at org.jboss.messaging.integration.transports.netty.NettyConnector.close(NettyConnector.java:363)
| - locked <0x00007fc50a6114d0> (a org.jboss.messaging.integration.transports.netty.NettyConnector)
| at org.jboss.messaging.core.client.impl.ConnectionManagerImpl.checkCloseConnections(ConnectionManagerImpl.java:761)
| at org.jboss.messaging.core.client.impl.ConnectionManagerImpl.returnConnection(ConnectionManagerImpl.java:852)
| at org.jboss.messaging.core.client.impl.ConnectionManagerImpl.removeSession(ConnectionManagerImpl.java:387)
| - locked <0x00007fc50a611708> (a java.lang.Object)
| - locked <0x00007fc50a6116f8> (a java.lang.Object)
| at org.jboss.messaging.core.client.impl.ClientSessionImpl.doCleanup(ClientSessionImpl.java:1313)
| at org.jboss.messaging.core.client.impl.ClientSessionImpl.close(ClientSessionImpl.java:761)
| at org.jboss.messaging.tests.stress.remote.PingStressTest$1LocalThread.run(PingStressTest.java:248)
|
|
| "Thread-1 (group:jbm-pinger-threads-896472140)" daemon prio=10 tid=0x00007fc500385c00 nid=0x6463 waiting for monitor entry [0x0000000045ece000..0x0000000045ecec00]
| java.lang.Thread.State: BLOCKED (on object monitor)
| at org.jboss.messaging.core.client.impl.ConnectionManagerImpl.failover(ConnectionManagerImpl.java:493)
| - waiting to lock <0x00007fc50a611708> (a java.lang.Object)
| at org.jboss.messaging.core.client.impl.ConnectionManagerImpl.connectionFailed(ConnectionManagerImpl.java:411)
| at org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.callListeners(RemotingConnectionImpl.java:530)
| at org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:421)
| at org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl$Pinger.run(RemotingConnectionImpl.java:1537)
| - locked <0x00007fc50a614508> (a org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl$Pinger)
| at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
| at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
| at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
| at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
| at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
| at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
| at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
| at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
| at java.lang.Thread.run(Thread.java:619)
|
|
|
| "LocalThread i = 10" prio=10 tid=0x00007fc50035c000 nid=0x6484 waiting on condition [0x0000000040849000..0x0000000040849c00]
| java.lang.Thread.State: TIMED_WAITING (parking)
| at sun.misc.Unsafe.park(Native Method)
| - parking to wait for <0x00007fc50a5eebe8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
| at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
| at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)
| at java.util.concurrent.ThreadPoolExecutor.awaitTermination(ThreadPoolExecutor.java:1244)
| at org.jboss.messaging.integration.transports.netty.NettyConnector.close(NettyConnector.java:363)
| - locked <0x00007fc50a6114d0> (a org.jboss.messaging.integration.transports.netty.NettyConnector)
| at org.jboss.messaging.core.client.impl.ConnectionManagerImpl.checkCloseConnections(ConnectionManagerImpl.java:761)
| at org.jboss.messaging.core.client.impl.ConnectionManagerImpl.returnConnection(ConnectionManagerImpl.java:852)
| at org.jboss.messaging.core.client.impl.ConnectionManagerImpl.removeSession(ConnectionManagerImpl.java:387)
| - locked <0x00007fc50a611708> (a java.lang.Object)
| - locked <0x00007fc50a6116f8> (a java.lang.Object)
| at org.jboss.messaging.core.client.impl.ClientSessionImpl.doCleanup(ClientSessionImpl.java:1313)
| at org.jboss.messaging.core.client.impl.ClientSessionImpl.close(ClientSessionImpl.java:761)
| at org.jboss.messaging.tests.stress.remote.PingStressTest$1LocalThread.run(PingStressTest.java:248)
|
|
NettyConnector.close is waiting the executor to finish,
But the Pinger is waiting the lock, while it holds the executor.
To fix this I would need to run the Pinger cleanup on a different thread. This problem was fixed with this fix:
| Index: src/main/org/jboss/messaging/core/remoting/impl/RemotingConnectionImpl.java
| ===================================================================
| --- src/main/org/jboss/messaging/core/remoting/impl/RemotingConnectionImpl.java (revision 6007)
| +++ src/main/org/jboss/messaging/core/remoting/impl/RemotingConnectionImpl.java (working copy)
| @@ -417,10 +417,17 @@
|
| log.warn("Connection failed, client " + client + " " + System.identityHashCode(this) + " " + me.getMessage(), me);
|
| - // Then call the listeners
| - callListeners(me);
| + // Using another thread to avoid a hang or dead lock (another thread waiting an executor to f
| inish, while it holds the lock)
| + new Thread()
| + {
| + public void run()
| + {
| + // Then call the listeners
| + callListeners(me);
|
| - internalClose();
| + internalClose();
| + }
| + }.start();
| }
|
| public void destroy()
|
|
I found this while investigating another deadlock.
It is realted to the waitToFinish change on the executors, but it s probably a differnt issue. (different from what I originally thought)
View the original post : http://www.jboss.org/index.html?module=bb&op=viewtopic&p=4215350#4215350
Reply to the post : http://www.jboss.org/index.html?module=bb&op=posting&mode=reply&p=4215350
17 years, 1 month
[Design of Messaging on JBoss (Messaging/JBoss)] - Re: Another DeadLock, something around InVM
by ataylor
Ive checked the logging, i log the packet being received, the lock before the response thread tries to lock it and just after it has obtained the lock (which doesn't appear) and the stacktrace from the createsession method prefixed with lots of *'s.
[junit] main 14:51:49,532 INFO [SecurityTest] ####################################################### Start IN-VM test: testLoginValidUserInvalidPassword
| [junit] New I/O server worker #6-2 14:51:49,537 ERROR [MessagingServerPacketHandler] Failed to create session
| [junit] MessagingException[errorCode=105 message=Unable to validate user: john]
| [junit] at org.jboss.messaging.core.security.impl.SecurityStoreImpl.authenticate(SecurityStoreImpl.java:119)
| [junit] at org.jboss.messaging.core.server.impl.MessagingServerImpl.doCreateSession(MessagingServerImpl.java:975)
| [junit] at org.jboss.messaging.core.server.impl.MessagingServerImpl.createSession(MessagingServerImpl.java:693)
| [junit] at org.jboss.messaging.core.server.impl.MessagingServerPacketHandler.doHandleCreateSession(MessagingServerPacketHandler.java:103)
| [junit] at org.jboss.messaging.core.server.impl.MessagingServerPacketHandler.handleCreateSession(MessagingServerPacketHandler.java:140)
| [junit] at org.jboss.messaging.core.server.impl.MessagingServerPacketHandler.handlePacket(MessagingServerPacketHandler.java:71)
| [junit] at org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl$ChannelImpl.handlePacket(RemotingConnectionImpl.java:1472)
| [junit] at org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl$ChannelImpl.access$400(RemotingConnectionImpl.java:845)
| [junit] at org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.bufferReceived(RemotingConnectionImpl.java:492)
| [junit] at org.jboss.messaging.core.remoting.server.impl.RemotingServiceImpl$DelegatingBufferHandler.bufferReceived(RemotingServiceImpl.java:361)
| [junit] at org.jboss.messaging.integration.transports.netty.MessagingChannelHandler.messageReceived(MessagingChannelHandler.java:62)
| [junit] at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:105)
| [junit] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:577)
| [junit] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:804)
| [junit] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:385)
| [junit] at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:279)
| [junit] at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:202)
| [junit] at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:105)
| [junit] at org.jboss.netty.handler.codec.frame.FrameDecoder.handleUpstream(FrameDecoder.java:162)
| [junit] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:577)
| [junit] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:572)
| [junit] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:342)
| [junit] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:329)
| [junit] at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:302)
| [junit] at org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:254)
| [junit] at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:171)
| [junit] at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:72)
| [junit] at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
| [junit] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
| [junit] at java.lang.Thread.run(Thread.java:619)
| [junit] New I/O client worker #317-1 14:51:49,538 INFO [RemotingConnectionImpl] packet = PACKET[type=20, channelID=1], exception= MessagingException[errorCode=105 message=Unable to validate user: john]]
| [junit] New I/O client worker #317-1 14:51:49,538 INFO [RemotingConnectionImpl] lock = java.util.concurrent.locks.ReentrantLock@12b71a3[Locked by thread main]
| [junit] main 14:51:49,538 INFO [ConnectionManagerImpl] ************************************************
| [junit] MessagingException[errorCode=105 message=Unable to validate user: john]
| [junit] at org.jboss.messaging.core.remoting.impl.wireformat.MessagingExceptionMessage.decodeBody(MessagingExceptionMessage.java:81)
| [junit] at org.jboss.messaging.core.remoting.impl.wireformat.PacketImpl.decode(PacketImpl.java:198)
| [junit] at org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.decode(RemotingConnectionImpl.java:836)
| [junit] at org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.bufferReceived(RemotingConnectionImpl.java:482)
| [junit] at org.jboss.messaging.core.client.impl.ConnectionManagerImpl$DelegatingBufferHandler.bufferReceived(ConnectionManagerImpl.java:944)
| [junit] at org.jboss.messaging.integration.transports.netty.MessagingChannelHandler.messageReceived(MessagingChannelHandler.java:62)
| [junit] at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:105)
| [junit] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:577)
| [junit] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:804)
| [junit] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:385)
| [junit] at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:279)
| [junit] at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:202)
| [junit] at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:105)
| [junit] at org.jboss.netty.handler.codec.frame.FrameDecoder.handleUpstream(FrameDecoder.java:162)
| [junit] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:577)
| [junit] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:572)
| [junit] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:342)
| [junit] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:329)
| [junit] at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:302)
| [junit] at org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:254)
| [junit] at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:171)
| [junit] at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:72)
| [junit] at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
| [junit] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
| [junit] at java.lang.Thread.run(Thread.java:619)
|
|
|
View the original post : http://www.jboss.org/index.html?module=bb&op=viewtopic&p=4215344#4215344
Reply to the post : http://www.jboss.org/index.html?module=bb&op=posting&mode=reply&p=4215344
17 years, 1 month
[Design of OSGi Integration] - Re: JBOSGI-38 and bundle deployment order
by adrian@jboss.org
"adrian(a)jboss.org" wrote :
| If you don't want the exception then you need to control the "required state"
| of the deployment such that it doesn't move to "Installed" and instead remains
| the unresolved state (Described?).
|
| There already is a method to change the "required state" of an existing deployment
| DeployerClient.change(deploymentName, DeploymentState);
| but there's nothing currently that lets you set it at initial deployment, e.g. something like
| DeployerClient.deploy(Deployment, DeploymentState); // DOES NOT EXIST
|
In principle to reproduce what other OSGi implementations do, we should make
the OSGi bundle deployments "On Demand" and only move them to the Installed state
when somebody asks for a class in one of the packages they export.
This would require some notification from the classloading layer.
I personally don't like this behaviour because it means services are not registered
until somebody tries to load one of the classes or you somehow manually work out which
bundle provides which service(s) and explicitly activate them.
The "somebody tries to load one of the classes" is almost certainly not going to
happen since the classes are usually implementation details while the caller uses
an interface from some other bundle/deployment.
View the original post : http://www.jboss.org/index.html?module=bb&op=viewtopic&p=4215336#4215336
Reply to the post : http://www.jboss.org/index.html?module=bb&op=posting&mode=reply&p=4215336
17 years, 1 month