[jboss-dev-forums] [Design of Messaging on JBoss (Messaging/JBoss)] - Programmatic ThreadDumps on the TestSuite

[clebert.suconic@jboss.com]

I have added a few calls to a method that I wrote on UnitTestCase (threadDump) around the testsuite.

In particular, I added one at ClusterTestBase::waitForBindings, and it fired a dump while running the testsuite on my box:


Most of the threads were idle waiting tasks on the thread pool, but few threads were working around ManagementService:


-.ManagementServiceImpl.sendNotification(ManagementServiceImpl.java:526) - this line is waiting for a lock:

    [junit] ===============================================================================
  |     [junit] Thread Thread[New I/O server worker #198-5,5,jbm-netty-acceptor-worker-threads-292661220] name = New I/O server worker #198-5 group = java.lang.ThreadGroup[name=jbm-netty-acceptor-worker-threads-292661220,maxpri=10]
  |     [junit] 
  |     [junit] org.jboss.messaging.core.management.impl.ManagementServiceImpl.sendNotification(ManagementServiceImpl.java:526)
  |     [junit] org.jboss.messaging.core.postoffice.impl.PostOfficeImpl.addBinding(PostOfficeImpl.java:510)
  |     [junit] org.jboss.messaging.core.server.impl.ServerSessionImpl.doHandleCreateQueue(ServerSessionImpl.java:1340)
  |     [junit] org.jboss.messaging.core.server.impl.ServerSessionImpl.handleCreateQueue(ServerSessionImpl.java:378)
  |     [junit] org.jboss.messaging.core.server.impl.ServerSessionPacketHandler.handlePacket(ServerSessionPacketHandler.java:126)
  |     [junit] org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl$ChannelImpl.handlePacket(RemotingConnectionImpl.java:1486)
  |     [junit] org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl$ChannelImpl.access$400(RemotingConnectionImpl.java:846)
  |     [junit] org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.bufferReceived(RemotingConnectionImpl.java:493)
  |     [junit] org.jboss.messaging.core.remoting.server.impl.RemotingServiceImpl$DelegatingBufferHandler.bufferReceived(RemotingServiceImpl.java:361)
  |     [junit] org.jboss.messaging.integration.transports.netty.MessagingChannelHandler.messageReceived(MessagingChannelHandler.java:73)
  |     [junit] org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:105)
  |     [junit] org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:577)
  |     [junit] org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:804)
  |     [junit] org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:385)
  |     [junit] org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:279)
  |     [junit] org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:202)
  |     [junit] org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:105)
  |     [junit] org.jboss.netty.handler.codec.frame.FrameDecoder.handleUpstream(FrameDecoder.java:162)
  |     [junit] org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:577)
  |     [junit] org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:572)
  |     [junit] org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:342)
  |     [junit] org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:329)
  |     [junit] org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:302)
  |     [junit] org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:254)
  |     [junit] org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:171)
  |     [junit] org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:72)
  |     [junit] java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
  |     [junit] java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
  |     [junit] java.lang.Thread.run(Thread.java:595)
  | 

Same thing here:

    [junit] ===============================================================================
  |     [junit] Thread Thread[New I/O server worker #198-3,5,jbm-netty-acceptor-worker-threads-292661220] name = New I/O server worker #198-3 group = java.lang.ThreadGroup[name=jbm-netty-acceptor-worker-threads-292661220,maxpri=10]
  |     [junit] 
  |     [junit] org.jboss.messaging.core.management.impl.ManagementServiceImpl.sendNotification(ManagementServiceImpl.java:526)
  |     [junit] org.jboss.messaging.core.server.impl.ServerSessionImpl.doHandleCreateConsumer(ServerSessionImpl.java:1262)
  |     [junit] org.jboss.messaging.core.server.impl.ServerSessionImpl.handleCreateConsumer(ServerSessionImpl.java:360)
  |     [junit] org.jboss.messaging.core.server.impl.ServerSessionPacketHandler.handlePacket(ServerSessionPacketHandler.java:120)
  |     [junit] org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl$ChannelImpl.handlePacket(RemotingConnectionImpl.java:1486)
  |     [junit] org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl$ChannelImpl.access$400(RemotingConnectionImpl.java:846)
  |     [junit] org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.bufferReceived(RemotingConnectionImpl.java:493)
  |     [junit] org.jboss.messaging.core.remoting.server.impl.RemotingServiceImpl$DelegatingBufferHandler.bufferReceived(RemotingServiceImpl.java:361)
  |     [junit] org.jboss.messaging.integration.transports.netty.MessagingChannelHandler.messageReceived(MessagingChannelHandler.java:73)
  |     [junit] org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:105)
  |     [junit] org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:577)
  |     [junit] org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:804)
  |     [junit] org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:385)
  |     [junit] org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:279)
  |     [junit] org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:202)
  |     [junit] org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:105)
  |     [junit] org.jboss.netty.handler.codec.frame.FrameDecoder.handleUpstream(FrameDecoder.java:162)
  |     [junit] org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:577)
  |     [junit] org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:572)
  |     [junit] org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:342)
  |     [junit] org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:329)
  |     [junit] org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:302)
  |     [junit] org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:254)
  |     [junit] org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:171)
  |     [junit] org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:72)
  |     [junit] java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
  |     [junit] java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
  |     [junit] java.lang.Thread.run(Thread.java:595)
  |     [junit] ===============================================================================
  | 


And this one is working:


  |     [junit] ===============================================================================
  |     [junit] Thread Thread[Thread-5 (group:jbm-client-session-threads-441502220),5,jbm-client-session-threads-441502220] name = Thread-5 (group:jbm-client-session-threads-441502220) group = java.lang.ThreadGroup[name=jbm-client-session-threads-441502220,maxpri=10]
  |     [junit]
  |     [junit] java.lang.Throwable.fillInStackTrace(Native Method)
  |     [junit] java.lang.Throwable.<init>(Throwable.java:196)
  |     [junit] java.lang.Exception.<init>(Exception.java:41)
  |     [junit] java.lang.RuntimeException.<init>(RuntimeException.java:43)
  |     [junit] java.lang.IndexOutOfBoundsException.<init>(IndexOutOfBoundsException.java:37)
  |     [junit] java.util.concurrent.CopyOnWriteArrayList.rangeCheck(CopyOnWriteArrayList.java:708)
  |     [junit] java.util.concurrent.CopyOnWriteArrayList.get(CopyOnWriteArrayList.java:328)
  |     [junit] org.jboss.messaging.core.postoffice.impl.BindingsImpl.route(BindingsImpl.java:307)
  |     [junit] org.jboss.messaging.core.postoffice.impl.PostOfficeImpl.route(PostOfficeImpl.java:635)
  |     [junit] org.jboss.messaging.core.management.impl.ManagementServiceImpl.sendNotification(ManagementServiceImpl.java:568)
  |     [junit] org.jboss.messaging.core.server.cluster.impl.ClusterConnectionImpl$MessageFlowRecordImpl.doConsumerCreated(ClusterConnectionImpl.java:784)
  |     [junit] org.jboss.messaging.core.server.cluster.impl.ClusterConnectionImpl$MessageFlowRecordImpl.onMessage(ClusterConnectionImpl.java:565)
  |     [junit] org.jboss.messaging.core.client.impl.ClientConsumerImpl.callOnMessage(ClientConsumerImpl.java:564)
  |     [junit] org.jboss.messaging.core.client.impl.ClientConsumerImpl.access$300(ClientConsumerImpl.java:45)
  |     [junit] org.jboss.messaging.core.client.impl.ClientConsumerImpl$Runner.run(ClientConsumerImpl.java:748)
  |     [junit] org.jboss.messaging.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:105)
  |     [junit] java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
  |     [junit] java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
  |     [junit] java.lang.Thread.run(Thread.java:595)
  |     [junit] ===============================================================================
  | 
  | 


I also see several timeouts while this is happening.

I will make another post with the complete dump. (which is somewhat big (5000 lines))




View the original post : http://www.jboss.org/index.html?module=bb&op=viewtopic&p=4217504#4217504

Reply to the post : http://www.jboss.org/index.html?module=bb&op=posting&mode=reply&p=4217504