[JBoss Messaging] - MDBs not getting messages after long run
by mpr_prabhu
We are running into strange problem with JBoss Messaging when we are doing performance tesing of our application after a long run (typically after 4-5 hours).
Our application is high volume application (200-250 TPS) and during load tests everything is fine until we encounter this issue when our MDBs seem to freeze.
When this happens the messages just pile up on the queues and MDBs do not seem to get any messages. Only way out of this sutuation is to restart the JBoss server
-- Everything on the server seems to be fine (We can access JMX Console, resource utilization is normal etc.)
-- There are no errors in the logs anywhere and we are sure all exceptions in application from the MDBs are handled (there are no un handled exceptions)
-- We have taken thread dumps that seem to indicate that worker threads are waiting for work but are not given anything (though there are messages in the queue)
We are running following configuration in our 3 node cluster
-- JBoss 4.2.0. EAP - CP01
-- JBoss Messaging 1.4.1 SP1
-- Recently patched concurrent.jar to 1.34 as we found in JIRA that there are some new fixes that are missing in our JBoss package.
Any ideas of what could be going on?
Thread dump from JMX Console
Total Threads: 251Total Thread Groups: 10Thread Group: system : max priority:10, demon:falseThread: Reference Handler : priority:10, demon:true, threadId:2, threadState:WAITING, lockName:java.lang.ref.Reference$Lock@b22549java.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)Thread: Finalizer : priority:8, demon:true, threadId:3, threadState:WAITING, lockName:java.lang.ref.ReferenceQueue$Lock@1255cdcjava.lang.Object.wait(Native Method)java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)Thread: Signal Dispatcher : priority:9, demon:true, threadId:5, threadState:RUNNABLE, lockName:nullThread: RMI TCP Accept-0 : priority:5, demon:true, threadId:12, threadState:RUNNABLE, lockName:nulljava.net.PlainSocketImpl.socketAccept(Native Method)java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)java.net.ServerSocket.implAccept(ServerSocket.java:450)java.net.ServerSocket.accept(ServerSocket.java:421)sun.rmi.transport.tcp.TCPTransport.run(TCPTransport.java:340)java.lang.Thread.run(Thread.java:595)Thread: RMI TCP Accept-9003 : priority:5, demon:true, threadId:13, threadState:RUNNABLE, lockName:nulljava.net.PlainSocketImpl.socketAccept(Native Method)java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)java.net.ServerSocket.implAccept(ServerSocket.java:450)java.net.ServerSocket.accept(ServerSocket.java:421)sun.rmi.transport.tcp.TCPTransport.run(TCPTransport.java:340)java.lang.Thread.run(Thread.java:595)Thread: RMI TCP Accept-1098 : priority:5, demon:true, threadId:21, threadState:RUNNABLE, lockName:nulljava.net.PlainSocketImpl.socketAccept(Native Method)java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)java.net.ServerSocket.implAccept(ServerSocket.java:450)java.net.ServerSocket.accept(ServerSocket.java:421)sun.rmi.transport.tcp.TCPTransport.run(TCPTransport.java:340)java.lang.Thread.run(Thread.java:595)Thread: RMI Reaper : priority:5, demon:false, threadId:22, threadState:WAITING, lockName:java.lang.ref.ReferenceQueue$Lock@11af8b0java.lang.Object.wait(Native Method)java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)sun.rmi.transport.ObjectTable$Reaper.run(ObjectTable.java:336)java.lang.Thread.run(Thread.java:595)Thread: GC Daemon : priority:2, demon:true, threadId:23, threadState:TIMED_WAITING, lockName:sun.misc.GC$LatencyLock@27db1cjava.lang.Object.wait(Native Method)sun.misc.GC$Daemon.run(GC.java:100)Thread: RMI TCP Accept-4444 : priority:5, demon:true, threadId:32, threadState:RUNNABLE, lockName:nulljava.net.PlainSocketImpl.socketAccept(Native Method)java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)java.net.ServerSocket.implAccept(ServerSocket.java:450)java.net.ServerSocket.accept(ServerSocket.java:421)sun.rmi.transport.tcp.TCPTransport.run(TCPTransport.java:340)java.lang.Thread.run(Thread.java:595)Thread: RMI TCP Accept-1101 : priority:5, demon:true, threadId:73, threadState:RUNNABLE, lockName:nulljava.net.PlainSocketImpl.socketAccept(Native Method)java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)java.net.ServerSocket.implAccept(ServerSocket.java:450)java.net.ServerSocket.accept(ServerSocket.java:421)sun.rmi.transport.tcp.TCPTransport.run(TCPTransport.java:340)java.lang.Thread.run(Thread.java:595)Thread: RMI TCP Accept-4447 : priority:5, demon:true, threadId:76, threadState:RUNNABLE, lockName:nulljava.net.PlainSocketImpl.socketAccept(Native Method)java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)java.net.ServerSocket.implAccept(ServerSocket.java:450)java.net.ServerSocket.accept(ServerSocket.java:421)sun.rmi.transport.tcp.TCPTransport.run(TCPTransport.java:340)java.lang.Thread.run(Thread.java:595)Thread: RMI LeaseChecker : priority:5, demon:true, threadId:158, threadState:TIMED_WAITING, lockName:nulljava.lang.Thread.sleep(Native Method)sun.rmi.transport.DGCImpl$LeaseChecker.run(DGCImpl.java:310)java.lang.Thread.run(Thread.java:595)Thread: RMI RenewClean-[172.31.25.56:1101] : priority:5, demon:true, threadId:182, threadState:TIMED_WAITING, lockName:java.lang.ref.ReferenceQueue$Lock@df10ejava.lang.Object.wait(Native Method)java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run(DGCClient.java:501)java.lang.Thread.run(Thread.java:595)Thread: RMI ConnectionExpiration-[172.31.25.56:1101] : priority:5, demon:true, threadId:3261, threadState:TIMED_WAITING, lockName:nulljava.lang.Thread.sleep(Native Method)sun.rmi.transport.tcp.TCPChannel$Reaper.run(TCPChannel.java:446)java.lang.Thread.run(Thread.java:595)Thread Group: main : max priority:10, demon:falseThread: Timer-0 : priority:5, demon:true, threadId:11, threadState:WAITING, lockName:java.util.TaskQueue@9c5dejava.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)java.util.TimerThread.mainLoop(Timer.java:483)java.util.TimerThread.run(Timer.java:462)Thread: DestroyJavaVM : priority:5, demon:false, threadId:15, threadState:RUNNABLE, lockName:nullThread Group: jboss : max priority:10, demon:falseThread: Timer-1 : priority:5, demon:true, threadId:17, threadState:TIMED_WAITING, lockName:java.util.TaskQueue@fa7d0cjava.lang.Object.wait(Native Method)java.util.TimerThread.mainLoop(Timer.java:509)java.util.TimerThread.run(Timer.java:462)Thread: ScannerThread : priority:5, demon:true, threadId:18, threadState:TIMED_WAITING, lockName:nulljava.lang.Thread.sleep(Native Method)org.jboss.deployment.scanner.AbstractDeploymentScanner$ScannerThread.loop(AbstractDeploymentScanner.java:280)org.jboss.deployment.scanner.AbstractDeploymentScanner$ScannerThread.run(AbstractDeploymentScanner.java:225)Thread: Listener:33909 : priority:5, demon:true, threadId:25, threadState:RUNNABLE, lockName:nulljava.net.PlainSocketImpl.socketAccept(Native Method)java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)java.net.ServerSocket.implAccept(ServerSocket.java:450)java.net.ServerSocket.accept(ServerSocket.java:421)com.arjuna.ats.internal.arjuna.recovery.Listener.run(Listener.java:119)Thread: Thread-4 : priority:5, demon:true, threadId:27, threadState:TIMED_WAITING, lockName:com.arjuna.ats.arjuna.coordinator.TransactionReaper@100893ajava.lang.Object.wait(Native Method)com.arjuna.ats.internal.arjuna.coordinator.ReaperThread.run(ReaperThread.java:98)Thread: Thread-5 : priority:5, demon:true, threadId:28, threadState:TIMED_WAITING, lockName:nulljava.lang.Thread.sleep(Native Method)com.arjuna.ats.internal.arjuna.recovery.ExpiredEntryMonitor.run(ExpiredEntryMonitor.java:237)Thread: Listener:49466 : priority:5, demon:true, threadId:31, threadState:RUNNABLE, lockName:nulljava.net.PlainSocketImpl.socketAccept(Native Method)java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)java.net.ServerSocket.implAccept(ServerSocket.java:450)java.net.ServerSocket.accept(ServerSocket.java:421)com.arjuna.ats.internal.arjuna.recovery.Listener.run(Listener.java:119)Thread: PooledInvokerAcceptor#0-4445 : priority:5, demon:false, threadId:33, threadState:RUNNABLE, lockName:nulljava.net.PlainSocketImpl.socketAccept(Native Method)java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)java.net.ServerSocket.implAccept(ServerSocket.java:450)java.net.ServerSocket.accept(ServerSocket.java:421)org.jboss.invocation.pooled.server.PooledInvoker.run(PooledInvoker.java:262)java.lang.Thread.run(Thread.java:595)Thread: AcceptorThread#0:4446 : priority:5, demon:false, threadId:34, threadState:RUNNABLE, lockName:nulljava.net.PlainSocketImpl.socketAccept(Native Method)java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)java.net.ServerSocket.implAccept(ServerSocket.java:450)java.net.ServerSocket.accept(ServerSocket.java:421)org.jboss.remoting.transport.socket.SocketServerInvoker.run(SocketServerInvoker.java:520)java.lang.Thread.run(Thread.java:595)Thread: ServerSocketRefresh : priority:5, demon:true, threadId:35, threadState:WAITING, lockName:java.lang.Object@10d43aajava.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)org.jboss.remoting.transport.socket.SocketServerInvoker$ServerSocketRefresh.run(SocketServerInvoker.java:704)Thread: AcceptorThread#0:3873 : priority:5, demon:false, threadId:36, threadState:RUNNABLE, lockName:nulljava.net.PlainSocketImpl.socketAccept(Native Method)java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)java.net.ServerSocket.implAccept(ServerSocket.java:450)java.net.ServerSocket.accept(ServerSocket.java:421)org.jboss.remoting.transport.socket.SocketServerInvoker.run(SocketServerInvoker.java:520)java.lang.Thread.run(Thread.java:595)Thread: ServerSocketRefresh : priority:5, demon:true, threadId:37, threadState:WAITING, lockName:java.lang.Object@3aa006java.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)org.jboss.remoting.transport.socket.SocketServerInvoker$ServerSocketRefresh.run(SocketServerInvoker.java:704)Thread: AcceptorThread#0:4457 : priority:5, demon:false, threadId:38, threadState:RUNNABLE, lockName:nulljava.net.PlainSocketImpl.socketAccept(Native Method)java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)java.net.ServerSocket.implAccept(ServerSocket.java:450)java.net.ServerSocket.accept(ServerSocket.java:421)org.jboss.remoting.transport.socket.SocketServerInvoker.run(SocketServerInvoker.java:520)java.lang.Thread.run(Thread.java:595)Thread: ServerSocketRefresh : priority:5, demon:true, threadId:39, threadState:WAITING, lockName:java.lang.Object@17e27d1java.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)org.jboss.remoting.transport.socket.SocketServerInvoker$ServerSocketRefresh.run(SocketServerInvoker.java:704)Thread: secondaryServerSocketThread : priority:5, demon:true, threadId:40, threadState:RUNNABLE, lockName:nulljava.net.PlainSocketImpl.socketAccept(Native Method)java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)java.net.ServerSocket.implAccept(ServerSocket.java:450)java.net.ServerSocket.accept(ServerSocket.java:421)org.jboss.remoting.transport.bisocket.BisocketServerInvoker$SecondaryServerSocketThread.run(BisocketServerInvoker.java:889)Thread: Timer-2 : priority:5, demon:true, threadId:41, threadState:TIMED_WAITING, lockName:java.util.TaskQueue@162f520java.lang.Object.wait(Native Method)java.util.TimerThread.mainLoop(Timer.java:509)java.util.TimerThread.run(Timer.java:462)Thread: SubscriptionWatchDog : priority:5, demon:false, threadId:52, threadState:TIMED_WAITING, lockName:nulljava.lang.Thread.sleep(Native Method)org.jboss.ws.extensions.eventing.mgmt.SubscriptionManager$WatchDog.run(SubscriptionManager.java:643)java.lang.Thread.run(Thread.java:595)Thread: SnmpPortal--1 : priority:5, demon:false, threadId:53, threadState:WAITING, lockName:java.util.LinkedList@2a2bcajava.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)org.opennms.protocols.snmp.SnmpPortal$Receiver.run(SnmpPortal.java:567)Thread: Timer-3 : priority:5, demon:false, threadId:55, threadState:WAITING, lockName:java.util.TaskQueue@1c6dd93java.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)java.util.TimerThread.mainLoop(Timer.java:483)java.util.TimerThread.run(Timer.java:462)Thread: main-FastReceiver : priority:5, demon:false, threadId:54, threadState:RUNNABLE, lockName:nulljava.net.PlainDatagramSocketImpl.receive0(Native Method)java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:136)java.net.DatagramSocket.receive(DatagramSocket.java:712)org.opennms.protocols.snmp.SnmpPortal$1.run(SnmpPortal.java:467)java.lang.Thread.run(Thread.java:595)Thread: SnmpPortal--1 : priority:5, demon:false, threadId:56, threadState:WAITING, lockName:java.util.LinkedList@1489fafjava.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)org.opennms.protocols.snmp.SnmpPortal$Receiver.run(SnmpPortal.java:567)Thread: main-FastReceiver : priority:5, demon:false, threadId:57, threadState:RUNNABLE, lockName:nulljava.net.PlainDatagramSocketImpl.receive0(Native Method)java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:136)java.net.DatagramSocket.receive(DatagramSocket.java:712)org.opennms.protocols.snmp.SnmpPortal$1.run(SnmpPortal.java:467)java.lang.Thread.run(Thread.java:595)Thread: SnmpTimer : priority:5, demon:true, threadId:58, threadState:TIMED_WAITING, lockName:java.util.TreeSet@65aadfjava.lang.Object.wait(Native Method)org.opennms.protocols.snmp.SnmpTimer$TimerLogic.run(SnmpTimer.java:148)java.lang.Thread.run(Thread.java:595)Thread: SnmpPortal--1 : priority:5, demon:false, threadId:59, threadState:WAITING, lockName:java.util.LinkedList@4d11c9java.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)org.opennms.protocols.snmp.SnmpPortal$Receiver.run(SnmpPortal.java:567)Thread: main-FastReceiver : priority:5, demon:false, threadId:60, threadState:RUNNABLE, lockName:nulljava.net.PlainDatagramSocketImpl.receive0(Native Method)java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:136)java.net.DatagramSocket.receive(DatagramSocket.java:712)org.opennms.protocols.snmp.SnmpPortal$1.run(SnmpPortal.java:467)java.lang.Thread.run(Thread.java:595)Thread: Timer-4 : priority:5, demon:true, threadId:61, threadState:TIMED_WAITING, lockName:java.util.TaskQueue@111e079java.lang.Object.wait(Native Method)java.util.TimerThread.mainLoop(Timer.java:509)java.util.TimerThread.run(Timer.java:462)Thread: AsynchKeyChangeHandler Thread : priority:5, demon:false, threadId:71, threadState:WAITING, lockName:java.lang.Object@6465b5java.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)org.jboss.ha.framework.server.AsynchEventHandler.run(AsynchEventHandler.java:102)java.lang.Thread.run(Thread.java:595)Thread: AsynchViewChangeHandler Thread : priority:5, demon:false, threadId:72, threadState:WAITING, lockName:java.lang.Object@1d99ee0java.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)org.jboss.ha.framework.server.AsynchEventHandler.run(AsynchEventHandler.java:102)java.lang.Thread.run(Thread.java:595)Thread: PooledInvokerAcceptor#0-4448 : priority:5, demon:false, threadId:77, threadState:RUNNABLE, lockName:nulljava.net.PlainSocketImpl.socketAccept(Native Method)java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)java.net.ServerSocket.implAccept(ServerSocket.java:450)java.net.ServerSocket.accept(ServerSocket.java:421)org.jboss.invocation.pooled.server.PooledInvoker.run(PooledInvoker.java:262)java.lang.Thread.run(Thread.java:595)Thread: Timer-5 : priority:5, demon:true, threadId:78, threadState:TIMED_WAITING, lockName:java.util.TaskQueue@1a81e0fjava.lang.Object.wait(Native Method)java.util.TimerThread.mainLoop(Timer.java:509)java.util.TimerThread.run(Timer.java:462)Thread: Timer-6 : priority:5, demon:false, threadId:88, threadState:TIMED_WAITING, lockName:java.util.TaskQueue@1913aedjava.lang.Object.wait(Native Method)java.util.TimerThread.mainLoop(Timer.java:509)java.util.TimerThread.run(Timer.java:462)Thread: Timer-7 : priority:5, demon:true, threadId:89, threadState:TIMED_WAITING, lockName:java.util.TaskQueue@115a52djava.lang.Object.wait(Native Method)java.util.TimerThread.mainLoop(Timer.java:509)java.util.TimerThread.run(Timer.java:462)Thread: Timer-8 : priority:5, demon:false, threadId:100, threadState:TIMED_WAITING, lockName:java.util.TaskQueue@71f2d5java.lang.Object.wait(Native Method)java.util.TimerThread.mainLoop(Timer.java:509)java.util.TimerThread.run(Timer.java:462)Thread: RequestController-1 : priority:10, demon:true, threadId:101, threadState:WAITING, lockName:java.lang.Object@70eb8bjava.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)org.jacorb.poa.RequestController.waitForQueue(Unknown Source)org.jacorb.poa.RequestController.run(Unknown Source)Thread: ServerSocketListener : priority:5, demon:true, threadId:102, threadState:RUNNABLE, lockName:nulljava.net.PlainSocketImpl.socketAccept(Native Method)java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)java.net.ServerSocket.implAccept(ServerSocket.java:450)java.net.ServerSocket.accept(ServerSocket.java:421)org.jacorb.orb.iiop.IIOPListener$Acceptor.run(Unknown Source)Thread: ORB thread : priority:5, demon:false, threadId:104, threadState:WAITING, lockName:java.lang.Object@86f3e9java.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)org.jacorb.orb.ORB.run(Unknown Source)org.jboss.iiop.CorbaORBService$1.run(CorbaORBService.java:274)java.lang.Thread.run(Thread.java:595)Thread: RequestController-2 : priority:10, demon:true, threadId:105, threadState:WAITING, lockName:java.lang.Object@167912djava.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)org.jacorb.poa.RequestController.waitForQueue(Unknown Source)org.jacorb.poa.RequestController.run(Unknown Source)Thread: RequestController-3 : priority:10, demon:true, threadId:107, threadState:WAITING, lockName:java.lang.Object@1db21bcjava.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)org.jacorb.poa.RequestController.waitForQueue(Unknown Source)org.jacorb.poa.RequestController.run(Unknown Source)Thread: RequestController-4 : priority:10, demon:true, threadId:108, threadState:WAITING, lockName:java.lang.Object@9350e4java.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)org.jacorb.poa.RequestController.waitForQueue(Unknown Source)org.jacorb.poa.RequestController.run(Unknown Source)Thread: RequestController-5 : priority:10, demon:true, threadId:112, threadState:WAITING, lockName:java.lang.Object@1eadcejava.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)org.jacorb.poa.RequestController.waitForQueue(Unknown Source)org.jacorb.poa.RequestController.run(Unknown Source)Thread: RequestController-6 : priority:10, demon:true, threadId:114, threadState:WAITING, lockName:java.lang.Object@cc54bcjava.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)org.jacorb.poa.RequestController.waitForQueue(Unknown Source)org.jacorb.poa.RequestController.run(Unknown Source)Thread: ClientMessageReceptor0 : priority:5, demon:true, threadId:116, threadState:WAITING, lockName:org.jacorb.orb.iiop.IIOPLoopbackInputStream@ca763djava.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)org.jacorb.orb.iiop.IIOPLoopbackInputStream.internalRead(Unknown Source)org.jacorb.orb.iiop.IIOPLoopbackInputStream.read(Unknown Source)org.jacorb.orb.etf.StreamConnectionBase.read(Unknown Source)org.jacorb.orb.giop.GIOPConnection.getMessage(Unknown Source)org.jacorb.orb.giop.GIOPConnection.receiveMessages(Unknown Source)org.jacorb.orb.giop.MessageReceptor.doWork(Unknown Source)org.jacorb.util.threadpool.ConsumerTie.run(Unknown Source)java.lang.Thread.run(Thread.java:595)Thread: ServerMessageReceptor0 : priority:5, demon:true, threadId:117, threadState:WAITING, lockName:org.jacorb.orb.iiop.IIOPLoopbackInputStream@a2f12java.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)org.jacorb.orb.iiop.IIOPLoopbackInputStream.internalRead(Unknown Source)org.jacorb.orb.iiop.IIOPLoopbackInputStream.read(Unknown Source)org.jacorb.orb.etf.StreamConnectionBase.read(Unknown Source)org.jacorb.orb.giop.GIOPConnection.getMessage(Unknown Source)org.jacorb.orb.giop.GIOPConnection.receiveMessages(Unknown Source)org.jacorb.orb.giop.MessageReceptor.doWork(Unknown Source)org.jacorb.util.threadpool.ConsumerTie.run(Unknown Source)java.lang.Thread.run(Thread.java:595)Thread: RequestProcessor-1 : priority:10, demon:true, threadId:118, threadState:WAITING, lockName:org.jacorb.poa.RequestProcessor@1e9d0ccjava.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)org.jacorb.poa.RequestProcessor.run(Unknown Source)Thread: RequestProcessor-2 : priority:10, demon:true, threadId:119, threadState:WAITING, lockName:org.jacorb.poa.RequestProcessor@18b0b4ajava.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)org.jacorb.poa.RequestProcessor.run(Unknown Source)Thread: ContainerBackgroundProcessor[StandardEngine[jboss.web]] : priority:5, demon:true, threadId:120, threadState:TIMED_WAITING, lockName:nulljava.lang.Thread.sleep(Native Method)org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1549)java.lang.Thread.run(Thread.java:595)Thread: Timer-9 : priority:5, demon:true, threadId:121, threadState:TIMED_WAITING, lockName:java.util.TaskQueue@10a9bf0java.lang.Object.wait(Native Method)java.util.TimerThread.mainLoop(Timer.java:509)java.util.TimerThread.run(Timer.java:462)Thread: Thread-28 : priority:5, demon:true, threadId:122, threadState:TIMED_WAITING, lockName:nulljava.lang.Thread.sleep(Native Method)org.jboss.console.plugins.AOPLister$1.run(AOPLister.java:882)java.lang.Thread.run(Thread.java:595)Thread: IdleRemover : priority:5, demon:true, threadId:123, threadState:TIMED_WAITING, lockName:java.util.ArrayList@12e9518java.lang.Object.wait(Native Method)org.jboss.resource.connectionmanager.IdleRemover$IdleRemoverRunnable.run(IdleRemover.java:161)java.lang.Thread.run(Thread.java:595)Thread: Thread-29 : priority:5, demon:true, threadId:124, threadState:TIMED_WAITING, lockName:nulljava.lang.Thread.sleep(Native Method)org.jboss.messaging.core.impl.memory.SimpleMemoryManager$MemoryRunnable.run(SimpleMemoryManager.java:126)java.lang.Thread.run(Thread.java:595)Thread: Timer-10 : priority:5, demon:true, threadId:125, threadState:TIMED_WAITING, lockName:java.util.TaskQueue@196a9d0java.lang.Object.wait(Native Method)java.util.TimerThread.mainLoop(Timer.java:509)java.util.TimerThread.run(Timer.java:462)Thread: Timer-11 : priority:5, demon:true, threadId:127, threadState:TIMED_WAITING, lockName:java.util.TaskQueue@157f1efjava.lang.Object.wait(Native Method)java.util.TimerThread.mainLoop(Timer.java:509)java.util.TimerThread.run(Timer.java:462)Thread: Scheduler main thread : priority:5, demon:true, threadId:131, threadState:WAITING, lockName:java.lang.Object@d9e701java.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)org.jgroups.util.Queue.peek(Queue.java:404)org.jgroups.util.Scheduler.run(Scheduler.java:77)java.lang.Thread.run(Thread.java:595)Thread: Timer-12 : priority:5, demon:true, threadId:154, threadState:TIMED_WAITING, lockName:java.util.TaskQueue@18a2803java.lang.Object.wait(Native Method)java.util.TimerThread.mainLoop(Timer.java:509)java.util.TimerThread.run(Timer.java:462)Thread: Thread-35 : priority:5, demon:false, threadId:160, threadState:WAITING, lockName:java.lang.Object@1270bc4java.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)java.lang.Thread.run(Thread.java:595)Thread: Connection Consumer for dest JBossQueue[IH_Core_Request_Queue] id=1 : priority:5, demon:false, threadId:161, threadState:WAITING, lockName:java.lang.Object@275db2java.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)org.jboss.jms.client.container.ClientConsumer.getMessage(ClientConsumer.java:797)org.jboss.jms.client.container.ClientConsumer.receive(ClientConsumer.java:416)org.jboss.jms.client.container.ConsumerAspect.handleReceive(ConsumerAspect.java:189)org.jboss.aop.advice.org.jboss.jms.client.container.ConsumerAspect35.invoke(ConsumerAspect35.java)org.jboss.jms.client.delegate.ClientConsumerDelegate$receive_N8299950230150603585.invokeNext(ClientConsumerDelegate$receive_N8299950230150603585.java)org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)org.jboss.jms.client.delegate.ClientConsumerDelegate$receive_N8299950230150603585.invokeNext(ClientConsumerDelegate$receive_N8299950230150603585.java)org.jboss.jms.client.delegate.ClientConsumerDelegate.receive(ClientConsumerDelegate.java)org.jboss.jms.client.JBossConnectionConsumer.run(JBossConnectionConsumer.java:248)java.lang.Thread.run(Thread.java:595)Thread: ScannerThread : priority:5, demon:true, threadId:162, threadState:TIMED_WAITING, lockName:nulljava.lang.Thread.sleep(Native Method)org.jboss.deployment.scanner.AbstractDeploymentScanner$ScannerThread.loop(AbstractDeploymentScanner.java:280)org.jboss.deployment.scanner.AbstractDeploymentScanner$ScannerThread.run(AbstractDeploymentScanner.java:225)Thread: JBossLifeThread : priority:5, demon:false, threadId:164, threadState:WAITING, lockName:java.lang.Object@1d34f56java.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)org.jboss.system.server.ServerImpl$LifeThread.run(ServerImpl.java:940)Thread: Thread-6 : priority:5, demon:false, threadId:29, threadState:TIMED_WAITING, lockName:nulljava.lang.Thread.sleep(Native Method)com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.doWork(PeriodicRecovery.java:248)com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.run(PeriodicRecovery.java:163)Thread: http-172.31.25.61-8080-Acceptor-0 : priority:5, demon:true, threadId:165, threadState:RUNNABLE, lockName:nulljava.net.PlainSocketImpl.socketAccept(Native Method)java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)java.net.ServerSocket.implAccept(ServerSocket.java:450)java.net.ServerSocket.accept(ServerSocket.java:421)org.apache.tomcat.util.net.DefaultServerSocketFactory.acceptSocket(DefaultServerSocketFactory.java:61)org.apache.tomcat.util.net.JIoEndpoint$Acceptor.run(JIoEndpoint.java:310)java.lang.Thread.run(Thread.java:595)Thread: ajp-172.31.25.61-8009-Acceptor-0 : priority:5, demon:true, threadId:166, threadState:RUNNABLE, lockName:nulljava.net.PlainSocketImpl.socketAccept(Native Method)java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)java.net.ServerSocket.implAccept(ServerSocket.java:450)java.net.ServerSocket.accept(ServerSocket.java:421)org.apache.tomcat.util.net.DefaultServerSocketFactory.acceptSocket(DefaultServerSocketFactory.java:61)org.apache.tomcat.util.net.JIoEndpoint$Acceptor.run(JIoEndpoint.java:310)java.lang.Thread.run(Thread.java:595)Thread: http-172.31.25.61-8080-1 : priority:5, demon:true, threadId:167, threadState:RUNNABLE, lockName:nullsun.management.ThreadImpl.getThreadInfo0(Native Method)sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:142)sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:120)sun.reflect.GeneratedMethodAccessor218.invoke(Unknown Source)sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)java.lang.reflect.Method.invoke(Method.java:585)org.jboss.system.server.ServerInfo.outputJdk5ThreadMXBeanInfo(ServerInfo.java:662)org.jboss.system.server.ServerInfo.getThreadGroupInfo(ServerInfo.java:631)org.jboss.system.server.ServerInfo.getThreadGroupInfo(ServerInfo.java:638)org.jboss.system.server.ServerInfo.getThreadGroupInfo(ServerInfo.java:638)org.jboss.system.server.ServerInfo.listThreadDump(ServerInfo.java:508)sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)java.lang.reflect.Method.invoke(Method.java:585)org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)org.jboss.mx.server.Invocation.invoke(Invocation.java:86)org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)org.jboss.system.server.jmx.LazyMBeanServer.invoke(LazyMBeanServer.java:291)org.jboss.jmx.adaptor.control.Server.invokeOpByName(Server.java:258)org.jboss.jmx.adaptor.control.Server.invokeOp(Server.java:223)org.jboss.jmx.adaptor.html.HtmlAdaptorServlet.invokeOp(HtmlAdaptorServlet.java:262)org.jboss.jmx.adaptor.html.HtmlAdaptorServlet.processRequest(HtmlAdaptorServlet.java:100)org.jboss.jmx.adaptor.html.HtmlAdaptorServlet.doPost(HtmlAdaptorServlet.java:82)javax.servlet.http.HttpServlet.service(HttpServlet.java:710)javax.servlet.http.HttpServlet.service(HttpServlet.java:803)org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230)org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:179)org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:525)org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:84)org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:104)org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:157)org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:241)org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844)org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:580)org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)java.lang.Thread.run(Thread.java:595)Thread: http-172.31.25.61-8080-2 : priority:5, demon:true, threadId:172, threadState:RUNNABLE, lockName:nulljava.net.SocketInputStream.socketRead0(Native Method)java.net.SocketInputStream.read(SocketInputStream.java:129)org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:700)org.apache.coyote.http11.InternalInputBuffer.parseRequestLine(InternalInputBuffer.java:366)org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:805)org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:580)org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)java.lang.Thread.run(Thread.java:595)Thread: ReusableThread : priority:5, demon:true, threadId:197, threadState:WAITING, lockName:org.jgroups.util.ReusableThread@1836b03java.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)org.jgroups.util.ReusableThread.run(ReusableThread.java:201)java.lang.Thread.run(Thread.java:595)Thread: WorkerThread#0[172.31.25.56:51553] : priority:5, demon:false, threadId:198, threadState:RUNNABLE, lockName:nulljava.net.SocketInputStream.socketRead0(Native Method)java.net.SocketInputStream.read(SocketInputStream.java:129)java.io.BufferedInputStream.fill(BufferedInputStream.java:218)java.io.BufferedInputStream.read(BufferedInputStream.java:235)java.io.FilterInputStream.read(FilterInputStream.java:66)org.jboss.remoting.transport.socket.ServerThread.readVersion(ServerThread.java:824)org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:510)org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:387)org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:166)Thread: Timer-13 : priority:5, demon:true, threadId:199, threadState:TIMED_WAITING, lockName:java.util.TaskQueue@8bdac1java.lang.Object.wait(Native Method)java.util.TimerThread.mainLoop(Timer.java:509)java.util.TimerThread.run(Timer.java:462)Thread: Timer-14 : priority:5, demon:true, threadId:200, threadState:TIMED_WAITING, lockName:java.util.TaskQueue@1d781d5java.lang.Object.wait(Native Method)java.util.TimerThread.mainLoop(Timer.java:509)java.util.TimerThread.run(Timer.java:462)Thread: Timer-15 : priority:5, demon:true, threadId:201, threadState:TIMED_WAITING, lockName:java.util.TaskQueue@160c68bjava.lang.Object.wait(Native Method)java.util.TimerThread.mainLoop(Timer.java:509)java.util.TimerThread.run(Timer.java:462)Thread: control: Socket[addr=/172.31.25.56,port=2750,localport=58074] : priority:5, demon:true, threadId:202, threadState:RUNNABLE, lockName:nulljava.net.SocketInputStream.socketRead0(Native Method)java.net.SocketInputStream.read(SocketInputStream.java:129)java.net.SocketInputStream.read(SocketInputStream.java:182)java.io.FilterInputStream.read(FilterInputStream.java:66)org.jboss.remoting.transport.bisocket.BisocketServerInvoker$ControlConnectionThread.run(BisocketServerInvoker.java:741)Thread: WorkerThread#1[172.31.24.54:41537] : priority:5, demon:false, threadId:225, threadState:RUNNABLE, lockName:nulljava.net.SocketInputStream.socketRead0(Native Method)java.net.SocketInputStream.read(SocketInputStream.java:129)java.io.BufferedInputStream.fill(BufferedInputStream.java:218)java.io.BufferedInputStream.read(BufferedInputStream.java:235)java.io.FilterInputStream.read(FilterInputStream.java:66)org.jboss.remoting.transport.socket.ServerThread.readVersion(ServerThread.java:824)org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:510)org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:387)org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:166)Thread: WorkerThread#2[172.31.24.54:41539] : priority:5, demon:false, threadId:226, threadState:RUNNABLE, lockName:nulljava.net.SocketInputStream.socketRead0(Native Method)java.net.SocketInputStream.read(SocketInputStream.java:129)java.io.BufferedInputStream.fill(BufferedInputStream.java:218)java.io.BufferedInputStream.read(BufferedInputStream.java:235)java.io.FilterInputStream.read(FilterInputStream.java:66)org.jboss.remoting.transport.socket.ServerThread.readVersion(ServerThread.java:824)org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:510)org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:387)org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:166)Thread: WorkerThread#3[172.31.24.54:41541] : priority:5, demon:false, threadId:227, threadState:RUNNABLE, lockName:nulljava.net.SocketInputStream.socketRead0(Native Method)java.net.SocketInputStream.read(SocketInputStream.java:129)java.io.BufferedInputStream.fill(BufferedInputStream.java:218)java.io.BufferedInputStream.read(BufferedInputStream.java:235)java.io.FilterInputStream.read(FilterInputStream.java:66)org.jboss.remoting.transport.socket.ServerThread.readVersion(ServerThread.java:824)org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:510)org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:387)org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:166)Thread: WorkerThread#4[172.31.24.54:41543] : priority:5, demon:false, threadId:228, threadState:RUNNABLE, lockName:nulljava.net.SocketInputStream.socketRead0(Native Method)java.net.SocketInputStream.read(SocketInputStream.java:129)java.io.BufferedInputStream.fill(BufferedInputStream.java:218)java.io.BufferedInputStream.read(BufferedInputStream.java:235)java.io.FilterInputStream.read(FilterInputStream.java:66)org.jboss.remoting.transport.socket.ServerThread.readVersion(ServerThread.java:824)org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:510)org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:387)org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:166)Thread: WorkerThread#5[172.31.24.54:41545] : priority:5, demon:false, threadId:233, threadState:RUNNABLE, lockName:nulljava.net.SocketInputStream.socketRead0(Native Method)java.net.SocketInputStream.read(SocketInputStream.java:129)java.io.BufferedInputStream.fill(BufferedInputStream.java:218)java.io.BufferedInputStream.read(BufferedInputStream.java:235)java.io.FilterInputStream.read(FilterInputStream.java:66)org.jboss.remoting.transport.socket.ServerThread.readVersion(ServerThread.java:824)org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:510)org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:387)org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:166)Thread: WorkerThread#6[172.31.24.54:41546] : priority:5, demon:false, threadId:234, threadState:RUNNABLE, lockName:nulljava.net.SocketInputStream.socketRead0(Native Method)java.net.SocketInputStream.read(SocketInputStream.java:129)java.io.BufferedInputStream.fill(BufferedInputStream.java:218)java.io.BufferedInputStream.read(BufferedInputStream.java:235)java.io.FilterInputStream.read(FilterInputStream.java:66)org.jboss.remoting.transport.socket.ServerThread.readVersion(ServerThread.java:824)org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:510)org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:387)org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:166)Thread: WorkerThread#7[172.31.24.54:41549] : priority:5, demon:false, threadId:235, threadState:RUNNABLE, lockName:nulljava.net.SocketInputStream.socketRead0(Native Method)java.net.SocketInputStream.read(SocketInputStream.java:129)java.io.BufferedInputStream.fill(BufferedInputStream.java:218)java.io.BufferedInputStream.read(BufferedInputStream.java:235)java.io.FilterInputStream.read(FilterInputStream.java:66)org.jboss.remoting.transport.socket.ServerThread.readVersion(ServerThread.java:824)org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:510)org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:387)org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:166)Thread: WorkerThread#8[172.31.24.54:41552] : priority:5, demon:false, threadId:236, threadState:RUNNABLE, lockName:nulljava.net.SocketInputStream.socketRead0(Native Method)java.net.SocketInputStream.read(SocketInputStream.java:129)java.io.BufferedInputStream.fill(BufferedInputStream.java:218)java.io.BufferedInputStream.read(BufferedInputStream.java:235)java.io.FilterInputStream.read(FilterInputStream.java:66)org.jboss.remoting.transport.socket.ServerThread.readVersion(ServerThread.java:824)org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:510)org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:387)org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:166)Thread: Thread-57 : priority:5, demon:false, threadId:237, threadState:WAITING, lockName:java.lang.Object@14a18c3java.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)java.lang.Thread.run(Thread.java:595)Thread: Thread-58 : priority:5, demon:false, threadId:238, threadState:WAITING, lockName:java.lang.Object@8d09bdjava.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)java.lang.Thread.run(Thread.java:595)Thread: Thread-59 : priority:5, demon:false, threadId:239, threadState:WAITING, lockName:java.lang.Object@157feb8java.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)java.lang.Thread.run(Thread.java:595)Thread: Thread-60 : priority:5, demon:false, threadId:240, threadState:WAITING, lockName:java.lang.Object@176b8d6java.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)java.lang.Thread.run(Thread.java:595)Thread: WorkerThread#9[172.31.25.56:36526] : priority:5, demon:false, threadId:242, threadState:RUNNABLE, lockName:nulljava.net.SocketInputStream.socketRead0(Native Method)java.net.SocketInputStream.read(SocketInputStream.java:129)java.io.BufferedInputStream.fill(BufferedInputStream.java:218)java.io.BufferedInputStream.read(BufferedInputStream.java:235)java.io.FilterInputStream.read(FilterInputStream.java:66)org.jboss.remoting.transport.socket.ServerThread.readVersion(ServerThread.java:824)org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:510)org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:387)org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:166)Thread: Thread-63 : priority:5, demon:false, threadId:243, threadState:WAITING, lockName:java.lang.Object@5d82a2java.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)java.lang.Thread.run(Thread.java:595)Thread: Thread-64 : priority:5, demon:false, threadId:244, threadState:WAITING, lockName:java.lang.Object@1774bb5java.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)java.lang.Thread.run(Thread.java:595)Thread: Thread-65 : priority:5, demon:false, threadId:245, threadState:WAITING, lockName:java.lang.Object@626e7ejava.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)java.lang.Thread.run(Thread.java:595)Thread: Thread-66 : priority:5, demon:false, threadId:246, threadState:WAITING, lockName:java.lang.Object@113ccc3java.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)java.lang.Thread.run(Thread.java:595)Thread: Thread-67 : priority:5, demon:false, threadId:247, threadState:WAITING, lockName:java.lang.Object@101fd42java.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)java.lang.Thread.run(Thread.java:595)Thread: Thread-68 : priority:5, demon:false, threadId:248, threadState:WAITING, lockName:java.lang.Object@bb0ef5java.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)java.lang.Thread.run(Thread.java:595)Thread: WorkerThread#10[172.31.25.56:36527] : priority:5, demon:false, threadId:249, threadState:RUNNABLE, lockName:nulljava.net.SocketInputStream.socketRead0(Native Method)java.net.SocketInputStream.read(SocketInputStream.java:129)java.io.BufferedInputStream.fill(BufferedInputStream.java:218)java.io.BufferedInputStream.read(BufferedInputStream.java:235)java.io.FilterInputStream.read(FilterInputStream.java:66)org.jboss.remoting.transport.socket.ServerThread.readVersion(ServerThread.java:824)org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:510)org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:387)org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:166)Thread: Thread-70 : priority:5, demon:false, threadId:250, threadState:WAITING, lockName:java.lang.Object@1b82a5fjava.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)java.lang.Thread.run(Thread.java:595)Thread: Thread-71 : priority:5, demon:false, threadId:251, threadState:WAITING, lockName:java.lang.Object@14676bjava.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)java.lang.Thread.run(Thread.java:595)Thread: Thread-72 : priority:5, demon:false, threadId:252, threadState:WAITING, lockName:java.lang.Object@54113java.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)java.lang.Thread.run(Thread.java:595)Thread: Thread-73 : priority:5, demon:false, threadId:253, threadState:WAITING, lockName:java.lang.Object@e1a39ajava.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)java.lang.Thread.run(Thread.java:595)Thread: Thread-74 : priority:5, demon:false, threadId:254, threadState:WAITING, lockName:java.lang.Object@18622eajava.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)java.lang.Thread.run(Thread.java:595)Thread: Thread-75 : priority:5, demon:false, threadId:256, threadState:WAITING, lockName:java.lang.Object@1de024java.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)java.lang.Thread.run(Thread.java:595)Thread: Timer-16 : priority:5, demon:true, threadId:1386, threadState:TIMED_WAITING, lockName:java.util.TaskQueue@941055java.lang.Object.wait(Native Method)java.util.TimerThread.mainLoop(Timer.java:509)java.util.TimerThread.run(Timer.java:462)Thread: http-172.31.25.61-8080-3 : priority:5, demon:true, threadId:1819, threadState:WAITING, lockName:org.apache.tomcat.util.net.JIoEndpoint$Worker@1e2b51cjava.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:416)org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:442)java.lang.Thread.run(Thread.java:595)Thread: http-172.31.25.61-8080-4 : priority:5, demon:true, threadId:3114, threadState:WAITING, lockName:org.apache.tomcat.util.net.JIoEndpoint$Worker@b8377ejava.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:416)org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:442)java.lang.Thread.run(Thread.java:595)Thread Group: JBoss Pooled Threads : max priority:10, demon:falseThread Group: System Threads : max priority:10, demon:falseThread: JBoss System Threads(1)-1 : priority:5, demon:true, threadId:20, threadState:RUNNABLE, lockName:nulljava.net.PlainSocketImpl.socketAccept(Native Method)java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)java.net.ServerSocket.implAccept(ServerSocket.java:450)java.net.ServerSocket.accept(ServerSocket.java:421)org.jboss.web.WebServer.run(WebServer.java:320)org.jboss.util.threadpool.RunnableTaskWrapper.run(RunnableTaskWrapper.java:148)EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:756)java.lang.Thread.run(Thread.java:595)Thread: JBoss System Threads(1)-2 : priority:5, demon:true, threadId:24, threadState:RUNNABLE, lockName:nulljava.net.PlainSocketImpl.socketAccept(Native Method)java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)java.net.ServerSocket.implAccept(ServerSocket.java:450)java.net.ServerSocket.accept(ServerSocket.java:421)org.jnp.server.Main$AcceptHandler.run(Main.java:481)org.jboss.util.threadpool.RunnableTaskWrapper.run(RunnableTaskWrapper.java:148)EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:756)java.lang.Thread.run(Thread.java:595)Thread: JBoss System Threads(1)-3 : priority:5, demon:true, threadId:74, threadState:RUNNABLE, lockName:nulljava.net.PlainSocketImpl.socketAccept(Native Method)java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)java.net.ServerSocket.implAccept(ServerSocket.java:450)java.net.ServerSocket.accept(ServerSocket.java:421)org.jboss.ha.jndi.DetachedHANamingService$AcceptHandler.run(DetachedHANamingService.java:711)org.jboss.util.threadpool.RunnableTaskWrapper.run(RunnableTaskWrapper.java:148)EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:756)java.lang.Thread.run(Thread.java:595)Thread: JBoss System Threads(1)-4 : priority:5, demon:true, threadId:75, threadState:RUNNABLE, lockName:nulljava.net.PlainDatagramSocketImpl.receive0(Native Method)java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:136)java.net.DatagramSocket.receive(DatagramSocket.java:712)org.jboss.ha.jndi.DetachedHANamingService$AutomaticDiscovery.run(DetachedHANamingService.java:615)org.jboss.util.threadpool.RunnableTaskWrapper.run(RunnableTaskWrapper.java:148)EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:756)java.lang.Thread.run(Thread.java:595)Thread Group: JGroups threads : max priority:10, demon:falseThread: DiagnosticsHandler (channel=Tomcat-Cluster) : priority:5, demon:true, threadId:43, threadState:RUNNABLE, lockName:nulljava.net.PlainDatagramSocketImpl.receive0(Native Method)java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:136)java.net.DatagramSocket.receive(DatagramSocket.java:712)org.jgroups.protocols.TP$DiagnosticsHandler.run(TP.java:1749)java.lang.Thread.run(Thread.java:595)Thread: IncomingPacketHandler (channel=Tomcat-Cluster) : priority:5, demon:true, threadId:44, threadState:WAITING, lockName:java.lang.Object@1a4be49java.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)org.jgroups.util.Queue.remove(Queue.java:263)org.jgroups.protocols.TP$IncomingPacketHandler.run(TP.java:1295)java.lang.Thread.run(Thread.java:595)Thread: UDP.UcastReceiverThread (channel=Tomcat-Cluster) : priority:5, demon:true, threadId:45, threadState:RUNNABLE, lockName:nulljava.net.PlainDatagramSocketImpl.receive0(Native Method)java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:136)java.net.DatagramSocket.receive(DatagramSocket.java:712)org.jgroups.protocols.UDP$UcastReceiver.run(UDP.java:952)java.lang.Thread.run(Thread.java:595)Thread: UDP mcast receiver (channel=Tomcat-Cluster) : priority:10, demon:true, threadId:46, threadState:RUNNABLE, lockName:nulljava.net.PlainDatagramSocketImpl.receive0(Native Method)java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:136)java.net.DatagramSocket.receive(DatagramSocket.java:712)org.jgroups.protocols.UDP.run(UDP.java:246)java.lang.Thread.run(Thread.java:595)Thread: ServerSocket acceptor thread (channel=Tomcat-Cluster) : priority:5, demon:true, threadId:47, threadState:RUNNABLE, lockName:nulljava.net.PlainSocketImpl.socketAccept(Native Method)java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)java.net.ServerSocket.implAccept(ServerSocket.java:450)java.net.ServerSocket.accept(ServerSocket.java:421)org.jgroups.protocols.FD_SOCK$ServerSocketHandler.run(FD_SOCK.java:1082)java.lang.Thread.run(Thread.java:595)Thread: FD_SOCK Ping thread (channel=Tomcat-Cluster) : priority:5, demon:true, threadId:50, threadState:RUNNABLE, lockName:nulljava.net.SocketInputStream.socketRead0(Native Method)java.net.SocketInputStream.read(SocketInputStream.java:129)java.net.SocketInputStream.read(SocketInputStream.java:182)org.jgroups.protocols.FD_SOCK.run(FD_SOCK.java:463)java.lang.Thread.run(Thread.java:595)Thread: ClientConnectionHandler : priority:5, demon:true, threadId:51, threadState:RUNNABLE, lockName:nulljava.net.SocketInputStream.socketRead0(Native Method)java.net.SocketInputStream.read(SocketInputStream.java:129)java.net.SocketInputStream.read(SocketInputStream.java:182)org.jgroups.protocols.FD_SOCK$ClientConnectionHandler.run(FD_SOCK.java:1146)Thread: DiagnosticsHandler (channel=Coreservices-Production) : priority:5, demon:true, threadId:63, threadState:RUNNABLE, lockName:nulljava.net.PlainDatagramSocketImpl.receive0(Native Method)java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:136)java.net.DatagramSocket.receive(DatagramSocket.java:712)org.jgroups.protocols.TP$DiagnosticsHandler.run(TP.java:1749)java.lang.Thread.run(Thread.java:595)Thread: IncomingPacketHandler (channel=Coreservices-Production) : priority:5, demon:true, threadId:64, threadState:WAITING, lockName:java.lang.Object@854097java.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)org.jgroups.util.Queue.remove(Queue.java:263)org.jgroups.protocols.TP$IncomingPacketHandler.run(TP.java:1295)java.lang.Thread.run(Thread.java:595)Thread: UDP.UcastReceiverThread (channel=Coreservices-Production) : priority:5, demon:true, threadId:65, threadState:RUNNABLE, lockName:nulljava.net.PlainDatagramSocketImpl.receive0(Native Method)java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:136)java.net.DatagramSocket.receive(DatagramSocket.java:712)org.jgroups.protocols.UDP$UcastReceiver.run(UDP.java:952)java.lang.Thread.run(Thread.java:595)Thread: UDP mcast receiver (channel=Coreservices-Production) : priority:10, demon:true, threadId:66, threadState:RUNNABLE, lockName:nulljava.net.PlainDatagramSocketImpl.receive0(Native Method)java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:136)java.net.DatagramSocket.receive(DatagramSocket.java:712)org.jgroups.protocols.UDP.run(UDP.java:246)java.lang.Thread.run(Thread.java:595)Thread: ServerSocket acceptor thread (channel=Coreservices-Production) : priority:5, demon:true, threadId:67, threadState:RUNNABLE, lockName:nulljava.net.PlainSocketImpl.socketAccept(Native Method)java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)java.net.ServerSocket.implAccept(ServerSocket.java:450)java.net.ServerSocket.accept(ServerSocket.java:421)org.jgroups.protocols.FD_SOCK$ServerSocketHandler.run(FD_SOCK.java:1082)java.lang.Thread.run(Thread.java:595)Thread: MERGE2.FindSubgroups thread (channel=Coreservices-Production) : priority:5, demon:true, threadId:70, threadState:TIMED_WAITING, lockName:nulljava.lang.Thread.sleep(Native Method)org.jgroups.util.Util.sleep(Util.java:914)org.jgroups.protocols.MERGE2$FindSubgroups.run(MERGE2.java:289)java.lang.Thread.run(Thread.java:595)Thread: DiagnosticsHandler (channel=DefaultPartition-SFSBCache) : priority:5, demon:true, threadId:80, threadState:RUNNABLE, lockName:nulljava.net.PlainDatagramSocketImpl.receive0(Native Method)java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:136)java.net.DatagramSocket.receive(DatagramSocket.java:712)org.jgroups.protocols.TP$DiagnosticsHandler.run(TP.java:1749)java.lang.Thread.run(Thread.java:595)Thread: IncomingPacketHandler (channel=DefaultPartition-SFSBCache) : priority:5, demon:true, threadId:81, threadState:WAITING, lockName:java.lang.Object@bf0463java.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)org.jgroups.util.Queue.remove(Queue.java:263)org.jgroups.protocols.TP$IncomingPacketHandler.run(TP.java:1295)java.lang.Thread.run(Thread.java:595)Thread: UDP.UcastReceiverThread (channel=DefaultPartition-SFSBCache) : priority:5, demon:true, threadId:82, threadState:RUNNABLE, lockName:nulljava.net.PlainDatagramSocketImpl.receive0(Native Method)java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:136)java.net.DatagramSocket.receive(DatagramSocket.java:712)org.jgroups.protocols.UDP$UcastReceiver.run(UDP.java:952)java.lang.Thread.run(Thread.java:595)Thread: UDP mcast receiver (channel=DefaultPartition-SFSBCache) : priority:10, demon:true, threadId:83, threadState:RUNNABLE, lockName:nulljava.net.PlainDatagramSocketImpl.receive0(Native Method)java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:136)java.net.DatagramSocket.receive(DatagramSocket.java:712)org.jgroups.protocols.UDP.run(UDP.java:246)java.lang.Thread.run(Thread.java:595)Thread: ServerSocket acceptor thread (channel=DefaultPartition-SFSBCache) : priority:5, demon:true, threadId:84, threadState:RUNNABLE, lockName:nulljava.net.PlainSocketImpl.socketAccept(Native Method)java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)java.net.ServerSocket.implAccept(ServerSocket.java:450)java.net.ServerSocket.accept(ServerSocket.java:421)org.jgroups.protocols.FD_SOCK$ServerSocketHandler.run(FD_SOCK.java:1082)java.lang.Thread.run(Thread.java:595)Thread: FD_SOCK Ping thread (channel=DefaultPartition-SFSBCache) : priority:5, demon:true, threadId:87, threadState:RUNNABLE, lockName:nulljava.net.SocketInputStream.socketRead0(Native Method)java.net.SocketInputStream.read(SocketInputStream.java:129)java.net.SocketInputStream.read(SocketInputStream.java:182)org.jgroups.protocols.FD_SOCK.run(FD_SOCK.java:463)java.lang.Thread.run(Thread.java:595)Thread: DiagnosticsHandler (channel=DefaultPartition-EntityCache) : priority:5, demon:true, threadId:91, threadState:RUNNABLE, lockName:nulljava.net.PlainDatagramSocketImpl.receive0(Native Method)java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:136)java.net.DatagramSocket.receive(DatagramSocket.java:712)org.jgroups.protocols.TP$DiagnosticsHandler.run(TP.java:1749)java.lang.Thread.run(Thread.java:595)Thread: IncomingPacketHandler (channel=DefaultPartition-EntityCache) : priority:5, demon:true, threadId:92, threadState:WAITING, lockName:java.lang.Object@c5aa1djava.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)org.jgroups.util.Queue.remove(Queue.java:263)org.jgroups.protocols.TP$IncomingPacketHandler.run(TP.java:1295)java.lang.Thread.run(Thread.java:595)Thread: UDP.UcastReceiverThread (channel=DefaultPartition-EntityCache) : priority:5, demon:true, threadId:93, threadState:RUNNABLE, lockName:nulljava.net.PlainDatagramSocketImpl.receive0(Native Method)java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:136)java.net.DatagramSocket.receive(DatagramSocket.java:712)org.jgroups.protocols.UDP$UcastReceiver.run(UDP.java:952)java.lang.Thread.run(Thread.java:595)Thread: UDP mcast receiver (channel=DefaultPartition-EntityCache) : priority:10, demon:true, threadId:94, threadState:RUNNABLE, lockName:nulljava.net.PlainDatagramSocketImpl.receive0(Native Method)java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:136)java.net.DatagramSocket.receive(DatagramSocket.java:712)org.jgroups.protocols.UDP.run(UDP.java:246)java.lang.Thread.run(Thread.java:595)Thread: ServerSocket acceptor thread (channel=DefaultPartition-EntityCache) : priority:5, demon:true, threadId:95, threadState:RUNNABLE, lockName:nulljava.net.PlainSocketImpl.socketAccept(Native Method)java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)java.net.ServerSocket.implAccept(ServerSocket.java:450)java.net.ServerSocket.accept(ServerSocket.java:421)org.jgroups.protocols.FD_SOCK$ServerSocketHandler.run(FD_SOCK.java:1082)java.lang.Thread.run(Thread.java:595)Thread: ClientConnectionHandler : priority:5, demon:true, threadId:98, threadState:RUNNABLE, lockName:nulljava.net.SocketInputStream.socketRead0(Native Method)java.net.SocketInputStream.read(SocketInputStream.java:129)java.net.SocketInputStream.read(SocketInputStream.java:182)org.jgroups.protocols.FD_SOCK$ClientConnectionHandler.run(FD_SOCK.java:1146)Thread: FD_SOCK Ping thread (channel=DefaultPartition-EntityCache) : priority:5, demon:true, threadId:99, threadState:RUNNABLE, lockName:nulljava.net.SocketInputStream.socketRead0(Native Method)java.net.SocketInputStream.read(SocketInputStream.java:129)java.net.SocketInputStream.read(SocketInputStream.java:182)org.jgroups.protocols.FD_SOCK.run(FD_SOCK.java:463)java.lang.Thread.run(Thread.java:595)Thread: ClientConnectionHandler : priority:5, demon:true, threadId:111, threadState:RUNNABLE, lockName:nulljava.net.SocketInputStream.socketRead0(Native Method)java.net.SocketInputStream.read(SocketInputStream.java:129)java.net.SocketInputStream.read(SocketInputStream.java:182)org.jgroups.protocols.FD_SOCK$ClientConnectionHandler.run(FD_SOCK.java:1146)Thread: DownHandler (MPING) : priority:5, demon:true, threadId:129, threadState:WAITING, lockName:java.lang.Object@1890c5ejava.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)org.jgroups.util.Queue.remove(Queue.java:263)org.jgroups.stack.DownHandler.run(Protocol.java:104)Thread: UpHandler (MPING) : priority:5, demon:true, threadId:130, threadState:WAITING, lockName:java.lang.Object@e7a8b8java.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)org.jgroups.util.Queue.remove(Queue.java:263)org.jgroups.stack.UpHandler.run(Protocol.java:49)Thread: DiagnosticsHandler (channel=MessagingPostOffice) : priority:5, demon:true, threadId:132, threadState:RUNNABLE, lockName:nulljava.net.PlainDatagramSocketImpl.receive0(Native Method)java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:136)java.net.DatagramSocket.receive(DatagramSocket.java:712)org.jgroups.protocols.TP$DiagnosticsHandler.run(TP.java:1749)java.lang.Thread.run(Thread.java:595)Thread: IncomingPacketHandler (channel=MessagingPostOffice) : priority:5, demon:true, threadId:133, threadState:WAITING, lockName:java.lang.Object@121cf89java.lang.Object.wait(Native Method)java.lang.Object.wait(Object.java:474)org.jgroups.util.Queue.remove(Queue.java:263)org.jgroups.protocols.TP$IncomingPacketHandler.run(TP.java:1295)java.lang.Thread.run(Thread.java:595)Thread: UDP.UcastReceiverThread (channel=MessagingPostOffice) : priority:5, demon:true, threadId:134, threadState:RUNNABLE, lockName:nulljava.net.PlainDatagramSocketImpl.receive0(Native Method)java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:136)java.net.DatagramSocket.receive(DatagramSocket.java:712)org.jgroups.protocols.UDP$UcastReceiver.run(UDP.java:952)java.lang.Thread.run(Thread.java:595)Thread: UDP mcast receiver (channel=MessagingPostOffice) : priority:10, demon:true, threadId:135, threadState:RUNNABLE, lockName:nulljava.net.PlainDatagramSocketImpl.receive0(Native Method)java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:136)java.net.DatagramSocket.receive(DatagramSocket.java:712)org.jgroups.protocols.UDP.run(UDP.java:246)java.lang.Thread.run(Thread.java:595)Thread: ServerSocket acceptor thread (channel=MessagingPostOffice) : priority:5, demon:true, threadId:137, threadState:RUNNABLE, lockName:nulljava.net.PlainSocketImpl.socketAccept(Native Method)java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)java.net.ServerSocket.implAccept(ServerSocket.java:450)java.net.ServerSocket.accept(ServerSocket.java:421)org.jgroups.protocols.FD_SOCK$ServerSocketHandler.run(FD_SOCK.java:1082)java.lang.Thread.run(Thread.java:595)Thread: MERGE2.FindSubgroups thread (channel=MessagingPostOffice) : priority:5, demon:true, threadId:141, threadState:TIMED_WAITING, lockName:nulljava.lang.Thread.sleep(Native Method)org.jgroups.util.Util.sleep(Util.java:914)org.jgroups.protocols.MERGE2$FindSubgroups.run(MERGE2.java:289)java.lang.Thread.run(Thread.java:595)Thread: ReceiverThread : priority:5, demon:true, threadId:142, threadState:RUNNABLE, lockName:nulljava.net.PlainDatagramSocketImpl.receive0(Native Method)java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:136)java.net.DatagramSocket.receive(DatagramSocket.java:712)org.jgroups.protocols.MPING.run(MPING.java:263)java.lang.Thread.run(Thread.java:595)Thread: DiagnosticsHandler (channel=MessagingPostOffice) : priority:5, demon:true, threadId:144, threadState:RUNNABLE, lockName:nulljava.net.PlainDatagramSocketImpl.receive0(Native Method)java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:136)java.net.DatagramSocket.receiv
View the original post : http://www.jboss.com/index.html?module=bb&op=viewtopic&p=4134130#4134130
Reply to the post : http://www.jboss.com/index.html?module=bb&op=posting&mode=reply&p=4134130
18 years, 1 month
- 1
- 0
[JBoss jBPM] - Re: Strange behaviour when using fork
by bungrudi
ok here is an important point for you who happen to stumble into this thread..
Action handlers _MAY NOT_ continue graph of execution. it means that you cannot do node.leave(), executionContext.leave(), or token.signal() from inside an action handlers.
There is one case where you are allowed to do those in action handler. In fact, you _MUST_ do one of those in this case. It is when the action handler is in a node of type "Node".
View the original post : http://www.jboss.com/index.html?module=bb&op=viewtopic&p=4134129#4134129
Reply to the post : http://www.jboss.com/index.html?module=bb&op=posting&mode=reply&p=4134129
18 years, 1 month
- 1
- 0
[Installation, Configuration & DEPLOYMENT] - jar deployment problem through ear
by mjitch
We shifted our application to Jboss-4.2.2GA version, from Jboss-3.2.8SP1 version. We have virtual instances instances of jboss running. Problem here is, in jboss 4.2.2.GA, a jar file abc.jar in .../WEB-INF/lib is not getting deployed properly. I get a ClassNotFoundException. Surprisingly, if I put the same jar file in the lib folder of the virtual server instance directly it works fine. We've the same ear deployment in Jboss 3.2.8SP1, it works fine without any issues.
I just wanted to know if this issue is specific to jboss 4.2.2.GA? Any loopholes in it?
Thanks
View the original post : http://www.jboss.com/index.html?module=bb&op=viewtopic&p=4134125#4134125
Reply to the post : http://www.jboss.com/index.html?module=bb&op=posting&mode=reply&p=4134125
18 years, 1 month
- 1
- 0
[JBoss jBPM] - Re: Strange behaviour when using fork
by bungrudi
http://www.jboss.com/index.html?module=bb&op=viewtopic&t=129225
looks like a clue to me :)
View the original post : http://www.jboss.com/index.html?module=bb&op=viewtopic&p=4134124#4134124
Reply to the post : http://www.jboss.com/index.html?module=bb&op=posting&mode=reply&p=4134124
18 years, 1 month
- 1
- 0
[JBoss jBPM] - Re: Strange behaviour when using fork
by bungrudi
"bungrudi" wrote :
|
| I have a flow that basically does the following:
| AO -> RO#1 -> ... -> HBS -> REV_ADM -> REV#1 -> ... -> REV#3 ->
| -> COM_ADM -> COM#1 -> ... ->COM#3 ->
|
| (continued)
| join1 -> dropped
|
|
to be clearer:
| AO -> RO#1 -> ... -> HBS -> fork1
|
| (path1)
| REV_ADM -> REV#1 -> ... -> REV#3 -> join1
| (path2)
| COM_ADM -> COM#1 -> ... ->COM#3 -> join1
|
| (continued after joined)
| join1 -> dropped
|
im starting to think that im using wrong perspective. is it okay to continue the execution from inside an action?
regards,
rudi
View the original post : http://www.jboss.com/index.html?module=bb&op=viewtopic&p=4134122#4134122
Reply to the post : http://www.jboss.com/index.html?module=bb&op=posting&mode=reply&p=4134122
18 years, 1 month
- 1
- 0
[JBossWS] - SSL and native libraries
by fastbob
Hopefully I have the correct forum section. While I've researched the issue a bit, I can't claim to fully understand what's going on.
On Windows XP, using JBoss 4.2.2 (no modifications) and Sun jdk 1.6, we enabled SSL by uncommenting the SSL connector in service.xml:
<Connector port="8443" protocol="HTTP/1.1" SSLEnabled="true"
| address="${jboss.bind.address}" maxThreads="100"
| scheme="https" secure="true" clientAuth="false"
| keystoreFile="${jboss.server.home.dir}/conf/my.keystore"
| keystorePass="something" sslProtocol = "TLS" />
This seemed to be handled fine during startup - no native library found but startup continued and SSL worked:
05:47:06,150 INFO [NativeServerConfig] JBoss Web Services - Native
| 05:47:06,150 INFO [NativeServerConfig] jbossws-native-2.0.1.SP2 (build=200710210837)
| 05:47:06,941 INFO [Embedded] Catalina naming disabled
| 05:47:07,281 INFO [AprLifecycleListener] The Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: C:\Java\jdk1.6.0\bin;.;C:\WINDOWS\Sun\Java\bin;C:\WINDOWS\system32;C:\WINDOWS;C:\Java\jdk1.6.0\bin;C:\Java\apache-ant-1.7.0\bin;C:\Java\jikes_1.22\bin;C:\Java\maven-2.0.7\bin;c:\tools\cygwin\bin;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\Program Files\ATI Technologies\ATI Control Panel;C:\Program Files\cvsnt;C:\Program Files\MySQL\MySQL Server 5.0\bin;c:\Program Files\Microsoft SQL Server\90\Tools\binn\;C:\Program Files\QuickTime\QTSystem\
| 05:47:07,352 INFO [Http11Protocol] Initializing Coyote HTTP/1.1 on http-0.0.0.0-8080
| 05:47:08,083 INFO [Http11Protocol] Initializing Coyote HTTP/1.1 on http-0.0.0.0-8443
| 05:47:08,093 INFO [AjpProtocol] Initializing Coyote AJP/1.3 on ajp-0.0.0.0-8009
|
On Fedora8 Linux, we copied the same JBoss directory to the system (same Sun jdk 1.6). However, the log showed the following during startup:
03:22:19,508 INFO [NativeServerConfig] JBoss Web Services - Native
| 03:22:19,508 INFO [NativeServerConfig] jbossws-native-2.0.1.SP2 (build=200710210837)
| 03:22:20,310 INFO [Embedded] Catalina naming disabled
| 03:22:20,462 WARN [Connector] Property debug not found on the protocol handler.
| 03:22:20,466 WARN [Connector] Property clientAuth not found on the protocol handler.
| 03:22:20,466 WARN [Connector] Property keystoreFile not found on the protocol handler.
| 03:22:20,467 WARN [Connector] Property keystorePass not found on the protocol handler.
| 03:22:20,747 INFO [AprLifecycleListener] Loaded Apache Tomcat Native library 1.1.10.
| 03:22:20,751 INFO [AprLifecycleListener] APR capabilities: IPv6 [true], sendfile [true], accept filters [false], random [true].
| 03:22:21,092 INFO [Http11AprProtocol] Initializing Coyote HTTP/1.1 on http-0.0.0.0-8080
| 03:22:21,094 ERROR [Http11AprProtocol] Error initializing endpoint
| java.lang.Exception: No Certificate file specified
| at org.apache.tomcat.jni.SSLContext.setCertificate(Native Method)
| at org.apache.tomcat.util.net.AprEndpoint.init(AprEndpoint.java:683)
| ...
So it appears that on Fedora the native library was loaded, but then the properties of the SSL connector were not loaded. Startup eventually failed due to the port being in use:
03:23:12,305 INFO [Http11AprProtocol] Starting Coyote HTTP/1.1 on http-0.0.0.0-8080
| 03:23:12,383 ERROR [Http11AprProtocol] Error starting endpoint
| java.lang.Exception: Socket bind failed: [98] Address already in use
| at org.apache.tomcat.util.net.AprEndpoint.init(AprEndpoint.java:611)
| at org.apache.tomcat.util.net.AprEndpoint.start(AprEndpoint.java:716)
| at org.apache.coyote.http11.Http11AprProtocol.start(Http11AprProtocol.java:137)
|
I'm not clear what we need to change in the configuration to get things working on Linux. It seems like removing the native library would work against better performance. Any pointers would be appreciated. Thanks.
Bob
View the original post : http://www.jboss.com/index.html?module=bb&op=viewtopic&p=4134121#4134121
Reply to the post : http://www.jboss.com/index.html?module=bb&op=posting&mode=reply&p=4134121
18 years, 1 month
- 1
- 0
[JBoss jBPM] - Strange behaviour when using fork
by bungrudi
Hello,
I have a flow that basically does the following:
AO -> RO#1 -> ... -> HBS -> REV_ADM -> REV#1 -> ... -> REV#3 ->
-> COM_ADM -> COM#1 -> ... ->COM#3 ->
(continued)
join1 -> dropped
on each node, it should be checked whether any user is currently assigned to it. if yes, create a task and assign. if not, continue execution.
my idea to this is to make each task-node "create-task=false", assign an "enter-node" handler that checks whether any user is assigned to the node, and then create task/continue execution.
This solution works fine when the execution path is straightforward (no forking and async execution involved).
but when "fork" come into play, the behaviour is far from what i expected.
the debug print (from my own code, to simplify the printout) is like the following:
http://www.2shared.com/file/2939237/171d5892/stdout_jbpm.html
as you can see after HBS node it hits the fork node, and it started to execute in a way beyond my comprehension.
yet more strange behaviour is that when the execution is stopped in HBS for a while (created a task and waited for the task to be completed), the printout is far shorter, but still far from what i expected and still beyond my comprehension:
| Token(/)264445391 calc auth level
| Token(/)264445391 review RO #1
| Token(/)264445391 review RO #2
| Token(/)264445391 review SRM
| Token(/)264445391 review GAM
| Token(/)264445391 review BM
| Token(/)264445391 review HOBS
| Token(/)264445391 review HA
| Token(/)264445391 review HR
| Token(/)264445391 review HBS
| Token(/to.pick compliances)264625293 pick compliances
| Token(/to.pick compliances)264625293 compliance #1
| Token(/to.pick compliances)264625293 compliance #2
| Token(/to.pick compliances)264625293 compliance #3
| Token(/to.pick compliances)264625293 compliance #3
| Token(/to.pick compliances)264625293 compliance #2
| Token(/to.pick compliances)264625293 compliance #3
| Token(/to.pick compliances)264625293 compliance #3
| Token(/to.pick compliances)264625293 compliance #1
| Token(/to.pick compliances)264625293 compliance #2
| Token(/to.pick compliances)264625293 compliance #3
| Token(/to.pick compliances)264625293 compliance #3
| Token(/to.pick compliances)264625293 compliance #2
| Token(/to.pick compliances)264625293 compliance #3
| Token(/to.pick compliances)264625293 compliance #3
| Token(/to.pick reviewer)264625334 pick reviewer
| Token(/to.pick reviewer)264625334 reviewer #1
| Token(/to.pick reviewer)264625334 reviewer #2
| Token(/to.pick reviewer)264625334 reviewer #3
| Token(/to.pick reviewer)264625334 reviewer #3
| Token(/to.pick reviewer)264625334 reviewer #2
| Token(/to.pick reviewer)264625334 reviewer #3
| Token(/to.pick reviewer)264625334 reviewer #3
| Token(/to.pick reviewer)264625334 reviewer #1
| Token(/to.pick reviewer)264625334 reviewer #2
| Token(/to.pick reviewer)264625334 reviewer #3
| Token(/to.pick reviewer)264625334 reviewer #3
| Token(/to.pick reviewer)264625334 reviewer #2
| Token(/to.pick reviewer)264625334 reviewer #3
| Token(/to.pick reviewer)264625334 reviewer #3
|
i'm expecting that after "review HBS", it should be like
Token(/to.pick compliances)264625293 pick compliances
Token(/to.pick compliances)264625293 compliance #1
Token(/to.pick compliances)264625293 compliance #2
Token(/to.pick compliances)264625293 compliance #3
Token(/to.pick reviewer)264625334 pick reviewer
Token(/to.pick reviewer)264625334 reviewer #1
Token(/to.pick reviewer)264625334 reviewer #2
Token(/to.pick reviewer)264625334 reviewer #3
and then join and finish.
did i do something wrong? is ctx.getNode().leave(ctx) is the wrong way to continue execution? or continuing execution from inside an action handler is not a supported way at all?
the unit test can be downloaded from
http://www.2shared.com/file/2939148/caa56d9d/fork_test.html
anyone has any insight???
regards,
rudi
View the original post : http://www.jboss.com/index.html?module=bb&op=viewtopic&p=4134120#4134120
Reply to the post : http://www.jboss.com/index.html?module=bb&op=posting&mode=reply&p=4134120
18 years, 1 month
- 1
- 0
[EJB 3.0] - Re: Storing user info on server
by jwcone
"jwcone" wrote : ...(with the Session Principal name as a key)?...
Oops -- that should have been the (JAAS) Subject Principal...
View the original post : http://www.jboss.com/index.html?module=bb&op=viewtopic&p=4134118#4134118
Reply to the post : http://www.jboss.com/index.html?module=bb&op=posting&mode=reply&p=4134118
18 years, 1 month
- 1
- 0
[EJB 3.0] - Re: Storing user info on server
by jwcone
Andrew,
Thank you for the quick response!
"ALRubinger" wrote : So above the parts bound to the webapp are the HttpSession for storing the SFSB and the Filter for placing it in Thread scope. To support another client type, just add the appropriate mechanisms to provide these behaviors.
This is where I'm wondering what direction to take. For a non-HTTP client, what would be the best place to store the SFSB proxy? Is there a client-agnostic equivalent to the HttpSession at the container-level? Maybe a HashMap accessed through JNDI (with the Session Principal name as a key)?
Sorry if I'm being dense -- I'm fairly new to EJBs.
View the original post : http://www.jboss.com/index.html?module=bb&op=viewtopic&p=4134115#4134115
Reply to the post : http://www.jboss.com/index.html?module=bb&op=posting&mode=reply&p=4134115
18 years, 1 month
- 1
- 0
[EJB 3.0] - Re: Storing user info on server
by ALRubinger
A pattern I've traditionally employed revolves around the notion of an "Application Session", which is stored in any of your client-specific access sessions (ie. Webapp, Flex, GUI). So the application deals with this alone, and is agnostic towards the type of client.
I'll outline for a webapp here as its probably the most widely used, but you can apply the techniques however you'd like.
---
* End user makes HTTP Request
* Servlet Filter intercepts before sending along the FilterChain. Grabbing the ApplicationSession, which is an SFSB proxy, from the HttpSession, places in a public static ThreadLocal variable before the invocation continues.
* There's now an ApplicationSession bound to this request/Thread's scope, usable from any component without having to pass it around or add it to every method signature.
* On the way out, Servlet Filter (in a finally block) clears the ThreadLocal var holding the SFSB proxy back to intialValue(), otherwise the ApplicationSession will remain tied to this Thread, which will be reused in other requests.
---
So above the parts bound to the webapp are the HttpSession for storing the SFSB and the Filter for placing it in Thread scope. To support another client type, just add the appropriate mechanisms to provide these behaviors.
S,
ALR
View the original post : http://www.jboss.com/index.html?module=bb&op=viewtopic&p=4134114#4134114
Reply to the post : http://www.jboss.com/index.html?module=bb&op=posting&mode=reply&p=4134114
18 years, 1 month
- 1
- 0