[jboss-dev] JBoss Embedded AS - JVM keeps running

Wed Apr 14 07:27:48 EDT 2010

While experimenting with the latest JBoss Embedded AS (against AS trunk) 
from within a standalone Java client we have been seeing that the JVM 
never shuts down. The relevant code looks like:

public class EmbeddedJBossASClient
{

   private JBossASEmbeddedServer jbossServer;

   public static void main(String[] args) throws Exception
   {
      EmbeddedJBossASClient embeddedASClient = new EmbeddedJBossASClient();
      try
      {
         embeddedASClient.startServer(JBOSS_HOME);
         embeddedASClient.deployEjb();
         // call ejb
         int result = embeddedASClient.invokeSumOnEjb(1, 2);
         System.out.println("Result is " + result);

      }
      finally
      {
         embeddedASClient.shutdownServer();

      }

   }
...
}   

Not much code - it just starts the embedded server within the current VM 
deploys an EJB and shuts down the server. So after 
embeddedASClient.shutdownServer() returns, it would be expected that the 
JVM shuts down. However, the JVM keeps running even after the call 
returns successfully. jstack util on this JVM process shows the 
following threads [2] which are still active. Most of the threads are 
"daemon" which means that they won't prevent the JVM from shutting down 
[1].

However, there are some non-daemon threads still active even after a 
server shutdown:

"pool-15-thread-1" prio=10 tid=0x8cb1ec00 nid=0x4d13 waiting on 
condition [0x8e7a6000]
   java.lang.Thread.State: TIMED_WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0xb20d6258> (a 
java.util.concurrent.SynchronousQueue$TransferStack)
    at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
    at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
    at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
    at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
    at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
    at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
    at java.lang.Thread.run(Thread.java:619)

...
"pool-5-thread-1" prio=10 tid=0x8cfc2800 nid=0x4ce0 waiting on condition 
[0x8cdad000]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x97ece7e8> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
    at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
    at java.lang.Thread.run(Thread.java:619)

These pool-x-thread-y threads prevent the JVM from shutting down. The 
thread names aren't very informative to figure out which service started 
those threads and why were those not shutdown. Anyone knows what these 
threads are for and why they are hanging around after a server shutdown?

Note that this issue isn't observed in a normal JBoss AS server 
environment because:

a) When the server is shutdown through a Ctrl+C, it leads to JVM 
termination and effectively these threads are nuked.
b) When the server is shutdown through shutdown.sh (or through 
jmx-console), it leads to 
org.jboss.system.server.jmx.JMXKernel.shutdown() which does a 
Runtime.exit() leading to JVM shutdown:

public void shutdown()
   {
...
      // JBAS-6916:
      //TODO This must be removed, VM exit is not a feature of server 
shutdown.
      // Now exit the Runtime, because AS has non-daemon Threads still
      // chugging away.
      new Thread()
      {
         public void run()
         {
            log.info("Server exit called, exiting the JVM now!");
            Runtime.getRuntime().exit(1);
         }
      }.start();
   }

[1] 
http://java.sun.com/j2se/1.4.2/docs/api/java/lang/Runtime.html#addShutdownHook%28java.lang.Thread%29 
-
"The Java virtual machine shuts down in response to two kinds of events:
    * The program exits normally, when the last non-daemon thread exits 
or when the exit (equivalently, System.exit) method is invoked, or
    * The virtual machine is terminated in response to a user interrupt, 
such as typing ^C, or a system-wide event, such as user logoff or system 
shutdown."

[2] Entire thread dump from jstack:

2010-04-13 23:15:11
Full thread dump Java HotSpot(TM) Server VM (14.2-b01 mixed mode):

"Attach Listener" daemon prio=10 tid=0x08da4000 nid=0x4d2a waiting on 
condition [0x00000000]
   java.lang.Thread.State: RUNNABLE

"DestroyJavaVM" prio=10 tid=0x8cf17000 nid=0x4cc8 waiting on condition 
[0x00000000]
   java.lang.Thread.State: RUNNABLE

"ConnectionValidator" daemon prio=10 tid=0x8cb7a000 nid=0x4d17 in 
Object.wait() [0x8b338000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0xb2108f88> (a java.util.concurrent.CopyOnWriteArrayList)
    at 
org.jboss.resource.connectionmanager.ConnectionValidator$ConnectionValidatorRunnable.run(ConnectionValidator.java:156)
    - locked <0xb2108f88> (a java.util.concurrent.CopyOnWriteArrayList)
    at java.lang.Thread.run(Thread.java:619)

"IdleRemover" daemon prio=10 tid=0x8c924800 nid=0x4d16 in Object.wait() 
[0x8b2e7000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0xb2108d90> (a java.util.concurrent.CopyOnWriteArrayList)
    at 
org.jboss.resource.connectionmanager.IdleRemover$IdleRemoverRunnable.run(IdleRemover.java:167)
    - locked <0xb2108d90> (a java.util.concurrent.CopyOnWriteArrayList)
    at java.lang.Thread.run(Thread.java:619)

"pool-15-thread-1" prio=10 tid=0x8cb1ec00 nid=0x4d13 waiting on 
condition [0x8e7a6000]
   java.lang.Thread.State: TIMED_WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0xb20d6258> (a 
java.util.concurrent.SynchronousQueue$TransferStack)
    at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
    at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
    at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
    at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
    at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
    at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
    at java.lang.Thread.run(Thread.java:619)

"ajp-127.0.0.1-8009-Poller" daemon prio=10 tid=0x8c92d000 nid=0x4d11 in 
Object.wait() [0x8b1f4000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0xb210f200> (a 
org.apache.tomcat.util.net.JIoEndpoint$Poller)
    at 
org.apache.tomcat.util.net.JIoEndpoint$Poller.run(JIoEndpoint.java:764)
    - locked <0xb210f200> (a org.apache.tomcat.util.net.JIoEndpoint$Poller)
    at java.lang.Thread.run(Thread.java:619)

"http-127.0.0.1-8080-Acceptor-0" daemon prio=10 tid=0x8c935000 
nid=0x4d10 waiting on condition [0x8b245000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
    at java.lang.Thread.sleep(Native Method)
    at 
org.apache.tomcat.util.net.JIoEndpoint$Acceptor.run(JIoEndpoint.java:335)
    at java.lang.Thread.run(Thread.java:619)

"http-127.0.0.1-8080-Poller" daemon prio=10 tid=0x8c936800 nid=0x4d0f in 
Object.wait() [0x8b296000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0xb210f650> (a 
org.apache.tomcat.util.net.JIoEndpoint$Poller)
    at 
org.apache.tomcat.util.net.JIoEndpoint$Poller.run(JIoEndpoint.java:764)
    - locked <0xb210f650> (a org.apache.tomcat.util.net.JIoEndpoint$Poller)
    at java.lang.Thread.run(Thread.java:619)

"JCA PoolFiller" daemon prio=10 tid=0x8cbb1800 nid=0x4d09 in 
Object.wait() [0x8b47c000]
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0xb2040a00> (a java.util.LinkedList)
    at java.lang.Object.wait(Object.java:485)
    at 
org.jboss.resource.connectionmanager.PoolFiller.run(PoolFiller.java:88)
    - locked <0xb2040a00> (a java.util.LinkedList)
    at java.lang.Thread.run(Thread.java:619)

"Timer-1" daemon prio=10 tid=0x8cb6fc00 nid=0x4d07 in Object.wait() 
[0x8b51e000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x989f04e8> (a java.util.TaskQueue)
    at java.util.TimerThread.mainLoop(Timer.java:509)
    - locked <0x989f04e8> (a java.util.TaskQueue)
    at java.util.TimerThread.run(Timer.java:462)

"Thread-9" daemon prio=10 tid=0x8cf3c400 nid=0x4cf8 in Object.wait() 
[0x8b9dd000]
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x98847c88> (a java.util.LinkedList)
    at java.lang.Object.wait(Object.java:485)
    at 
com.arjuna.ats.arjuna.coordinator.TransactionReaper.waitForCancellations(TransactionReaper.java:421)
    - locked <0x98847c88> (a java.util.LinkedList)
    at 
com.arjuna.ats.internal.arjuna.coordinator.ReaperWorkerThread.run(ReaperWorkerThread.java:74)

"Thread-8" daemon prio=10 tid=0x8cf3b800 nid=0x4cf7 in Object.wait() 
[0x8ba2e000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x988471f8> (a 
com.arjuna.ats.arjuna.coordinator.TransactionReaper)
    at 
com.arjuna.ats.internal.arjuna.coordinator.ReaperThread.run(ReaperThread.java:104)
    - locked <0x988471f8> (a 
com.arjuna.ats.arjuna.coordinator.TransactionReaper)

"Listener:4713" daemon prio=10 tid=0x8cbc8000 nid=0x4cf2 runnable 
[0x8bb72000]
   java.lang.Thread.State: RUNNABLE
    at java.net.PlainSocketImpl.socketAccept(Native Method)
    at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)
    - locked <0x98824280> (a java.net.SocksSocketImpl)
    at java.net.ServerSocket.implAccept(ServerSocket.java:453)
    at java.net.ServerSocket.accept(ServerSocket.java:421)
    at 
com.arjuna.ats.internal.arjuna.recovery.Listener.run(Listener.java:127)

"Timer-0" daemon prio=10 tid=0x8cbfbc00 nid=0x4ce4 in Object.wait() 
[0x8ccba000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x98449890> (a java.util.TaskQueue)
    at java.util.TimerThread.mainLoop(Timer.java:509)
    - locked <0x98449890> (a java.util.TaskQueue)
    at java.util.TimerThread.run(Timer.java:462)

"pool-5-thread-1" prio=10 tid=0x8cfc2800 nid=0x4ce0 waiting on condition 
[0x8cdad000]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x97ece7e8> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
    at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
    at java.lang.Thread.run(Thread.java:619)

"Low Memory Detector" daemon prio=10 tid=0x08e3f000 nid=0x4cd1 runnable 
[0x00000000]
   java.lang.Thread.State: RUNNABLE

"CompilerThread1" daemon prio=10 tid=0x08e3d000 nid=0x4cd0 waiting on 
condition [0x00000000]
   java.lang.Thread.State: RUNNABLE

"CompilerThread0" daemon prio=10 tid=0x08e3a400 nid=0x4ccf waiting on 
condition [0x00000000]
   java.lang.Thread.State: RUNNABLE

"Signal Dispatcher" daemon prio=10 tid=0x08e38c00 nid=0x4cce runnable 
[0x00000000]
   java.lang.Thread.State: RUNNABLE

"Finalizer" daemon prio=10 tid=0x08e29000 nid=0x4ccd in Object.wait() 
[0x90545000]
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x94be4528> (a java.lang.ref.ReferenceQueue$Lock)
    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
    - locked <0x94be4528> (a java.lang.ref.ReferenceQueue$Lock)
    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
    at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)

"Reference Handler" daemon prio=10 tid=0x08e24400 nid=0x4ccc in 
Object.wait() [0x90596000]
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x94b398d8> (a java.lang.ref.Reference$Lock)
    at java.lang.Object.wait(Object.java:485)
    at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
    - locked <0x94b398d8> (a java.lang.ref.Reference$Lock)

"VM Thread" prio=10 tid=0x08e20400 nid=0x4ccb runnable

"GC task thread#0 (ParallelGC)" prio=10 tid=0x08daac00 nid=0x4cc9 runnable

"GC task thread#1 (ParallelGC)" prio=10 tid=0x08dac000 nid=0x4cca runnable

"VM Periodic Task Thread" prio=10 tid=0x08e9cc00 nid=0x4cd2 waiting on 
condition

JNI global references: 678

regards,
-Jaikiran