[jboss-dev] JBoss Embedded AS - JVM keeps running

[Jaikiran Pai]

Jaikiran Pai wrote:
>
> However, there are some non-daemon threads still active even after a 
> server shutdown:
>
> "pool-15-thread-1" prio=10 tid=0x8cb1ec00 nid=0x4d13 waiting on 
> condition [0x8e7a6000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
>     at sun.misc.Unsafe.park(Native Method)
>     - parking to wait for  <0xb20d6258> (a 
> java.util.concurrent.SynchronousQueue$TransferStack)
>     at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>     at 
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>     at 
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>     at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>     at 
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>     at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>     at java.lang.Thread.run(Thread.java:619)
>
> ...
> "pool-5-thread-1" prio=10 tid=0x8cfc2800 nid=0x4ce0 waiting on condition 
> [0x8cdad000]
>    java.lang.Thread.State: WAITING (parking)
>     at sun.misc.Unsafe.park(Native Method)
>     - parking to wait for  <0x97ece7e8> (a 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>     at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>     at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
>     at 
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
>     at 
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>     at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>     at java.lang.Thread.run(Thread.java:619)
>
>
> These pool-x-thread-y threads prevent the JVM from shutting down. The 
> thread names aren't very informative to figure out which service started 
> those threads and why were those not shutdown. Anyone knows what these 
> threads are for and why they are hanging around after a server shutdown?
>
Thanks to Byteman (http://www.jboss.org/byteman), I was able to figure 
out what was creating these threads and why they were not cleaned up on 
shutdown.

Out the two threads, one is from ShrinkWrap 
(http://www.jboss.org/shrinkwrap) which uses an ExecutorService to 
create a cached thread pool. The thread created out of this pool was 
staying around idle for 60 seconds (default) which was preventing (or 
rather contributing to one minute delay in) the standalone client shutdown.

However, the main culprit turns out to be NamingServer in AS which uses 
an Exceutor backed by the default thread pool factory which creates 
non-daemon threads 
http://anonsvn.jboss.org/repos/jbossas/trunk/server/src/etc/deploy/naming-jboss-beans.xml:

<bean name="LookupPool">
<constructor factoryMethod="newFixedThreadPool"
factoryClass="java.util.concurrent.Executors">
<!-- At least 2 threads are required -->
<parameter>2</parameter>
</constructor>
</bean>

<bean name="RemoteNamingBean" class="org.jnp.server.Main">
...
<!-- The thread pool service used to control the bootstrap lookups -->
<property name="lookupExector"><inject bean="LookupPool"/></property>
</bean>

Locally, I have fixed this by passing a JBossThreadFactory (configured 
to create daemon threads) to the Executor, so that it doesn't use the 
default one. I have some questions around the configuration options for 
JBossThreadFactory, for which I'll create a forum thread. I'll commit 
the fix after that discussion.

By the way, as ALR pointed out on IRC, changing any random thread in AS 
to mark them as daemon isn't always the right solution. But in case of 
the NamingServer, IMO, changing it to daemon makes sense. If anyone has 
objections, feel free to raise them.

-Jaikiran