Stuck in an infinite loop inside Netty when shutting down NIO client

[tsuna]

In my application, I'm calling
NioClientSocketChannelFactory.releaseExternalResources() from a brand
new thread, and I'm still getting in cases where my application is
unable to shutdown as it gets stuck in an infinite loop in Netty.

Here's the stack trace of the thread that's shutting down Netty:
"HBaseClient at 647057258 shutdown" prio=10 tid=0x00000000402ea000
nid=0x60a1 waiting on condition [0x0000000040927000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00002aab5fea21a8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
	at java.util.concurrent.ThreadPoolExecutor.awaitTermination(ThreadPoolExecutor.java:1253)
	at org.jboss.netty.util.internal.ExecutorUtil.terminate(ExecutorUtil.java:87)
	at org.jboss.netty.channel.socket.nio.NioClientSocketChannelFactory.releaseExternalResources(NioClientSocketChannelFactory.java:143)
	at org.hbase.async.HBaseClient$1ShutdownThread.run(HBaseClient.java:502)


Here are the two lingering threads from Netty that don't wanna go away:

"New I/O client worker #1-3" prio=10 tid=0x00000000402e7000 nid=0x60a0
runnable [0x00000000405ab000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.IOUtil.drain(Native Method)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:76)
	- locked <0x00002aab5feaa3f8> (a java.lang.Object)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
	- locked <0x00002aab5feaa130> (a sun.nio.ch.Util$1)
	- locked <0x00002aab5feaa118> (a java.util.Collections$UnmodifiableSet)
	- locked <0x00002aab5fea3190> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
	at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:38)
	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:164)
	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
	at org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:46)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	at java.lang.Thread.run(Thread.java:662)

"New I/O client worker #1-2" prio=10 tid=0x0000000040113000 nid=0x609f
runnable [0x0000000040c8b000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.IOUtil.drain(Native Method)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:76)
	- locked <0x00002aab5fea85d0> (a java.lang.Object)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
	- locked <0x00002aab5fea8308> (a sun.nio.ch.Util$1)
	- locked <0x00002aab5fea82f0> (a java.util.Collections$UnmodifiableSet)
	- locked <0x00002aab5fea6658> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
	at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:38)
	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:164)
	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
	at org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:46)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	at java.lang.Thread.run(Thread.java:662)

Full stack trace is in the comment at the end of
https://gist.github.com/709554#file_async_h_base_put.java
The code that's shutting down Netty is at
https://github.com/stumbleupon/asynchbase/blob/def453b61935f9b93a8bc58e720e31ab2969482f/src/HBaseClient.java#L502

Any idea what could be causing this?  Am I doing something wrong?

-- 
Benoit "tsuna" Sigoure
Software Engineer @ www.StumbleUpon.com