Cached thread pools going away

[Jeff G.]

I'm having an odd problem that I haven't encountered before, and which I'm
sure is thread-related, but I'm kind of stuck on how to isolate it.

Background: I'm writing a client library to be used inside a server, and I'm
using Netty client stuff to write to multiple hosts in a cluster
simultaneously (via ChannelGroups), talking to a bunch of Redis instances on
the back-end.  I'm sharing one pair of cached thread pools among the client
bootstraps I create.  So, I've got a collection of objects that represent
the Redis instances that make up my cluster, each having its own
ClientBootstrap.

Everything is working fine in my testing until I let the server go idle for
a while.  In the past, with other servers I've written using Netty, the
behavior I've seen with the cached thread pools is for some number of
threads to be created when the application starts, then, when it goes idle,
the pools resize down to one thread each, which hangs around waiting for
requests to come in, then more threads get created as needed for the traffic
level.  In this case, what's happening is that when the Jetty server that my
client is running inside goes idle, after a few minutes, the cached thread
pools I create (via Executors.newCachedThreadPool()) and share among all my
ClientBootstraps shrink to one thread each, as expected, but then a few
seconds later, the pools go away entirely.  Then, when I send more traffic
to the Jetty server, the Jetty acceptor threads deadlock trying to obtain a
lock on something I haven't been able to track down yet (but which I have
reason to suspect might be *my* thread pools that I created, even though I
never pass them out to the server; we've already determined that Jetty is
actively shutting down other threads it shouldn't have access to, so I can't
dismiss the possibility).

I'm of the opinion that there is something weird going on with the
interaction of the various threads (Jetty is starting around 100 threads for
itself when the server starts up), but I've tried every trick I know to find
where the interaction is going awry, with no luck.  Further, my code doesn't
attempt to lock on any object that is not local to the method scope (i.e.,
nothing passed in or that exists external to the current method), so I'm
really baffled.  I have no control over the server part (that belongs to
someone else), so I don't have the option of just re-writing the server on
top of Netty to see if that makes any difference, but any other suggestions
people have are definitely welcome.  I'm using Netty 3.2.4-Final; the server
is built on Jetty 7.4.1.

Anyone who knows threading inside and out have any suggestions?

--
View this message in context: http://netty-forums-and-mailing-lists.685743.n2.nabble.com/Cached-thread-pools-going-away-tp6378919p6378919.html
Sent from the Netty User Group mailing list archive at Nabble.com.