Cached thread pools going away

[Marc-André Laverdière]

Hello,

I'm decent at threads, and I NEVER heard of something like this.
I am thinking... is there any way you can try with other thread pools?
Like MATPE?
Otherwise, is there any way you can extract your netty code from Jetty
and run that in isolation (with some stubbing)? At least you could
confirm if the interaction is responsible for that.

On 5/18/11, Jeff G. <jeff at steelhouse.com> wrote:
> I'm having an odd problem that I haven't encountered before, and which I'm
> sure is thread-related, but I'm kind of stuck on how to isolate it.
>
> Background: I'm writing a client library to be used inside a server, and I'm
> using Netty client stuff to write to multiple hosts in a cluster
> simultaneously (via ChannelGroups), talking to a bunch of Redis instances on
> the back-end.  I'm sharing one pair of cached thread pools among the client
> bootstraps I create.  So, I've got a collection of objects that represent
> the Redis instances that make up my cluster, each having its own
> ClientBootstrap.
>
> Everything is working fine in my testing until I let the server go idle for
> a while.  In the past, with other servers I've written using Netty, the
> behavior I've seen with the cached thread pools is for some number of
> threads to be created when the application starts, then, when it goes idle,
> the pools resize down to one thread each, which hangs around waiting for
> requests to come in, then more threads get created as needed for the traffic
> level.  In this case, what's happening is that when the Jetty server that my
> client is running inside goes idle, after a few minutes, the cached thread
> pools I create (via Executors.newCachedThreadPool()) and share among all my
> ClientBootstraps shrink to one thread each, as expected, but then a few
> seconds later, the pools go away entirely.  Then, when I send more traffic
> to the Jetty server, the Jetty acceptor threads deadlock trying to obtain a
> lock on something I haven't been able to track down yet (but which I have
> reason to suspect might be *my* thread pools that I created, even though I
> never pass them out to the server; we've already determined that Jetty is
> actively shutting down other threads it shouldn't have access to, so I can't
> dismiss the possibility).
>
> I'm of the opinion that there is something weird going on with the
> interaction of the various threads (Jetty is starting around 100 threads for
> itself when the server starts up), but I've tried every trick I know to find
> where the interaction is going awry, with no luck.  Further, my code doesn't
> attempt to lock on any object that is not local to the method scope (i.e.,
> nothing passed in or that exists external to the current method), so I'm
> really baffled.  I have no control over the server part (that belongs to
> someone else), so I don't have the option of just re-writing the server on
> top of Netty to see if that makes any difference, but any other suggestions
> people have are definitely welcome.  I'm using Netty 3.2.4-Final; the server
> is built on Jetty 7.4.1.
>
> Anyone who knows threading inside and out have any suggestions?
>
> --
> View this message in context:
> http://netty-forums-and-mailing-lists.685743.n2.nabble.com/Cached-thread-pools-going-away-tp6378919p6378919.html
> Sent from the Netty User Group mailing list archive at Nabble.com.
> _______________________________________________
> netty-users mailing list
> netty-users at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/netty-users
>


-- 
Marc-André LAVERDIÈRE
"Perseverance must finish its work so that you may be mature and complete,
not lacking anything." -James 1:4
http://asimplediscipleslife.blogspot.com/
mlaverd.theunixplace.com