[jboss-user] [Installation, Configuration & Deployment] - MaxThread issue with Apache mod_proxy

[gdriggs]

Hi folks,

Pardon my verbosity but I want to include as much info as possible...

We're a mid-size software company and we've recently launched a web forum using JForum and JBoss -- we hadn't known about JBoss's own forum that's part of JBoss Portal(?) or we might have considered it first. Anyhow, after issuing a press release yesterday, our fora were swamped with traffic and at 3am went offline. Here's what I'm seeing in the logs earlier in the day but nothing of the sort when it finally went offline in the middle of the night...

12:48:16,806 ERROR [ThreadPool] All threads (250) are currently busy, waiting. Increase maxThreads (250) or check the servlet status
12:49:24,829 WARN  [MsgContext] Error sending end packet
java.net.SocketException: Broken pipe
        at java.net.SocketOutputStream.socketWrite0(Native Method)
        at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
        at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
        at org.apache.jk.common.ChannelSocket.send(ChannelSocket.java:518)
        at org.apache.jk.common.JkInputStream.endMessage(JkInputStream.java:112)
        at org.apache.jk.core.MsgContext.action(MsgContext.java:293)
        at org.apache.coyote.Response.action(Response.java:182)
        at org.apache.coyote.Response.finish(Response.java:304)
        at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:204)
        at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:282)
        at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:754)
        at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:684)
        at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:876)
        at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
        at java.lang.Thread.run(Thread.java:595)
12:49:24,830 WARN  [ChannelSocket] processCallbacks status 2
12:50:49,812 WARN  [MsgContext] Error sending end packet

We're using Apache 2.2.2 with mod_proxy and the following settings:

ProxyTimeout 10
 <Proxy balancer://cluster>
  BalancerMember ajp://app2.example.com:8009
 
 <Location /jforum>
   ProxyPass balancer://cluster/jforum
 

I've found the likely place to change the thread count in jboss.web in the jk-8009 ThreadPool but I want to know if my Apache settings are causing problems as well. I started out with a 60 second time out and lowering has helped considerably but it only manages to postpone our crashes. Until we upgrade in a few weeks, we're running all pieces on RedHat Enterprise Linux 2.1 with a 2.4.9-e.59custom kernel.

I've been monitoring the web server and noticed that after dropping the ProxyTimeout value, the number of connections in a wait state have dropped considerably so it seems to be helping (the following output is snipped):

$ netstat -a|grep 8009
tcp        0      0 forums.example.com:41174   app2.extensis.com:8009  ESTABLISHED 
tcp        0      0 forums.example.com:41180   app2.example.com:8009  ESTABLISHED 
tcp        0      0 forums.example.com:41163   app2.example.com:8009  ESTABLISHED 
tcp        0      0 forums.example.com:41205   app2.example.com:8009  TIME_WAIT   
tcp        0      0 forums.example.com:41204   app2.example.com:8009  ESTABLISHED 
tcp        0      0 forums.example.com:41207   app2.example.com:8009  TIME_WAIT   
tcp        0      0 forums.example.com:41206   app2.example.com:8009  ESTABLISHED 

Any help would be most appreciated.

-Gary

View the original post : http://www.jboss.com/index.html?module=bb&op=viewtopic&p=3967317#3967317

Reply to the post : http://www.jboss.com/index.html?module=bb&op=posting&mode=reply&p=3967317