[jboss-jira] [JBoss JIRA] Resolved: (JBWEB-82) JBossWeb leaves hung ajp connections with mod_jk

JBossWeb leaves hung ajp connections with mod_jk ------------------------------------------------ Key: JBWEB-82 URL: http://jira.jboss.com/jira/browse/JBWEB-82 Project: JBoss Web Issue Type: Bug Security Level: Public(Everyone can see) Components: Tomcat Module Environment: Apache 2.2.3, mod_jk 1.2, JBoss 4.2.0.GA Reporter: Devon Hillard Assigned To: Mladen Turk I have a Debian (stable) server which is running Apache 2.2.3, and using mod_jk to connect to my application running within JBoss. I have been running with this using JBoss 4.0.5 for many months, without issue. I just upgraded to JBoss 4.2.0. Things start off fine, however after running for a while (last night it took 6.5 hours) the mod_jk logs start showing these errors: Code: [Wed Jun 06 07:45:00 2007][29336:14016] [error] ajp_connection_tcp_get_message::jk_ajp_common.c (948 ): (node1) can't receive the response message from tomcat, network problems or tomcat is down (*MyIP *:8009), err=-104 [Wed Jun 06 07:45:00 2007][29336:14016] [error] ajp_get_reply::jk_ajp_common.c (1566): (node1) Tomca t is down or refused connection. No response has been sent to the client (yet) [Wed Jun 06 07:45:00 2007][29336:14016] [error] ajp_service::jk_ajp_common.c (1928): (node1) Connect ing to tomcat failed. Tomcat is probably not started or is listening on the wrong port These errors do NOT happen with each request. They happen perhaps one out of five, although it seems to fluctuate. Most requests work, but many don't. There are NO errors in the JBoss logs. I have tried using 127.0.0.1 and my actual non-loopback IP address and the same thing happens. I can fix it by restarting JBoss, then everything works, for a few hours at least, before the problem starts up again. ----------------- More information: This appears to be related to open connections. mod_jk was holding connections open to the JBoss web server on 8009, and the number of ESTABLISHED connections kept growing until tomcat started rejecting new connections. Some of the connections would get reused, but not that many. I set the connection_pool_timeout in the mod_jk config to 10 seconds. This keeps the number of established connections down around 50-60. However, the connections that have timed out of the pool, appear to persist forever. The connection from the mod_jk TO JBossWeb sticks in FIN_WAIT1 or FIN_WAIT2: 10.10.77.194:57587 10.10.77.194:8009 FIN_WAIT1 The connection view from JBossWeb gets stuck in CLOSE_WAIT: 10.10.77.194:8009 10.10.77.194:57587 CLOSE_WAIT Once the number of connections hits 1024 (or thereabouts) everything falls apart, of course, and I have to kill -9 jboss and start it up again. This never happened with JBoss 4.0.5. Apache and mod_jk haven't changed at all, so this has to be a change between Tomcat 5.5 or JBossWeb. I can't be the only person to run into this.....