Hi,
We have a big application which uses JBoss 5.1 and is running on Linux servers. Our production environments has 20 web servers (which runs on Apache/Tomcat) which talks to 4 app servers (which runs on JBoss 5.1 in one cluster) with a Oracle 9i database.
As far as the JBoss configuration is concerned, we are using pooledha invoker as our EJB calls are remote calls. Also <max-pool-size> in the oracle-ds.xml is set to 275. The ha-legacy-jboss-beans.xml has the following setup for the pooledInvokerHA:
<bean name="PooledInvokerHA"
class="org.jboss.invocation.pooled.server.PVOPooledInvokerHA">
<depends>jboss:service=TransactionManager</depends>
<depends>HAPartition</depends>
<annotation>@org.jboss.aop.microcontainer.aspects.jmx.JMX(name="jboss:service=invoker,type=pooledha", exposedInterface=org.jboss.invocation.pooled.server.PooledInvokerMBean.class, registerDirectly=true)</annotation>
<property name="numAcceptThreads">1</property>
<property name="maxPoolSize">800</property>
<property name="clientMaxPoolSize">300</property>
<property name="socketTimeout">150000</property>
<property name="serverBindAddress">
<value-factory bean="ServiceBindingManager" method="getStringBinding"
parameter="jboss:service=invoker,type=pooledha"/>
</property>
<property name="serverBindPort">
<value-factory bean="ServiceBindingManager" method="getIntBinding"
parameter="jboss:service=invoker,type=pooledha"/>
</property>
<property name="clientConnectAddress">${jboss.bind.address}</property>
<property name="clientConnectPort">0</property>
<property name="enableTcpNoDelay">false</property>
<property name="transactionManagerService">jboss:service=TransactionManager</property>
</bean>
The issue we are facing is that when we have an average/high load, we are reaching the "InUseConnectionCount" to 275 very quickly. When we look at the database side, we really don't have any traffic. So once we reached the maximum connection mark, the app servers started to get slow and then FULL GC and then it will crash. An interesting point is that during this situation just recycling the app servers doesn't kill the connections though.i.e we end up in recycling the complete web servers so that all the connections will be released. We spend many hours trying to figure out the issue, but no luck. So any thoughts on this issue will be appreciated.
Thanks
Anil Mathew