[jbossseam-issues] [JBoss JIRA] Assigned: (JBSEAM-1332) Init.instance() causes (unecessary) lock contention under heavy load.

Init.instance() causes (unecessary) lock contention under heavy load. --------------------------------------------------------------------- Key: JBSEAM-1332 URL: http://jira.jboss.com/jira/browse/JBSEAM-1332 Project: JBoss Seam Issue Type: Bug Components: Core Affects Versions: 1.2.1.GA, 1.2.0.GA Environment: Jboss AS 4.0.5GA , Sun jdk 1.5.0_11 , JSF RI 1.2 b4, Linux FC 5 running 2.5.15.2054_FC5 SMP , Dual Intel Xeons w/ HT enabled. (linux sees 4 cpus); Note that the tomcat instance is modified as described in the associated forum post. Reporter: Radu Aghinii Assigned To: Shane Bryzak Under heavy load the variation of response time for identical seam requests gets very large. For my app, for example, under heavy load some requests will take 5 seconds to finish while others take 5 minutes. Check the associated forum post for a more narrative story that motivated this investigation but the long and the short of it is that my app ( and i suspect any seam app) calls Init.instance() a LOT ( ~100K times per request). The current implementation of it just: Init init = (Init) Contexts.getApplicationContext().get(Init.class); which looks up the seam component name for the Init class and then looks up the component by name. The problem is that the name lookup uses java.lang.Class.getAnnotation() which internally calls the synchronized method "initAnnotationsIfNecessary()". Under heavy load i found most of the jboss request threads waiting to lock that lock. Replacing the body of Init.instance() to skip the name lookup and just do: Init init = (Init) Contexts.getApplicationContext().get("org.jboss.seam.core.init") resulted in much improved behavior. Before this change, once the machine was loaded to the point where requests were taking generally taking 5-10 seconds SOME connections would take a few minutes (capped at 300 seconds due to some timeout either in jboss or apache). Now that performance is pretty bad for a web app, but even transient conditions such as GC, traffic spikes and DB locks would induce the condition. Further, especially when this happened under moderate general load, the machine wouldn't recover, but rather the number of concurrent connections would climb, the performance per requests would fall further and eventually the jvm would run out of memory. With the change in place the machines recover. Even when the average requests are taking 20-30 seconds (a very high load situation, but we see it due to the transients mentioned above) the slowest requests are in the 1minute to 2 minute range and, most importantly, the machine recovers pretty quickly.