RuntimeSupport.find2Methods a perf hotspot when proxy's methods are called at higher concurrency ------------------------------------------------------------------------------------------------ Key: JASSIST-163 URL: https://issues.jboss.org/browse/JASSIST-163 Project: Javassist Issue Type: Enhancement Affects Versions: 3.15.0-GA, 3.16.1-GA Environment: hibernate-core 3.6.10.Final Reporter: Nikita Tovstoles Assignee: Shigeru Chiba Fix For: 3.17.0-GA Attachments: Bean_$$_bulkaccess_0.txt, Bean_$$_bulkaccess_0_Two.txt, blocked-threads.png, BulkAccessorFactory.java.diff, BulkAccessorFactory.java.diff2, BulkAccessorFactory.patch, find2methods-hotspot.png, jassist-163-fix.patch, monitor-backtraces.png, monitor-backtraces.png, Product.java, Product_$$_javassist_0-post-patch.java, Product_$$_javassist_0.java, Tomcat-2012-03-28(2).zip We've been profiling our Hibernate 3.6.10-based app and noticed a perf bottleneck in javassist.util.proxy.RuntimeSupport.find2methods. Unfortunately, this method, which has a synch. block, is being called on every invocation of every proxied entity method (see javassist.util.proxy.ProxyFactory.makeForwarder(), called indirectly by ProxyFactory.createClass()). In our testing, the result is that our service call's latency increases from 33 to 55, 260, 400ms as concurrency increases 1-10-20-30 users on a 4-core CPU. At 20 and 30 users 51% of CPU time is spent contending for a monitor in RuntimeSupport.find2methods: {code} synchronized (methods) { if (methods[index] == null) { methods[index + 1] = thisMethod == null ? null : findMethod(self, thisMethod, desc); methods[index] = findSuperMethod(self, superMethod, desc); } } {code} Since find2methods merely interrogates class metadata, seems like its return values should be cached (in a ConcurrentMap?) instead of repeatedly executing the above synchronized statement. Instead, currently, it's being called every time (?) a proxied method is executed - see *Invocation Count* in this screen shot: https://issues.jboss.org/secure/attachment/12353025/monitor-backtraces.png Full [YourKit profile|http://yourkit.com] is [attached as a ZIP archive|^Tomcat-2012-03-28(2).zip]; key screen shots from the snapshot also attached separately