[ https://jira.jboss.org/browse/EJBTHREE-2042?page=com.atlassian.jira.plugi... ] Remi Nodet commented on EJBTHREE-2042: -------------------------------------- We are having as well performance problems related to security. Since we have migrated from JBoss 4.2 to JBoss EAP 5.0.1 we have seen a noticeable performance decrease, both during our nightly automated tests and on production. Doing a quick profiling during our nightly load tests (sampling) shows that there is a bottleneck in this method: 20.4% - 32,520 ms java.security.AccessController.doPrivileged(java.security.PrivilegedAction) It is called from a lot of different places: 2.8% - 4,479 ms org.jboss.ejb.plugins.security.SecurityActions.getSecurityContext 2.7% - 4,254 ms org.jboss.ejb.plugins.SecurityActions.getSecurityContext 2.3% - 3,662 ms org.jboss.ejb.plugins.security.SecurityActions.trace(org.jboss.security.SecurityContext) 2.0% - 3,234 ms org.jboss.ejb.plugins.security.SecurityActions.trace(org.jboss.security.SecurityIdentity) 2.0% - 3,121 ms org.jboss.ejb.SecurityActions.setContextID 1.7% - 2,783 ms org.jboss.ejb.plugins.SecurityActions.authorize 1.0% - 1,545 ms org.jboss.security.SimplePrincipal.equals 0.9% - 1,381 ms org.jboss.security.jacc.SubjectPolicyContextHandler.getContext 0.8% - 1,232 ms org.jboss.ejb.SecurityActions.getSecurityContext 0.7% - 1,173 ms org.jboss.ejb.plugins.SecurityActions.isValid 0.6% - 989 ms org.jboss.ejb.plugins.SecurityActions.popSubjectContext 0.3% - 464 ms org.jboss.security.authorization.modules.SecurityActions.getContextClassLoader 0.2% - 397 ms org.jboss.ejb.plugins.security.SecurityActions.setSecurityDomain 0.2% - 348 ms org.jboss.ejb.plugins.PerTxEntityInstanceCache.removeTimerServiceIfAllCancelledOrExpired 0.2% - 339 ms org.jboss.ejb.plugins.security.SecurityActions.getSecurityIdentity Also noticeable 12.3% - 19,653 ms java.lang.StringBuilder.append 7.1% - 11,243 ms javax.security.auth.Subject.toString 7.1% - 11,243 ms javax.security.auth.Subject.toString 7.1% - 11,243 ms java.lang.String.valueOf -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/browse/EJBTHREE-2042?page=com.atlassian.jira.plugi... ] Remi Nodet updated EJBTHREE-2042: --------------------------------- Attachment: ProfilingHotSpots.zip Profiling hotspots in an HTML file -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/browse/EJBTHREE-2042?page=com.atlassian.jira.plugi... ] Remi Nodet commented on EJBTHREE-2042: -------------------------------------- It is a little difficult to provide you with the bean code since this is gotten from a large application with a lot of stateless session beans and entity beans. However it looks to me like every single call to any method (remote, local, home or business interface) on a session bean or entity bean (I am using EJB 2.1) results in calls to doPrivileged(). Since we have a lot of calls to local entity beans generated for each call to our remote stateless session beans (facade pattern) the doPrivileged() method seems to turn out to be a nasty bottleneck. I'll try to make a simple example with only a session bean and an entity bean to get a simpler scenario for the profiling -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/browse/EJBTHREE-2042?page=com.atlassian.jira.plugi... ] Maros Bajtos updated EJBTHREE-2042: ----------------------------------- Attachment: hotspots.tar.gz blocking hotspots -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/browse/EJBTHREE-2042?page=com.atlassian.jira.plugi... ] Maros Bajtos edited comment on EJBTHREE-2042 at 12/3/10 3:32 AM: ----------------------------------------------------------------- I've attached blocking hotspots from my yesterday's profiling. Please see also my thread about terrible performance with concurrent ejb calls (which is more descriptive) - http://community.jboss.org/message/574096 . To sum it up, I've found out that if there is lot of (hundreds) concurrent ejb calls, they'll start to block each other on ClassLoader.loadClass method which is called from hotspots I've attached. ClassLoader.loadClass is synchronized method - so only one thread at given time may be executing it. Sample trace calling it: org.jboss.ejb3.security.RoleBasedAuthorizationInterceptorv2.invoke SecurityHelperFactory.getEJBAuthorizationHelper SecurityActions.loadClass ClassLoader.loadClass The application I profiled is composed of lot of ejbs and most of the ejb calls are really fast - therefore with one request from web, there is maybe hundreds fast ejb calls - as soon as more users start to use application, these small concurrent ejb calls start to block each other because of synchronized method. was (Author: maros.bajtos): I've added blocking hotspots from my yesterday's profiling. Please see also my thread about terrible performance with concurrent ejb calls (which is more descriptive) - http://community.jboss.org/message/574096 . To sum it up, I've found out that if there is lot of (hundreds) concurrent ejb calls, they'll start to block each other on ClassLoader.loadClass method which is called from hotspots I've attached. ClassLoader.loadClass is synchronized method - so only one thread at given time may be executing it. Sample trace calling it: org.jboss.ejb3.security.RoleBasedAuthorizationInterceptorv2.invoke SecurityHelperFactory.getEJBAuthorizationHelper SecurityActions.loadClass ClassLoader.loadClass The application I profiled is composed of lot of ejbs and most of the ejb calls are really fast - therefore with one request from web, there is maybe hundreds fast ejb calls - as soon as more users start to use application, these small concurrent ejb calls start to block each other because of synchronized method. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira