[jboss-jira] [JBoss JIRA] Commented: (JBAS-7468) Memory leak in org.jboss.security.plugins.authorization.JBossAuthorizationContext

Thu Apr 1 05:59:37 EDT 2010

    [ https://jira.jboss.org/jira/browse/JBAS-7468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12523359#action_12523359 ] 

Michal Borowiecki commented on JBAS-7468:
-----------------------------------------

Thanks for the workaround.
This issue can hit you quite suddenly. Our monitoring kept showing 100MB of free heap space and then OutOfMemoryErrors started appearing, monitoring still showing 100MB of free heap.
"controlFlags" must have grown to over 100MB by that time, and on .add() a new enlarged copy of the underlying array was being allocated resulting in OutOfMemoryError while heap usage remained at 100MB below max.
Here is the stack trace for reference, so that someone can google it when it hits them ;)

2010-03-31 20:13:06,791 ERROR [org.apache.catalina.connector.CoyoteAdapter] An exception or error occurred in the container during the request processing
java.lang.OutOfMemoryError: Java heap space
        at java.util.Arrays.copyOf(Arrays.java:2734)
        at java.util.ArrayList.ensureCapacity(ArrayList.java:167)
        at java.util.ArrayList.add(ArrayList.java:351)
        at org.jboss.security.plugins.authorization.JBossAuthorizationContext.initializeModules(JBossAuthorizationContext.java:198)
        at org.jboss.security.plugins.authorization.JBossAuthorizationContext.authorize(JBossAuthorizationContext.java:139)
        at org.jboss.security.plugins.JBossAuthorizationManager.internalAuthorization(JBossAuthorizationManager.java:474)
        at org.jboss.security.plugins.JBossAuthorizationManager.authorize(JBossAuthorizationManager.java:124)
        at org.jboss.security.plugins.javaee.WebAuthorizationHelper.hasUserDataPermission(WebAuthorizationHelper.java:201)
        at org.jboss.web.tomcat.security.JBossWebRealm.hasUserDataPermission(JBossWebRealm.java:643)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:461)
        at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:92)
        at org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.process(SecurityContextEstablishmentValve.java:126)
        at org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.invoke(SecurityContextEstablishmentValve.java:70)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
        at org.apache.catalina.authenticator.SingleSignOn.invoke(SingleSignOn.java:383)
        at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:567)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:330)
        at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:436)
        at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:384)
        at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
        at java.lang.Thread.run(Thread.java:619)

> Memory leak in org.jboss.security.plugins.authorization.JBossAuthorizationContext
> ---------------------------------------------------------------------------------
>
>                 Key: JBAS-7468
>                 URL: https://jira.jboss.org/jira/browse/JBAS-7468
>             Project: JBoss Application Server
>          Issue Type: Bug
>      Security Level: Public(Everyone can see) 
>          Components: Security
>    Affects Versions: JBossAS-5.1.0.GA
>         Environment: JBoss Version: jboss-5.1.0.GA, OS: Linux (2.6.18-164.el5), Architecture: amd64 64bit,  JVM: Java HotSpot(TM) 64-Bit Server VM (14.0-b16, mixed mode)
>            Reporter: Ganesh Ingle
>            Assignee: Anil Saldhana
>             Fix For: JBossAS-6.0.0.M1
>
>         Attachments: HeapMemoryOldGen.png, JBossAuthorizationContext.java, JBossAuthorizationContext_MemLeak.png
>
>
> Our use case (only security related portion is mentioned here):
> Axis 1.4 webservice, standard J2EE declarative security through WEB-INF/web.xml, a http client sends soap request and BASIC auth information, the JBoss server performs authentication and authorization as per WEB-INF/web.xml configuration.
> We did a performance/stability test on above web service. After 8.5 million requests the server gone out of memory. We did heap dump analysis using VisualVM tool and found that the class org.jboss.security.plugins.authorization.JBossAuthorizationContext is consuming most of the memory. This class has a memer array named "controlFlags", this array was showing 25.7 million ControlFlag entries.
> When we investigated the code we found that there is one instance of JBossAuthorizationManager per security domain and this manager has one instance of JBossAuthorizationContext. For every authorization the JBossAuthorizationContext initializes authorization modules and pushes their control flags (instances of class ControlFlag) in member arrays. When the authorization is complete, a commit/abort is invoked on all modules and finally the "modules" array is cleared. However, the "controlFlags" array is not cleared. We checked the entire class, this array never gets cleared.
> We changed the code to clear both "modules" and "controlFlags" array in a finally block in method JBossAuthorizationContext.authorize(final Resource resource, final Subject subject, final RoleGroup callerRoles). We ran a 50million test after this fix, the test was successful which proves the fix worked.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira