[undertow-dev] InSessionMemoryManager synchronized methods

Tue Jul 7 14:11:02 EDT 2015

We're still using 1.1.x to keep code churn to a minimum - internal deadlines on this end have required me to back-port to 1.1.x and build - Looking at your 1.1.x version, it's functionally identical to the 1.1.x I prepared locally, I'll keep with our version, there is no need to cut a 1.1.x for us.

For others encountering UNDERTOW-479, if they are using seam I'd suggest they look at JBSEAM-4861

Fresh data:
It looks like we ran into https://issues.jboss.org/browse/JBSEAM-4861 which set us up for the deadlock in Undertow by causing multiple threads to pile up on the org.jboss.seam.Component.createFromFactory() call.
- Looks like Seam  versions prior to 2.3.0.CR1 help create the Undertow hang.  This includes the Seam 2.2.x (which we use).

The fix to Undertow makes recovery from long blocks in createFromFactory() possible.
The fix to JBSEAM-4861 makes the initial failure unlikely, and appears to introduce new deadlock problems.  Threading in complex settings is never easy.  

//EricP

-----Original Message-----
From: Stuart Douglas [mailto:sdouglas at redhat.com] 
Sent: Tuesday, July 07, 2015 10:57 AM
To: Eric Peters
Cc: undertow-dev at lists.jboss.org
Subject: Re: [undertow-dev] InSessionMemoryManager synchronized methods

This has been backported to the 1.2 and 1.1 branches. 

Which branch are you on? I will try and get a release out this week.

Stuart

----- Original Message -----
> From: "Eric Peters" <epeters at epicor.com>
> To: undertow-dev at lists.jboss.org
> Sent: Wednesday, 1 July, 2015 8:44:46 PM
> Subject: Re: [undertow-dev] InSessionMemoryManager synchronized 
> methods
> 
> 
> 
> I’m still trying to gauge the release process for Undertow, and where 
> in that process this issue sits, so as to decide how to roll things 
> out to customers on my end:
> 
> - I see UNDERTOW-479 was entered for this deadlock.
> 
> - I see a patch for this was pulled into Undertow 1.3.0 Beta 3
> 
> - The patch has not been back-ported to prior releases.
> 
> - The Jira case is still open.
> 
> 
> 
> What can I expect to see in the coming days? I need an answer to give 
> management for the question of ‘How soon will the patch be deemed 
> production-ready?’, but I’d appreciate any additional insight that can 
> be provided.
> 
> 
> 
> //EricP
> 
> 
> 
> 
> From: Eric Peters
> Sent: Thursday, June 25, 2015 5:23 PM
> To: Eric Peters; undertow-dev at lists.jboss.org
> Subject: RE: InSessionMemoryManager synchronized methods
> 
> 
> 
> 
> I stand corrected – This is 50% of what brought our server down.
> 
> 
> 
> The other half of the deadlock belongs to Seam:
> 
> 
> 
>     *
>     io.undertow.server.session.InMemorySessionManager$SessionImpl.bumpTimeout()
>     (line: 271)
>     *
>     io.undertow.server.session.InMemorySessionManager$SessionImpl.getAttribute(java.lang.String)
>     (line: 355)
>     * io.undertow.servlet.spec.HttpSessionImpl.getAttribute(java.lang.String)
>     * com.sun.faces.context.SessionMap.get(java.lang.Object)
>     * org.jboss.seam.contexts.ServerConversationContext.get(java.lang.String)
>     (line: 110)
>     *
>     org.jboss.seam.contexts.Contexts.lookupInStatefulContexts(java.lang.String)
>     (line: 189)
>     * org.jboss.seam.Component.getInstanceFromFactory(java.lang.String,
>     org.jboss.seam.ScopeType) (line: 2088)
>     * org.jboss.seam.Component.getInstance(java.lang.String, boolean,
>     boolean, java.lang.Object, org.jboss.seam.ScopeType) (line: 2014)
>     * org.jboss.seam.Component.getInstance(java.lang.String, boolean,
>     boolean) (line: 1986)
>     * org.jboss.seam.Component.getInstance(java.lang.String, boolean)
>     * org.jboss.seam.Namespace.getComponentInstance(java.lang.String,
>     boolean)
> 
> 
> 
> 
> org.jboss.seam.Component.getInstanceFromFactory() obtains a lock that 
> has static scope, so nothing else can invoke getInstanceFromFacotry() 
> while the lock is held.It then blocks on
> InMemorySessionManager$SessionImpl.bumpTimeout()
> 
> Meanwhile, the invalidate() method blocks against the static method in
> getInstanceFromFactory()
> 
> //EricP
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> From: undertow-dev-bounces at lists.jboss.org [ 
> mailto:undertow-dev-bounces at lists.jboss.org ] On Behalf Of Eric Peters
> Sent: Thursday, June 25, 2015 4:39 PM
> To: undertow-dev at lists.jboss.org
> Subject: [undertow-dev] InSessionMemoryManager synchronized methods
> 
> 
> 
> 
> Chasing after another problem I found several blocked task threads, 
> best as I can determine:
> 
> InMemorySessionmanager$SessionImpl.invalidate() is invoked.
> 
> This method is synchronized, and the call blocks inside
> SesssionListeners.sessionDestroyed() (separate issue there…)
> 
> 
> 
> Later, InMemorySessionmanager$SessionImpl.bumpTimeout() is invoked 
> from another thread.
> 
> This method is also synchronized, and blocks waiting for invalidate() 
> to complete. As the first thread is indefinitely blocked inside 
> invalidate(), this second thread also remains blocked indefinitely.
> 
> 
> 
> In my environment it appears to make an existing problem slightly more 
> severe.
> 
> 
> 
> Found in 1.1.0, but it looks like it’s in both 1.1.7 and the master 
> branch as well.
> 
> Using RHEL, Java 7, Wildfly 8.2
> 
> 
> 
> invalidate() Thread trace (excerpt):
> 
>     * (blocked) - sun.misc.Unsafe.park(boolean, long)
>     * …
>     *
>     org.jboss.seam.contexts.ServletLifecycle.endSession(javax.servlet.http.HttpSession)
>     (line: 187)
>     *
>     org.jboss.seam.servlet.SeamListener.sessionDestroyed(javax.servlet.http.HttpSessionEvent)
>     (line: 59)
>     *
>     io.undertow.servlet.core.ApplicationListeners.sessionDestroyed(javax.servlet.http.HttpSession)
>     (line: 264)
>     *
>     io.undertow.servlet.core.SessionListenerBridge.sessionDestroyed(io.undertow.server.session.Session,
>     io.undertow.server.HttpServerExchange,
>     io.undertow.server.session.SessionListener$SessionDestroyedReason)
>     (line: 67)
>     *
>     io.undertow.server.session.SessionListeners.sessionDestroyed(io.undertow.server.session.Session,
>     io.undertow.server.HttpServerExchange,
>     io.undertow.server.session.SessionListener$SessionDestroyedReason)
>     (line: 61)
>     *
>     io.undertow.server.session.InMemorySessionManager$SessionImpl.invalidate(io.undertow.server.HttpServerExchange,
>     io.undertow.server.session.SessionListener$SessionDestroyedReason)
>     (line: 413)
>     *
>     io.undertow.server.session.InMemorySessionManager$SessionImpl.invalidate(io.undertow.server.HttpServerExchange)
>     (line: 399)
>     * io.undertow.servlet.spec.HttpSessionImpl.invalidate() (line: 197)
>     * …
> 
> 
> 
> 
> bumpTimeout() Thread trace:
> 
>     *
>     io.undertow.server.session.InMemorySessionManager$SessionImpl.bumpTimeout()
>     (line: 271)
>     *
>     io.undertow.server.session.InMemorySessionManager$SessionImpl.getAttribute(java.lang.String)
>     (line: 355)
>     *
>     io.undertow.servlet.util.SavedRequest.tryRestoreRequest(io.undertow.server.HttpServerExchange,
>     javax.servlet.http.HttpSession) (line: 124)
>     *
>     io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(io.undertow.server.HttpServerExchange)
>     (line: 67)
>     *
>     io.undertow.security.handlers.SecurityInitialHandler.handleRequest(io.undertow.server.HttpServerExchange)
>     (line: 76)
>     *
>     io.undertow.server.handlers.PredicateHandler.handleRequest(io.undertow.server.HttpServerExchange)
>     (line: 43)
>     *
>     org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(io.undertow.server.HttpServerExchange)
>     (line: 61)
>     *
>     io.undertow.server.handlers.PredicateHandler.handleRequest(io.undertow.server.HttpServerExchange)
>     (line: 43)
>     *
>     io.undertow.server.handlers.PredicateHandler.handleRequest(io.undertow.server.HttpServerExchange)
>     (line: 43)
>     *
>     io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(io.undertow.server.HttpServerExchange,
>     io.undertow.servlet.handlers.ServletChain,
>     io.undertow.servlet.handlers.ServletRequestContext,
>     javax.servlet.ServletRequest, javax.servlet.ServletResponse) (line: 261)
>     *
>     io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(io.undertow.server.HttpServerExchange,
>     io.undertow.servlet.handlers.ServletRequestContext,
>     io.undertow.servlet.handlers.ServletChain, javax.servlet.DispatcherType)
>     (line: 247)
>     *
>     io.undertow.servlet.handlers.ServletInitialHandler.access$000(io.undertow.servlet.handlers.ServletInitialHandler,
>     io.undertow.server.HttpServerExchange,
>     io.undertow.servlet.handlers.ServletRequestContext,
>     io.undertow.servlet.handlers.ServletChain, javax.servlet.DispatcherType)
>     *
>     io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(io.undertow.server.HttpServerExchange)
>     (line: 166)
>     *
>     io.undertow.server.Connectors.executeRootHandler(io.undertow.server.HttpHandler,
>     io.undertow.server.HttpServerExchange) (line: 197)
>     * io.undertow.server.HttpServerExchange$1.run() (line: 759)
>     *
>     java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
>     (line: 1145)
>     * java.util.concurrent.ThreadPoolExecutor$Worker.run() (line: 615)
>     * java.lang.Thread.run() (line: 745)
> 
> 
> //EricP
> 
> 
> 
> 
> 
> This message has been scanned for malware by Websense. 
> www.websense.com
> 
> 
> 
> Click here to report this email as spam.
> 
> _______________________________________________
> undertow-dev mailing list
> undertow-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/undertow-dev