[jboss-jira] [JBoss JIRA] (WFLY-5727) After day or two Wildfly hangs and consumes 100% CPU
Petri Tuomaala (JIRA)
issues at jboss.org
Thu Nov 26 02:05:00 EST 2015
[ https://issues.jboss.org/browse/WFLY-5727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13134027#comment-13134027 ]
Petri Tuomaala commented on WFLY-5727:
--------------------------------------
I could try it in our development environments but we can't reproduce this on that environment either on 9.01 => not sure does that verify anything. We believe that this needs some kind of slow request etc to kick this on but haven't found the way yet. In environments where this happens has real users and we need to verify that our rather big legacy application works in 10 before we can try it => slow process.
Any workaround tips or root cause for this that we could make the env more stable? We have tried many conf changes and currently figuring out should we handle SSL in proxy (our app has restrictions for this) or run the server on windows
> After day or two Wildfly hangs and consumes 100% CPU
> ----------------------------------------------------
>
> Key: WFLY-5727
> URL: https://issues.jboss.org/browse/WFLY-5727
> Project: WildFly
> Issue Type: Bug
> Components: IO, Web (Undertow)
> Affects Versions: 8.2.0.Final, 9.0.1.Final
> Environment: Wildfly 9.01 Final, Wildfly 8.2
> CentOS Linux release 7.1.1503 (Core)
> java version "1.8.0_66"
> Java(TM) SE Runtime Environment (build 1.8.0_66-b17)
> Java HotSpot(TM) 64-Bit Server VM (build 25.66-b17, mixed mode)
> Reporter: Petri Tuomaala
> Assignee: Stuart Douglas
> Priority: Blocker
> Labels: Linux, io, ssl, undertow, wildfly, xnio
> Fix For: 9.x.x TBD
>
> Attachments: jstack.22263.102919.280420771, jstack.22263.102925.032513958, jstack.22263.102930.660792783, jstack.22263.102936.697001192, jstack.22263.102942.535009068, jstack.22263.102948.169314576, jstack.22263.102954.185475598, jstack.22263.102959.848031929, jstack.22263.103005.599986837, jstack.22263.103011.261542540, monitor.png, pid.png
>
>
> First disclaimer, this looks pointing to undertow/xnio but I am not sure for the root cause so I'm filing this for Wildfly.
> After a day or two Wildfly just hangs and consumes the cpu. We have noticed that the site traffic itself has no impact on this: it has happened during peak traffic and also almost in idle situations.
> We are handling SSL in Wildfly and example this https://issues.jboss.org/browse/UNDERTOW-282 looks similar to this but it has been resolved over year ago so it should be in Wildfly 9.01.
> Our SSL conf in standalone.xml:
> <https-listener name="https" socket-binding="https" security-realm="UndertowRealm" enabled-cipher-suites="TLS_DHE_DSS_WITH_AES_128_CBC_SHA,SSL_RSA_WITH_3DES_EDE_CBC_SHA,SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA" max-parameters="5000"/>
> I have narrowed down the thread that hangs. Or at least tried, my first time to study thread dumps. PID for thread that is jammed is 22599 => 0x5847 as hex and it points to the thread end of this description. I attach the actual thread dumps to this ticket. Any workaround for this is preciated, we currently need to reboot the servers when the Wildfly hangs (almost every nigth).
> "default task-20" #285 prio=5 os_prio=0 tid=0x00007f862c328800 nid=0x5847 runnable [0x00007f85f5ad6000]
> java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
> at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
> at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
> at sun.nio.ch.IOUtil.read(IOUtil.java:192)
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
> - locked <0x00000006f6065210> (a java.lang.Object)
> at org.xnio.nio.NioSocketConduit.read(NioSocketConduit.java:282)
> at org.xnio.ssl.JsseSslConduitEngine.handleUnwrapResult(JsseSslConduitEngine.java:737)
> - locked <0x00000006f606b708> (a org.xnio.ByteBufferSlicePool$PooledByteBuffer)
> at org.xnio.ssl.JsseSslConduitEngine.unwrap(JsseSslConduitEngine.java:620)
> - locked <0x00000006f606b708> (a org.xnio.ByteBufferSlicePool$PooledByteBuffer)
> at org.xnio.ssl.JsseSslConduitEngine.unwrap(JsseSslConduitEngine.java:574)
> at org.xnio.ssl.JsseSslStreamSourceConduit.read(JsseSslStreamSourceConduit.java:89)
> at org.xnio.conduits.AbstractStreamSourceConduit.read(AbstractStreamSourceConduit.java:51)
> at io.undertow.conduits.ReadDataStreamSourceConduit.read(ReadDataStreamSourceConduit.java:67)
> at io.undertow.conduits.FixedLengthStreamSourceConduit.read(FixedLengthStreamSourceConduit.java:249)
> at org.xnio.conduits.ConduitStreamSourceChannel.read(ConduitStreamSourceChannel.java:127)
> at io.undertow.channels.DetachableStreamSourceChannel.read(DetachableStreamSourceChannel.java:209)
> at io.undertow.server.HttpServerExchange$ReadDispatchChannel.read(HttpServerExchange.java:2136)
> at io.undertow.server.handlers.form.FormEncodedDataDefinition$FormEncodedDataParser.doParse(FormEncodedDataDefinition.java:133)
> at io.undertow.server.handlers.form.FormEncodedDataDefinition$FormEncodedDataParser.parseBlocking(FormEncodedDataDefinition.java:251)
> at io.undertow.servlet.spec.HttpServletRequestImpl.parseFormData(HttpServletRequestImpl.java:753)
> at io.undertow.servlet.spec.HttpServletRequestImpl.getParameter(HttpServletRequestImpl.java:627)
> at org.apache.myfaces.context.servlet.RequestParameterMap.getAttribute(RequestParameterMap.java:45)
> at org.apache.myfaces.context.servlet.RequestParameterMap.getAttribute(RequestParameterMap.java:33)
> at org.apache.myfaces.util.AbstractAttributeMap.containsKey(AbstractAttributeMap.java:62)
> at org.apache.myfaces.renderkit.html.HtmlResponseStateManager.isPostback(HtmlResponseStateManager.java:303)
> at org.apache.myfaces.lifecycle.DefaultRestoreViewSupport.isPostback(DefaultRestoreViewSupport.java:242)
> at org.apache.myfaces.lifecycle.RestoreViewExecutor.execute(RestoreViewExecutor.java:120)
> at org.apache.myfaces.lifecycle.LifecycleImpl.executePhase(LifecycleImpl.java:170)
> at org.apache.myfaces.lifecycle.LifecycleImpl.execute(LifecycleImpl.java:117)
> at javax.faces.webapp.FacesServlet.service(FacesServlet.java:197)
> at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:86)
> at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:130)
> at fi.elbit.skills.loginview.DefaultsFilter.doFilter(DefaultsFilter.java:117)
> at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:60)
> at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:132)
> at org.apache.myfaces.webapp.filter.ExtensionsFilter.doFilter(ExtensionsFilter.java:357)
> at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:60)
> at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:132)
> at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:85)
> at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)
> at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)
> at org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)
> at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
> at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:131)
> at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)
> at io.undertow.server.handlers.DisableCacheHandler.handleRequest(DisableCacheHandler.java:33)
> at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
> at io.undertow.security.handlers.AuthenticationConstraintHandler.handleRequest(AuthenticationConstraintHandler.java:51)
> at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)
> at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)
> at io.undertow.servlet.handlers.security.ServletSecurityConstraintHandler.handleRequest(ServletSecurityConstraintHandler.java:56)
> at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:58)
> at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:72)
> at io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)
> at io.undertow.security.handlers.SecurityInitialHandler.handleRequest(SecurityInitialHandler.java:76)
> at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
> at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)
> at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
> at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
> at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:282)
> at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:261)
> at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:80)
> at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:172)
> at io.undertow.server.Connectors.executeRootHandler(Connectors.java:199)
> at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:774)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Locked ownable synchronizers:
> - <0x00000006f20bb6c8> (a java.util.concurrent.ThreadPoolExecutor$Worker)
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
More information about the jboss-jira
mailing list