[jboss-jira] [JBoss JIRA] (WFLY-5727) After day or two Wildfly hangs and consumes 100% CPU
Petri Tuomaala (JIRA)
issues at jboss.org
Wed Nov 25 05:29:00 EST 2015
Petri Tuomaala created WFLY-5727:
------------------------------------
Summary: After day or two Wildfly hangs and consumes 100% CPU
Key: WFLY-5727
URL: https://issues.jboss.org/browse/WFLY-5727
Project: WildFly
Issue Type: Bug
Components: IO, Web (Undertow)
Affects Versions: 9.0.1.Final, 8.2.0.Final
Environment: Wildfly 9.01 Final, Wildfly 8.2
CentOS Linux release 7.1.1503 (Core)
java version "1.8.0_66"
Java(TM) SE Runtime Environment (build 1.8.0_66-b17)
Java HotSpot(TM) 64-Bit Server VM (build 25.66-b17, mixed mode)
Reporter: Petri Tuomaala
Assignee: Stuart Douglas
Priority: Blocker
Fix For: 9.x.x TBD
Attachments: jstack.22263.102919.280420771, jstack.22263.102925.032513958, jstack.22263.102930.660792783, jstack.22263.102936.697001192, jstack.22263.102942.535009068, jstack.22263.102948.169314576, jstack.22263.102954.185475598, jstack.22263.102959.848031929, jstack.22263.103005.599986837, jstack.22263.103011.261542540, monitor.png, pid.png
First disclaimer, this looks pointing to undertow/xnio but I am not sure for the root cause so I'm filing this for Wildfly.
After a day or two Wildfly just hangs and consumes the cpu. We have noticed that the site traffic itself has no impact on this: it has happened during peak traffic and also almost in idle situations.
We are handling SSL in Wildfly and example this https://issues.jboss.org/browse/UNDERTOW-282 looks similar to this but it has been resolved over year ago so it should be in Wildfly 9.01.
Our SSL conf in standalone.xml:
<https-listener name="https" socket-binding="https" security-realm="UndertowRealm" enabled-cipher-suites="TLS_DHE_DSS_WITH_AES_128_CBC_SHA,SSL_RSA_WITH_3DES_EDE_CBC_SHA,SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA" max-parameters="5000"/>
I have narrowed down the thread that hangs. Or at least tried, my first time to study thread dumps. PID for thread that is jammed is 22599 => 0x5847 as hex and it points to the thread end of this description. I attach the actual thread dumps to this ticket. Any workaround for this is preciated, we currently need to reboot the servers when the Wildfly hangs (almost every nigth).
"default task-20" #285 prio=5 os_prio=0 tid=0x00007f862c328800 nid=0x5847 runnable [0x00007f85f5ad6000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:192)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
- locked <0x00000006f6065210> (a java.lang.Object)
at org.xnio.nio.NioSocketConduit.read(NioSocketConduit.java:282)
at org.xnio.ssl.JsseSslConduitEngine.handleUnwrapResult(JsseSslConduitEngine.java:737)
- locked <0x00000006f606b708> (a org.xnio.ByteBufferSlicePool$PooledByteBuffer)
at org.xnio.ssl.JsseSslConduitEngine.unwrap(JsseSslConduitEngine.java:620)
- locked <0x00000006f606b708> (a org.xnio.ByteBufferSlicePool$PooledByteBuffer)
at org.xnio.ssl.JsseSslConduitEngine.unwrap(JsseSslConduitEngine.java:574)
at org.xnio.ssl.JsseSslStreamSourceConduit.read(JsseSslStreamSourceConduit.java:89)
at org.xnio.conduits.AbstractStreamSourceConduit.read(AbstractStreamSourceConduit.java:51)
at io.undertow.conduits.ReadDataStreamSourceConduit.read(ReadDataStreamSourceConduit.java:67)
at io.undertow.conduits.FixedLengthStreamSourceConduit.read(FixedLengthStreamSourceConduit.java:249)
at org.xnio.conduits.ConduitStreamSourceChannel.read(ConduitStreamSourceChannel.java:127)
at io.undertow.channels.DetachableStreamSourceChannel.read(DetachableStreamSourceChannel.java:209)
at io.undertow.server.HttpServerExchange$ReadDispatchChannel.read(HttpServerExchange.java:2136)
at io.undertow.server.handlers.form.FormEncodedDataDefinition$FormEncodedDataParser.doParse(FormEncodedDataDefinition.java:133)
at io.undertow.server.handlers.form.FormEncodedDataDefinition$FormEncodedDataParser.parseBlocking(FormEncodedDataDefinition.java:251)
at io.undertow.servlet.spec.HttpServletRequestImpl.parseFormData(HttpServletRequestImpl.java:753)
at io.undertow.servlet.spec.HttpServletRequestImpl.getParameter(HttpServletRequestImpl.java:627)
at org.apache.myfaces.context.servlet.RequestParameterMap.getAttribute(RequestParameterMap.java:45)
at org.apache.myfaces.context.servlet.RequestParameterMap.getAttribute(RequestParameterMap.java:33)
at org.apache.myfaces.util.AbstractAttributeMap.containsKey(AbstractAttributeMap.java:62)
at org.apache.myfaces.renderkit.html.HtmlResponseStateManager.isPostback(HtmlResponseStateManager.java:303)
at org.apache.myfaces.lifecycle.DefaultRestoreViewSupport.isPostback(DefaultRestoreViewSupport.java:242)
at org.apache.myfaces.lifecycle.RestoreViewExecutor.execute(RestoreViewExecutor.java:120)
at org.apache.myfaces.lifecycle.LifecycleImpl.executePhase(LifecycleImpl.java:170)
at org.apache.myfaces.lifecycle.LifecycleImpl.execute(LifecycleImpl.java:117)
at javax.faces.webapp.FacesServlet.service(FacesServlet.java:197)
at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:86)
at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:130)
at fi.elbit.skills.loginview.DefaultsFilter.doFilter(DefaultsFilter.java:117)
at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:60)
at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:132)
at org.apache.myfaces.webapp.filter.ExtensionsFilter.doFilter(ExtensionsFilter.java:357)
at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:60)
at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:132)
at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:85)
at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)
at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)
at org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:131)
at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)
at io.undertow.server.handlers.DisableCacheHandler.handleRequest(DisableCacheHandler.java:33)
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at io.undertow.security.handlers.AuthenticationConstraintHandler.handleRequest(AuthenticationConstraintHandler.java:51)
at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)
at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)
at io.undertow.servlet.handlers.security.ServletSecurityConstraintHandler.handleRequest(ServletSecurityConstraintHandler.java:56)
at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:58)
at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:72)
at io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)
at io.undertow.security.handlers.SecurityInitialHandler.handleRequest(SecurityInitialHandler.java:76)
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:282)
at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:261)
at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:80)
at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:172)
at io.undertow.server.Connectors.executeRootHandler(Connectors.java:199)
at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:774)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- <0x00000006f20bb6c8> (a java.util.concurrent.ThreadPoolExecutor$Worker)
!pid.png|thumbnail!
!monitor.png|thumbnail!
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
More information about the jboss-jira
mailing list