[jboss-jira] [JBoss JIRA] (WFLY-5727) After day or two Wildfly hangs and consumes 100% CPU

After day or two Wildfly hangs and consumes 100% CPU ---------------------------------------------------- Key: WFLY-5727 URL: https://issues.jboss.org/browse/WFLY-5727 Project: WildFly Issue Type: Bug Components: IO, Web (Undertow) Affects Versions: 8.2.0.Final, 9.0.1.Final Environment: Wildfly 9.01 Final, Wildfly 8.2 CentOS Linux release 7.1.1503 (Core) java version "1.8.0_66" Java(TM) SE Runtime Environment (build 1.8.0_66-b17) Java HotSpot(TM) 64-Bit Server VM (build 25.66-b17, mixed mode) Reporter: Petri Tuomaala Assignee: Stuart Douglas Priority: Blocker Labels: Linux, io, ssl, undertow, wildfly, xnio Fix For: 9.x.x TBD Attachments: jstack.22263.102919.280420771, jstack.22263.102925.032513958, jstack.22263.102930.660792783, jstack.22263.102936.697001192, jstack.22263.102942.535009068, jstack.22263.102948.169314576, jstack.22263.102954.185475598, jstack.22263.102959.848031929, jstack.22263.103005.599986837, jstack.22263.103011.261542540, monitor.png, pid.png First disclaimer, this looks pointing to undertow/xnio but I am not sure for the root cause so I'm filing this for Wildfly. After a day or two Wildfly just hangs and consumes the cpu. We have noticed that the site traffic itself has no impact on this: it has happened during peak traffic and also almost in idle situations. We are handling SSL in Wildfly and example this https://issues.jboss.org/browse/UNDERTOW-282 looks similar to this but it has been resolved over year ago so it should be in Wildfly 9.01. Our SSL conf in standalone.xml: <https-listener name="https" socket-binding="https" security-realm="UndertowRealm" enabled-cipher-suites="TLS_DHE_DSS_WITH_AES_128_CBC_SHA,SSL_RSA_WITH_3DES_EDE_CBC_SHA,SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA" max-parameters="5000"/> I have narrowed down the thread that hangs. Or at least tried, my first time to study thread dumps. PID for thread that is jammed is 22599 => 0x5847 as hex and it points to the thread end of this description. I attach the actual thread dumps to this ticket. Any workaround for this is preciated, we currently need to reboot the servers when the Wildfly hangs (almost every nigth). "default task-20" #285 prio=5 os_prio=0 tid=0x00007f862c328800 nid=0x5847 runnable [0x00007f85f5ad6000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.FileDispatcherImpl.read0(Native Method) at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) at sun.nio.ch.IOUtil.read(IOUtil.java:192) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) - locked <0x00000006f6065210> (a java.lang.Object) at org.xnio.nio.NioSocketConduit.read(NioSocketConduit.java:282) at org.xnio.ssl.JsseSslConduitEngine.handleUnwrapResult(JsseSslConduitEngine.java:737) - locked <0x00000006f606b708> (a org.xnio.ByteBufferSlicePool$PooledByteBuffer) at org.xnio.ssl.JsseSslConduitEngine.unwrap(JsseSslConduitEngine.java:620) - locked <0x00000006f606b708> (a org.xnio.ByteBufferSlicePool$PooledByteBuffer) at org.xnio.ssl.JsseSslConduitEngine.unwrap(JsseSslConduitEngine.java:574) at org.xnio.ssl.JsseSslStreamSourceConduit.read(JsseSslStreamSourceConduit.java:89) at org.xnio.conduits.AbstractStreamSourceConduit.read(AbstractStreamSourceConduit.java:51) at io.undertow.conduits.ReadDataStreamSourceConduit.read(ReadDataStreamSourceConduit.java:67) at io.undertow.conduits.FixedLengthStreamSourceConduit.read(FixedLengthStreamSourceConduit.java:249) at org.xnio.conduits.ConduitStreamSourceChannel.read(ConduitStreamSourceChannel.java:127) at io.undertow.channels.DetachableStreamSourceChannel.read(DetachableStreamSourceChannel.java:209) at io.undertow.server.HttpServerExchange$ReadDispatchChannel.read(HttpServerExchange.java:2136) at io.undertow.server.handlers.form.FormEncodedDataDefinition$FormEncodedDataParser.doParse(FormEncodedDataDefinition.java:133) at io.undertow.server.handlers.form.FormEncodedDataDefinition$FormEncodedDataParser.parseBlocking(FormEncodedDataDefinition.java:251) at io.undertow.servlet.spec.HttpServletRequestImpl.parseFormData(HttpServletRequestImpl.java:753) at io.undertow.servlet.spec.HttpServletRequestImpl.getParameter(HttpServletRequestImpl.java:627) at org.apache.myfaces.context.servlet.RequestParameterMap.getAttribute(RequestParameterMap.java:45) at org.apache.myfaces.context.servlet.RequestParameterMap.getAttribute(RequestParameterMap.java:33) at org.apache.myfaces.util.AbstractAttributeMap.containsKey(AbstractAttributeMap.java:62) at org.apache.myfaces.renderkit.html.HtmlResponseStateManager.isPostback(HtmlResponseStateManager.java:303) at org.apache.myfaces.lifecycle.DefaultRestoreViewSupport.isPostback(DefaultRestoreViewSupport.java:242) at org.apache.myfaces.lifecycle.RestoreViewExecutor.execute(RestoreViewExecutor.java:120) at org.apache.myfaces.lifecycle.LifecycleImpl.executePhase(LifecycleImpl.java:170) at org.apache.myfaces.lifecycle.LifecycleImpl.execute(LifecycleImpl.java:117) at javax.faces.webapp.FacesServlet.service(FacesServlet.java:197) at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:86) at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:130) at fi.elbit.skills.loginview.DefaultsFilter.doFilter(DefaultsFilter.java:117) at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:60) at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:132) at org.apache.myfaces.webapp.filter.ExtensionsFilter.doFilter(ExtensionsFilter.java:357) at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:60) at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:132) at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:85) at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62) at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36) at org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78) at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:131) at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57) at io.undertow.server.handlers.DisableCacheHandler.handleRequest(DisableCacheHandler.java:33) at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at io.undertow.security.handlers.AuthenticationConstraintHandler.handleRequest(AuthenticationConstraintHandler.java:51) at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46) at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64) at io.undertow.servlet.handlers.security.ServletSecurityConstraintHandler.handleRequest(ServletSecurityConstraintHandler.java:56) at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:58) at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:72) at io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50) at io.undertow.security.handlers.SecurityInitialHandler.handleRequest(SecurityInitialHandler.java:76) at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61) at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:282) at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:261) at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:80) at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:172) at io.undertow.server.Connectors.executeRootHandler(Connectors.java:199) at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:774) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Locked ownable synchronizers: - <0x00000006f20bb6c8> (a java.util.concurrent.ThreadPoolExecutor$Worker)