[jboss-jira] [JBoss JIRA] (WFLY-5727) After day or two Wildfly hangs and consumes 100% CPU
Stuart Douglas (JIRA)
issues at jboss.org
Fri Dec 4 01:57:00 EST 2015
[ https://issues.jboss.org/browse/WFLY-5727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13136949#comment-13136949 ]
Stuart Douglas commented on WFLY-5727:
--------------------------------------
I think this is a known bug that was fixed in Wildfly 10.
If you want to fix it for WF9 you could try building from this branch: https://github.com/stuartwdouglas/wildfly/tree/9-SSL-fixes
You should also update the version of Undertow to the most recent 1.2.x version (1.2.12.Final).
> After day or two Wildfly hangs and consumes 100% CPU
> ----------------------------------------------------
>
> Key: WFLY-5727
> URL: https://issues.jboss.org/browse/WFLY-5727
> Project: WildFly
> Issue Type: Bug
> Components: IO, Web (Undertow)
> Affects Versions: 8.2.0.Final, 9.0.1.Final
> Environment: Wildfly 9.01 Final, Wildfly 8.2
> CentOS Linux release 7.1.1503 (Core)
> java version "1.8.0_66"
> Java(TM) SE Runtime Environment (build 1.8.0_66-b17)
> Java HotSpot(TM) 64-Bit Server VM (build 25.66-b17, mixed mode)
> Reporter: Petri Tuomaala
> Assignee: Stuart Douglas
> Priority: Blocker
> Labels: Linux, io, ssl, undertow, wildfly, xnio
> Fix For: 9.x.x TBD
>
> Attachments: jstack.22263.102919.280420771, jstack.22263.102925.032513958, jstack.22263.102930.660792783, jstack.22263.102936.697001192, jstack.22263.102942.535009068, jstack.22263.102948.169314576, jstack.22263.102954.185475598, jstack.22263.102959.848031929, jstack.22263.103005.599986837, jstack.22263.103011.261542540, monitor.png, pid.png
>
>
> First disclaimer, this looks pointing to undertow/xnio but I am not sure for the root cause so I'm filing this for Wildfly.
> After a day or two Wildfly just hangs and consumes the cpu. We have noticed that the site traffic itself has no impact on this: it has happened during peak traffic and also almost in idle situations.
> We are handling SSL in Wildfly and example this https://issues.jboss.org/browse/UNDERTOW-282 looks similar to this but it has been resolved over year ago so it should be in Wildfly 9.01.
> Our SSL conf in standalone.xml:
> <https-listener name="https" socket-binding="https" security-realm="UndertowRealm" enabled-cipher-suites="TLS_DHE_DSS_WITH_AES_128_CBC_SHA,SSL_RSA_WITH_3DES_EDE_CBC_SHA,SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA" max-parameters="5000"/>
> I have narrowed down the thread that hangs. Or at least tried, my first time to study thread dumps. PID for thread that is jammed is 22599 => 0x5847 as hex and it points to the thread end of this description. I attach the actual thread dumps to this ticket. Any workaround for this is preciated, we currently need to reboot the servers when the Wildfly hangs (almost every nigth).
> "default task-20" #285 prio=5 os_prio=0 tid=0x00007f862c328800 nid=0x5847 runnable [0x00007f85f5ad6000]
> java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
> at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
> at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
> at sun.nio.ch.IOUtil.read(IOUtil.java:192)
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
> - locked <0x00000006f6065210> (a java.lang.Object)
> at org.xnio.nio.NioSocketConduit.read(NioSocketConduit.java:282)
> at org.xnio.ssl.JsseSslConduitEngine.handleUnwrapResult(JsseSslConduitEngine.java:737)
> - locked <0x00000006f606b708> (a org.xnio.ByteBufferSlicePool$PooledByteBuffer)
> at org.xnio.ssl.JsseSslConduitEngine.unwrap(JsseSslConduitEngine.java:620)
> - locked <0x00000006f606b708> (a org.xnio.ByteBufferSlicePool$PooledByteBuffer)
> at org.xnio.ssl.JsseSslConduitEngine.unwrap(JsseSslConduitEngine.java:574)
> at org.xnio.ssl.JsseSslStreamSourceConduit.read(JsseSslStreamSourceConduit.java:89)
> at org.xnio.conduits.AbstractStreamSourceConduit.read(AbstractStreamSourceConduit.java:51)
> at io.undertow.conduits.ReadDataStreamSourceConduit.read(ReadDataStreamSourceConduit.java:67)
> at io.undertow.conduits.FixedLengthStreamSourceConduit.read(FixedLengthStreamSourceConduit.java:249)
> at org.xnio.conduits.ConduitStreamSourceChannel.read(ConduitStreamSourceChannel.java:127)
> at io.undertow.channels.DetachableStreamSourceChannel.read(DetachableStreamSourceChannel.java:209)
> at io.undertow.server.HttpServerExchange$ReadDispatchChannel.read(HttpServerExchange.java:2136)
> at io.undertow.server.handlers.form.FormEncodedDataDefinition$FormEncodedDataParser.doParse(FormEncodedDataDefinition.java:133)
> at io.undertow.server.handlers.form.FormEncodedDataDefinition$FormEncodedDataParser.parseBlocking(FormEncodedDataDefinition.java:251)
> at io.undertow.servlet.spec.HttpServletRequestImpl.parseFormData(HttpServletRequestImpl.java:753)
> at io.undertow.servlet.spec.HttpServletRequestImpl.getParameter(HttpServletRequestImpl.java:627)
> at org.apache.myfaces.context.servlet.RequestParameterMap.getAttribute(RequestParameterMap.java:45)
> at org.apache.myfaces.context.servlet.RequestParameterMap.getAttribute(RequestParameterMap.java:33)
> at org.apache.myfaces.util.AbstractAttributeMap.containsKey(AbstractAttributeMap.java:62)
> at org.apache.myfaces.renderkit.html.HtmlResponseStateManager.isPostback(HtmlResponseStateManager.java:303)
> at org.apache.myfaces.lifecycle.DefaultRestoreViewSupport.isPostback(DefaultRestoreViewSupport.java:242)
> at org.apache.myfaces.lifecycle.RestoreViewExecutor.execute(RestoreViewExecutor.java:120)
> at org.apache.myfaces.lifecycle.LifecycleImpl.executePhase(LifecycleImpl.java:170)
> at org.apache.myfaces.lifecycle.LifecycleImpl.execute(LifecycleImpl.java:117)
> at javax.faces.webapp.FacesServlet.service(FacesServlet.java:197)
> at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:86)
> at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:130)
> at fi.elbit.skills.loginview.DefaultsFilter.doFilter(DefaultsFilter.java:117)
> at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:60)
> at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:132)
> at org.apache.myfaces.webapp.filter.ExtensionsFilter.doFilter(ExtensionsFilter.java:357)
> at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:60)
> at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:132)
> at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:85)
> at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)
> at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)
> at org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)
> at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
> at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:131)
> at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)
> at io.undertow.server.handlers.DisableCacheHandler.handleRequest(DisableCacheHandler.java:33)
> at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
> at io.undertow.security.handlers.AuthenticationConstraintHandler.handleRequest(AuthenticationConstraintHandler.java:51)
> at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)
> at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)
> at io.undertow.servlet.handlers.security.ServletSecurityConstraintHandler.handleRequest(ServletSecurityConstraintHandler.java:56)
> at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:58)
> at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:72)
> at io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)
> at io.undertow.security.handlers.SecurityInitialHandler.handleRequest(SecurityInitialHandler.java:76)
> at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
> at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)
> at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
> at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
> at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:282)
> at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:261)
> at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:80)
> at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:172)
> at io.undertow.server.Connectors.executeRootHandler(Connectors.java:199)
> at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:774)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Locked ownable synchronizers:
> - <0x00000006f20bb6c8> (a java.util.concurrent.ThreadPoolExecutor$Worker)
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
More information about the jboss-jira
mailing list