[JBoss JIRA] (WFLY-10923) Server stop blocked by DistributableSessionManager after invalidation during async request
by Jan Stourac (Jira)
[ https://issues.jboss.org/browse/WFLY-10923?page=com.atlassian.jira.plugin... ]
Jan Stourac updated WFLY-10923:
-------------------------------
Steps to Reproduce:
1. deploy servlet-async.war
2. request /servlet-async/hi.jsp to establish session
3. request /servlet-async/ to start repeated long async requests
4. In new tab, request /servlet-async/invalidate.jsp
5. Close out /servlet-async/ tab, attempt shutdown, and note hang
was:
1. deploy servlet-async.war
2. request /servet-async/hi.jsp to establish session
3. request /servlet-async/ to start repeated long async requests
4. In new tab, request /servlet-async/invalidate.jsp
5. Close out /servlet-async/ tab, attempt shutdown, and note hang
> Server stop blocked by DistributableSessionManager after invalidation during async request
> ------------------------------------------------------------------------------------------
>
> Key: WFLY-10923
> URL: https://issues.jboss.org/browse/WFLY-10923
> Project: WildFly
> Issue Type: Bug
> Components: Clustering
> Affects Versions: 14.0.0.Beta2
> Reporter: Aaron Ogburn
> Assignee: Paul Ferraro
> Priority: Major
> Fix For: 14.0.0.Final
>
> Attachments: servlet-async.war
>
>
> After WFLY-10621, a DistributableSessionManager.stop hang can still occur in the following scenario.
> 1. async request a starts
> 2. sync request b invalidates session
> 3. async request a completes
> Upon completion, the async request does not unlock and throws following exception:
> {code}
> 16:04:58,260 ERROR [stderr] (default task-18) Exception in thread "default task-18" java.lang.RuntimeException: java.lang.IllegalStateException: WFLYCLWEBUT0001: Session 8adSKXK1li8Qo15QnaIGrd31DrB6B83jEL4blzq1 is invalid
> 16:04:58,260 ERROR [stderr] (default task-18) at io.undertow.servlet.spec.ServletContextImpl.invokeRunnable(ServletContextImpl.java:944)
> 16:04:58,260 ERROR [stderr] (default task-18) at io.undertow.servlet.spec.AsyncContextImpl.onAsyncComplete(AsyncContextImpl.java:594)
> 16:04:58,260 ERROR [stderr] (default task-18) at io.undertow.servlet.spec.AsyncContextImpl.access$100(AsyncContextImpl.java:71)
> 16:04:58,261 ERROR [stderr] (default task-18) at io.undertow.servlet.spec.AsyncContextImpl$3.run(AsyncContextImpl.java:315)
> 16:04:58,261 ERROR [stderr] (default task-18) at io.undertow.servlet.spec.AsyncContextImpl$6.run(AsyncContextImpl.java:473)
> 16:04:58,261 ERROR [stderr] (default task-18) at io.undertow.servlet.spec.AsyncContextImpl$TaskDispatchRunnable.run(AsyncContextImpl.java:584)
> 16:04:58,261 ERROR [stderr] (default task-18) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> 16:04:58,261 ERROR [stderr] (default task-18) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 16:04:58,261 ERROR [stderr] (default task-18) at java.lang.Thread.run(Thread.java:748)
> 16:04:58,261 ERROR [stderr] (default task-18) Caused by: java.lang.IllegalStateException: WFLYCLWEBUT0001: Session 8adSKXK1li8Qo15QnaIGrd31DrB6B83jEL4blzq1 is invalid
> 16:04:58,261 ERROR [stderr] (default task-18) at org.wildfly.clustering.web.undertow.session.DistributableSession.validate(DistributableSession.java:57)
> 16:04:58,261 ERROR [stderr] (default task-18) at org.wildfly.clustering.web.undertow.session.DistributableSession.getAttribute(DistributableSession.java:157)
> 16:04:58,261 ERROR [stderr] (default task-18) at io.undertow.servlet.spec.HttpSessionImpl.getAttribute(HttpSessionImpl.java:122)
> 16:04:58,261 ERROR [stderr] (default task-18) at org.jboss.weld.context.http.HttpSessionContextImpl.checkBeanIdentifierIndexConsistency(HttpSessionContextImpl.java:98)
> 16:04:58,261 ERROR [stderr] (default task-18) at org.jboss.weld.context.http.HttpSessionContextImpl.associate(HttpSessionContextImpl.java:47)
> 16:04:58,261 ERROR [stderr] (default task-18) at org.jboss.weld.context.http.HttpSessionContextImpl.associate(HttpSessionContextImpl.java:23)
> 16:04:58,261 ERROR [stderr] (default task-18) at org.jboss.weld.servlet.HttpContextLifecycle.requestInitialized(HttpContextLifecycle.java:237)
> 16:04:58,261 ERROR [stderr] (default task-18) at org.jboss.weld.servlet.WeldInitialListener.requestInitialized(WeldInitialListener.java:152)
> 16:04:58,261 ERROR [stderr] (default task-18) at io.undertow.servlet.core.ApplicationListeners.requestInitialized(ApplicationListeners.java:246)
> 16:04:58,261 ERROR [stderr] (default task-18) at io.undertow.servlet.spec.AsyncContextImpl.setupRequestContext(AsyncContextImpl.java:678)
> 16:04:58,261 ERROR [stderr] (default task-18) at io.undertow.servlet.spec.AsyncContextImpl.access$700(AsyncContextImpl.java:71)
> 16:04:58,261 ERROR [stderr] (default task-18) at io.undertow.servlet.spec.AsyncContextImpl$7.run(AsyncContextImpl.java:599)
> 16:04:58,261 ERROR [stderr] (default task-18) at io.undertow.servlet.spec.ServletContextImpl$2.call(ServletContextImpl.java:177)
> 16:04:58,262 ERROR [stderr] (default task-18) at io.undertow.servlet.spec.ServletContextImpl$2.call(ServletContextImpl.java:174)
> 16:04:58,262 ERROR [stderr] (default task-18) at io.undertow.servlet.spec.ServletContextImpl.invokeRunnable(ServletContextImpl.java:942)
> 16:04:58,262 ERROR [stderr] (default task-18) ... 8 more
> {code}
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
7 years, 8 months
[JBoss JIRA] (JGRP-2298) InputStream is processed after the different version error
by Bela Ban (Jira)
[ https://issues.jboss.org/browse/JGRP-2298?page=com.atlassian.jira.plugin.... ]
Bela Ban commented on JGRP-2298:
--------------------------------
Hmm... at connection setup, the peer sends both a cookie _and_ the version number, and if either of them doesn't match, the connection is closed.
This means, you actually got *past* this point and a peer connection turned sour during normal communication? Is this a real scenario? The only 2 things I could think of that could cause this are (1) peer crash after sending half of a message and (2) malicious code.
(1) would be handled as the connection would be terminated. Do you suspect (2)?
> InputStream is processed after the different version error
> ----------------------------------------------------------
>
> Key: JGRP-2298
> URL: https://issues.jboss.org/browse/JGRP-2298
> Project: JGroups
> Issue Type: Bug
> Affects Versions: 4.0.11, 4.0.13
> Environment: OS: Debian 8
> Reporter: Boris Kantor
> Assignee: Bela Ban
> Priority: Major
> Fix For: 4.0.16
>
>
> Occasionally JGroups shows up a warning when receiving a message with a different JGroups version:
> {{JGRP000010: packet from 10.10.10.130:7800 has different version (0.0.0) than ours (4.0.11); packet is discarded | (Log4J2LogImpl.java:91)}}
> Although the message is then discarded, the socket connection will be kept open. The input stream still contains data that will be processed and falsly interpreted as the next message.
> If you have bad luck, the next bytes will match the correct version number. This leads then to following errors:
> {code:java}
> 2018-07-17 10:15:34.052 [Connect | ] WARN TCP - JGRP000010: packet from 10.10.10.130:7800 has different version (0.0.0) than ours (4.0.11); packet is discarded | (Log4J2LogImpl.java:91)
> 2018-07-17 10:15:34.059 [Connect | ] ERROR TCP - JGRP000030: jgm113: failed handling incoming message | (Log4J2LogImpl.java:116)
> java.lang.ClassNotFoundException: Class for magic number 8237 cannot be found
> at org.jgroups.conf.ClassConfigurator.create(ClassConfigurator.java:118)
> at org.jgroups.Message.readHeader(Message.java:869)
> at org.jgroups.Message.readFrom(Message.java:742)
> at org.jgroups.protocols.TP.handleSingleMessage(TP.java:1346)
> at org.jgroups.protocols.TP.receive(TP.java:1323)
> at org.jgroups.blocks.cs.BaseServer.receive(BaseServer.java:171)
> at org.jgroups.blocks.cs.TcpConnection$Receiver.run(TcpConnection.java:290)
> at eu.application.nac.cluster.jgroups.JGroupsTask.execute(JGroupsTask.java:41)
> at eu.application.nac.engine.scheduling.Task.run(Task.java:339)
> at java.lang.Thread.run(Thread.java:745)
> 2018-07-17 10:15:34.065 [Connect | ] ERROR TCP - JGRP000030: jgm113: failed handling incoming message | (Log4J2LogImpl.java:116)
> java.lang.ClassNotFoundException: Class for magic number 8237 cannot be found
> at org.jgroups.conf.ClassConfigurator.create(ClassConfigurator.java:118)
> at org.jgroups.Message.readHeader(Message.java:869)
> at org.jgroups.Message.readFrom(Message.java:742)
> at org.jgroups.protocols.TP.handleSingleMessage(TP.java:1346)
> at org.jgroups.protocols.TP.receive(TP.java:1323)
> at org.jgroups.blocks.cs.BaseServer.receive(BaseServer.java:171)
> at org.jgroups.blocks.cs.TcpConnection$Receiver.run(TcpConnection.java:290)
> at eu.application.nac.cluster.jgroups.JGroupsTask.execute(JGroupsTask.java:41)
> at eu.application.nac.engine.scheduling.Task.run(Task.java:339)
> at java.lang.Thread.run(Thread.java:745)
> 2018-07-17 10:15:34.072 [Connect | ] ERROR TCP - JGRP000030: jgm113: failed handling incoming message | (Log4J2LogImpl.java:116)
> java.io.IOException: length has to be 4 or 16 bytes (was 47 bytes)
> at org.jgroups.stack.IpAddress.readFrom(IpAddress.java:171)
> at org.jgroups.util.Util.readAddress(Util.java:1379)
> at org.jgroups.Message.readFrom(Message.java:731)
> at org.jgroups.protocols.TP.handleSingleMessage(TP.java:1346)
> at org.jgroups.protocols.TP.receive(TP.java:1323)
> at org.jgroups.blocks.cs.BaseServer.receive(BaseServer.java:171)
> at org.jgroups.blocks.cs.TcpConnection$Receiver.run(TcpConnection.java:290)
> at eu.application.nac.cluster.jgroups.JGroupsTask.execute(JGroupsTask.java:41)
> at eu.application.nac.engine.scheduling.Task.run(Task.java:339)
> at java.lang.Thread.run(Thread.java:745)
> 2018-07-17 10:15:34.076 [Connect | ] ERROR TCP - JGRP000030: jgm113: failed handling incoming message | (Log4J2LogImpl.java:116)
> java.lang.ClassNotFoundException: Class for magic number 28789 cannot be found
> at org.jgroups.conf.ClassConfigurator.create(ClassConfigurator.java:118)
> at org.jgroups.Message.readHeader(Message.java:869)
> at org.jgroups.Message.readFrom(Message.java:742)
> at org.jgroups.protocols.TP.handleSingleMessage(TP.java:1346)
> at org.jgroups.protocols.TP.receive(TP.java:1323)
> at org.jgroups.blocks.cs.BaseServer.receive(BaseServer.java:171)
> at org.jgroups.blocks.cs.TcpConnection$Receiver.run(TcpConnection.java:290)
> at eu.application.nac.cluster.jgroups.JGroupsTask.execute(JGroupsTask.java:41)
> at eu.application.nac.engine.scheduling.Task.run(Task.java:339)
> at java.lang.Thread.run(Thread.java:745)
> 2018-07-17 10:15:34.077 [Connect | ] ERROR TCP - JGRP000030: jgm113: failed handling incoming message | (Log4J2LogImpl.java:116)
> java.lang.ClassNotFoundException: Class for magic number 8237 cannot be found
> at org.jgroups.conf.ClassConfigurator.create(ClassConfigurator.java:118)
> at org.jgroups.Message.readHeader(Message.java:869)
> at org.jgroups.Message.readFrom(Message.java:742)
> at org.jgroups.protocols.TP.handleSingleMessage(TP.java:1346)
> at org.jgroups.protocols.TP.receive(TP.java:1323)
> at org.jgroups.blocks.cs.BaseServer.receive(BaseServer.java:171)
> at org.jgroups.blocks.cs.TcpConnection$Receiver.run(TcpConnection.java:290)
> at eu.application.nac.cluster.jgroups.JGroupsTask.execute(JGroupsTask.java:41)
> at eu.application.nac.engine.scheduling.Task.run(Task.java:339)
> at java.lang.Thread.run(Thread.java:745)
> 2018-07-17 10:15:34.078 [Connect | ] ERROR TCP - JGRP000030: jgm113: failed handling incoming message | (Log4J2LogImpl.java:116)
> java.io.IOException: length has to be 4 or 16 bytes (was 47 bytes)
> at org.jgroups.stack.IpAddress.readFrom(IpAddress.java:171)
> at org.jgroups.util.Util.readAddress(Util.java:1379)
> at org.jgroups.Message.readFrom(Message.java:731)
> at org.jgroups.protocols.TP.handleSingleMessage(TP.java:1346)
> at org.jgroups.protocols.TP.receive(TP.java:1323)
> at org.jgroups.blocks.cs.BaseServer.receive(BaseServer.java:171)
> at org.jgroups.blocks.cs.TcpConnection$Receiver.run(TcpConnection.java:290)
> at eu.application.nac.cluster.jgroups.JGroupsTask.execute(JGroupsTask.java:41)
> at eu.application.nac.engine.scheduling.Task.run(Task.java:339)
> at java.lang.Thread.run(Thread.java:745)
> 2018-07-17 10:15:34.078 [Connect | ] ERROR TCP - JGRP000030: jgm113: failed handling incoming message | (Log4J2LogImpl.java:116)
> java.lang.ClassNotFoundException: Class for magic number 8237 cannot be found
> at org.jgroups.conf.ClassConfigurator.create(ClassConfigurator.java:118)
> at org.jgroups.Message.readHeader(Message.java:869)
> at org.jgroups.Message.readFrom(Message.java:742)
> at org.jgroups.protocols.TP.handleSingleMessage(TP.java:1346)
> at org.jgroups.protocols.TP.receive(TP.java:1323)
> at org.jgroups.blocks.cs.BaseServer.receive(BaseServer.java:171)
> at org.jgroups.blocks.cs.TcpConnection$Receiver.run(TcpConnection.java:290)
> at eu.application.nac.cluster.jgroups.JGroupsTask.execute(JGroupsTask.java:41)
> at eu.application.nac.engine.scheduling.Task.run(Task.java:339)
> at java.lang.Thread.run(Thread.java:745)
> 2018-07-17 10:15:34.079 [Connect | ] ERROR TCP - JGRP000030: jgm113: failed handling incoming message | (Log4J2LogImpl.java:116)
> java.io.IOException: length has to be 4 or 16 bytes (was 47 bytes)
> at org.jgroups.stack.IpAddress.readFrom(IpAddress.java:171)
> at org.jgroups.util.Util.readAddress(Util.java:1379)
> at org.jgroups.Message.readFrom(Message.java:731)
> at org.jgroups.protocols.TP.handleSingleMessage(TP.java:1346)
> at org.jgroups.protocols.TP.receive(TP.java:1323)
> at org.jgroups.blocks.cs.BaseServer.receive(BaseServer.java:171)
> at org.jgroups.blocks.cs.TcpConnection$Receiver.run(TcpConnection.java:290)
> at eu.application.nac.cluster.jgroups.JGroupsTask.execute(JGroupsTask.java:41)
> at eu.application.nac.engine.scheduling.Task.run(Task.java:339)
> at java.lang.Thread.run(Thread.java:745)
> 2018-07-17 10:15:34.080 [Connect | ] ERROR TCP - JGRP000030: jgm113: failed handling incoming message | (Log4J2LogImpl.java:116)
> java.lang.ClassNotFoundException: Class for magic number 12320 cannot be found
> at org.jgroups.conf.ClassConfigurator.create(ClassConfigurator.java:118)
> at org.jgroups.Message.readHeader(Message.java:869)
> at org.jgroups.Message.readFrom(Message.java:742)
> at org.jgroups.protocols.TP.handleSingleMessage(TP.java:1346)
> at org.jgroups.protocols.TP.receive(TP.java:1323)
> at org.jgroups.blocks.cs.BaseServer.receive(BaseServer.java:171)
> at org.jgroups.blocks.cs.TcpConnection$Receiver.run(TcpConnection.java:290)
> at eu.application.nac.cluster.jgroups.JGroupsTask.execute(JGroupsTask.java:41)
> at eu.application.nac.engine.scheduling.Task.run(Task.java:339)
> at java.lang.Thread.run(Thread.java:745)
> 2018-07-17 10:15:34.089 [Connect | ] ERROR TCP - JGRP000030: jgm113: failed handling incoming message | (Log4J2LogImpl.java:116)
> java.lang.ClassNotFoundException: Class for magic number 25193 cannot be found
> at org.jgroups.conf.ClassConfigurator.create(ClassConfigurator.java:118)
> at org.jgroups.Message.readHeader(Message.java:869)
> at org.jgroups.Message.readFrom(Message.java:742)
> at org.jgroups.protocols.TP.handleSingleMessage(TP.java:1346)
> at org.jgroups.protocols.TP.receive(TP.java:1323)
> at org.jgroups.blocks.cs.BaseServer.receive(BaseServer.java:171)
> at org.jgroups.blocks.cs.TcpConnection$Receiver.run(TcpConnection.java:290)
> at eu.application.nac.cluster.jgroups.JGroupsTask.execute(JGroupsTask.java:41)
> at eu.application.nac.engine.scheduling.Task.run(Task.java:339)
> at java.lang.Thread.run(Thread.java:745)
> 2018-07-17 10:15:34.107 [Connect | ] ERROR TCP - JGRP000030: jgm113: failed handling incoming message | (Log4J2LogImpl.java:116)
> java.lang.ClassNotFoundException: Class for magic number 25970 cannot be found
> at org.jgroups.conf.ClassConfigurator.create(ClassConfigurator.java:118)
> at org.jgroups.Message.readHeader(Message.java:869)
> at org.jgroups.Message.readFrom(Message.java:742)
> at org.jgroups.util.Util.readMessageBatch(Util.java:1193)
> at org.jgroups.protocols.TP.handleMessageBatch(TP.java:1329)
> at org.jgroups.protocols.TP.receive(TP.java:1321)
> at org.jgroups.blocks.cs.BaseServer.receive(BaseServer.java:171)
> at org.jgroups.blocks.cs.TcpConnection$Receiver.run(TcpConnection.java:290)
> at eu.application.nac.cluster.jgroups.JGroupsTask.execute(JGroupsTask.java:41)
> at eu.application.nac.engine.scheduling.Task.run(Task.java:339)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> When it comes to the payload of the message the size of the payload can be as big as 2GB ({{len=in.readInt();}}). On small machines this will lead to an OutOfMemoryError.
> There is a need for some handling of such misinterpreted messages. Like closing and reopening a socket connection or completely ignoring the following bytes until a new message begins or an other algorithm that would prevent the processing of the remaining bytes.
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
7 years, 8 months
[JBoss JIRA] (WFWIP-140) Server shutdown or microprofile-metrics-smallrye subsystem removal causes INFO log entry for each metric
by Jeff Mesnil (Jira)
[ https://issues.jboss.org/browse/WFWIP-140?page=com.atlassian.jira.plugin.... ]
Jeff Mesnil commented on WFWIP-140:
-----------------------------------
fix applied to smallrye-metrics in https://github.com/smallrye/smallrye-metrics/pull/19
> Server shutdown or microprofile-metrics-smallrye subsystem removal causes INFO log entry for each metric
> --------------------------------------------------------------------------------------------------------
>
> Key: WFWIP-140
> URL: https://issues.jboss.org/browse/WFWIP-140
> Project: WildFly WIP
> Issue Type: Bug
> Components: MP Metrics
> Reporter: Rostislav Svoboda
> Assignee: Jeff Mesnil
> Priority: Blocker
>
> Server shutdown causes INFO log entry for each metric.
> microprofile-metrics-smallrye subsystem removal causes INFO log entry for each metric.
> I think this should be decreased to DEBUG level.
> {code}
> 12:09:38,411 INFO [io.smallrye.metrics.MetricsRegistryImpl] (MSC service thread 1-8) Remove metric [name: classloader.currentLoadedClass.count]
> 12:09:38,412 INFO [io.smallrye.metrics.MetricsRegistryImpl] (MSC service thread 1-8) Remove metric [name: classloader.totalLoadedClass.count]
> 12:09:38,412 INFO [io.smallrye.metrics.MetricsRegistryImpl] (MSC service thread 1-8) Remove metric [name: classloader.totalUnloadedClass.count]
> 12:09:38,412 INFO [io.smallrye.metrics.MetricsRegistryImpl] (MSC service thread 1-8) Remove metric [name: cpu.availableProcessors]
> 12:09:38,412 INFO [io.smallrye.metrics.MetricsRegistryImpl] (MSC service thread 1-8) Remove metric [name: cpu.systemLoadAverage]
> 12:09:38,412 INFO [io.smallrye.metrics.MetricsRegistryImpl] (MSC service thread 1-8) Remove metric [name: gc.G1 Old Generation.count]
> ...
> {code}
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
7 years, 8 months