[JBoss JIRA] (WFLY-10389) [pretesting] NPE during server shutdown when using scattered cache
by Radoslav Husar (JIRA)
[ https://issues.jboss.org/browse/WFLY-10389?page=com.atlassian.jira.plugin... ]
Radoslav Husar commented on WFLY-10389:
---------------------------------------
With WFLY-9664 merged, moving issues found in pretesting to the correct tracker.
> [pretesting] NPE during server shutdown when using scattered cache
> ------------------------------------------------------------------
>
> Key: WFLY-10389
> URL: https://issues.jboss.org/browse/WFLY-10389
> Project: WildFly
> Issue Type: Bug
> Components: Clustering
> Reporter: Michal Vinkler
> Assignee: Paul Ferraro
>
> We hit NPE when running tests for RFE EAP7-867.
> EAP distribution was built from https://github.com/pferraro/wildfly/tree/scattered .
> Test description: Positive stress test (no failover), 4-node EAP cluster, clients: starting with 400 clients in the beginning, raising the number of clients to 6000 in the end of the test.
> During clean server shutdown in the end of the test, server logged NPE and got stuck:
> {code}
> [JBossINF] [0m[31m07:55:57,643 ERROR [org.infinispan.scattered.impl.ScatteredStateConsumerImpl] (thread-200,ejb,dev214) ISPN000471: Failed processing values received from remote node during rebalance.: java.lang.NullPointerException
> [JBossINF] at org.infinispan.scattered.impl.ScatteredStateConsumerImpl.applyValues(ScatteredStateConsumerImpl.java:505)
> [JBossINF] at org.infinispan.scattered.impl.ScatteredStateConsumerImpl.lambda$getValuesAndApply$8(ScatteredStateConsumerImpl.java:475)
> [JBossINF] at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760)
> [JBossINF] at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736)
> [JBossINF] at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
> [JBossINF] at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
> [JBossINF] at org.infinispan.remoting.transport.AbstractRequest.complete(AbstractRequest.java:66)
> [JBossINF] at org.infinispan.remoting.transport.impl.SingleTargetRequest.receiveResponse(SingleTargetRequest.java:56)
> [JBossINF] at org.infinispan.remoting.transport.impl.SingleTargetRequest.onResponse(SingleTargetRequest.java:35)
> [JBossINF] at org.infinispan.remoting.transport.impl.RequestRepository.addResponse(RequestRepository.java:53)
> [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.processResponse(JGroupsTransport.java:1304)
> [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.processMessage(JGroupsTransport.java:1207)
> [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.access$200(JGroupsTransport.java:123)
> [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport$ChannelCallbacks.receive(JGroupsTransport.java:1342)
> [JBossINF] at org.jgroups.JChannel.up(JChannel.java:819)
> [JBossINF] at org.jgroups.fork.ForkProtocolStack.up(ForkProtocolStack.java:134)
> [JBossINF] at org.jgroups.stack.Protocol.up(Protocol.java:340)
> [JBossINF] at org.jgroups.protocols.FORK.up(FORK.java:134)
> [JBossINF] at org.jgroups.protocols.FRAG3.up(FRAG3.java:166)
> [JBossINF] at org.jgroups.protocols.FlowControl.up(FlowControl.java:343)
> [JBossINF] at org.jgroups.protocols.FlowControl.up(FlowControl.java:343)
> [JBossINF] at org.jgroups.protocols.pbcast.GMS.up(GMS.java:864)
> [JBossINF] at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:240)
> [JBossINF] at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1002)
> [JBossINF] at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:728)
> [JBossINF] at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:383)
> [JBossINF] at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:600)
> [JBossINF] at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:119)
> [JBossINF] at org.jgroups.protocols.FD_ALL.up(FD_ALL.java:199)
> [JBossINF] at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:252)
> [JBossINF] at org.jgroups.protocols.MERGE3.up(MERGE3.java:276)
> [JBossINF] at org.jgroups.protocols.Discovery.up(Discovery.java:267)
> [JBossINF] at org.jgroups.protocols.TP.passMessageUp(TP.java:1248)
> [JBossINF] at org.jgroups.util.SubmitToThreadPool$SingleMessageHandler.run(SubmitToThreadPool.java:87)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> [JBossINF] at org.jboss.as.clustering.jgroups.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:52)
> [JBossINF] at java.lang.Thread.run(Thread.java:748)
> [JBossINF]
> {code}
> Scattered cache was configured with bias-lifespan="0".
> Server configuration:
> http://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/eap-7x-stress-ses...
> Server link:
> http://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/eap-7x-stress-ses...
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
8 years, 2 months
[JBoss JIRA] (WFLY-10389) [pretesting] NPE during server shutdown when using scattered cache
by Radoslav Husar (JIRA)
[ https://issues.jboss.org/browse/WFLY-10389?page=com.atlassian.jira.plugin... ]
Radoslav Husar moved WFWIP-33 to WFLY-10389:
--------------------------------------------
Project: WildFly (was: WildFly WIP)
Key: WFLY-10389 (was: WFWIP-33)
Component/s: Clustering
(was: Clustering)
> [pretesting] NPE during server shutdown when using scattered cache
> ------------------------------------------------------------------
>
> Key: WFLY-10389
> URL: https://issues.jboss.org/browse/WFLY-10389
> Project: WildFly
> Issue Type: Bug
> Components: Clustering
> Reporter: Michal Vinkler
> Assignee: Paul Ferraro
>
> We hit NPE when running tests for RFE EAP7-867.
> EAP distribution was built from https://github.com/pferraro/wildfly/tree/scattered .
> Test description: Positive stress test (no failover), 4-node EAP cluster, clients: starting with 400 clients in the beginning, raising the number of clients to 6000 in the end of the test.
> During clean server shutdown in the end of the test, server logged NPE and got stuck:
> {code}
> [JBossINF] [0m[31m07:55:57,643 ERROR [org.infinispan.scattered.impl.ScatteredStateConsumerImpl] (thread-200,ejb,dev214) ISPN000471: Failed processing values received from remote node during rebalance.: java.lang.NullPointerException
> [JBossINF] at org.infinispan.scattered.impl.ScatteredStateConsumerImpl.applyValues(ScatteredStateConsumerImpl.java:505)
> [JBossINF] at org.infinispan.scattered.impl.ScatteredStateConsumerImpl.lambda$getValuesAndApply$8(ScatteredStateConsumerImpl.java:475)
> [JBossINF] at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760)
> [JBossINF] at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736)
> [JBossINF] at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
> [JBossINF] at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
> [JBossINF] at org.infinispan.remoting.transport.AbstractRequest.complete(AbstractRequest.java:66)
> [JBossINF] at org.infinispan.remoting.transport.impl.SingleTargetRequest.receiveResponse(SingleTargetRequest.java:56)
> [JBossINF] at org.infinispan.remoting.transport.impl.SingleTargetRequest.onResponse(SingleTargetRequest.java:35)
> [JBossINF] at org.infinispan.remoting.transport.impl.RequestRepository.addResponse(RequestRepository.java:53)
> [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.processResponse(JGroupsTransport.java:1304)
> [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.processMessage(JGroupsTransport.java:1207)
> [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.access$200(JGroupsTransport.java:123)
> [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport$ChannelCallbacks.receive(JGroupsTransport.java:1342)
> [JBossINF] at org.jgroups.JChannel.up(JChannel.java:819)
> [JBossINF] at org.jgroups.fork.ForkProtocolStack.up(ForkProtocolStack.java:134)
> [JBossINF] at org.jgroups.stack.Protocol.up(Protocol.java:340)
> [JBossINF] at org.jgroups.protocols.FORK.up(FORK.java:134)
> [JBossINF] at org.jgroups.protocols.FRAG3.up(FRAG3.java:166)
> [JBossINF] at org.jgroups.protocols.FlowControl.up(FlowControl.java:343)
> [JBossINF] at org.jgroups.protocols.FlowControl.up(FlowControl.java:343)
> [JBossINF] at org.jgroups.protocols.pbcast.GMS.up(GMS.java:864)
> [JBossINF] at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:240)
> [JBossINF] at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1002)
> [JBossINF] at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:728)
> [JBossINF] at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:383)
> [JBossINF] at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:600)
> [JBossINF] at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:119)
> [JBossINF] at org.jgroups.protocols.FD_ALL.up(FD_ALL.java:199)
> [JBossINF] at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:252)
> [JBossINF] at org.jgroups.protocols.MERGE3.up(MERGE3.java:276)
> [JBossINF] at org.jgroups.protocols.Discovery.up(Discovery.java:267)
> [JBossINF] at org.jgroups.protocols.TP.passMessageUp(TP.java:1248)
> [JBossINF] at org.jgroups.util.SubmitToThreadPool$SingleMessageHandler.run(SubmitToThreadPool.java:87)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> [JBossINF] at org.jboss.as.clustering.jgroups.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:52)
> [JBossINF] at java.lang.Thread.run(Thread.java:748)
> [JBossINF]
> {code}
> Scattered cache was configured with bias-lifespan="0".
> Server configuration:
> http://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/eap-7x-stress-ses...
> Server link:
> http://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/eap-7x-stress-ses...
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
8 years, 2 months
[JBoss JIRA] (WFWIP-13) Regression in messaging cluster tests with network failures
by Radoslav Husar (JIRA)
[ https://issues.jboss.org/browse/WFWIP-13?page=com.atlassian.jira.plugin.s... ]
Radoslav Husar updated WFWIP-13:
--------------------------------
Summary: Regression in messaging cluster tests with network failures (was: Regression in cluster tests with network failures)
> Regression in messaging cluster tests with network failures
> -----------------------------------------------------------
>
> Key: WFWIP-13
> URL: https://issues.jboss.org/browse/WFWIP-13
> Project: WildFly WIP
> Issue Type: Bug
> Components: Artemis
> Reporter: Erich Duda
> Assignee: Yong Hao Gao
> Priority: Blocker
> Labels: feature-branch-blocker
>
> *Scenario*
> * There are two Artemis brokers configured to form cluster
> * There is a producer sending messages to broker 1 and receiver receiving messages from broker 2
> * Between the brokers there is a proxy which simulates network failure
> * The proxy is several times stopped and restarted to simulate the network failure
> * The test expects that all messages sent to broker 1 will be received by receiver from broker 2 (despite the network failures)
> *Reality:* After the proxy is stopped and restarted, the cluster is not able to form again. Both brokers try to reconnect to their opposites but with no luck.
> *Customer scenario:* Messaging cluster is not able to recover after network failures.
> *Investigation of issue*
> I investigated why brokers are not able to reconnect and I found out that always when they try to reconnect, they give it up because there is no topology record for {{nodeId}} where they try to connect. So the re-connection attempt ends here \[1\].
> I compared the behavior with Artemis 1.x and I found out that Artemis 2.x removes the topology member when connection failure is detected, but Artemis 1.x doesn't. When I commented the line \[2\] it fixed the issue. This line is not present in 1.x.
> \[1\] https://github.com/apache/activemq-artemis/blob/b66d0f7ac40001cce14ca7146...
> \[2\] https://github.com/apache/activemq-artemis/blob/b66d0f7ac40001cce14ca7146...
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
8 years, 2 months
[JBoss JIRA] (WFWIP-36) [pretesting] IllegalStateException: WFLYCLWEBUT0002: Session X already exists in session offloading scenario with hotrod store configured
by Radoslav Husar (JIRA)
[ https://issues.jboss.org/browse/WFWIP-36?page=com.atlassian.jira.plugin.s... ]
Radoslav Husar resolved WFWIP-36.
---------------------------------
Resolution: Done
> [pretesting] IllegalStateException: WFLYCLWEBUT0002: Session X already exists in session offloading scenario with hotrod store configured
> -----------------------------------------------------------------------------------------------------------------------------------------
>
> Key: WFWIP-36
> URL: https://issues.jboss.org/browse/WFWIP-36
> Project: WildFly WIP
> Issue Type: Bug
> Components: Clustering
> Reporter: Michal Vinkler
> Assignee: Radoslav Husar
> Priority: Critical
>
> We hit an issue when running tests for RFE EAP7-942.
> Wildfly was built from https://github.com/rhusar/wildfly/tree/WFLY-9294
> Scenario:
> 2-node Wildfly cluster + 2-node JDG 7.2.0.CR1 cluster.
> Wildfly servers are configured to use hotrod store which is backed up by JDG cluster.
> Each Wildfly node is gracefully shut down and then started up again.
> Wildfly configuration:
> http://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/eap-7x-failover-r...
> JDG configuration:
> http://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/eap-7x-failover-r...
> When perf18 shut down, perf19 logged one time:
> {code}
> ERROR [io.undertow.request] (default task-5) UT005023: Exception handling request to /clusterbench-offload/session: java.lang.IllegalStateException: WFLYCLWEBUT0002: Session M4qPQhyKwIG6vYTdLTxuyO_wqcKUABbRVtLtkEy_ already exists
> [JBossINF] at org.wildfly.clustering.web.undertow.session.DistributableSessionManager.createSession(DistributableSessionManager.java:124)
> [JBossINF] at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:918)
> [JBossINF] at io.undertow.servlet.spec.HttpServletRequestImpl.getSession(HttpServletRequestImpl.java:412)
> [JBossINF] at org.jboss.test.clusterbench.common.session.CommonHttpSessionServlet.doGet(CommonHttpSessionServlet.java:38)
> [JBossINF] at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
> [JBossINF] at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
> [JBossINF] at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:74)
> [JBossINF] at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)
> [JBossINF] at io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:67)
> [JBossINF] at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)
> [JBossINF] at org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)
> [JBossINF] at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
> [JBossINF] at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:132)
> [JBossINF] at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)
> [JBossINF] at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
> [JBossINF] at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)
> [JBossINF] at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)
> [JBossINF] at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)
> [JBossINF] at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)
> [JBossINF] at io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)
> [JBossINF] at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)
> [JBossINF] at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
> [JBossINF] at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)
> [JBossINF] at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
> [JBossINF] at org.wildfly.extension.undertow.deployment.GlobalRequestControllerHandler.handleRequest(GlobalRequestControllerHandler.java:68)
> [JBossINF] at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:292)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)
> [JBossINF] at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
> [JBossINF] at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
> [JBossINF] at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)
> [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514)
> [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514)
> [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514)
> [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104)
> [JBossINF] at io.undertow.server.Connectors.executeRootHandler(Connectors.java:360)
> [JBossINF] at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830)
> [JBossINF] at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
> [JBossINF] at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985)
> [JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487)
> [JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378)
> [JBossINF] at java.lang.Thread.run(Thread.java:748)
> {code}
> Server log:
> https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/eap-7x-failover-...
> Please note, that in this test run, number of clients was lowered to 10 (!). With the standard number of clients (2000), the exception occurs around 100 times, as can be seen in this run:
> https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/eap-7x-failover-...
> On the client, this issue (and maybe others as well) manifests as *very stale data* received:
> {code}
> 2018/05/11 02:35:17:934 EDT [WARN ][Runner - 16] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:test - Error sampling data: <org.jboss.smartfrog.loaddriver.RequestProcessingException: Stale session data received. Expected 30, received 1, Runner: 16>
> org.jboss.smartfrog.loaddriver.RequestProcessingException: Stale session data received. Expected 30, received 1, Runner: 16
> at org.jboss.smartfrog.loaddriver.http.AbstractSerialNumberValidatorFactoryImpl$SerialNumberValidator.processRequest(AbstractSerialNumberValidatorFactoryImpl.java:133)
> at org.jboss.smartfrog.loaddriver.CompoundRequestProcessorFactoryImpl$CompoundRequestProcessor.processRequest(CompoundRequestProcessorFactoryImpl.java:52)
> at org.jboss.smartfrog.loaddriver.Runner.run(Runner.java:103)
> at java.lang.Thread.run(Thread.java:748)
> {code}
> Client log:
> https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/eap-7x-failover-...
> There were only 14 sampling errors in the run with 10 clients. However, with 2000 clients, there are around 1700 sampling errors (just for 2 failovers happening in the test):
> https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/eap-7x-failover-...
> The JDG uses default cache. When trying to add transactions for the cache, I hit another issue: WFWIP-37.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
8 years, 2 months
[JBoss JIRA] (WFWIP-36) [pretesting] IllegalStateException: WFLYCLWEBUT0002: Session X already exists in session offloading scenario with hotrod store configured
by Radoslav Husar (JIRA)
[ https://issues.jboss.org/browse/WFWIP-36?page=com.atlassian.jira.plugin.s... ]
Radoslav Husar commented on WFWIP-36:
-------------------------------------
This is caused by Infinispan issue, see ISPN-9168.
> [pretesting] IllegalStateException: WFLYCLWEBUT0002: Session X already exists in session offloading scenario with hotrod store configured
> -----------------------------------------------------------------------------------------------------------------------------------------
>
> Key: WFWIP-36
> URL: https://issues.jboss.org/browse/WFWIP-36
> Project: WildFly WIP
> Issue Type: Bug
> Components: Clustering
> Reporter: Michal Vinkler
> Assignee: Radoslav Husar
> Priority: Critical
>
> We hit an issue when running tests for RFE EAP7-942.
> Wildfly was built from https://github.com/rhusar/wildfly/tree/WFLY-9294
> Scenario:
> 2-node Wildfly cluster + 2-node JDG 7.2.0.CR1 cluster.
> Wildfly servers are configured to use hotrod store which is backed up by JDG cluster.
> Each Wildfly node is gracefully shut down and then started up again.
> Wildfly configuration:
> http://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/eap-7x-failover-r...
> JDG configuration:
> http://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/eap-7x-failover-r...
> When perf18 shut down, perf19 logged one time:
> {code}
> ERROR [io.undertow.request] (default task-5) UT005023: Exception handling request to /clusterbench-offload/session: java.lang.IllegalStateException: WFLYCLWEBUT0002: Session M4qPQhyKwIG6vYTdLTxuyO_wqcKUABbRVtLtkEy_ already exists
> [JBossINF] at org.wildfly.clustering.web.undertow.session.DistributableSessionManager.createSession(DistributableSessionManager.java:124)
> [JBossINF] at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:918)
> [JBossINF] at io.undertow.servlet.spec.HttpServletRequestImpl.getSession(HttpServletRequestImpl.java:412)
> [JBossINF] at org.jboss.test.clusterbench.common.session.CommonHttpSessionServlet.doGet(CommonHttpSessionServlet.java:38)
> [JBossINF] at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
> [JBossINF] at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
> [JBossINF] at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:74)
> [JBossINF] at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)
> [JBossINF] at io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:67)
> [JBossINF] at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)
> [JBossINF] at org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)
> [JBossINF] at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
> [JBossINF] at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:132)
> [JBossINF] at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)
> [JBossINF] at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
> [JBossINF] at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)
> [JBossINF] at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)
> [JBossINF] at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)
> [JBossINF] at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)
> [JBossINF] at io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)
> [JBossINF] at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)
> [JBossINF] at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
> [JBossINF] at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)
> [JBossINF] at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
> [JBossINF] at org.wildfly.extension.undertow.deployment.GlobalRequestControllerHandler.handleRequest(GlobalRequestControllerHandler.java:68)
> [JBossINF] at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:292)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)
> [JBossINF] at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
> [JBossINF] at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
> [JBossINF] at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)
> [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514)
> [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514)
> [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514)
> [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104)
> [JBossINF] at io.undertow.server.Connectors.executeRootHandler(Connectors.java:360)
> [JBossINF] at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830)
> [JBossINF] at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
> [JBossINF] at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985)
> [JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487)
> [JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378)
> [JBossINF] at java.lang.Thread.run(Thread.java:748)
> {code}
> Server log:
> https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/eap-7x-failover-...
> Please note, that in this test run, number of clients was lowered to 10 (!). With the standard number of clients (2000), the exception occurs around 100 times, as can be seen in this run:
> https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/eap-7x-failover-...
> On the client, this issue (and maybe others as well) manifests as *very stale data* received:
> {code}
> 2018/05/11 02:35:17:934 EDT [WARN ][Runner - 16] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:test - Error sampling data: <org.jboss.smartfrog.loaddriver.RequestProcessingException: Stale session data received. Expected 30, received 1, Runner: 16>
> org.jboss.smartfrog.loaddriver.RequestProcessingException: Stale session data received. Expected 30, received 1, Runner: 16
> at org.jboss.smartfrog.loaddriver.http.AbstractSerialNumberValidatorFactoryImpl$SerialNumberValidator.processRequest(AbstractSerialNumberValidatorFactoryImpl.java:133)
> at org.jboss.smartfrog.loaddriver.CompoundRequestProcessorFactoryImpl$CompoundRequestProcessor.processRequest(CompoundRequestProcessorFactoryImpl.java:52)
> at org.jboss.smartfrog.loaddriver.Runner.run(Runner.java:103)
> at java.lang.Thread.run(Thread.java:748)
> {code}
> Client log:
> https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/eap-7x-failover-...
> There were only 14 sampling errors in the run with 10 clients. However, with 2000 clients, there are around 1700 sampling errors (just for 2 failovers happening in the test):
> https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/eap-7x-failover-...
> The JDG uses default cache. When trying to add transactions for the cache, I hit another issue: WFWIP-37.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
8 years, 2 months
[JBoss JIRA] (WFWIP-36) [pretesting] IllegalStateException: WFLYCLWEBUT0002: Session X already exists in session offloading scenario with hotrod store configured
by Radoslav Husar (JIRA)
[ https://issues.jboss.org/browse/WFWIP-36?page=com.atlassian.jira.plugin.s... ]
Radoslav Husar edited comment on WFWIP-36 at 5/17/18 3:24 AM:
--------------------------------------------------------------
This is caused by Infinispan issue, see ISPN-9168. The workaround is to define caches for deployments explicitly until fixed.
was (Author: rhusar):
This is caused by Infinispan issue, see ISPN-9168.
> [pretesting] IllegalStateException: WFLYCLWEBUT0002: Session X already exists in session offloading scenario with hotrod store configured
> -----------------------------------------------------------------------------------------------------------------------------------------
>
> Key: WFWIP-36
> URL: https://issues.jboss.org/browse/WFWIP-36
> Project: WildFly WIP
> Issue Type: Bug
> Components: Clustering
> Reporter: Michal Vinkler
> Assignee: Radoslav Husar
> Priority: Critical
>
> We hit an issue when running tests for RFE EAP7-942.
> Wildfly was built from https://github.com/rhusar/wildfly/tree/WFLY-9294
> Scenario:
> 2-node Wildfly cluster + 2-node JDG 7.2.0.CR1 cluster.
> Wildfly servers are configured to use hotrod store which is backed up by JDG cluster.
> Each Wildfly node is gracefully shut down and then started up again.
> Wildfly configuration:
> http://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/eap-7x-failover-r...
> JDG configuration:
> http://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/eap-7x-failover-r...
> When perf18 shut down, perf19 logged one time:
> {code}
> ERROR [io.undertow.request] (default task-5) UT005023: Exception handling request to /clusterbench-offload/session: java.lang.IllegalStateException: WFLYCLWEBUT0002: Session M4qPQhyKwIG6vYTdLTxuyO_wqcKUABbRVtLtkEy_ already exists
> [JBossINF] at org.wildfly.clustering.web.undertow.session.DistributableSessionManager.createSession(DistributableSessionManager.java:124)
> [JBossINF] at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:918)
> [JBossINF] at io.undertow.servlet.spec.HttpServletRequestImpl.getSession(HttpServletRequestImpl.java:412)
> [JBossINF] at org.jboss.test.clusterbench.common.session.CommonHttpSessionServlet.doGet(CommonHttpSessionServlet.java:38)
> [JBossINF] at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
> [JBossINF] at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
> [JBossINF] at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:74)
> [JBossINF] at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)
> [JBossINF] at io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:67)
> [JBossINF] at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)
> [JBossINF] at org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)
> [JBossINF] at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
> [JBossINF] at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:132)
> [JBossINF] at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)
> [JBossINF] at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
> [JBossINF] at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)
> [JBossINF] at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)
> [JBossINF] at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)
> [JBossINF] at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)
> [JBossINF] at io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)
> [JBossINF] at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)
> [JBossINF] at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
> [JBossINF] at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)
> [JBossINF] at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
> [JBossINF] at org.wildfly.extension.undertow.deployment.GlobalRequestControllerHandler.handleRequest(GlobalRequestControllerHandler.java:68)
> [JBossINF] at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:292)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)
> [JBossINF] at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
> [JBossINF] at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
> [JBossINF] at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)
> [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514)
> [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514)
> [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514)
> [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104)
> [JBossINF] at io.undertow.server.Connectors.executeRootHandler(Connectors.java:360)
> [JBossINF] at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830)
> [JBossINF] at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
> [JBossINF] at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985)
> [JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487)
> [JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378)
> [JBossINF] at java.lang.Thread.run(Thread.java:748)
> {code}
> Server log:
> https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/eap-7x-failover-...
> Please note, that in this test run, number of clients was lowered to 10 (!). With the standard number of clients (2000), the exception occurs around 100 times, as can be seen in this run:
> https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/eap-7x-failover-...
> On the client, this issue (and maybe others as well) manifests as *very stale data* received:
> {code}
> 2018/05/11 02:35:17:934 EDT [WARN ][Runner - 16] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:test - Error sampling data: <org.jboss.smartfrog.loaddriver.RequestProcessingException: Stale session data received. Expected 30, received 1, Runner: 16>
> org.jboss.smartfrog.loaddriver.RequestProcessingException: Stale session data received. Expected 30, received 1, Runner: 16
> at org.jboss.smartfrog.loaddriver.http.AbstractSerialNumberValidatorFactoryImpl$SerialNumberValidator.processRequest(AbstractSerialNumberValidatorFactoryImpl.java:133)
> at org.jboss.smartfrog.loaddriver.CompoundRequestProcessorFactoryImpl$CompoundRequestProcessor.processRequest(CompoundRequestProcessorFactoryImpl.java:52)
> at org.jboss.smartfrog.loaddriver.Runner.run(Runner.java:103)
> at java.lang.Thread.run(Thread.java:748)
> {code}
> Client log:
> https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/eap-7x-failover-...
> There were only 14 sampling errors in the run with 10 clients. However, with 2000 clients, there are around 1700 sampling errors (just for 2 failovers happening in the test):
> https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/eap-7x-failover-...
> The JDG uses default cache. When trying to add transactions for the cache, I hit another issue: WFWIP-37.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
8 years, 2 months
[JBoss JIRA] (WFWIP-36) [pretesting] IllegalStateException: WFLYCLWEBUT0002: Session X already exists in session offloading scenario with hotrod store configured
by Radoslav Husar (JIRA)
[ https://issues.jboss.org/browse/WFWIP-36?page=com.atlassian.jira.plugin.s... ]
Radoslav Husar edited comment on WFWIP-36 at 5/17/18 3:24 AM:
--------------------------------------------------------------
This is caused by Infinispan issue, see ISPN-9168. Caches need to be defined explicitly or use 'cache-configuration' to choose a specific template until it is fixed.
was (Author: rhusar):
This is caused by Infinispan issue, see ISPN-9168. The workaround is to define caches for deployments explicitly until fixed.
> [pretesting] IllegalStateException: WFLYCLWEBUT0002: Session X already exists in session offloading scenario with hotrod store configured
> -----------------------------------------------------------------------------------------------------------------------------------------
>
> Key: WFWIP-36
> URL: https://issues.jboss.org/browse/WFWIP-36
> Project: WildFly WIP
> Issue Type: Bug
> Components: Clustering
> Reporter: Michal Vinkler
> Assignee: Radoslav Husar
> Priority: Critical
>
> We hit an issue when running tests for RFE EAP7-942.
> Wildfly was built from https://github.com/rhusar/wildfly/tree/WFLY-9294
> Scenario:
> 2-node Wildfly cluster + 2-node JDG 7.2.0.CR1 cluster.
> Wildfly servers are configured to use hotrod store which is backed up by JDG cluster.
> Each Wildfly node is gracefully shut down and then started up again.
> Wildfly configuration:
> http://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/eap-7x-failover-r...
> JDG configuration:
> http://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/eap-7x-failover-r...
> When perf18 shut down, perf19 logged one time:
> {code}
> ERROR [io.undertow.request] (default task-5) UT005023: Exception handling request to /clusterbench-offload/session: java.lang.IllegalStateException: WFLYCLWEBUT0002: Session M4qPQhyKwIG6vYTdLTxuyO_wqcKUABbRVtLtkEy_ already exists
> [JBossINF] at org.wildfly.clustering.web.undertow.session.DistributableSessionManager.createSession(DistributableSessionManager.java:124)
> [JBossINF] at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:918)
> [JBossINF] at io.undertow.servlet.spec.HttpServletRequestImpl.getSession(HttpServletRequestImpl.java:412)
> [JBossINF] at org.jboss.test.clusterbench.common.session.CommonHttpSessionServlet.doGet(CommonHttpSessionServlet.java:38)
> [JBossINF] at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
> [JBossINF] at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
> [JBossINF] at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:74)
> [JBossINF] at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)
> [JBossINF] at io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:67)
> [JBossINF] at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)
> [JBossINF] at org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)
> [JBossINF] at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
> [JBossINF] at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:132)
> [JBossINF] at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)
> [JBossINF] at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
> [JBossINF] at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)
> [JBossINF] at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)
> [JBossINF] at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)
> [JBossINF] at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)
> [JBossINF] at io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)
> [JBossINF] at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)
> [JBossINF] at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
> [JBossINF] at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)
> [JBossINF] at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
> [JBossINF] at org.wildfly.extension.undertow.deployment.GlobalRequestControllerHandler.handleRequest(GlobalRequestControllerHandler.java:68)
> [JBossINF] at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:292)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)
> [JBossINF] at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
> [JBossINF] at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
> [JBossINF] at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)
> [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514)
> [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514)
> [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514)
> [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104)
> [JBossINF] at io.undertow.server.Connectors.executeRootHandler(Connectors.java:360)
> [JBossINF] at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830)
> [JBossINF] at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
> [JBossINF] at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985)
> [JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487)
> [JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378)
> [JBossINF] at java.lang.Thread.run(Thread.java:748)
> {code}
> Server log:
> https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/eap-7x-failover-...
> Please note, that in this test run, number of clients was lowered to 10 (!). With the standard number of clients (2000), the exception occurs around 100 times, as can be seen in this run:
> https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/eap-7x-failover-...
> On the client, this issue (and maybe others as well) manifests as *very stale data* received:
> {code}
> 2018/05/11 02:35:17:934 EDT [WARN ][Runner - 16] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:test - Error sampling data: <org.jboss.smartfrog.loaddriver.RequestProcessingException: Stale session data received. Expected 30, received 1, Runner: 16>
> org.jboss.smartfrog.loaddriver.RequestProcessingException: Stale session data received. Expected 30, received 1, Runner: 16
> at org.jboss.smartfrog.loaddriver.http.AbstractSerialNumberValidatorFactoryImpl$SerialNumberValidator.processRequest(AbstractSerialNumberValidatorFactoryImpl.java:133)
> at org.jboss.smartfrog.loaddriver.CompoundRequestProcessorFactoryImpl$CompoundRequestProcessor.processRequest(CompoundRequestProcessorFactoryImpl.java:52)
> at org.jboss.smartfrog.loaddriver.Runner.run(Runner.java:103)
> at java.lang.Thread.run(Thread.java:748)
> {code}
> Client log:
> https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/eap-7x-failover-...
> There were only 14 sampling errors in the run with 10 clients. However, with 2000 clients, there are around 1700 sampling errors (just for 2 failovers happening in the test):
> https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/eap-7x-failover-...
> The JDG uses default cache. When trying to add transactions for the cache, I hit another issue: WFWIP-37.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
8 years, 2 months
[JBoss JIRA] (WFLY-10388) Persistent EJB timer timed_object_id naming discrepancy
by Boris Pran (JIRA)
Boris Pran created WFLY-10388:
---------------------------------
Summary: Persistent EJB timer timed_object_id naming discrepancy
Key: WFLY-10388
URL: https://issues.jboss.org/browse/WFLY-10388
Project: WildFly
Issue Type: Bug
Components: EJB
Affects Versions: 12.0.0.Final
Reporter: Boris Pran
Persistent EJB timers created declaratively (with @Schedule) create timed_object_id without version information while timers created using TimerService API include version info from deployment unit.
This way after version upgrade @Schedule timers survive but TimerService API created ones are lost and need to be created again.
I also seems that old timers remain created or at least their definitions can be seen in jboss_ejb_timer table.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
8 years, 2 months
[JBoss JIRA] (WFLY-10182) Unstable work of the persistent EJB timers with Oracle DB. ORA-08177: can't serialize access for this transaction
by Maxim Karavaev (JIRA)
[ https://issues.jboss.org/browse/WFLY-10182?page=com.atlassian.jira.plugin... ]
Maxim Karavaev updated WFLY-10182:
----------------------------------
Git Pull Request: https://github.com/wildfly/wildfly/pull/11100, https://github.com/wildfly/wildfly/pull/11242 (was: https://github.com/wildfly/wildfly/pull/11100)
> Unstable work of the persistent EJB timers with Oracle DB. ORA-08177: can't serialize access for this transaction
> -----------------------------------------------------------------------------------------------------------------
>
> Key: WFLY-10182
> URL: https://issues.jboss.org/browse/WFLY-10182
> Project: WildFly
> Issue Type: Bug
> Components: EJB
> Affects Versions: 10.1.0.Final, 11.0.0.Final, 12.0.0.Final
> Environment: WildFly cluster in domain mode.
> Oracle 12c DB
> Reporter: Maxim Karavaev
> Attachments: Stable_persistence_logic_of_EJB_Timers.patch
>
>
> The main signal is an exception in logs:
> {noformat}
> ORA-08177: can't serialize access for this transaction
> {noformat}
> When one cluster node is halted (the one on the witch "Timer" is actually working) then the other nodes are still can't catch up this "timer" from the DB due to the mentioned exception. Even worth scenario - when all nodes work perfectly, but occasionally all nodes start to decline, that timer is in the state IN_TIMEOUT and it really is due to the DB state, just because the last timer's invocation did not manage to commit final state after processing due to the same reason (SQL exception).
> So, why Oracle DB is so special? That's because it never manages transactions by themselves. It's full responsibility of the client.
> Looking into the code of the class
> {noformat}
> org.jboss.as.ejb3.timerservice.persistence.database.DatabaseTimerPersistence
> {noformat}
> I didn't find transaction management, except one method, and even in it, I found, that it probably doesn't manage transaction as desired, due to the lack of resource participation.
> After I've made some changes to enable transactions for that functionality I have managed to stable EJB Timer logic on the server. Let's have a look at the attached patch (it's for 11 version, but it also works for 12).
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
8 years, 2 months