[jboss-jira] [JBoss JIRA] (WFWIP-36) [pretesting] IllegalStateException: WFLYCLWEBUT0002: Session X already exists in session offloading scenario with hotrod store configured

[pretesting] IllegalStateException: WFLYCLWEBUT0002: Session X already exists in session offloading scenario with hotrod store configured ----------------------------------------------------------------------------------------------------------------------------------------- Key: WFWIP-36 URL: https://issues.jboss.org/browse/WFWIP-36 Project: WildFly WIP Issue Type: Bug Components: Clustering Reporter: Michal Vinkler Assignee: Radoslav Husar Priority: Critical We hit an issue when running tests for RFE EAP7-942. Wildfly was built from https://github.com/rhusar/wildfly/tree/WFLY-9294 Scenario: 2-node Wildfly cluster + 2-node JDG 7.2.0.CR1 cluster. Wildfly servers are configured to use hotrod store which is backed up by JDG cluster. Each Wildfly node is gracefully shut down and then started up again. Wildfly configuration: http://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/eap-7x-failover-r... JDG configuration: http://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/eap-7x-failover-r... When perf18 shut down, perf19 logged one time: {code} ERROR [io.undertow.request] (default task-5) UT005023: Exception handling request to /clusterbench-offload/session: java.lang.IllegalStateException: WFLYCLWEBUT0002: Session M4qPQhyKwIG6vYTdLTxuyO_wqcKUABbRVtLtkEy_ already exists [JBossINF] at org.wildfly.clustering.web.undertow.session.DistributableSessionManager.createSession(DistributableSessionManager.java:124) [JBossINF] at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:918) [JBossINF] at io.undertow.servlet.spec.HttpServletRequestImpl.getSession(HttpServletRequestImpl.java:412) [JBossINF] at org.jboss.test.clusterbench.common.session.CommonHttpSessionServlet.doGet(CommonHttpSessionServlet.java:38) [JBossINF] at javax.servlet.http.HttpServlet.service(HttpServlet.java:687) [JBossINF] at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) [JBossINF] at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:74) [JBossINF] at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62) [JBossINF] at io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:67) [JBossINF] at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36) [JBossINF] at org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78) [JBossINF] at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [JBossINF] at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:132) [JBossINF] at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57) [JBossINF] at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [JBossINF] at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46) [JBossINF] at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64) [JBossINF] at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60) [JBossINF] at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77) [JBossINF] at io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50) [JBossINF] at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43) [JBossINF] at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [JBossINF] at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61) [JBossINF] at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [JBossINF] at org.wildfly.extension.undertow.deployment.GlobalRequestControllerHandler.handleRequest(GlobalRequestControllerHandler.java:68) [JBossINF] at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:292) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135) [JBossINF] at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48) [JBossINF] at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43) [JBossINF] at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105) [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514) [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514) [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514) [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81) [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104) [JBossINF] at io.undertow.server.Connectors.executeRootHandler(Connectors.java:360) [JBossINF] at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830) [JBossINF] at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35) [JBossINF] at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985) [JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487) [JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378) [JBossINF] at java.lang.Thread.run(Thread.java:748) {code} Server log: https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/eap-7x-failover-... Please note, that in this test run, number of clients was lowered to 10 (!). With the standard number of clients (2000), the exception occurs around 100 times, as can be seen in this run: https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/eap-7x-failover-... On the client, this issue (and maybe others as well) manifests as *very stale data* received: {code} 2018/05/11 02:35:17:934 EDT [WARN ][Runner - 16] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:test - Error sampling data: <org.jboss.smartfrog.loaddriver.RequestProcessingException: Stale session data received. Expected 30, received 1, Runner: 16> org.jboss.smartfrog.loaddriver.RequestProcessingException: Stale session data received. Expected 30, received 1, Runner: 16 at org.jboss.smartfrog.loaddriver.http.AbstractSerialNumberValidatorFactoryImpl$SerialNumberValidator.processRequest(AbstractSerialNumberValidatorFactoryImpl.java:133) at org.jboss.smartfrog.loaddriver.CompoundRequestProcessorFactoryImpl$CompoundRequestProcessor.processRequest(CompoundRequestProcessorFactoryImpl.java:52) at org.jboss.smartfrog.loaddriver.Runner.run(Runner.java:103) at java.lang.Thread.run(Thread.java:748) {code} Client log: https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/eap-7x-failover-... There were only 14 sampling errors in the run with 10 clients. However, with 2000 clients, there are around 1700 sampling errors (just for 2 failovers happening in the test): https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/eap-7x-failover-... The JDG uses default cache. When trying to add transactions for the cache, I hit another issue: WFWIP-37.