[ https://issues.redhat.com/browse/WFLY-13328?page=com.atlassian.jira.plugi... ] Tommasso Borgato updated WFLY-13328: ------------------------------------ Description: The error happens in WildFly Clustering tests where EAP is connected to a JDG Cluster using the HotRod-based distributed session manager and WildFly is using a non transactional cache; The error is pretty bad because it affects every client call; Here is the EAP nodes configuration cli script: {noformat} embed-server --server-config=standalone-ha.xml /subsystem=jgroups/channel=ee:write-attribute(name=stack,value=udp) /subsystem=transactions:write-attribute(name=node-identifier,value=wildfly1) /socket-binding-group=standard-sockets/remote-destination-outbound-socket-binding=remote-jdg-server1:add(host=10.16.176.58, port=11222) /socket-binding-group=standard-sockets/remote-destination-outbound-socket-binding=remote-jdg-server2:add(host=10.16.176.56, port=11222) batch /subsystem=infinispan/remote-cache-container=session_data_cc:add(default-remote-cluster=jdg-server-cluster, module=org.wildfly.clustering.web.hotrod) /subsystem=infinispan/remote-cache-container=session_data_cc/remote-cluster=jdg-server-cluster:add(socket-bindings=[remote-jdg-server1,remote-jdg-server2]) run-batch /subsystem=distributable-web/hotrod-session-management=sm_offload:add(remote-cache-container=session_data_cc, granularity=ATTRIBUTE) /subsystem=distributable-web/hotrod-session-management=sm_offload/affinity=local:add() /subsystem=distributable-web/hotrod-session-management=sm_offload_granular:add(remote-cache-container=session_data_cc, granularity=ATTRIBUTE) /subsystem=distributable-web/hotrod-session-management=sm_offload_granular/affinity=local:add() /subsystem=infinispan/remote-cache-container=session_data_cc/near-cache=invalidation:add(max-entries=1000) /subsystem=distributable-web:write-attribute(name=default-session-management,value=sm_offload_granular) {noformat} The error happens on the client side when trying to retrieve a session attribute from the cache: {code:java} SerialBean bean = (SerialBean) session.getAttribute(KEY); int serial = bean.getSerial(); // bean is null!!! {code} The error is: {noformat} 2020-04-02 12:50:05,189 ERROR [io.undertow.request] (default task-68) UT005023: Exception handling request to /clusterbench/session: java.lang.NullPointerException at deployment.clusterbench-ee8.ear//org.jboss.test.clusterbench.common.session.CommonHttpSessionServlet.doGet(CommonHttpSessionServlet.java:64) at javax.servlet.api@2.0.0.Final//javax.servlet.http.HttpServlet.service(HttpServlet.java:503) at javax.servlet.api@2.0.0.Final//javax.servlet.http.HttpServlet.service(HttpServlet.java:590) at io.undertow.servlet@2.0.30.Final//io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:74) at io.undertow.servlet@2.0.30.Final//io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:129) at io.opentracing.contrib.opentracing-jaxrs2//io.opentracing.contrib.jaxrs2.server.SpanFinishingFilter.doFilter(SpanFinishingFilter.java:52) at io.undertow.servlet@2.0.30.Final//io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) at io.undertow.servlet@2.0.30.Final//io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) at io.undertow.servlet@2.0.30.Final//io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84) at io.undertow.servlet@2.0.30.Final//io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62) at io.undertow.servlet@2.0.30.Final//io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:68) at io.undertow.servlet@2.0.30.Final//io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36) at org.wildfly.extension.undertow@20.0.0.Beta1-SNAPSHOT//org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78) at io.undertow.core@2.0.30.Final//io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at io.undertow.servlet@2.0.30.Final//io.undertow.servlet.handlers.RedirectDirHandler.handleRequest(RedirectDirHandler.java:68) at io.undertow.servlet@2.0.30.Final//io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:132) at io.undertow.servlet@2.0.30.Final//io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57) at io.undertow.core@2.0.30.Final//io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at io.undertow.core@2.0.30.Final//io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46) at io.undertow.servlet@2.0.30.Final//io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64) at io.undertow.core@2.0.30.Final//io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60) at io.undertow.servlet@2.0.30.Final//io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77) at io.undertow.core@2.0.30.Final//io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50) at io.undertow.core@2.0.30.Final//io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43) at io.undertow.core@2.0.30.Final//io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at org.wildfly.extension.undertow@20.0.0.Beta1-SNAPSHOT//org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61) at io.undertow.core@2.0.30.Final//io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at org.wildfly.extension.undertow@20.0.0.Beta1-SNAPSHOT//org.wildfly.extension.undertow.deployment.GlobalRequestControllerHandler.handleRequest(GlobalRequestControllerHandler.java:68) at io.undertow.core@2.0.30.Final//io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at io.undertow.servlet@2.0.30.Final//io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:269) at io.undertow.servlet@2.0.30.Final//io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:78) at io.undertow.servlet@2.0.30.Final//io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:133) at io.undertow.servlet@2.0.30.Final//io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:130) at io.undertow.servlet@2.0.30.Final//io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48) at io.undertow.servlet@2.0.30.Final//io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43) at org.wildfly.extension.undertow@20.0.0.Beta1-SNAPSHOT//org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105) at org.wildfly.extension.undertow@20.0.0.Beta1-SNAPSHOT//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1541) at org.wildfly.extension.undertow@20.0.0.Beta1-SNAPSHOT//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1541) at org.wildfly.extension.undertow@20.0.0.Beta1-SNAPSHOT//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1541) at org.wildfly.extension.undertow@20.0.0.Beta1-SNAPSHOT//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1541) at io.undertow.servlet@2.0.30.Final//io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:249) at io.undertow.servlet@2.0.30.Final//io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:78) at io.undertow.servlet@2.0.30.Final//io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:99) at io.undertow.core@2.0.30.Final//io.undertow.server.Connectors.executeRootHandler(Connectors.java:376) at io.undertow.core@2.0.30.Final//io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830) at org.jboss.threads@2.3.3.Final//org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35) at org.jboss.threads@2.3.3.Final//org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1982) at org.jboss.threads@2.3.3.Final//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486) at org.jboss.threads@2.3.3.Final//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1348) at java.base/java.lang.Thread.run(Thread.java:834) {noformat} The error disappears when using a transactional cache (see run [eap-7.x-clustering-jdg-session-shutdown-haproxy-hotrod#63|https://eap-qe-...]): {noformat} /subsystem=infinispan/remote-cache-container=session_data_cc/component=transaction:add(mode=BATCH) {noformat} Complete run here: [eap-7.x-clustering-stress-session-dist-nginx-jdg-hotrod#7|https://eap-qe-...]; Logs attached; was: The error happens in EAP Clustering tests where EAP is connected to a JDG Cluster using the HotRod-based distributed session manager and is using a non transactional cache; The error is pretty bad because it affects every client call; Here is the EAP nodes configuration cli script: {noformat} embed-server --server-config=standalone-ha.xml /subsystem=jgroups/channel=ee:write-attribute(name=stack,value=udp) /subsystem=transactions:write-attribute(name=node-identifier,value=wildfly1) /socket-binding-group=standard-sockets/remote-destination-outbound-socket-binding=remote-jdg-server1:add(host=10.16.176.58, port=11222) /socket-binding-group=standard-sockets/remote-destination-outbound-socket-binding=remote-jdg-server2:add(host=10.16.176.56, port=11222) batch /subsystem=infinispan/remote-cache-container=session_data_cc:add(default-remote-cluster=jdg-server-cluster, module=org.wildfly.clustering.web.hotrod) /subsystem=infinispan/remote-cache-container=session_data_cc/remote-cluster=jdg-server-cluster:add(socket-bindings=[remote-jdg-server1,remote-jdg-server2]) run-batch /subsystem=distributable-web/hotrod-session-management=sm_offload:add(remote-cache-container=session_data_cc, granularity=ATTRIBUTE) /subsystem=distributable-web/hotrod-session-management=sm_offload/affinity=local:add() /subsystem=distributable-web/hotrod-session-management=sm_offload_granular:add(remote-cache-container=session_data_cc, granularity=ATTRIBUTE) /subsystem=distributable-web/hotrod-session-management=sm_offload_granular/affinity=local:add() /subsystem=infinispan/remote-cache-container=session_data_cc/near-cache=invalidation:add(max-entries=1000) /subsystem=distributable-web:write-attribute(name=default-session-management,value=sm_offload_granular) {noformat} The error happens on the client side when trying to retrieve a session attribute from the cache: {code:java} SerialBean bean = (SerialBean) session.getAttribute(KEY); int serial = bean.getSerial(); // bean is null!!! {code} The error is: {noformat} 2020-04-02 12:50:05,189 ERROR [io.undertow.request] (default task-68) UT005023: Exception handling request to /clusterbench/session: java.lang.NullPointerException at deployment.clusterbench-ee8.ear//org.jboss.test.clusterbench.common.session.CommonHttpSessionServlet.doGet(CommonHttpSessionServlet.java:64) at javax.servlet.api@2.0.0.Final//javax.servlet.http.HttpServlet.service(HttpServlet.java:503) at javax.servlet.api@2.0.0.Final//javax.servlet.http.HttpServlet.service(HttpServlet.java:590) at io.undertow.servlet@2.0.30.Final//io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:74) at io.undertow.servlet@2.0.30.Final//io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:129) at io.opentracing.contrib.opentracing-jaxrs2//io.opentracing.contrib.jaxrs2.server.SpanFinishingFilter.doFilter(SpanFinishingFilter.java:52) at io.undertow.servlet@2.0.30.Final//io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) at io.undertow.servlet@2.0.30.Final//io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) at io.undertow.servlet@2.0.30.Final//io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84) at io.undertow.servlet@2.0.30.Final//io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62) at io.undertow.servlet@2.0.30.Final//io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:68) at io.undertow.servlet@2.0.30.Final//io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36) at org.wildfly.extension.undertow@20.0.0.Beta1-SNAPSHOT//org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78) at io.undertow.core@2.0.30.Final//io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at io.undertow.servlet@2.0.30.Final//io.undertow.servlet.handlers.RedirectDirHandler.handleRequest(RedirectDirHandler.java:68) at io.undertow.servlet@2.0.30.Final//io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:132) at io.undertow.servlet@2.0.30.Final//io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57) at io.undertow.core@2.0.30.Final//io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at io.undertow.core@2.0.30.Final//io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46) at io.undertow.servlet@2.0.30.Final//io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64) at io.undertow.core@2.0.30.Final//io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60) at io.undertow.servlet@2.0.30.Final//io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77) at io.undertow.core@2.0.30.Final//io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50) at io.undertow.core@2.0.30.Final//io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43) at io.undertow.core@2.0.30.Final//io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at org.wildfly.extension.undertow@20.0.0.Beta1-SNAPSHOT//org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61) at io.undertow.core@2.0.30.Final//io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at org.wildfly.extension.undertow@20.0.0.Beta1-SNAPSHOT//org.wildfly.extension.undertow.deployment.GlobalRequestControllerHandler.handleRequest(GlobalRequestControllerHandler.java:68) at io.undertow.core@2.0.30.Final//io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at io.undertow.servlet@2.0.30.Final//io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:269) at io.undertow.servlet@2.0.30.Final//io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:78) at io.undertow.servlet@2.0.30.Final//io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:133) at io.undertow.servlet@2.0.30.Final//io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:130) at io.undertow.servlet@2.0.30.Final//io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48) at io.undertow.servlet@2.0.30.Final//io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43) at org.wildfly.extension.undertow@20.0.0.Beta1-SNAPSHOT//org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105) at org.wildfly.extension.undertow@20.0.0.Beta1-SNAPSHOT//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1541) at org.wildfly.extension.undertow@20.0.0.Beta1-SNAPSHOT//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1541) at org.wildfly.extension.undertow@20.0.0.Beta1-SNAPSHOT//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1541) at org.wildfly.extension.undertow@20.0.0.Beta1-SNAPSHOT//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1541) at io.undertow.servlet@2.0.30.Final//io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:249) at io.undertow.servlet@2.0.30.Final//io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:78) at io.undertow.servlet@2.0.30.Final//io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:99) at io.undertow.core@2.0.30.Final//io.undertow.server.Connectors.executeRootHandler(Connectors.java:376) at io.undertow.core@2.0.30.Final//io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830) at org.jboss.threads@2.3.3.Final//org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35) at org.jboss.threads@2.3.3.Final//org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1982) at org.jboss.threads@2.3.3.Final//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486) at org.jboss.threads@2.3.3.Final//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1348) at java.base/java.lang.Thread.run(Thread.java:834) {noformat} The error disappears when using a transactional cache (see run [eap-7.x-clustering-jdg-session-shutdown-haproxy-hotrod#63|https://eap-qe-...]): {noformat} /subsystem=infinispan/remote-cache-container=session_data_cc/component=transaction:add(mode=BATCH) {noformat} Complete run here: [eap-7.x-clustering-stress-session-dist-nginx-jdg-hotrod#7|https://eap-qe-...]; Logs attached; -- This message was sent by Atlassian Jira (v7.13.8#713008)