When a node left and rejoined the cluster, the authentication failed to sync. ----------------------------------------------------------------------------- Key: WFLY-4249 URL: https://issues.jboss.org/browse/WFLY-4249 Project: WildFly Issue Type: Bug Components: Clustering Affects Versions: 8.1.0.Final, 8.2.0.Final, 9.0.0.Alpha1 Environment: Fedora 18, Java 1.7.0_45-b18 64Bit Reporter: Jixing Wang Assignee: Paul Ferraro Priority: Critical Hi there, When I try to use cluster features, I got an issue. With SSO enabled, when a node left and rejoined the cluster, the authentication failed to sync. I enabled SSO in standalone-ha.xml <single-sign-on/> Node 3: $JBOSS_HOME/bin/standalone.sh -c standalone-ha.xml -b=10.0.0.120 -u 230.0.0.4 -Djboss.node.name=node3 -Djboss.socket.binding.port-offset=100 Node 4: $JBOSS_HOME/bin/standalone.sh -c standalone-ha.xml -b=10.0.0.120 -u 230.0.0.4 -Djboss.node.name=node4 -Djboss.socket.binding.port-offset=200 I first started both nodes and logged in. Then I shut down node 3, and refresh the page. Everything worked fine until now. After this, I started node 3 again but I found I was logged out. When I shut down the node3, it worked again. I also tried different versions of WildFly, 8.1, 8.2 and 9.0 and it failed in all versions. The log from node 4 below happens sometimes but NOT always. 14:56:44,874 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-18,shared=udp) ISPN000094: Received new cluster view: [node4/web|3] (2) [node4/web, node3/web] 14:56:45,117 ERROR [org.infinispan.statetransfer.OutboundTransferTask] (transport-thread-12) Failed to send entries to node node3/web : ISPN000217: Received exception from node3/web, see cause for remote stack trace: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from node3/web, see cause for remote stack trace at org.infinispan.remoting.transport.AbstractTransport.checkResponse(AbstractTransport.java:41) at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.processSingleCall(CommandAwareRpcDispatcher.java:362) at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:167) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:521) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:281) at org.infinispan.statetransfer.OutboundTransferTask.sendEntries(OutboundTransferTask.java:233) at org.infinispan.statetransfer.OutboundTransferTask.run(OutboundTransferTask.java:174) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_45] at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_45] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_45] at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_45] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_45] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_45] at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_45] Caused by: org.infinispan.commons.CacheException: Problems invoking command. at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:221) at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:460) at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:377) at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:247) at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:667) at org.jgroups.blocks.mux.MuxUpHandler.up(MuxUpHandler.java:130) at org.jgroups.JChannel.up(JChannel.java:708) at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1015) at org.jgroups.protocols.RSVP.up(RSVP.java:171) at org.jgroups.protocols.FRAG2.up(FRAG2.java:165) at org.jgroups.protocols.FlowControl.up(FlowControl.java:381) at org.jgroups.protocols.FlowControl.up(FlowControl.java:370) at org.jgroups.protocols.pbcast.GMS.up(GMS.java:1010) at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:234) at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:695) at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:382) at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:600) at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:147) at org.jgroups.protocols.FD_ALL.up(FD_ALL.java:185) at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:301) at org.jgroups.protocols.MERGE3.up(MERGE3.java:303) at org.jgroups.protocols.Discovery.up(Discovery.java:379) at org.jgroups.protocols.TP$ProtocolAdapter.up(TP.java:2622) at org.jgroups.protocols.TP.passMessageUp(TP.java:1412) at org.jgroups.protocols.TP$MyHandler.run(TP.java:1598) ... 3 more Caused by: java.io.InvalidClassException: com.britensw.nyx.core.security.NyxPrincipal; Module load failed at org.jboss.marshalling.ModularClassResolver.resolveClass(ModularClassResolver.java:104) [jboss-marshalling-1.4.8.Final.jar:1.4.8.Final] at org.jboss.marshalling.river.RiverUnmarshaller.doReadClassDescriptor(RiverUnmarshaller.java:949) at org.jboss.marshalling.river.RiverUnmarshaller.doReadNewObject(RiverUnmarshaller.java:1256) at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:276) at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:224) at org.jboss.marshalling.river.RiverUnmarshaller.readFields(RiverUnmarshaller.java:1746) at org.jboss.marshalling.river.RiverUnmarshaller.doInitSerializable(RiverUnmarshaller.java:1659) at org.jboss.marshalling.river.RiverUnmarshaller.doReadNewObject(RiverUnmarshaller.java:1286) at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:276) at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:224) at org.jboss.marshalling.river.RiverUnmarshaller.readFields(RiverUnmarshaller.java:1746) at org.jboss.marshalling.river.RiverUnmarshaller.doInitSerializable(RiverUnmarshaller.java:1659) at org.jboss.marshalling.river.RiverUnmarshaller.doReadNewObject(RiverUnmarshaller.java:1286) at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:276) at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209) at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:41) [jboss-marshalling-1.4.8.Final.jar:1.4.8.Final] at org.wildfly.clustering.web.infinispan.sso.coarse.CoarseAuthenticationEntryExternalizer.readObject(CoarseAuthenticationEntryExternalizer.java:57) at org.wildfly.clustering.web.infinispan.sso.coarse.CoarseAuthenticationEntryExternalizer.readObject(CoarseAuthenticationEntryExternalizer.java:36) at org.infinispan.marshall.core.ExternalizerTable$ExternalizerAdapter.readObject(ExternalizerTable.java:389) at org.infinispan.marshall.core.ExternalizerTable.readObject(ExternalizerTable.java:205) at org.infinispan.marshall.core.JBossMarshaller$ExternalizerTableProxy.readObject(JBossMarshaller.java:152) at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:355) at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209) at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:41) [jboss-marshalling-1.4.8.Final.jar:1.4.8.Final] at org.infinispan.container.entries.ImmortalCacheEntry$Externalizer.readObject(ImmortalCacheEntry.java:152) at org.infinispan.container.entries.ImmortalCacheEntry$Externalizer.readObject(ImmortalCacheEntry.java:142) at org.infinispan.marshall.core.ExternalizerTable$ExternalizerAdapter.readObject(ExternalizerTable.java:389) at org.infinispan.marshall.core.ExternalizerTable.readObject(ExternalizerTable.java:205) at org.infinispan.marshall.core.JBossMarshaller$ExternalizerTableProxy.readObject(JBossMarshaller.java:152) at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:355) at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209) at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:41) [jboss-marshalling-1.4.8.Final.jar:1.4.8.Final] at org.infinispan.marshall.exts.ListExternalizer.readObject(ListExternalizer.java:62) at org.infinispan.marshall.exts.ListExternalizer.readObject(ListExternalizer.java:26) at org.infinispan.marshall.core.ExternalizerTable$ExternalizerAdapter.readObject(ExternalizerTable.java:389) at org.infinispan.marshall.core.ExternalizerTable.readObject(ExternalizerTable.java:205) at org.infinispan.marshall.core.JBossMarshaller$ExternalizerTableProxy.readObject(JBossMarshaller.java:152) at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:355) at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209) at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:41) [jboss-marshalling-1.4.8.Final.jar:1.4.8.Final] at org.infinispan.statetransfer.StateChunk$Externalizer.readObject(StateChunk.java:88) at org.infinispan.statetransfer.StateChunk$Externalizer.readObject(StateChunk.java:65) at org.infinispan.marshall.core.ExternalizerTable$ExternalizerAdapter.readObject(ExternalizerTable.java:389) at org.infinispan.marshall.core.ExternalizerTable.readObject(ExternalizerTable.java:205) at org.infinispan.marshall.core.JBossMarshaller$ExternalizerTableProxy.readObject(JBossMarshaller.java:152) at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:355) at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209) at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:41) [jboss-marshalling-1.4.8.Final.jar:1.4.8.Final] at org.infinispan.marshall.exts.ListExternalizer.readObject(ListExternalizer.java:62) at org.infinispan.marshall.exts.ListExternalizer.readObject(ListExternalizer.java:26) at org.infinispan.marshall.core.ExternalizerTable$ExternalizerAdapter.readObject(ExternalizerTable.java:389) at org.infinispan.marshall.core.ExternalizerTable.readObject(ExternalizerTable.java:205) at org.infinispan.marshall.core.JBossMarshaller$ExternalizerTableProxy.readObject(JBossMarshaller.java:152) at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:355) at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209) at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:41) [jboss-marshalling-1.4.8.Final.jar:1.4.8.Final] at org.infinispan.marshall.exts.ReplicableCommandExternalizer.readParameters(ReplicableCommandExternalizer.java:100) at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.readObject(CacheRpcCommandExternalizer.java:146) at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.readObject(CacheRpcCommandExternalizer.java:59) at org.infinispan.marshall.core.ExternalizerTable$ExternalizerAdapter.readObject(ExternalizerTable.java:389) at org.infinispan.marshall.core.ExternalizerTable.readObject(ExternalizerTable.java:205) at org.infinispan.marshall.core.JBossMarshaller$ExternalizerTableProxy.readObject(JBossMarshaller.java:152) at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:355) at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209) at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:41) [jboss-marshalling-1.4.8.Final.jar:1.4.8.Final] at org.infinispan.commons.marshall.jboss.AbstractJBossMarshaller.objectFromObjectStream(AbstractJBossMarshaller.java:136) at org.infinispan.marshall.core.VersionAwareMarshaller.objectFromByteBuffer(VersionAwareMarshaller.java:101) at org.infinispan.commons.marshall.AbstractDelegatingMarshaller.objectFromByteBuffer(AbstractDelegatingMarshaller.java:80) at org.infinispan.remoting.transport.jgroups.MarshallerAdapter.objectFromBuffer(MarshallerAdapter.java:28) at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:206) ... 27 more Caused by: org.jboss.modules.ModuleNotFoundException: deployment.nyx.war:main at org.jboss.modules.ModuleLoader.loadModule(ModuleLoader.java:240) [jboss-modules.jar:1.3.4.Final] at org.jboss.marshalling.ModularClassResolver.resolveClass(ModularClassResolver.java:102) [jboss-marshalling-1.4.8.Final.jar:1.4.8.Final] ... 96 more Caused by: an exception which occurred: in field principal in object of type org.wildfly.extension.undertow.security.AccountImpl in field account in object of type io.undertow.security.api.AuthenticatedSessionManager$AuthenticatedSession 14:56:49,679 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-1,shared=udp) ISPN000094: Received new cluster view: [node4/ejb|3] (2) [node4/ejb, node3/ejb] 14:56:51,515 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-15,shared=udp) ISPN000094: Received new cluster view: [node4/hibernate|3] (2) [node4/hibernate, node3/hibernate] Apache configuration: <<mod_cluster.conf>> LoadModule proxy_module modules/mod_proxy.so LoadModule proxy_http_module modules/mod_proxy_http.so LoadModule proxy_ajp_module modules/mod_proxy_ajp.so LoadModule slotmem_module modules/mod_slotmem.so LoadModule manager_module modules/mod_manager.so LoadModule proxy_cluster_module modules/mod_proxy_cluster.so LoadModule advertise_module modules/mod_advertise.so MemManagerFile /var/cache/httpd <<cluster.conf>> Listen 10001 <VirtualHost *:10001> ServerAdmin root(a)example.com ServerName localhost <Directory /> Require all granted </Directory> <Location /mod_cluster_manager> SetHandler mod_cluster-manager Require all granted </Location> AdvertiseFrequency 5 KeepAliveTimeout 60 MaxKeepAliveRequests 0 AllowDisplay On ManagerBalancerName mycluster ServerAdvertise On EnableMCPMReceive ErrorLog /var/log/httpd/error_log TransferLog /var/log/httpd/access_log </VirtualHost>