[ https://issues.jboss.org/browse/ISPN-1806?page=com.atlassian.jira.plugin.... ] Paul Ferraro updated ISPN-1806: ------------------------------- Attachment: org.jboss.as.test.clustering.unmanaged.singleton.SingletonTestCase-output.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/ISPN-1806?page=com.atlassian.jira.plugin.... ] Manik Surtani updated ISPN-1806: -------------------------------- Fix Version/s: (was: 5.1.1.FINAL) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/ISPN-1806?page=com.atlassian.jira.plugin.... ] Paul Ferraro commented on ISPN-1806: ------------------------------------ Hmm - this may be related to ISPN-1731. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/ISPN-1806?page=com.atlassian.jira.plugin.... ] Dan Berindei commented on ISPN-1806: ------------------------------------ According to the attached log there there is no problem with the commit/StateTransferInProgressException - the transaction thread is blocked because CacheViewsManagerImpl is not able to install a new cache view. I created a separate issue to describe the problem: ISPN-1814. There is another problem apparent in the log: JGroups apparently didn't exclude the killed cluster member from the view even after it failed to ACK the new view: {noformat} 20:22:49,552 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-13,null) ISPN000094: Received new cluster view: [node-udp-0/cluster|4] [node-udp-0/cluster, node-udp-1/cluster, node-udp-1/cluster] 20:22:52,497 WARNING [org.jgroups.protocols.pbcast.GMS] (pool-5-thread-1) JOIN(node-udp-1/cluster) sent to node-udp-0/cluster timed out (after 3000 ms), retrying 20:22:54,549 WARNING [org.jgroups.protocols.pbcast.GMS] (ViewHandler,cluster,node-udp-0/cluster) node-udp-0/cluster: failed to collect all ACKs (expected=2) for view [node-udp-0/cluster|4] after 5000ms, missing ACKs from [node-udp-1/cluster] 20:22:54,835 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (pool-9-thread-1) ISPN000094: Received new cluster view: [node-udp-0/cluster|4] [node-udp-0/cluster, 1903f28a-f7d1-1488-62a3-03a2df0f5b62, node-udp-1/cluster] ... 20:23:20,174 INFO [org.jboss.as.clustering.CoreGroupCommunicationService.cluster] (VERIFY_SUSPECT.TimerThread,cluster,node-udp-1/cluster) JBAS010254: Suspected member: 1903f28a-f7d1-1488-62a3-03a2df0f5b62 {noformat} I'm not sure why this happens - is FD/FD_ALL enabled in the JGroups configuration? And finally, there are some issues with the log itself: * The log messages from Incoming threads have a 'null' instead of the node name. Not sure if it's related to the (caught) exception in {{CoreGroupCommunicationService$MembershipListenerImpl.viewAccepted}}, since the OOB threads look fine. * The restarted node should start each time with a different name, it's hard to understand what's happening with two {{node-udp-1/cluster}} s in the same cluster. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/ISPN-1806?page=com.atlassian.jira.plugin.... ] Dan Berindei commented on ISPN-1806: ------------------------------------ FD_SOCK should have suspected the killed member immediately, since you killed the process the OS would have automatically closed all its sockets. I'll see if I can produce a simpler test case and ask Bela about it. Address.toString() *is* the logical name - the UUID is only printed if JGroups doesn't have a logical name for the address. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/ISPN-1806?page=com.atlassian.jira.plugin.... ] Dan Berindei commented on ISPN-1806: ------------------------------------ I missed something in the log, in fact FD_SOCK did suspect `node-udp-1` and it got kicked out of the cluster. However, it appears as if the node re-joins the cluster 50ms after it was killed: {noformat} 20:22:36,602 INFO [org.jboss.as.clustering.CoreGroupCommunicationService.lifecycle.cluster] (Incoming-9,null) JBAS010247: New cluster view for partition cluster (id: 2, delta: -1, merge: false) : [node-udp-0/cluster] 20:22:36,603 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-9,null) ISPN000094: Received new cluster view: [node-udp-0/cluster|2] [node-udp-0/cluster] 20:22:36,659 INFO [org.jboss.as.clustering.CoreGroupCommunicationService.lifecycle.cluster] (Incoming-11,null) JBAS010247: New cluster view for partition cluster (id: 3, delta: 1, merge: false) : [node-udp-0/cluster, node-udp-1/cluster] 20:22:36,659 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-11,null) ISPN000094: Received new cluster view: [node-udp-0/cluster|3] [node-udp-0/cluster, node-udp-1/cluster] {noformat} After 10 seconds `node-udp-1` is restarted and it properly joins the cluster the second time. The "ghost" of the previous `node-udp-1` instance is still there: {noformat} 20:22:49,479 INFO [stdout] (pool-5-thread-1) GMS: address=node-udp-1/cluster, cluster=cluster, physical address=127.0.0.1:55300 20:22:49,549 INFO [org.jboss.as.clustering.CoreGroupCommunicationService.lifecycle.cluster] (Incoming-13,null) JBAS010247: New cluster view for partition cluster (id: 4, delta: 1, merge: false) : [node-udp-0/cluster, node-udp-1/cluster, node-udp-1/cluster] 20:22:49,552 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-13,null) ISPN000094: Received new cluster view: [node-udp-0/cluster|4] [node-udp-0/cluster, node-udp-1/cluster, node-udp-1/cluster {noformat} 30 seconds after `node-udp-1` was killed, FD kicks it out of the cluster again: {noformat} 20:23:20,224 INFO [org.jboss.as.clustering.CoreGroupCommunicationService.lifecycle.cluster] (Incoming-15,null) JBAS010247: New cluster view for partition cluster (id: 5, delta: -1, merge: false) : [node-udp-0/cluster, node-udp-1/cluster] {noformat} This looks like it could be a JGroups issue, but I don't think it could have caused the hang-up - the cause is still ISPN-1814. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/ISPN-1806?page=com.atlassian.jira.plugin.... ] Dan Berindei resolved ISPN-1806. -------------------------------- Fix Version/s: 5.2.0.FINAL 5.1.1.FINAL (was: 5.1.2.CR1) (was: 5.1.2.FINAL) Resolution: Done The specific problem in the attached log has been fixed with ISPN-1814. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira