Infinispan cluster topology rebalance ------------------------------------- Key: WFLY-5135 URL: https://issues.jboss.org/browse/WFLY-5135 Project: WildFly Issue Type: Bug Affects Versions: 9.0.1.Final Environment: CentOS 7, Java 8u51, Wildfly 9.0.1.Final, TCP/TCPPING cluster Reporter: Gabor Auth Assignee: Jason Greene Configured TCPPING based TCP transport: {code} <stack name="tcpping"> <transport type="TCP" socket-binding="jgroups-tcp"/> <protocol type="TCPPING"> <property name="port_range">0</property> <property name="initial_hosts">...</property> </protocol> <protocol type="MERGE2"/> <protocol type="FD_SOCK" socket-binding="jgroups-tcp-fd"/> <protocol type="FD"/> <protocol type="VERIFY_SUSPECT"/> <protocol type="BARRIER"/> <protocol type="pbcast.NAKACK"/> <protocol type="UNICAST2"/> <protocol type="pbcast.STABLE"/> <protocol type="pbcast.GMS"> <property name="join_timeout">3000</property> </protocol> <protocol type="MFC"/> <protocol type="FRAG2"/> <protocol type="RSVP"/> </stack> {code} The cluster works but after some node restart, one of the node try to acquire the coordinator role and fail (the previous coordinator was the dc02-wild01): {code} 2015-08-19 10:00:41,597 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-16,ee,dc01-wild01) ISPN000094: Received new cluster view for channel gacivsContainer: [dc02-wild02|13] (4) [dc02-wild02, dc01-wild01, dc01-wild02, dc02-wild01] 2015-08-19 10:00:41,598 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-16,ee,dc01-wild01) ISPN000094: Received new cluster view for channel web: [dc02-wild02|13] (4) [dc02-wild02, dc01-wild01, dc01-wild02, dc02-wild01] 2015-08-19 10:00:41,598 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-16,ee,dc01-wild01) ISPN000094: Received new cluster view for channel ejb: [dc02-wild02|13] (4) [dc02-wild02, dc01-wild01, dc01-wild02, dc02-wild01] 2015-08-19 10:00:46,448 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-19,ee,dc01-wild01) ISPN000093: Received new, MERGED cluster view for channel gacivsContainer: MergeView::[dc01-wild01|14] (4) [dc01-wild01, dc02-wild01, dc01-wild02, dc02-wild02], 2 subgroups: [dc02-wild02|12] (3) [dc02-wild02, dc01-wild01, dc01-wild02], [dc02-wild02|13] (4) [dc02-wild02, dc01-wild01, dc01-wild02, dc02-wild01] 2015-08-19 10:00:46,450 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-19,ee,dc01-wild01) ISPN000093: Received new, MERGED cluster view for channel web: MergeView::[dc01-wild01|14] (4) [dc01-wild01, dc02-wild01, dc01-wild02, dc02-wild02], 2 subgroups: [dc02-wild02|12] (3) [dc02-wild02, dc01-wild01, dc01-wild02], [dc02-wild02|13] (4) [dc02-wild02, dc01-wild01, dc01-wild02, dc02-wild01] 2015-08-19 10:00:46,450 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-19,ee,dc01-wild01) ISPN000093: Received new, MERGED cluster view for channel ejb: MergeView::[dc01-wild01|14] (4) [dc01-wild01, dc02-wild01, dc01-wild02, dc02-wild02], 2 subgroups: [dc02-wild02|12] (3) [dc02-wild02, dc01-wild01, dc01-wild02], [dc02-wild02|13] (4) [dc02-wild02, dc01-wild01, dc01-wild02, dc02-wild01] 2015-08-19 10:00:46,637 ERROR [org.infinispan.topology.ClusterTopologyManagerImpl] (transport-thread--p4-t8) ISPN000196: Failed to recover cluster state after the current node became the coordinator: org.infinispan.commons.CacheException: Unsuccessful response received from node dc02-wild01: CacheNotFoundResponse at org.infinispan.topology.ClusterTopologyManagerImpl.executeOnClusterSync(ClusterTopologyManagerImpl.java:482) at org.infinispan.topology.ClusterTopologyManagerImpl.recoverClusterStatus(ClusterTopologyManagerImpl.java:350) at org.infinispan.topology.ClusterTopologyManagerImpl.handleClusterView(ClusterTopologyManagerImpl.java:286) at org.infinispan.topology.ClusterTopologyManagerImpl$ClusterViewListener$1.run(ClusterTopologyManagerImpl.java:590) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) {code} And the cluster gone... {code} 2015-08-19 10:10:17,312 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (remote-thread--p5-t49) ISPN000136: Execution error: org.infinispan.util.concurrent.TimeoutException: ISPN000299: Unable to acquire lock after 10 seconds for key 113481734232139653919 and requestor GlobalTransaction:<dc01-wild01>:628:remote. Lock is held by GlobalTransaction:<dc01-wild01>:602:remote, while request came from dc01-wild01 [...] 2015-08-19 10:13:32,295 WARNING [org.jgroups.protocols.TCP] (TransferQueueBundler,ee,dc01-wild01) JGRP000032: dc01-wild01: no physical address for dc02-wild01, dropping message [...] 2015-08-19 10:11:32,707 WARN [org.infinispan.transaction.impl.TransactionTable] (TxCleanupService,dist,dc01-wild02) ISPN000326: Remote transaction GlobalTransaction:<dc02-wild02>:1240:remote timed out. Rolling back after 84623 ms {code}