[jboss-jira] [JBoss JIRA] (WFLY-5135) Infinispan cluster topology rebalance

[Gabor Auth (JIRA)]

Gabor Auth created WFLY-5135:
--------------------------------

             Summary: Infinispan cluster topology rebalance
                 Key: WFLY-5135
                 URL: https://issues.jboss.org/browse/WFLY-5135
             Project: WildFly
          Issue Type: Bug
    Affects Versions: 9.0.1.Final
         Environment: CentOS 7, Java 8u51, Wildfly 9.0.1.Final, TCP/TCPPING cluster
            Reporter: Gabor Auth
            Assignee: Jason Greene


Configured TCPPING based TCP transport:
{code}
<stack name="tcpping">
  <transport type="TCP" socket-binding="jgroups-tcp"/>
  <protocol type="TCPPING">
    <property name="port_range">0</property>
    <property name="initial_hosts">...</property>
  </protocol>
  <protocol type="MERGE2"/>
  <protocol type="FD_SOCK" socket-binding="jgroups-tcp-fd"/>
  <protocol type="FD"/>
  <protocol type="VERIFY_SUSPECT"/>
  <protocol type="BARRIER"/>
  <protocol type="pbcast.NAKACK"/>
  <protocol type="UNICAST2"/>
  <protocol type="pbcast.STABLE"/>
  <protocol type="pbcast.GMS">
    <property name="join_timeout">3000</property>
  </protocol>
  <protocol type="MFC"/>
  <protocol type="FRAG2"/>
  <protocol type="RSVP"/>
</stack>
{code}

The cluster works but after some node restart, one of the node try to acquire the coordinator role and fail (the previous coordinator was the dc02-wild01):
{code}
2015-08-19 10:00:41,597 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-16,ee,dc01-wild01) ISPN000094: Received new cluster view for channel gacivsContainer: [dc02-wild02|13] (4) [dc02-wild02, dc01-wild01, dc01-wild02, dc02-wild01]
2015-08-19 10:00:41,598 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-16,ee,dc01-wild01) ISPN000094: Received new cluster view for channel web: [dc02-wild02|13] (4) [dc02-wild02, dc01-wild01, dc01-wild02, dc02-wild01]
2015-08-19 10:00:41,598 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-16,ee,dc01-wild01) ISPN000094: Received new cluster view for channel ejb: [dc02-wild02|13] (4) [dc02-wild02, dc01-wild01, dc01-wild02, dc02-wild01]
2015-08-19 10:00:46,448 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-19,ee,dc01-wild01) ISPN000093: Received new, MERGED cluster view for channel gacivsContainer: MergeView::[dc01-wild01|14] (4) [dc01-wild01, dc02-wild01, dc01-wild02, dc02-wild02], 2 subgroups: [dc02-wild02|12] (3) [dc02-wild02, dc01-wild01, dc01-wild02], [dc02-wild02|13] (4) [dc02-wild02, dc01-wild01, dc01-wild02, dc02-wild01]
2015-08-19 10:00:46,450 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-19,ee,dc01-wild01) ISPN000093: Received new, MERGED cluster view for channel web: MergeView::[dc01-wild01|14] (4) [dc01-wild01, dc02-wild01, dc01-wild02, dc02-wild02], 2 subgroups: [dc02-wild02|12] (3) [dc02-wild02, dc01-wild01, dc01-wild02], [dc02-wild02|13] (4) [dc02-wild02, dc01-wild01, dc01-wild02, dc02-wild01]
2015-08-19 10:00:46,450 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-19,ee,dc01-wild01) ISPN000093: Received new, MERGED cluster view for channel ejb: MergeView::[dc01-wild01|14] (4) [dc01-wild01, dc02-wild01, dc01-wild02, dc02-wild02], 2 subgroups: [dc02-wild02|12] (3) [dc02-wild02, dc01-wild01, dc01-wild02], [dc02-wild02|13] (4) [dc02-wild02, dc01-wild01, dc01-wild02, dc02-wild01]
2015-08-19 10:00:46,637 ERROR [org.infinispan.topology.ClusterTopologyManagerImpl] (transport-thread--p4-t8) ISPN000196: Failed to recover cluster state after the current node became the coordinator: org.infinispan.commons.CacheException: Unsuccessful response received from node dc02-wild01: CacheNotFoundResponse
        at org.infinispan.topology.ClusterTopologyManagerImpl.executeOnClusterSync(ClusterTopologyManagerImpl.java:482)
        at org.infinispan.topology.ClusterTopologyManagerImpl.recoverClusterStatus(ClusterTopologyManagerImpl.java:350)
        at org.infinispan.topology.ClusterTopologyManagerImpl.handleClusterView(ClusterTopologyManagerImpl.java:286)
        at org.infinispan.topology.ClusterTopologyManagerImpl$ClusterViewListener$1.run(ClusterTopologyManagerImpl.java:590)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
{code}

And the cluster gone...
{code}
2015-08-19 10:10:17,312 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (remote-thread--p5-t49) ISPN000136: Execution error: org.infinispan.util.concurrent.TimeoutException: ISPN000299: Unable to acquire lock after 10 seconds for key 113481734232139653919 and requestor GlobalTransaction:<dc01-wild01>:628:remote. Lock is held by GlobalTransaction:<dc01-wild01>:602:remote, while request came from dc01-wild01
[...]
2015-08-19 10:13:32,295 WARNING [org.jgroups.protocols.TCP] (TransferQueueBundler,ee,dc01-wild01) JGRP000032: dc01-wild01: no physical address for dc02-wild01, dropping message
[...]
2015-08-19 10:11:32,707 WARN  [org.infinispan.transaction.impl.TransactionTable] (TxCleanupService,dist,dc01-wild02) ISPN000326: Remote transaction GlobalTransaction:<dc02-wild02>:1240:remote timed out. Rolling back after 84623 ms
{code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)