[infinispan-dev] Put issues with newly joining node

Hey Dan/Adrian, Re: https://issues.jboss.org/browse/ISPN-2541 I'm looking at this intermittent failure, and it seems to be caused by the fact that the test does not wait for the cluster to be formed when the new node is started, which can lead a replication timeout failure from the new joining node. The test can easily be fixed by waiting for cluster to form, and then do the call. But, I think there's an underlying issue, which I'm not sure if it has been fixed already: Node A sends a put to Node-B and newly joined Node-C: test.log:11206:2012-11-23 09:11:29,925 148390 TRACE [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (testng-MarshallExternalPojosTest:org.infinispan.marshall.MarshallExternalPojosTest) dests=null, command=SingleRpcCommand{cacheName='org.infinispan.marshall.MarshallExternalPojosTest', command=PutKeyValueCommand{key=k0-testReplicateJBossExternalizePojoToNewJoiningNode, value=org.infinispan.marshall.PojoWithJBossExternalize@413f7ce9, flags=null, putIfAbsent=false, lifespanMillis=-1, maxIdleTimeMillis=-1}}, mode=SYNCHRONOUS, timeout=15000 Node-C receives the put request but it's in the middle of state transfer, so it forwards it to node B: 2012-11-23 09:11:29,960 148425 TRACE [org.infinispan.statetransfer.StateTransferManagerImpl] (OOB-2,ISPN,MarshallExternalPojosTest-NodeC-10724:org.infinispan.marshall.MarshallExternalPojosTest) Forwarding command PutKeyValueCommand{key=k0-testReplicateJBossExternalizePojoToNewJoiningNode, value=org.infinispan.marshall.PojoWithJBossExternalize@413f7ce9, flags=null, putIfAbsent=false, lifespanMillis=-1, maxIdleTimeMillis=-1} to new targets [MarshallExternalPojosTest-NodeB-22814] 2012-11-23 09:11:29,960 148425 TRACE [org.infinispan.remoting.rpc.RpcManagerImpl] (OOB-2,ISPN,MarshallExternalPojosTest-NodeC-10724:org.infinispan.marshall.MarshallExternalPojosTest) MarshallExternalPojosTest-NodeC-10724 broadcasting call PutKeyValueCommand{key=k0-testReplicateJBossExternalizePojoToNewJoiningNode, value=org.infinispan.marshall.PojoWithJBossExternalize@413f7ce9, flags=null, putIfAbsent=false, lifespanMillis=-1, maxIdleTimeMillis=-1} to recipient list [MarshallExternalPojosTest-NodeB-22814] At the same time, Node-B is forwarding the request to Node-C: 2012-11-23 09:11:29,965 148430 TRACE [org.infinispan.statetransfer.StateTransferManagerImpl] (OOB-2,ISPN,MarshallExternalPojosTest-NodeB-22814:org.infinispan.marshall.MarshallExternalPojosTest) Forwarding command PutKeyValueCommand{key=k0-testReplicateJBossExternalizePojoToNewJoiningNode, value=org.infinispan.marshall.PojoWithJBossExternalize@413f7ce9, flags=null, putIfAbsent=false, lifespanMillis=-1, maxIdleTimeMillis=-1} to new targets [MarshallExternalPojosTest-NodeC-10724] In the end, Node-A does not get a reply from Node-C: 2012-11-23 09:11:44,960 163425 TRACE [org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher] (testng-MarshallExternalPojosTest:org.infinispan.marshall.MarshallExternalPojosTest) Responses: [sender=MarshallExternalPojosTest-NodeC-10724, received=false, suspected=false] Does this ring a bell? Cheers, -- Galder Zamarreño galder(a)redhat.com twitter.com/galderz Project Lead, Escalante http://escalante.io Engineer, Infinispan http://infinispan.org