[infinispan-dev] Put issues with newly joining node

[Galder Zamarreño]

Hey Dan/Adrian,

Re: https://issues.jboss.org/browse/ISPN-2541

I'm looking at this intermittent failure, and it seems to be caused by the fact that the test does not wait for the cluster to be formed when the new node is started, which can lead a replication timeout failure from the new joining node.

The test can easily be fixed by waiting for cluster to form, and then do the call.

But, I think there's an underlying issue, which I'm not sure if it has been fixed already:

Node A sends a put to Node-B and newly joined Node-C:
test.log:11206:2012-11-23 09:11:29,925 148390 TRACE [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (testng-MarshallExternalPojosTest:org.infinispan.marshall.MarshallExternalPojosTest) dests=null, command=SingleRpcCommand{cacheName='org.infinispan.marshall.MarshallExternalPojosTest', command=PutKeyValueCommand{key=k0-testReplicateJBossExternalizePojoToNewJoiningNode, value=org.infinispan.marshall.PojoWithJBossExternalize at 413f7ce9, flags=null, putIfAbsent=false, lifespanMillis=-1, maxIdleTimeMillis=-1}}, mode=SYNCHRONOUS, timeout=15000

Node-C receives the put request but it's in the middle of state transfer, so it forwards it to node B:
2012-11-23 09:11:29,960 148425 TRACE [org.infinispan.statetransfer.StateTransferManagerImpl] (OOB-2,ISPN,MarshallExternalPojosTest-NodeC-10724:org.infinispan.marshall.MarshallExternalPojosTest) Forwarding command PutKeyValueCommand{key=k0-testReplicateJBossExternalizePojoToNewJoiningNode, value=org.infinispan.marshall.PojoWithJBossExternalize at 413f7ce9, flags=null, putIfAbsent=false, lifespanMillis=-1, maxIdleTimeMillis=-1} to new targets [MarshallExternalPojosTest-NodeB-22814]
2012-11-23 09:11:29,960 148425 TRACE [org.infinispan.remoting.rpc.RpcManagerImpl] (OOB-2,ISPN,MarshallExternalPojosTest-NodeC-10724:org.infinispan.marshall.MarshallExternalPojosTest) MarshallExternalPojosTest-NodeC-10724 broadcasting call PutKeyValueCommand{key=k0-testReplicateJBossExternalizePojoToNewJoiningNode, value=org.infinispan.marshall.PojoWithJBossExternalize at 413f7ce9, flags=null, putIfAbsent=false, lifespanMillis=-1, maxIdleTimeMillis=-1} to recipient list [MarshallExternalPojosTest-NodeB-22814]

At the same time, Node-B is forwarding the request to Node-C:
2012-11-23 09:11:29,965 148430 TRACE [org.infinispan.statetransfer.StateTransferManagerImpl] (OOB-2,ISPN,MarshallExternalPojosTest-NodeB-22814:org.infinispan.marshall.MarshallExternalPojosTest) Forwarding command PutKeyValueCommand{key=k0-testReplicateJBossExternalizePojoToNewJoiningNode, value=org.infinispan.marshall.PojoWithJBossExternalize at 413f7ce9, flags=null, putIfAbsent=false, lifespanMillis=-1, maxIdleTimeMillis=-1} to new targets [MarshallExternalPojosTest-NodeC-10724]

In the end, Node-A does not get a reply from Node-C:
2012-11-23 09:11:44,960 163425 TRACE [org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher] (testng-MarshallExternalPojosTest:org.infinispan.marshall.MarshallExternalPojosTest) Responses: [sender=MarshallExternalPojosTest-NodeC-10724, received=false, suspected=false]

Does this ring a bell?

Cheers,
--
Galder Zamarreño
galder at redhat.com
twitter.com/galderz

Project Lead, Escalante
http://escalante.io

Engineer, Infinispan
http://infinispan.org