[infinispan-issues] [JBoss JIRA] (ISPN-3366) Data loss when entry forwarding to primary owner and primary owner shutdown

[Mircea Markus (JIRA)]

    [ https://issues.jboss.org/browse/ISPN-3366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12793154#comment-12793154 ] 

Mircea Markus commented on ISPN-3366:
-------------------------------------

Interesting enough, the hash code of the key in N4 is different than the hash code of the key in N1:
N1:
{quote}
2013-07-25:13:26:02,242 TRACE [org.infinispan.interceptors.InvocationContextInterceptor] (HotRodServerWorker-15) Invoked with command PutKeyValueCommand{key=ByteArrayKey{data=ByteArray{size=16, hashCode=*574ff563*, array=*0x033e0d7468726561..*}}, value=CacheValue{data=ByteArray{size=8, array=0x033e0576616c7565..}, version=844437815034540
{quote}

N4:
{quote}
2013-07-25:13:26:01,999 TRACE [org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher] (OOB-188,shared=tcp) Attempting to execute command: SingleRpcCommand{cacheName='1', command=PutKeyValueCommand{key=ByteArrayKey{data=ByteArray{size=16, hashCode=*19e17f1d*, array=*0x033e0d7468726561..*}}, ...
{quote}
                
> Data loss when entry forwarding to primary owner and primary owner shutdown
> ---------------------------------------------------------------------------
>
>                 Key: ISPN-3366
>                 URL: https://issues.jboss.org/browse/ISPN-3366
>             Project: Infinispan
>          Issue Type: Bug
>          Components: Distributed Cache
>    Affects Versions: 5.2.4.Final, 6.0.0.Alpha1
>            Reporter: Takayoshi Kimura
>            Assignee: Dan Berindei
>            Priority: Critical
>         Attachments: ISPN-3366-logs.zip
>
>
> Looks like a problem in entry forwarding.
> Here is test scenario:
> * DIST numOwners=2, start with 4 nodes cluster then normal shutdown 1 node during load
> * HotRod putIfAbsent accesses from 40 threads (1 process, 1 remote cache instance), 40000 entries total
> After the test run, the numberOfEntries on each node are:
> * node1: 26608
> * node2: 26622
> * node3: 26746
> * node4: 0
> Total is 79976 and HotRod client received 11 errors, so 79976 + (11 * 2) = 79998. It means 1 entry is completely missing.
> Let's take a look at the missing entry, hash(thread16key59) = 574ff563.
> Current CH: owners(574ff563) are [node4, node1]
> The events sequence is:
> * hotrod -> node1
> * node1 forwarding it to primary owner node4
> * node4 doesn't process the forwarded entry, shutdown
> Result owners(7c29bccb) is [] empty. This entry is completely lost without any errors.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira