[infinispan-issues] [JBoss JIRA] (ISPN-3366) Data loss when entry forwarding to primary owner and primary owner shutdown

[Radim Vansa (JIRA)]

    [ https://issues.jboss.org/browse/ISPN-3366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12794538#comment-12794538 ] 

Radim Vansa commented on ISPN-3366:
-----------------------------------

I have a test-case in library mode (ispn 5.2.4) showing this bug - 4 nodes up, 3 of them loading data (100000 entries in total) via normal PUT operations. During the loading one of the nodes is restarted. If the put operation on some of the loading nodes gets exception, the put is repeated, but even with that in the end I end up with few keys returning null on one of the nodes.

[~dan.berindei]: please let me know as you'll have some fix for this issue, I'll try to test it.
                
> Data loss when entry forwarding to primary owner and primary owner shutdown
> ---------------------------------------------------------------------------
>
>                 Key: ISPN-3366
>                 URL: https://issues.jboss.org/browse/ISPN-3366
>             Project: Infinispan
>          Issue Type: Bug
>          Components: Distributed Cache
>    Affects Versions: 5.2.4.Final, 6.0.0.Alpha1
>            Reporter: Takayoshi Kimura
>            Assignee: Dan Berindei
>            Priority: Critical
>         Attachments: ISPN-3366-logs.zip
>
>
>   Looks like a problem in entry forwarding.
> Here is test scenario:
> * DIST numOwners=2, start with 4 nodes cluster then normal shutdown 1 node during load
> * HotRod putIfAbsent accesses from 40 threads (1 process, 1 remote cache instance), 40000 entries total
> After the test run, the numberOfEntries on each node are:
> * node1: 26608
> * node2: 26622
> * node3: 26746
> * node4: 0
> Total is 79976 and HotRod client received 11 errors, so 79976 + (11 * 2) = 79998. It means 1 entry is completely missing.
> Let's take a look at the missing entry, hash(thread16key59) = 574ff563.
> Current CH: owners(574ff563) are [node4, node1]
> The events sequence is:
> * hotrod -> node1
> * node1 forwarding it to primary owner node4
> * node4 doesn't process the forwarded entry, shutdown
> Result owners(7c29bccb) is [] empty. This entry is completely lost without any errors.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira