[ https://issues.jboss.org/browse/ISPN-3918?page=com.atlassian.jira.plugin.... ] Dan Berindei edited comment on ISPN-3918 at 8/10/17 4:20 AM: ------------------------------------------------------------- I found another failure mode while trying to work around this issue in {{NonTxPutIfAbsentDuringLeaveStressTest}} (I was also trying to merge it with {{NonTxPutIfAbsentDuringJoinStressTest}} into a {{NonTxPutIfAbsentDuringRebalanceStressTest}}, so the stack traces in the attached log _NonTxPutIfAbsentDuringRebalanceStressTest.testPutIfAbsentDuringJoin_1.log.gz_ don't match master). Say {{owners(k) = AB}} in topology {{t}}, and {{owners(k) = BA}} in topology {{t+1}}. In the following scenario, {{putIfAbsent}} fails in thread _C-app1_ and _C-app2_ with different values: # _C-app1_: start putIfAbsent(k, v1) # _C-app1_: send putIfAbsent(k, v1) to A (primary) # _A-remote1_: receive putIfAbsent(k, v1) # _A-remote1_: send backup request for putIfAbsent(k, v1) to B # _A-remote1_: write k=v1 # _C-remote1_: receive null for putIfAbsent(k, v1) # _C-app2_: start putIfAbsent(k, v2) # _C-app2_: send putIfAbsent(k, v2) to A (primary) # _A-remote2_: receive putIfAbsent(k, v1) # _A-remote2_: read v1 from data container, fail the command # _C-remote2_: receive v1 for putIfAbsent(k, v2) # *_C-app2_: return v1 (put was unsuccessful)* # _B-remote1_: install topology t+1, B is now primary owner # _B-remote1_: receive backup request for putIfAbsent(k, v1) # _B-remote1_: check topology, send OutdatedTopologyException ack to C # _C-remote3_: install topology t+1 # _C-app3_: start putIfAbsent(k, v3) # _C-app3_: send putIfAbsent(k, v3) to B (primary) # _B-remote3_: receive putIfAbsent(k, v3) # _B-remote3_: send backup request for putIfAbsent(k, 3) to A # _B-remote3_: write k=v3 # _C-remote3_: receive null for putIfAbsent(k, v3) # _A-remote3_: receive backup request for putIfAbsent(k, v3) # _A-remote3_: write k=v3 # _A-remote3_: send backup ack for putIfAbsent(k, v3) to C # _C-remote3_: receive backup ack for putIfAbsent(k, v3) # *_C-app3_: return null (put was successful)* # _B-remote1_: retry, B is now primary owner # _B-remote1_: read v3 from data container, fail the command # _C-remote1_: receive v3 for putIfAbsent(k, v1) # *_C-app1_: return v3 (put was unsuccessful)* I think both this scenario and the previous one are worse than the initial report of seeing {{null}}, because we're not respecting the {{READ_COMMITTED}} isolation level (a transaction is seeing a value that was never committed). was (Author: dan.berindei): I found another failure mode while trying to work around this issue in {{NonTxPutIfAbsentDuringLeaveStressTest}} (I was also trying to merge it with {{NonTxPutIfAbsentDuringJoinStressTest}} into a {{NonTxPutIfAbsentDuringRebalanceStressTest}}, so the stack traces in the attached log _NonTxPutIfAbsentDuringRebalanceStressTest.testPutIfAbsentDuringJoin_1.log.gz_ don't match master). Say {{owners(k) = AB}} in topology {{t}}, and {{owners(k) = BA}} in topology {{t+1}}. In the following scenario, {{putIfAbsent}} fails in thread _C-app1_ and _C-app2_ with different values: # _C-app1_: start putIfAbsent(k, v1) # _C-app1_: send putIfAbsent(k, v1) to A (primary) # _A-remote1_: receive putIfAbsent(k, v1) # _A-remote1_: send backup request for putIfAbsent(k, v1) to B # _A-remote1_: write k=v1 # _C-remote1_: receive null for putIfAbsent(k, v1) # _C-app2_: start putIfAbsent(k, v2) # _C-app2_: send putIfAbsent(k, v2) to A (primary) # _A-remote2_: receive putIfAbsent(k, v1) # _A-remote2_: read v1 from data container, fail the command # _C-remote2_: receive v1 for putIfAbsent(k, v2) # *_C-app2_: return v1 (put was unsuccessful)* # _B-remote1_: install topology t+1, B is now primary owner # _B-remote1_: receive backup request for putIfAbsent(k, v1) # _B-remote1_: check topology, send OutdatedTopologyException ack to C # _C-remote3_: install topology t+1 # _C-app3_: start putIfAbsent(k, v3) # _C-app3_: send putIfAbsent(k, v3) to B (primary) # _B-remote3_: receive putIfAbsent(k, v3) # _B-remote3_: send backup request for putIfAbsent(k, 3) to A # _B-remote3_: write k=v3 # _C-remote3_: receive null for putIfAbsent(k, v3) # _A-remote3_: receive backup request for putIfAbsent(k, v3) # _A-remote3_: write k=v3 # _A-remote3_: send backup ack for putIfAbsent(k, v3) to C # _C-remote3_: receive backup ack for putIfAbsent(k, v3) # *_C-app3_: return null (put was successful)* # _B-remote1_: retry, B is now primary owner # _B-remote1_: read v2 from data container, fail the command # _C-remote1_: receive v2 for putIfAbsent(k, v1) # *_C-app1_: return v2 (put was unsuccessful)* I think both this scenario and the previous one are worse than the initial report of seeing {{null}}, because we're not respecting the {{READ_COMMITTED}} isolation level (a transaction is seeing a value that was never committed). -- This message was sent by Atlassian JIRA (v7.2.3#72005)