[JBoss JIRA] (ISPN-2736) ConcurrentModificationException in StateRequestCommand
by Adrian Nistor (JIRA)
[ https://issues.jboss.org/browse/ISPN-2736?page=com.atlassian.jira.plugin.... ]
Adrian Nistor updated ISPN-2736:
--------------------------------
Status: Pull Request Sent (was: Coding In Progress)
Git Pull Request: https://github.com/infinispan/infinispan/pull/1600
> ConcurrentModificationException in StateRequestCommand
> -------------------------------------------------------
>
> Key: ISPN-2736
> URL: https://issues.jboss.org/browse/ISPN-2736
> Project: Infinispan
> Issue Type: Bug
> Components: State transfer
> Affects Versions: 5.2.0.CR2
> Reporter: Radim Vansa
> Assignee: Adrian Nistor
> Priority: Critical
> Fix For: 5.2.0.Final
>
>
> I have tried async replication scenario with async marshalling (optimistic transactions) and this has occurred:
> {code}
> 04:34:15,182 ERROR [org.infinispan.remoting.rpc.RpcManagerImpl] (OOB-77,edg-perf11-48420) ISPN000073: Unexpected error while replicating
> java.util.ConcurrentModificationException
> at java.util.HashMap$HashIterator.nextEntry(HashMap.java:793)
> at java.util.HashMap$KeyIterator.next(HashMap.java:828)
> at org.infinispan.statetransfer.StateProviderImpl.collectTransactionsToTransfer(StateProviderImpl.java:239)
> at org.infinispan.statetransfer.StateProviderImpl.getTransactionsForSegments(StateProviderImpl.java:200)
> at org.infinispan.statetransfer.StateRequestCommand.perform(StateRequestCommand.java:88)
> at org.infinispan.remoting.InboundInvocationHandlerImpl.handleInternal(InboundInvocationHandlerImpl.java:101)
> at org.infinispan.remoting.InboundInvocationHandlerImpl.handleWithWaitForBlocks(InboundInvocationHandlerImpl.java:122)
> at org.infinispan.remoting.InboundInvocationHandlerImpl.handle(InboundInvocationHandlerImpl.java:86)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommandFromLocalCluster(CommandAwareRpcDispatcher.java:245)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:218)
> at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:484)
> at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:391)
> at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:249)
> at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:598)
> at org.jgroups.JChannel.up(JChannel.java:707)
> {code}
> Seems that StateRequestCommand is iterating through all transactions' locked keys but the collections of these locked keys can be modified by interfering prepare/commit.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 11 months
[JBoss JIRA] (ISPN-2589) NPE in InvalidateL1Command
by Michal Linhard (JIRA)
[ https://issues.jboss.org/browse/ISPN-2589?page=com.atlassian.jira.plugin.... ]
Michal Linhard commented on ISPN-2589:
--------------------------------------
No probs. Does the link http://dev39.mw.lab.eng.bos.redhat.com/~mlinhard/hyperion3/run0037-startu... work for you ? if not I'll paste here
> NPE in InvalidateL1Command
> --------------------------
>
> Key: ISPN-2589
> URL: https://issues.jboss.org/browse/ISPN-2589
> Project: Infinispan
> Issue Type: Bug
> Components: Core API, State transfer
> Affects Versions: 5.2.0.Beta5
> Reporter: Thomas Fromm
> Assignee: Dan Berindei
> Fix For: 5.3.0.Final
>
> Attachments: standalone_node0001.xml
>
>
> The used cache is an REPL_SYNC one w/o L1 and Optimistic TX.
> Can't provide unit test, just saw it in my logs.
> WARN 05.12.12 20:08:22,746 [OOB-173,IBIS-2147] CacheTopologyControlCommand ISPN000071: Caught exception when handling command CacheTopologyControlCommand{cache=MODULE_PROPERTIES_VERSION, type=CH_UPDATE, sender=IBIS-57850, joinInfo=null, topologyId=14, currentCH=ReplicatedConsistentHash{members=[IBIS-57850, IBIS-15651, IBIS-14611, IBIS-7942, IBIS-4256, IBIS-25472, IBIS-32523]}, pendingCH=null, throwable=null, viewId=8}
> java.lang.NullPointerException
> at org.infinispan.commands.write.InvalidateL1Command.perform(InvalidateL1Command.java:109)
> at org.infinispan.interceptors.CallInterceptor.handleDefault(CallInterceptor.java:110)
> at org.infinispan.commands.AbstractVisitor.visitInvalidateCommand(AbstractVisitor.java:141)
> at org.infinispan.commands.AbstractVisitor.visitInvalidateL1Command(AbstractVisitor.java:146)
> at org.infinispan.commands.write.InvalidateL1Command.acceptVisitor(InvalidateL1Command.java:192)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
> at org.infinispan.commands.AbstractVisitor.visitInvalidateCommand(AbstractVisitor.java:141)
> at org.infinispan.commands.AbstractVisitor.visitInvalidateL1Command(AbstractVisitor.java:146)
> at org.infinispan.commands.write.InvalidateL1Command.acceptVisitor(InvalidateL1Command.java:192)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
> at org.infinispan.commands.AbstractVisitor.visitInvalidateCommand(AbstractVisitor.java:141)
> at org.infinispan.commands.AbstractVisitor.visitInvalidateL1Command(AbstractVisitor.java:146)
> at org.infinispan.commands.write.InvalidateL1Command.acceptVisitor(InvalidateL1Command.java:192)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.CacheLoaderInterceptor.visitInvalidateCommand(CacheLoaderInterceptor.java:127)
> at org.infinispan.commands.AbstractVisitor.visitInvalidateL1Command(AbstractVisitor.java:146)
> at org.infinispan.commands.write.InvalidateL1Command.acceptVisitor(InvalidateL1Command.java:192)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.EntryWrappingInterceptor.invokeNextAndApplyChanges(EntryWrappingInterceptor.java:211)
> at org.infinispan.interceptors.EntryWrappingInterceptor.visitInvalidateL1Command(EntryWrappingInterceptor.java:143)
> at org.infinispan.commands.write.InvalidateL1Command.acceptVisitor(InvalidateL1Command.java:192)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitInvalidateL1Command(AbstractLockingInterceptor.java:99)
> at org.infinispan.commands.write.InvalidateL1Command.acceptVisitor(InvalidateL1Command.java:192)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
> at org.infinispan.commands.AbstractVisitor.visitInvalidateCommand(AbstractVisitor.java:141)
> at org.infinispan.commands.AbstractVisitor.visitInvalidateL1Command(AbstractVisitor.java:146)
> at org.infinispan.commands.write.InvalidateL1Command.acceptVisitor(InvalidateL1Command.java:192)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.TxInterceptor.enlistWriteAndInvokeNext(TxInterceptor.java:256)
> at org.infinispan.interceptors.TxInterceptor.visitInvalidateCommand(TxInterceptor.java:224)
> at org.infinispan.commands.AbstractVisitor.visitInvalidateL1Command(AbstractVisitor.java:146)
> at org.infinispan.commands.write.InvalidateL1Command.acceptVisitor(InvalidateL1Command.java:192)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.statetransfer.StateTransferInterceptor.visitInvalidateL1Command(StateTransferInterceptor.java:172)
> at org.infinispan.commands.write.InvalidateL1Command.acceptVisitor(InvalidateL1Command.java:192)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
> at org.infinispan.commands.AbstractVisitor.visitInvalidateCommand(AbstractVisitor.java:141)
> at org.infinispan.commands.AbstractVisitor.visitInvalidateL1Command(AbstractVisitor.java:146)
> at org.infinispan.commands.write.InvalidateL1Command.acceptVisitor(InvalidateL1Command.java:192)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:128)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:92)
> at org.infinispan.commands.AbstractVisitor.visitInvalidateCommand(AbstractVisitor.java:141)
> at org.infinispan.commands.AbstractVisitor.visitInvalidateL1Command(AbstractVisitor.java:146)
> at org.infinispan.commands.write.InvalidateL1Command.acceptVisitor(InvalidateL1Command.java:192)
> at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:343)
> at org.infinispan.statetransfer.StateConsumerImpl.invalidateSegments(StateConsumerImpl.java:592)
> at org.infinispan.statetransfer.StateConsumerImpl.onTopologyUpdate(StateConsumerImpl.java:239)
> at org.infinispan.statetransfer.StateTransferManagerImpl.doTopologyUpdate(StateTransferManagerImpl.java:191)
> at org.infinispan.statetransfer.StateTransferManagerImpl.access$000(StateTransferManagerImpl.java:58)
> at org.infinispan.statetransfer.StateTransferManagerImpl$1.updateConsistentHash(StateTransferManagerImpl.java:117)
> at org.infinispan.topology.LocalTopologyManagerImpl.handleConsistentHashUpdate(LocalTopologyManagerImpl.java:194)
> at org.infinispan.topology.CacheTopologyControlCommand.doPerform(CacheTopologyControlCommand.java:165)
> at org.infinispan.topology.CacheTopologyControlCommand.perform(CacheTopologyControlCommand.java:137)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommandFromLocalCluster(CommandAwareRpcDispatcher.java:252)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:219)
> at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:483)
> at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:390)
> at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:248)
> at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:598)
> at org.jgroups.JChannel.up(JChannel.java:703)
> at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1020)
> at org.jgroups.protocols.RSVP.up(RSVP.java:172)
> at org.jgroups.protocols.FRAG2.up(FRAG2.java:181)
> at org.jgroups.protocols.FC.up(FC.java:479)
> at org.jgroups.protocols.pbcast.GMS.up(GMS.java:896)
> at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:244)
> at org.jgroups.protocols.UNICAST2.up(UNICAST2.java:432)
> at org.jgroups.protocols.pbcast.NAKACK2.handleMessage(NAKACK2.java:721)
> at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:574)
> at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:143)
> at org.jgroups.protocols.FD_ALL.up(FD_ALL.java:187)
> at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:288)
> at org.jgroups.protocols.MERGE3.up(MERGE3.java:290)
> at org.jgroups.protocols.Discovery.up(Discovery.java:359)
> at org.jgroups.protocols.TP.passMessageUp(TP.java:1287)
> at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1850)
> at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1823)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 11 months
[JBoss JIRA] (ISPN-2688) BaseDistributionInterceptor and TxDistributionInterceptor do not detect properly if the key needs to be fetched remotely
by Adrian Nistor (JIRA)
[ https://issues.jboss.org/browse/ISPN-2688?page=com.atlassian.jira.plugin.... ]
Adrian Nistor updated ISPN-2688:
--------------------------------
Summary: BaseDistributionInterceptor and TxDistributionInterceptor do not detect properly if the key needs to be fetched remotely (was: StateTransferLargeObjectTest.testForFailure still failing randomly)
Description:
BaseDistributionInterceptor.shouldFetchFromRemote() and TxDistributionInterceptor.remoteGetAndStoreInL1() can mistakenly decide not to fetch remotely because they check the presence of the key in data container. The key may be there _now_ but it was not there before the local execution if state transfer was in progress for this key. So it should be re-fetched rather than use the null result.
This makes StateTransferLargeObjectTest.testForFailure fail randomly.
The failure appears because the state transfer has not finished, yet the distribution interceptor doesn't go remotely for the key:
{noformat}
14:06:45,872 TRACE (asyncTransportThread-1,NodeA:___defaultcache) [StateTransferManagerImpl] Installing new cache topology CacheTopology{id=7, currentCH=DefaultConsistentHash{numSegments=60, numOwners=3, members=[NodeA-52814, NodeB-62397, NodeC-63995], owners={0: 0 1 2, 1: 0 1 2, 2: 0 1 2, 3: 0 1 2, 4: 0 1 2, 5: 0 1 2, 6: 0 1 2, 7: 0 1 2, 8: 0 1 2, 9: 0 1 2, 10: 0 1 2, 11: 0 1 2, 12: 0 2, 13: 0 2, 14: 0 2, 15: 0 1, 16: 0 1, 17: 0 1, 18: 0 1, 19: 0 1, 20: 2 0, 21: 2 0, 22: 2 0, 23: 2 0, 24: 2 0, 25: 2 0, 26: 2 0, 27: 2 0, 28: 2 0, 29: 2 0, 30: 1 0 2, 31: 1 0 2, 32: 1 0 2, 33: 1 2, 34: 1 0, 35: 1 2, 36: 1 0, 37: 1 2, 38: 1 0, 39: 1 2, 40: 1 0, 41: 1 2, 42: 1 0, 43: 1 2, 44: 1 0, 45: 1 0, 46: 1 0, 47: 1 0, 48: 1 0, 49: 1 2, 50: 2 1, 51: 2 1, 52: 2 1, 53: 2 1, 54: 2 1, 55: 2 1, 56: 2 1, 57: 2 1, 58: 2 0, 59: 2 0}, pendingCH=DefaultConsistentHash{numSegments=60, numOwners=3, members=[NodeA-52814, NodeB-62397, NodeC-63995], owners={0: 0 1 2, 1: 0 1 2, 2: 0 1 2, 3: 0 1 2, 4: 0 1 2, 5: 0 1 2, 6: 0 1 2, 7: 0 1 2, 8: 0 1 2, 9: 0 1 2, 10: 0 1 2, 11: 0 1 2, 12: 0 2 1, 13: 0 2 1, 14: 0 2 1, 15: 0 1 2, 16: 0 1 2, 17: 0 1 2, 18: 0 1 2, 19: 0 1 2, 20: 2 0 1, 21: 2 0 1, 22: 2 0 1, 23: 2 0 1, 24: 2 0 1, 25: 2 0 1, 26: 2 0 1, 27: 2 0 1, 28: 2 0 1, 29: 2 0 1, 30: 1 0 2, 31: 1 0 2, 32: 1 0 2, 33: 1 2 0, 34: 1 0 2, 35: 1 2 0, 36: 1 0 2, 37: 1 2 0, 38: 1 0 2, 39: 1 2 0, 40: 1 0 2, 41: 1 2 0, 42: 1 0 2, 43: 1 2 0, 44: 1 0 2, 45: 1 0 2, 46: 1 0 2, 47: 1 0 2, 48: 1 0 2, 49: 1 2 0, 50: 2 1 0, 51: 2 1 0, 52: 2 1 0, 53: 2 1 0, 54: 2 1 0, 55: 2 1 0, 56: 2 1 0, 57: 2 1 0, 58: 2 0 1, 59: 2 0 1}} on cache ___defaultcache
14:06:46,287 TRACE (OOB-9,ISPN,NodeA-52814:___defaultcache) [StateConsumerImpl] Received keys [0, 2, 10, 94, 103, 117, 187, 189, 288, 305, 307, 376, 481, 487, 502, 729, 771, 994] for segment 50 of cache ___defaultcache from node NodeB-62397
14:06:46,338 INFO (testng-StateTransferLargeObjectTest:) [StateTransferLargeObjectTest] ----Running a get on 10
14:06:46,351 TRACE (OOB-9,ISPN,NodeA-52814:___defaultcache ___defaultcache) [InvocationContextInterceptor] Invoked with command PutKeyValueCommand{key=10, value=org.infinispan.statetransfer.BigObject@6ed3126d, flags=[CACHE_MODE_LOCAL, SKIP_REMOTE_LOOKUP, PUT_FOR_STATE_TRANSFER, SKIP_SHARED_CACHE_STORE, SKIP_OWNERSHIP_CHECK, IGNORE_RETURN_VALUES, SKIP_XSITE_BACKUP], putIfAbsent=false, lifespanMillis=-1, maxIdleTimeMillis=-1, successful=true} and InvocationContext [org.infinispan.context.impl.LocalTxInvocationContext@2c6dd013]
14:06:46,358 TRACE (testng-StateTransferLargeObjectTest:___defaultcache) [GetKeyValueCommand] Entry not found
14:06:46,358 TRACE (testng-StateTransferLargeObjectTest:___defaultcache) [BaseDistributionInterceptor] Not doing a remote get for key 10 since entry is mapped to current node (NodeA-52814), or is in L1. Owners are [NodeC-63995, NodeB-62397, NodeA-52814]
14:06:46,359 ERROR (testng-StateTransferLargeObjectTest:) [UnitTestTestNGListener] Test testForFailure(org.infinispan.statetransfer.StateTransferLargeObjectTest) failed.
java.lang.AssertionError: expected object to not be null
at org.testng.Assert.fail(Assert.java:89)
at org.testng.Assert.assertNotNull(Assert.java:399)
at org.testng.Assert.assertNotNull(Assert.java:384)
at org.infinispan.statetransfer.StateTransferLargeObjectTest.assertValue(StateTransferLargeObjectTest.java:145)
at org.infinispan.statetransfer.StateTransferLargeObjectTest.testForFailure(StateTransferLargeObjectTest.java:115)
{noformat}
was:
The failure appears because the state transfer has not finished, yet the distribution interceptor doesn't go remotely for the key:
{noformat}
14:06:45,872 TRACE (asyncTransportThread-1,NodeA:___defaultcache) [StateTransferManagerImpl] Installing new cache topology CacheTopology{id=7, currentCH=DefaultConsistentHash{numSegments=60, numOwners=3, members=[NodeA-52814, NodeB-62397, NodeC-63995], owners={0: 0 1 2, 1: 0 1 2, 2: 0 1 2, 3: 0 1 2, 4: 0 1 2, 5: 0 1 2, 6: 0 1 2, 7: 0 1 2, 8: 0 1 2, 9: 0 1 2, 10: 0 1 2, 11: 0 1 2, 12: 0 2, 13: 0 2, 14: 0 2, 15: 0 1, 16: 0 1, 17: 0 1, 18: 0 1, 19: 0 1, 20: 2 0, 21: 2 0, 22: 2 0, 23: 2 0, 24: 2 0, 25: 2 0, 26: 2 0, 27: 2 0, 28: 2 0, 29: 2 0, 30: 1 0 2, 31: 1 0 2, 32: 1 0 2, 33: 1 2, 34: 1 0, 35: 1 2, 36: 1 0, 37: 1 2, 38: 1 0, 39: 1 2, 40: 1 0, 41: 1 2, 42: 1 0, 43: 1 2, 44: 1 0, 45: 1 0, 46: 1 0, 47: 1 0, 48: 1 0, 49: 1 2, 50: 2 1, 51: 2 1, 52: 2 1, 53: 2 1, 54: 2 1, 55: 2 1, 56: 2 1, 57: 2 1, 58: 2 0, 59: 2 0}, pendingCH=DefaultConsistentHash{numSegments=60, numOwners=3, members=[NodeA-52814, NodeB-62397, NodeC-63995], owners={0: 0 1 2, 1: 0 1 2, 2: 0 1 2, 3: 0 1 2, 4: 0 1 2, 5: 0 1 2, 6: 0 1 2, 7: 0 1 2, 8: 0 1 2, 9: 0 1 2, 10: 0 1 2, 11: 0 1 2, 12: 0 2 1, 13: 0 2 1, 14: 0 2 1, 15: 0 1 2, 16: 0 1 2, 17: 0 1 2, 18: 0 1 2, 19: 0 1 2, 20: 2 0 1, 21: 2 0 1, 22: 2 0 1, 23: 2 0 1, 24: 2 0 1, 25: 2 0 1, 26: 2 0 1, 27: 2 0 1, 28: 2 0 1, 29: 2 0 1, 30: 1 0 2, 31: 1 0 2, 32: 1 0 2, 33: 1 2 0, 34: 1 0 2, 35: 1 2 0, 36: 1 0 2, 37: 1 2 0, 38: 1 0 2, 39: 1 2 0, 40: 1 0 2, 41: 1 2 0, 42: 1 0 2, 43: 1 2 0, 44: 1 0 2, 45: 1 0 2, 46: 1 0 2, 47: 1 0 2, 48: 1 0 2, 49: 1 2 0, 50: 2 1 0, 51: 2 1 0, 52: 2 1 0, 53: 2 1 0, 54: 2 1 0, 55: 2 1 0, 56: 2 1 0, 57: 2 1 0, 58: 2 0 1, 59: 2 0 1}} on cache ___defaultcache
14:06:46,287 TRACE (OOB-9,ISPN,NodeA-52814:___defaultcache) [StateConsumerImpl] Received keys [0, 2, 10, 94, 103, 117, 187, 189, 288, 305, 307, 376, 481, 487, 502, 729, 771, 994] for segment 50 of cache ___defaultcache from node NodeB-62397
14:06:46,338 INFO (testng-StateTransferLargeObjectTest:) [StateTransferLargeObjectTest] ----Running a get on 10
14:06:46,351 TRACE (OOB-9,ISPN,NodeA-52814:___defaultcache ___defaultcache) [InvocationContextInterceptor] Invoked with command PutKeyValueCommand{key=10, value=org.infinispan.statetransfer.BigObject@6ed3126d, flags=[CACHE_MODE_LOCAL, SKIP_REMOTE_LOOKUP, PUT_FOR_STATE_TRANSFER, SKIP_SHARED_CACHE_STORE, SKIP_OWNERSHIP_CHECK, IGNORE_RETURN_VALUES, SKIP_XSITE_BACKUP], putIfAbsent=false, lifespanMillis=-1, maxIdleTimeMillis=-1, successful=true} and InvocationContext [org.infinispan.context.impl.LocalTxInvocationContext@2c6dd013]
14:06:46,358 TRACE (testng-StateTransferLargeObjectTest:___defaultcache) [GetKeyValueCommand] Entry not found
14:06:46,358 TRACE (testng-StateTransferLargeObjectTest:___defaultcache) [BaseDistributionInterceptor] Not doing a remote get for key 10 since entry is mapped to current node (NodeA-52814), or is in L1. Owners are [NodeC-63995, NodeB-62397, NodeA-52814]
14:06:46,359 ERROR (testng-StateTransferLargeObjectTest:) [UnitTestTestNGListener] Test testForFailure(org.infinispan.statetransfer.StateTransferLargeObjectTest) failed.
java.lang.AssertionError: expected object to not be null
at org.testng.Assert.fail(Assert.java:89)
at org.testng.Assert.assertNotNull(Assert.java:399)
at org.testng.Assert.assertNotNull(Assert.java:384)
at org.infinispan.statetransfer.StateTransferLargeObjectTest.assertValue(StateTransferLargeObjectTest.java:145)
at org.infinispan.statetransfer.StateTransferLargeObjectTest.testForFailure(StateTransferLargeObjectTest.java:115)
{noformat}
> BaseDistributionInterceptor and TxDistributionInterceptor do not detect properly if the key needs to be fetched remotely
> ------------------------------------------------------------------------------------------------------------------------
>
> Key: ISPN-2688
> URL: https://issues.jboss.org/browse/ISPN-2688
> Project: Infinispan
> Issue Type: Bug
> Components: State transfer
> Affects Versions: 5.2.0.Beta6
> Reporter: Dan Berindei
> Assignee: Mircea Markus
> Priority: Critical
> Fix For: 5.2.0.Final
>
> Attachments: stlot.log.gz
>
>
> BaseDistributionInterceptor.shouldFetchFromRemote() and TxDistributionInterceptor.remoteGetAndStoreInL1() can mistakenly decide not to fetch remotely because they check the presence of the key in data container. The key may be there _now_ but it was not there before the local execution if state transfer was in progress for this key. So it should be re-fetched rather than use the null result.
> This makes StateTransferLargeObjectTest.testForFailure fail randomly.
> The failure appears because the state transfer has not finished, yet the distribution interceptor doesn't go remotely for the key:
> {noformat}
> 14:06:45,872 TRACE (asyncTransportThread-1,NodeA:___defaultcache) [StateTransferManagerImpl] Installing new cache topology CacheTopology{id=7, currentCH=DefaultConsistentHash{numSegments=60, numOwners=3, members=[NodeA-52814, NodeB-62397, NodeC-63995], owners={0: 0 1 2, 1: 0 1 2, 2: 0 1 2, 3: 0 1 2, 4: 0 1 2, 5: 0 1 2, 6: 0 1 2, 7: 0 1 2, 8: 0 1 2, 9: 0 1 2, 10: 0 1 2, 11: 0 1 2, 12: 0 2, 13: 0 2, 14: 0 2, 15: 0 1, 16: 0 1, 17: 0 1, 18: 0 1, 19: 0 1, 20: 2 0, 21: 2 0, 22: 2 0, 23: 2 0, 24: 2 0, 25: 2 0, 26: 2 0, 27: 2 0, 28: 2 0, 29: 2 0, 30: 1 0 2, 31: 1 0 2, 32: 1 0 2, 33: 1 2, 34: 1 0, 35: 1 2, 36: 1 0, 37: 1 2, 38: 1 0, 39: 1 2, 40: 1 0, 41: 1 2, 42: 1 0, 43: 1 2, 44: 1 0, 45: 1 0, 46: 1 0, 47: 1 0, 48: 1 0, 49: 1 2, 50: 2 1, 51: 2 1, 52: 2 1, 53: 2 1, 54: 2 1, 55: 2 1, 56: 2 1, 57: 2 1, 58: 2 0, 59: 2 0}, pendingCH=DefaultConsistentHash{numSegments=60, numOwners=3, members=[NodeA-52814, NodeB-62397, NodeC-63995], owners={0: 0 1 2, 1: 0 1 2, 2: 0 1 2, 3: 0 1 2, 4: 0 1 2, 5: 0 1 2, 6: 0 1 2, 7: 0 1 2, 8: 0 1 2, 9: 0 1 2, 10: 0 1 2, 11: 0 1 2, 12: 0 2 1, 13: 0 2 1, 14: 0 2 1, 15: 0 1 2, 16: 0 1 2, 17: 0 1 2, 18: 0 1 2, 19: 0 1 2, 20: 2 0 1, 21: 2 0 1, 22: 2 0 1, 23: 2 0 1, 24: 2 0 1, 25: 2 0 1, 26: 2 0 1, 27: 2 0 1, 28: 2 0 1, 29: 2 0 1, 30: 1 0 2, 31: 1 0 2, 32: 1 0 2, 33: 1 2 0, 34: 1 0 2, 35: 1 2 0, 36: 1 0 2, 37: 1 2 0, 38: 1 0 2, 39: 1 2 0, 40: 1 0 2, 41: 1 2 0, 42: 1 0 2, 43: 1 2 0, 44: 1 0 2, 45: 1 0 2, 46: 1 0 2, 47: 1 0 2, 48: 1 0 2, 49: 1 2 0, 50: 2 1 0, 51: 2 1 0, 52: 2 1 0, 53: 2 1 0, 54: 2 1 0, 55: 2 1 0, 56: 2 1 0, 57: 2 1 0, 58: 2 0 1, 59: 2 0 1}} on cache ___defaultcache
> 14:06:46,287 TRACE (OOB-9,ISPN,NodeA-52814:___defaultcache) [StateConsumerImpl] Received keys [0, 2, 10, 94, 103, 117, 187, 189, 288, 305, 307, 376, 481, 487, 502, 729, 771, 994] for segment 50 of cache ___defaultcache from node NodeB-62397
> 14:06:46,338 INFO (testng-StateTransferLargeObjectTest:) [StateTransferLargeObjectTest] ----Running a get on 10
> 14:06:46,351 TRACE (OOB-9,ISPN,NodeA-52814:___defaultcache ___defaultcache) [InvocationContextInterceptor] Invoked with command PutKeyValueCommand{key=10, value=org.infinispan.statetransfer.BigObject@6ed3126d, flags=[CACHE_MODE_LOCAL, SKIP_REMOTE_LOOKUP, PUT_FOR_STATE_TRANSFER, SKIP_SHARED_CACHE_STORE, SKIP_OWNERSHIP_CHECK, IGNORE_RETURN_VALUES, SKIP_XSITE_BACKUP], putIfAbsent=false, lifespanMillis=-1, maxIdleTimeMillis=-1, successful=true} and InvocationContext [org.infinispan.context.impl.LocalTxInvocationContext@2c6dd013]
> 14:06:46,358 TRACE (testng-StateTransferLargeObjectTest:___defaultcache) [GetKeyValueCommand] Entry not found
> 14:06:46,358 TRACE (testng-StateTransferLargeObjectTest:___defaultcache) [BaseDistributionInterceptor] Not doing a remote get for key 10 since entry is mapped to current node (NodeA-52814), or is in L1. Owners are [NodeC-63995, NodeB-62397, NodeA-52814]
> 14:06:46,359 ERROR (testng-StateTransferLargeObjectTest:) [UnitTestTestNGListener] Test testForFailure(org.infinispan.statetransfer.StateTransferLargeObjectTest) failed.
> java.lang.AssertionError: expected object to not be null
> at org.testng.Assert.fail(Assert.java:89)
> at org.testng.Assert.assertNotNull(Assert.java:399)
> at org.testng.Assert.assertNotNull(Assert.java:384)
> at org.infinispan.statetransfer.StateTransferLargeObjectTest.assertValue(StateTransferLargeObjectTest.java:145)
> at org.infinispan.statetransfer.StateTransferLargeObjectTest.testForFailure(StateTransferLargeObjectTest.java:115)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 11 months
[JBoss JIRA] (ISPN-2589) NPE in InvalidateL1Command
by Adrian Nistor (JIRA)
[ https://issues.jboss.org/browse/ISPN-2589?page=com.atlassian.jira.plugin.... ]
Adrian Nistor commented on ISPN-2589:
-------------------------------------
And the NPE is thrown from exactly the same line from InvalidateL1Command? (sorry to ask so many details - cannot access the qa machine for logs)
> NPE in InvalidateL1Command
> --------------------------
>
> Key: ISPN-2589
> URL: https://issues.jboss.org/browse/ISPN-2589
> Project: Infinispan
> Issue Type: Bug
> Components: Core API, State transfer
> Affects Versions: 5.2.0.Beta5
> Reporter: Thomas Fromm
> Assignee: Dan Berindei
> Fix For: 5.3.0.Final
>
> Attachments: standalone_node0001.xml
>
>
> The used cache is an REPL_SYNC one w/o L1 and Optimistic TX.
> Can't provide unit test, just saw it in my logs.
> WARN 05.12.12 20:08:22,746 [OOB-173,IBIS-2147] CacheTopologyControlCommand ISPN000071: Caught exception when handling command CacheTopologyControlCommand{cache=MODULE_PROPERTIES_VERSION, type=CH_UPDATE, sender=IBIS-57850, joinInfo=null, topologyId=14, currentCH=ReplicatedConsistentHash{members=[IBIS-57850, IBIS-15651, IBIS-14611, IBIS-7942, IBIS-4256, IBIS-25472, IBIS-32523]}, pendingCH=null, throwable=null, viewId=8}
> java.lang.NullPointerException
> at org.infinispan.commands.write.InvalidateL1Command.perform(InvalidateL1Command.java:109)
> at org.infinispan.interceptors.CallInterceptor.handleDefault(CallInterceptor.java:110)
> at org.infinispan.commands.AbstractVisitor.visitInvalidateCommand(AbstractVisitor.java:141)
> at org.infinispan.commands.AbstractVisitor.visitInvalidateL1Command(AbstractVisitor.java:146)
> at org.infinispan.commands.write.InvalidateL1Command.acceptVisitor(InvalidateL1Command.java:192)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
> at org.infinispan.commands.AbstractVisitor.visitInvalidateCommand(AbstractVisitor.java:141)
> at org.infinispan.commands.AbstractVisitor.visitInvalidateL1Command(AbstractVisitor.java:146)
> at org.infinispan.commands.write.InvalidateL1Command.acceptVisitor(InvalidateL1Command.java:192)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
> at org.infinispan.commands.AbstractVisitor.visitInvalidateCommand(AbstractVisitor.java:141)
> at org.infinispan.commands.AbstractVisitor.visitInvalidateL1Command(AbstractVisitor.java:146)
> at org.infinispan.commands.write.InvalidateL1Command.acceptVisitor(InvalidateL1Command.java:192)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.CacheLoaderInterceptor.visitInvalidateCommand(CacheLoaderInterceptor.java:127)
> at org.infinispan.commands.AbstractVisitor.visitInvalidateL1Command(AbstractVisitor.java:146)
> at org.infinispan.commands.write.InvalidateL1Command.acceptVisitor(InvalidateL1Command.java:192)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.EntryWrappingInterceptor.invokeNextAndApplyChanges(EntryWrappingInterceptor.java:211)
> at org.infinispan.interceptors.EntryWrappingInterceptor.visitInvalidateL1Command(EntryWrappingInterceptor.java:143)
> at org.infinispan.commands.write.InvalidateL1Command.acceptVisitor(InvalidateL1Command.java:192)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitInvalidateL1Command(AbstractLockingInterceptor.java:99)
> at org.infinispan.commands.write.InvalidateL1Command.acceptVisitor(InvalidateL1Command.java:192)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
> at org.infinispan.commands.AbstractVisitor.visitInvalidateCommand(AbstractVisitor.java:141)
> at org.infinispan.commands.AbstractVisitor.visitInvalidateL1Command(AbstractVisitor.java:146)
> at org.infinispan.commands.write.InvalidateL1Command.acceptVisitor(InvalidateL1Command.java:192)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.TxInterceptor.enlistWriteAndInvokeNext(TxInterceptor.java:256)
> at org.infinispan.interceptors.TxInterceptor.visitInvalidateCommand(TxInterceptor.java:224)
> at org.infinispan.commands.AbstractVisitor.visitInvalidateL1Command(AbstractVisitor.java:146)
> at org.infinispan.commands.write.InvalidateL1Command.acceptVisitor(InvalidateL1Command.java:192)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.statetransfer.StateTransferInterceptor.visitInvalidateL1Command(StateTransferInterceptor.java:172)
> at org.infinispan.commands.write.InvalidateL1Command.acceptVisitor(InvalidateL1Command.java:192)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
> at org.infinispan.commands.AbstractVisitor.visitInvalidateCommand(AbstractVisitor.java:141)
> at org.infinispan.commands.AbstractVisitor.visitInvalidateL1Command(AbstractVisitor.java:146)
> at org.infinispan.commands.write.InvalidateL1Command.acceptVisitor(InvalidateL1Command.java:192)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:128)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:92)
> at org.infinispan.commands.AbstractVisitor.visitInvalidateCommand(AbstractVisitor.java:141)
> at org.infinispan.commands.AbstractVisitor.visitInvalidateL1Command(AbstractVisitor.java:146)
> at org.infinispan.commands.write.InvalidateL1Command.acceptVisitor(InvalidateL1Command.java:192)
> at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:343)
> at org.infinispan.statetransfer.StateConsumerImpl.invalidateSegments(StateConsumerImpl.java:592)
> at org.infinispan.statetransfer.StateConsumerImpl.onTopologyUpdate(StateConsumerImpl.java:239)
> at org.infinispan.statetransfer.StateTransferManagerImpl.doTopologyUpdate(StateTransferManagerImpl.java:191)
> at org.infinispan.statetransfer.StateTransferManagerImpl.access$000(StateTransferManagerImpl.java:58)
> at org.infinispan.statetransfer.StateTransferManagerImpl$1.updateConsistentHash(StateTransferManagerImpl.java:117)
> at org.infinispan.topology.LocalTopologyManagerImpl.handleConsistentHashUpdate(LocalTopologyManagerImpl.java:194)
> at org.infinispan.topology.CacheTopologyControlCommand.doPerform(CacheTopologyControlCommand.java:165)
> at org.infinispan.topology.CacheTopologyControlCommand.perform(CacheTopologyControlCommand.java:137)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommandFromLocalCluster(CommandAwareRpcDispatcher.java:252)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:219)
> at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:483)
> at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:390)
> at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:248)
> at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:598)
> at org.jgroups.JChannel.up(JChannel.java:703)
> at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1020)
> at org.jgroups.protocols.RSVP.up(RSVP.java:172)
> at org.jgroups.protocols.FRAG2.up(FRAG2.java:181)
> at org.jgroups.protocols.FC.up(FC.java:479)
> at org.jgroups.protocols.pbcast.GMS.up(GMS.java:896)
> at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:244)
> at org.jgroups.protocols.UNICAST2.up(UNICAST2.java:432)
> at org.jgroups.protocols.pbcast.NAKACK2.handleMessage(NAKACK2.java:721)
> at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:574)
> at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:143)
> at org.jgroups.protocols.FD_ALL.up(FD_ALL.java:187)
> at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:288)
> at org.jgroups.protocols.MERGE3.up(MERGE3.java:290)
> at org.jgroups.protocols.Discovery.up(Discovery.java:359)
> at org.jgroups.protocols.TP.passMessageUp(TP.java:1287)
> at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1850)
> at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1823)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 11 months
[JBoss JIRA] (ISPN-2736) ConcurrentModificationException in StateRequestCommand
by Adrian Nistor (JIRA)
[ https://issues.jboss.org/browse/ISPN-2736?page=com.atlassian.jira.plugin.... ]
Adrian Nistor updated ISPN-2736:
--------------------------------
Fix Version/s: 5.2.0.Final
> ConcurrentModificationException in StateRequestCommand
> -------------------------------------------------------
>
> Key: ISPN-2736
> URL: https://issues.jboss.org/browse/ISPN-2736
> Project: Infinispan
> Issue Type: Bug
> Components: State transfer
> Affects Versions: 5.2.0.CR2
> Reporter: Radim Vansa
> Assignee: Adrian Nistor
> Priority: Critical
> Fix For: 5.2.0.Final
>
>
> I have tried async replication scenario with async marshalling (optimistic transactions) and this has occurred:
> {code}
> 04:34:15,182 ERROR [org.infinispan.remoting.rpc.RpcManagerImpl] (OOB-77,edg-perf11-48420) ISPN000073: Unexpected error while replicating
> java.util.ConcurrentModificationException
> at java.util.HashMap$HashIterator.nextEntry(HashMap.java:793)
> at java.util.HashMap$KeyIterator.next(HashMap.java:828)
> at org.infinispan.statetransfer.StateProviderImpl.collectTransactionsToTransfer(StateProviderImpl.java:239)
> at org.infinispan.statetransfer.StateProviderImpl.getTransactionsForSegments(StateProviderImpl.java:200)
> at org.infinispan.statetransfer.StateRequestCommand.perform(StateRequestCommand.java:88)
> at org.infinispan.remoting.InboundInvocationHandlerImpl.handleInternal(InboundInvocationHandlerImpl.java:101)
> at org.infinispan.remoting.InboundInvocationHandlerImpl.handleWithWaitForBlocks(InboundInvocationHandlerImpl.java:122)
> at org.infinispan.remoting.InboundInvocationHandlerImpl.handle(InboundInvocationHandlerImpl.java:86)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommandFromLocalCluster(CommandAwareRpcDispatcher.java:245)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:218)
> at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:484)
> at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:391)
> at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:249)
> at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:598)
> at org.jgroups.JChannel.up(JChannel.java:707)
> {code}
> Seems that StateRequestCommand is iterating through all transactions' locked keys but the collections of these locked keys can be modified by interfering prepare/commit.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 11 months