[JBoss JIRA] (ISPN-6304) NPE during ClearCommand
by Pedro Ruivo (JIRA)
[ https://issues.jboss.org/browse/ISPN-6304?page=com.atlassian.jira.plugin.... ]
Pedro Ruivo updated ISPN-6304:
------------------------------
Status: Open (was: New)
> NPE during ClearCommand
> -----------------------
>
> Key: ISPN-6304
> URL: https://issues.jboss.org/browse/ISPN-6304
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Affects Versions: 8.2.0.CR1
> Reporter: Gustavo Fernandes
> Assignee: Pedro Ruivo
>
> The logs can be seen on https://travis-ci.org/infinispan/infinispan-hadoop/builds/113090728
> The NPE happens intermittently.
> {code}
> 0:52:28,139 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (HotRodServerWorker-7-1) ISPN000136: Error executing command ClearCommand, writing keys []: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from node1, see cause for remote stack trace
> at org.infinispan.remoting.transport.AbstractTransport.checkResponse(AbstractTransport.java:44)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:796)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$83(JGroupsTransport.java:633)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport$$Lambda$46/308539050.apply(Unknown Source)
> at java.util.concurrent.CompletableFuture$ThenApply.run(CompletableFuture.java:717)
> at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:193)
> at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2345)
> at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.futureDone(SingleResponseFuture.java:30)
> at org.jgroups.blocks.Request.checkCompletion(Request.java:162)
> at org.jgroups.blocks.UnicastRequest.receiveResponse(UnicastRequest.java:81)
> at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:373)
> at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:237)
> at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:695)
> at org.jgroups.JChannel.up(JChannel.java:738)
> at org.jgroups.fork.ForkProtocolStack.up(ForkProtocolStack.java:120)
> at org.jgroups.stack.Protocol.up(Protocol.java:379)
> at org.jgroups.protocols.FORK.up(FORK.java:114)
> at org.jgroups.protocols.FRAG2.up(FRAG2.java:165)
> at org.jgroups.protocols.FlowControl.up(FlowControl.java:392)
> at org.jgroups.protocols.FlowControl.up(FlowControl.java:381)
> at org.jgroups.protocols.pbcast.GMS.up(GMS.java:1043)
> at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:234)
> at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1064)
> at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:779)
> at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:426)
> at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:652)
> at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:155)
> at org.jgroups.protocols.FD_ALL.up(FD_ALL.java:200)
> at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:310)
> at org.jgroups.protocols.MERGE3.up(MERGE3.java:285)
> at org.jgroups.protocols.Discovery.up(Discovery.java:296)
> at org.jgroups.protocols.TP.passMessageUp(TP.java:1590)
> at org.jgroups.protocols.TP$SingleMessageHandler.run(TP.java:1802)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.infinispan.commons.CacheException: java.lang.NullPointerException
> at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleFromCluster(GlobalInboundInvocationHandler.java:89)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommandFromLocalCluster(CommandAwareRpcDispatcher.java:196)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:166)
> at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:402)
> at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:352)
> ... 25 more
> Caused by: java.lang.NullPointerException
> at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleCacheRpcCommand(GlobalInboundInvocationHandler.java:126)
> at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleFromCluster(GlobalInboundInvocationHandler.java:75)
> ... 29 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
10 years, 1 month
[JBoss JIRA] (ISPN-6304) NPE during ClearCommand
by Pedro Ruivo (JIRA)
[ https://issues.jboss.org/browse/ISPN-6304?page=com.atlassian.jira.plugin.... ]
Pedro Ruivo reassigned ISPN-6304:
---------------------------------
Assignee: Pedro Ruivo
> NPE during ClearCommand
> -----------------------
>
> Key: ISPN-6304
> URL: https://issues.jboss.org/browse/ISPN-6304
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Affects Versions: 8.2.0.CR1
> Reporter: Gustavo Fernandes
> Assignee: Pedro Ruivo
>
> The logs can be seen on https://travis-ci.org/infinispan/infinispan-hadoop/builds/113090728
> The NPE happens intermittently.
> {code}
> 0:52:28,139 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (HotRodServerWorker-7-1) ISPN000136: Error executing command ClearCommand, writing keys []: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from node1, see cause for remote stack trace
> at org.infinispan.remoting.transport.AbstractTransport.checkResponse(AbstractTransport.java:44)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:796)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$83(JGroupsTransport.java:633)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport$$Lambda$46/308539050.apply(Unknown Source)
> at java.util.concurrent.CompletableFuture$ThenApply.run(CompletableFuture.java:717)
> at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:193)
> at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2345)
> at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.futureDone(SingleResponseFuture.java:30)
> at org.jgroups.blocks.Request.checkCompletion(Request.java:162)
> at org.jgroups.blocks.UnicastRequest.receiveResponse(UnicastRequest.java:81)
> at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:373)
> at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:237)
> at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:695)
> at org.jgroups.JChannel.up(JChannel.java:738)
> at org.jgroups.fork.ForkProtocolStack.up(ForkProtocolStack.java:120)
> at org.jgroups.stack.Protocol.up(Protocol.java:379)
> at org.jgroups.protocols.FORK.up(FORK.java:114)
> at org.jgroups.protocols.FRAG2.up(FRAG2.java:165)
> at org.jgroups.protocols.FlowControl.up(FlowControl.java:392)
> at org.jgroups.protocols.FlowControl.up(FlowControl.java:381)
> at org.jgroups.protocols.pbcast.GMS.up(GMS.java:1043)
> at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:234)
> at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1064)
> at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:779)
> at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:426)
> at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:652)
> at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:155)
> at org.jgroups.protocols.FD_ALL.up(FD_ALL.java:200)
> at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:310)
> at org.jgroups.protocols.MERGE3.up(MERGE3.java:285)
> at org.jgroups.protocols.Discovery.up(Discovery.java:296)
> at org.jgroups.protocols.TP.passMessageUp(TP.java:1590)
> at org.jgroups.protocols.TP$SingleMessageHandler.run(TP.java:1802)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.infinispan.commons.CacheException: java.lang.NullPointerException
> at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleFromCluster(GlobalInboundInvocationHandler.java:89)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommandFromLocalCluster(CommandAwareRpcDispatcher.java:196)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:166)
> at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:402)
> at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:352)
> ... 25 more
> Caused by: java.lang.NullPointerException
> at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleCacheRpcCommand(GlobalInboundInvocationHandler.java:126)
> at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleFromCluster(GlobalInboundInvocationHandler.java:75)
> ... 29 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
10 years, 1 month
[JBoss JIRA] (ISPN-6254) State transfer hangs after CacheNotFoundResponse
by Dan Berindei (JIRA)
[ https://issues.jboss.org/browse/ISPN-6254?page=com.atlassian.jira.plugin.... ]
Dan Berindei reopened ISPN-6254:
--------------------------------
Fix is incomplete, always logs this warning:
{noformat}
15:46:50,970 WARN [org.infinispan.statetransfer.InboundTransferTask] (stateTransferExecutor-thread--p7-t8) ISPN000210: Failed to request segments [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239, 240, 241, 242, 243, 244, 245, 246, 247, 248, 249, 250, 251, 252, 253, 254, 255] of cache ___hotRodTopologyCache from node node0 (node will not be retried): org.infinispan.commons.CacheException: SuccessfulResponse{responseValue=true}
at org.infinispan.statetransfer.InboundTransferTask.requestSegments(InboundTransferTask.java:150)
at org.infinispan.statetransfer.StateConsumerImpl$4.call(StateConsumerImpl.java:1048)
at org.infinispan.statetransfer.StateConsumerImpl$4.call(StateConsumerImpl.java:1045)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.infinispan.executors.SemaphoreCompletionService$QueueingTask.runInternal(SemaphoreCompletionService.java:172)
at org.infinispan.executors.SemaphoreCompletionService$QueueingTask.run(SemaphoreCompletionService.java:151)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
{noformat}
> State transfer hangs after CacheNotFoundResponse
> ------------------------------------------------
>
> Key: ISPN-6254
> URL: https://issues.jboss.org/browse/ISPN-6254
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Affects Versions: 8.2.0.Beta2, 8.1.2.Final
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Priority: Critical
> Labels: testsuite_stability
> Fix For: 8.2.0.CR1, 8.2.0.Final
>
>
> When {{StateConsumerImpl.addTransfer()}} sends a {{StateRequestCommand(START_STATE_TRANSFER)}}, it doesn't block the thread until it receives all the state (ISPN-5019). Instead, it calls {{SemaphoreCompletionService.continueTaskInBackground()}}, and then it calls {{SemaphoreCompletionService.backgroundTaskFinished()}} when it receives the response.
> However, if the request wasn't successful, e.g. because the node providing the state is shutting down and replies with a {{CacheNotFoundResponse}}, {{SemaphoreCompletionService.backgroundTaskFinished()}} isn't called, and state transfer cannot make any progress.
> This shows up in the test suite as random failures in {{NumOwnersNodeCrashInSequenceTest}} and {{NumOwnersNodeStopInSequenceTest}}:
> {noformat}
> 17:49:16,960 ERROR (testng-NumOwnersNodeCrashInSequenceTest:) [UnitTestTestNGListener] Test testNodeCrashedBeforeStFinished0(org.infinispan.partitionhandling.NumOwnersNodeCrashInSequenceTest) failed.
> org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.Exception on object of type StateTransferManagerImpl
> at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:172) ~[infinispan-commons-8.2.0-SNAPSHOT.jar:8.2.0-SNAPSHOT]
> at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:887) ~[classes/:?]
> at org.infinispan.factories.AbstractComponentRegistry.invokeStartMethods(AbstractComponentRegistry.java:656) ~[classes/:?]
> at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:645) ~[classes/:?]
> at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:548) ~[classes/:?]
> at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:222) ~[classes/:?]
> at org.infinispan.cache.impl.CacheImpl.start(CacheImpl.java:849) ~[classes/:?]
> at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:624) ~[classes/:?]
> at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:577) ~[classes/:?]
> at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:445) ~[classes/:?]
> at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:431) ~[classes/:?]
> at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:413) ~[classes/:?]
> at org.infinispan.test.MultipleCacheManagersTest.getCaches(MultipleCacheManagersTest.java:211) ~[test-classes/:?]
> at org.infinispan.test.MultipleCacheManagersTest.waitForClusterToForm(MultipleCacheManagersTest.java:220) ~[test-classes/:?]
> at org.infinispan.test.MultipleCacheManagersTest.waitForClusterToForm(MultipleCacheManagersTest.java:229) ~[test-classes/:?]
> at org.infinispan.partitionhandling.NumOwnersNodeCrashInSequenceTest.testNodeCrashedBeforeStFinished(NumOwnersNodeCrashInSequenceTest.java:105) ~[test-classes/:?]
> at org.infinispan.partitionhandling.NumOwnersNodeCrashInSequenceTest.testNodeCrashedBeforeStFinished0(NumOwnersNodeCrashInSequenceTest.java:67) ~[test-classes/:?]
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
10 years, 1 month