[JBoss JIRA] (ISPN-3552) HotRod Rolling Upgrades NPE when disabling cache store on target node
by RH Bugzilla Integration (JIRA)
[ https://issues.jboss.org/browse/ISPN-3552?page=com.atlassian.jira.plugin.... ]
RH Bugzilla Integration commented on ISPN-3552:
-----------------------------------------------
Tristan Tarrant <ttarrant(a)redhat.com> changed the Status of [bug 1011499|https://bugzilla.redhat.com/show_bug.cgi?id=1011499] from NEW to ON_QA
> HotRod Rolling Upgrades NPE when disabling cache store on target node
> ---------------------------------------------------------------------
>
> Key: ISPN-3552
> URL: https://issues.jboss.org/browse/ISPN-3552
> Project: Infinispan
> Issue Type: Bug
> Affects Versions: 6.0.0.Beta1
> Reporter: Tomas Sykora
> Assignee: Pedro Ruivo
> Priority: Critical
> Labels: 620
> Fix For: 6.0.0.Final
>
> Attachments: standalone-source-node-rollups52-dist.xml, standalone-target-node-rollups-dist.xml
>
>
> After Tristan's fix for https://issues.jboss.org/browse/ISPN-3183 we can move successfully through recordKnownGlobalKeyset and synchronizeData operations.
> However, when we want to issue disconnectSource operation on target node, it is failing with given error:
> javax.management.MBeanException
> at org.infinispan.jmx.ResourceDMBean.invoke(ResourceDMBean.java:273)
> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819)
> at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:791)
> at org.jboss.as.jmx.PluggableMBeanServerImpl$TcclMBeanServer.invoke(PluggableMBeanServerImpl.java:527)
> at org.jboss.as.jmx.PluggableMBeanServerImpl.invoke(PluggableMBeanServerImpl.java:263)
> at org.jboss.remotingjmx.protocol.v1.ServerProxy$InvokeHandler.handle(ServerProxy.java:1058)
> at org.jboss.remotingjmx.protocol.v1.ServerProxy$MessageReciever$1.run(ServerProxy.java:225)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:601)
> at org.infinispan.jmx.ResourceDMBean.invoke(ResourceDMBean.java:271)
> ... 9 more
> Caused by: java.lang.NullPointerException
> at org.infinispan.persistence.manager.PersistenceManagerImpl.disableStore(PersistenceManagerImpl.java:253)
> at org.infinispan.persistence.remote.upgrade.HotRodTargetMigrator.disconnectSource(HotRodTargetMigrator.java:101)
> at org.infinispan.upgrade.RollingUpgradeManager.disconnectSource(RollingUpgradeManager.java:71)
> ... 14 more
> Short input from Tristan: "problem is that after removing the store, it determines there are no more stores left so it tries to remove the loader/writer interceptors which for some reason are missing"
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 1 month
[JBoss JIRA] (ISPN-3685) HotRod Rolling Upgrades ClassNotFoundException: org.infinispan.CacheException during recordKnownGlobalKeyset
by RH Bugzilla Integration (JIRA)
[ https://issues.jboss.org/browse/ISPN-3685?page=com.atlassian.jira.plugin.... ]
RH Bugzilla Integration commented on ISPN-3685:
-----------------------------------------------
Tristan Tarrant <ttarrant(a)redhat.com> changed the Status of [bug 1026862|https://bugzilla.redhat.com/show_bug.cgi?id=1026862] from NEW to ON_QA
> HotRod Rolling Upgrades ClassNotFoundException: org.infinispan.CacheException during recordKnownGlobalKeyset
> -------------------------------------------------------------------------------------------------------------
>
> Key: ISPN-3685
> URL: https://issues.jboss.org/browse/ISPN-3685
> Project: Infinispan
> Issue Type: Bug
> Components: Server
> Affects Versions: 6.0.0.CR1
> Reporter: Tomas Sykora
> Assignee: Pedro Ruivo
> Labels: 620
> Fix For: 6.1.0.Final
>
>
> During process of HotRod Rolling Upgrades from cluster of 2 JDG 6.1.GA nodes to 2 JDG 6.2.ER3.2 nodes, we are blocked by this exception while calling recordKnownGlobalKeyset operation on "old" cluster:
> java.io.IOException: java.lang.ClassNotFoundException: org.infinispan.CacheException
> at org.jboss.remotingjmx.protocol.v1.ClientConnection$BaseResponseHandler.handle(ClientConnection.java:1775)
> at org.jboss.remotingjmx.protocol.v1.ClientConnection$MessageReceiver$1.run(ClientConnection.java:455)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)
> Caused by: java.lang.ClassNotFoundException: org.infinispan.CacheException
> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:423)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:356)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:264)
> at org.jboss.marshalling.AbstractClassResolver.loadClass(AbstractClassResolver.java:135)
> at org.jboss.marshalling.AbstractClassResolver.resolveClass(AbstractClassResolver.java:116)
> at org.jboss.marshalling.river.RiverUnmarshaller.doReadClassDescriptor(RiverUnmarshaller.java:947)
> at org.jboss.marshalling.river.RiverUnmarshaller.doReadNewObject(RiverUnmarshaller.java:1259)
> at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:276)
> at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:213)
> at org.jboss.marshalling.river.RiverUnmarshaller.readFields(RiverUnmarshaller.java:1732)
> at org.jboss.marshalling.river.RiverUnmarshaller.doInitSerializable(RiverUnmarshaller.java:1648)
> at org.jboss.marshalling.river.RiverUnmarshaller.doReadNewObject(RiverUnmarshaller.java:1290)
> at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:276)
> at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:213)
> at org.jboss.marshalling.river.RiverUnmarshaller.readFields(RiverUnmarshaller.java:1732)
> at org.jboss.marshalling.river.RiverUnmarshaller.doInitSerializable(RiverUnmarshaller.java:1648)
> at org.jboss.marshalling.river.RiverUnmarshaller.doReadNewObject(RiverUnmarshaller.java:1290)
> at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:276)
> at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:213)
> at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:72)
> at org.jboss.remotingjmx.protocol.v1.ClientConnection$BaseResponseHandler.handle(ClientConnection.java:1766)
> ... 4 more
> This is also the same for Infinispan Server build from upstream, so it shouldn't be product specific.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 1 month
[JBoss JIRA] (ISPN-3375) Map/Reduce jobs with small value sizes fail because of timeouts putting intermediate keys into the cache
by RH Bugzilla Integration (JIRA)
[ https://issues.jboss.org/browse/ISPN-3375?page=com.atlassian.jira.plugin.... ]
RH Bugzilla Integration commented on ISPN-3375:
-----------------------------------------------
Tristan Tarrant <ttarrant(a)redhat.com> changed the Status of [bug 1013001|https://bugzilla.redhat.com/show_bug.cgi?id=1013001] from NEW to ON_QA
> Map/Reduce jobs with small value sizes fail because of timeouts putting intermediate keys into the cache
> --------------------------------------------------------------------------------------------------------
>
> Key: ISPN-3375
> URL: https://issues.jboss.org/browse/ISPN-3375
> Project: Infinispan
> Issue Type: Bug
> Components: Distributed Execution and Map/Reduce
> Affects Versions: 5.3.0.Final
> Reporter: Alan Field
> Assignee: Dan Berindei
> Priority: Critical
> Labels: 620
> Fix For: 6.0.0.Final
>
> Attachments: jgroups-udp-mr.xml, stdout.zip
>
>
> The following exception occurs on the primary node running a Map/Reduce job with 1KB values:
> {noformat}
> 12:57:32,755 ERROR [org.radargun.stages.MapReduceStage] (pool-1-thread-1) executeMapReduceTask() returned an exception
> org.infinispan.CacheException: java.util.concurrent.ExecutionException: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from edg-perf03-15313, see cause for remote stack trace
> at org.infinispan.distexec.mapreduce.MapReduceTask.execute(MapReduceTask.java:369)
> at org.radargun.cachewrappers.InfinispanMapReduceWrapper.executeMapReduceTask(InfinispanMapReduceWrapper.java:113)
> at org.radargun.stages.MapReduceStage.executeMapReduceTask(MapReduceStage.java:207)
> at org.radargun.stages.MapReduceStage.executeOnSlave(MapReduceStage.java:149)
> at org.radargun.Slave$2.run(Slave.java:103)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:724)
> Caused by: java.util.concurrent.ExecutionException: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from edg-perf03-15313, see cause for remote stack trace
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
> at java.util.concurrent.FutureTask.get(FutureTask.java:111)
> at org.infinispan.distexec.mapreduce.MapReduceTask$TaskPart.get(MapReduceTask.java:867)
> at org.infinispan.distexec.mapreduce.MapReduceTask.executeMapPhase(MapReduceTask.java:461)
> at org.infinispan.distexec.mapreduce.MapReduceTask.execute(MapReduceTask.java:363)
> ... 10 more
> Caused by: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from edg-perf03-15313, see cause for remote stack trace
> at org.infinispan.remoting.transport.AbstractTransport.checkResponse(AbstractTransport.java:70)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.processSingleCall(CommandAwareRpcDispatcher.java:384)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:189)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:531)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:303)
> at org.infinispan.remoting.rpc.RpcManagerImpl$2.call(RpcManagerImpl.java:340)
> ... 5 more
> Caused by: org.infinispan.CacheException: org.infinispan.CacheException: Could not move intermediate keys/values for M/R task edc7a51c-d25a-4650-b39a-8d8a87d7cd61
> at org.infinispan.distexec.mapreduce.MapReduceManagerImpl.mapAndCombineForDistributedReduction(MapReduceManagerImpl.java:114)
> at org.infinispan.commands.read.MapCombineCommand.perform(MapCombineCommand.java:93)
> at org.infinispan.remoting.InboundInvocationHandlerImpl.handleInternal(InboundInvocationHandlerImpl.java:122)
> at org.infinispan.remoting.InboundInvocationHandlerImpl.access$000(InboundInvocationHandlerImpl.java:68)
> at org.infinispan.remoting.InboundInvocationHandlerImpl$2.run(InboundInvocationHandlerImpl.java:194)
> ... 3 more
> Caused by: org.infinispan.CacheException: Could not move intermediate keys/values for M/R task edc7a51c-d25a-4650-b39a-8d8a87d7cd61
> at org.infinispan.distexec.mapreduce.MapReduceManagerImpl.combine(MapReduceManagerImpl.java:331)
> at org.infinispan.distexec.mapreduce.MapReduceManagerImpl.mapAndCombineForDistributedReduction(MapReduceManagerImpl.java:112)
> ... 7 more
> Caused by: org.infinispan.util.concurrent.TimeoutException: Node edg-perf02-56077 timed out
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:196)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:531)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:303)
> at org.infinispan.interceptors.distribution.BaseDistributionInterceptor.handleNonTxWriteCommand(BaseDistributionInterceptor.java:151)
> at org.infinispan.interceptors.distribution.NonTxDistributionInterceptor.visitPutKeyValueCommand(NonTxDistributionInterceptor.java:91)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
> at org.infinispan.interceptors.EntryWrappingInterceptor.invokeNextAndApplyChanges(EntryWrappingInterceptor.java:290)
> at org.infinispan.interceptors.EntryWrappingInterceptor.visitPutKeyValueCommand(EntryWrappingInterceptor.java:157)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
> at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitPutKeyValueCommand(AbstractLockingInterceptor.java:70)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:134)
> at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:54)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:134)
> at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:54)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
> at org.infinispan.statetransfer.StateTransferInterceptor.handleTopologyAffectedCommand(StateTransferInterceptor.java:216)
> at org.infinispan.statetransfer.StateTransferInterceptor.handleWriteCommand(StateTransferInterceptor.java:194)
> at org.infinispan.statetransfer.StateTransferInterceptor.visitPutKeyValueCommand(StateTransferInterceptor.java:136)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
> at org.infinispan.interceptors.CacheMgmtInterceptor.visitPutKeyValueCommand(CacheMgmtInterceptor.java:160)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:128)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:92)
> at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:54)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
> at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:343)
> at org.infinispan.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1337)
> at org.infinispan.CacheImpl.putInternal(CacheImpl.java:898)
> at org.infinispan.CacheImpl.put(CacheImpl.java:890)
> at org.infinispan.CacheImpl.put(CacheImpl.java:1390)
> at org.infinispan.CacheImpl.put(CacheImpl.java:229)
> at org.infinispan.distexec.mapreduce.MapReduceManagerImpl.combine(MapReduceManagerImpl.java:326)
> ... 8 more
> Caused by: org.jgroups.TimeoutException: timeout sending message to edg-perf02-56077
> at org.jgroups.blocks.MessageDispatcher.sendMessage(MessageDispatcher.java:419)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.processSingleCall(CommandAwareRpcDispatcher.java:375)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:189)
> ... 48 more
> {noformat}
> And this exception is occurring on the other node in the cluster:
> {noformat}
> 12:57:32,484 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (remote-thread-0) ISPN000136: Execution error
> org.infinispan.util.concurrent.TimeoutException: Node edg-perf02-56077 timed out
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:196)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:531)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:303)
> at org.infinispan.interceptors.distribution.BaseDistributionInterceptor.handleNonTxWriteCommand(BaseDistributionInterceptor.java:151)
> at org.infinispan.interceptors.distribution.NonTxDistributionInterceptor.visitPutKeyValueCommand(NonTxDistributionInterceptor.java:91)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
> at org.infinispan.interceptors.EntryWrappingInterceptor.invokeNextAndApplyChanges(EntryWrappingInterceptor.java:290)
> at org.infinispan.interceptors.EntryWrappingInterceptor.visitPutKeyValueCommand(EntryWrappingInterceptor.java:157)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
> at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitPutKeyValueCommand(AbstractLockingInterceptor.java:70)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:134)
> at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:54)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:134)
> at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:54)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
> at org.infinispan.statetransfer.StateTransferInterceptor.handleTopologyAffectedCommand(StateTransferInterceptor.java:216)
> at org.infinispan.statetransfer.StateTransferInterceptor.handleWriteCommand(StateTransferInterceptor.java:194)
> at org.infinispan.statetransfer.StateTransferInterceptor.visitPutKeyValueCommand(StateTransferInterceptor.java:136)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
> at org.infinispan.interceptors.CacheMgmtInterceptor.visitPutKeyValueCommand(CacheMgmtInterceptor.java:160)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:128)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:92)
> at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:54)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
> at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:343)
> at org.infinispan.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1337)
> at org.infinispan.CacheImpl.putInternal(CacheImpl.java:898)
> at org.infinispan.CacheImpl.put(CacheImpl.java:890)
> at org.infinispan.CacheImpl.put(CacheImpl.java:1390)
> at org.infinispan.CacheImpl.put(CacheImpl.java:229)
> at org.infinispan.distexec.mapreduce.MapReduceManagerImpl.combine(MapReduceManagerImpl.java:326)
> at org.infinispan.distexec.mapreduce.MapReduceManagerImpl.mapAndCombineForDistributedReduction(MapReduceManagerImpl.java:112)
> at org.infinispan.commands.read.MapCombineCommand.perform(MapCombineCommand.java:93)
> at org.infinispan.remoting.InboundInvocationHandlerImpl.handleInternal(InboundInvocationHandlerImpl.java:122)
> at org.infinispan.remoting.InboundInvocationHandlerImpl.access$000(InboundInvocationHandlerImpl.java:68)
> at org.infinispan.remoting.InboundInvocationHandlerImpl$2.run(InboundInvocationHandlerImpl.java:194)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:724)
> Caused by: org.jgroups.TimeoutException: timeout sending message to edg-perf02-56077
> at org.jgroups.blocks.MessageDispatcher.sendMessage(MessageDispatcher.java:419)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.processSingleCall(CommandAwareRpcDispatcher.java:375)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:189)
> ... 48 more
> 12:57:32,494 ERROR [org.infinispan.remoting.InboundInvocationHandlerImpl] (remote-thread-0) Exception executing command
> org.infinispan.CacheException: org.infinispan.CacheException: Could not move intermediate keys/values for M/R task edc7a51c-d25a-4650-b39a-8d8a87d7cd61
> at org.infinispan.distexec.mapreduce.MapReduceManagerImpl.mapAndCombineForDistributedReduction(MapReduceManagerImpl.java:114)
> at org.infinispan.commands.read.MapCombineCommand.perform(MapCombineCommand.java:93)
> at org.infinispan.remoting.InboundInvocationHandlerImpl.handleInternal(InboundInvocationHandlerImpl.java:122)
> at org.infinispan.remoting.InboundInvocationHandlerImpl.access$000(InboundInvocationHandlerImpl.java:68)
> at org.infinispan.remoting.InboundInvocationHandlerImpl$2.run(InboundInvocationHandlerImpl.java:194)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:724)
> Caused by: org.infinispan.CacheException: Could not move intermediate keys/values for M/R task edc7a51c-d25a-4650-b39a-8d8a87d7cd61
> at org.infinispan.distexec.mapreduce.MapReduceManagerImpl.combine(MapReduceManagerImpl.java:331)
> at org.infinispan.distexec.mapreduce.MapReduceManagerImpl.mapAndCombineForDistributedReduction(MapReduceManagerImpl.java:112)
> ... 7 more
> Caused by: org.infinispan.util.concurrent.TimeoutException: Node edg-perf02-56077 timed out
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:196)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:531)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:303)
> at org.infinispan.interceptors.distribution.BaseDistributionInterceptor.handleNonTxWriteCommand(BaseDistributionInterceptor.java:151)
> at org.infinispan.interceptors.distribution.NonTxDistributionInterceptor.visitPutKeyValueCommand(NonTxDistributionInterceptor.java:91)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
> at org.infinispan.interceptors.EntryWrappingInterceptor.invokeNextAndApplyChanges(EntryWrappingInterceptor.java:290)
> at org.infinispan.interceptors.EntryWrappingInterceptor.visitPutKeyValueCommand(EntryWrappingInterceptor.java:157)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
> at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitPutKeyValueCommand(AbstractLockingInterceptor.java:70)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:134)
> at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:54)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:134)
> at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:54)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
> at org.infinispan.statetransfer.StateTransferInterceptor.handleTopologyAffectedCommand(StateTransferInterceptor.java:216)
> at org.infinispan.statetransfer.StateTransferInterceptor.handleWriteCommand(StateTransferInterceptor.java:194)
> at org.infinispan.statetransfer.StateTransferInterceptor.visitPutKeyValueCommand(StateTransferInterceptor.java:136)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
> at org.infinispan.interceptors.CacheMgmtInterceptor.visitPutKeyValueCommand(CacheMgmtInterceptor.java:160)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:128)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:92)
> at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:54)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
> at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:343)
> at org.infinispan.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1337)
> at org.infinispan.CacheImpl.putInternal(CacheImpl.java:898)
> at org.infinispan.CacheImpl.put(CacheImpl.java:890)
> at org.infinispan.CacheImpl.put(CacheImpl.java:1390)
> at org.infinispan.CacheImpl.put(CacheImpl.java:229)
> at org.infinispan.distexec.mapreduce.MapReduceManagerImpl.combine(MapReduceManagerImpl.java:326)
> ... 8 more
> Caused by: org.jgroups.TimeoutException: timeout sending message to edg-perf02-56077
> at org.jgroups.blocks.MessageDispatcher.sendMessage(MessageDispatcher.java:419)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.processSingleCall(CommandAwareRpcDispatcher.java:375)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:189)
> ... 48 more
> {noformat}
> ISPN-2836 added a timeout for the entire MapReduceTask, but this timeout is happening when the intermediate keys are put into the cache by the map phase. This error does not occur when the value size is 8KB or larger.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 1 month
[JBoss JIRA] (ISPN-3404) ClusterCacheLoader NPE when ClusteredGetCommand response value is null
by RH Bugzilla Integration (JIRA)
[ https://issues.jboss.org/browse/ISPN-3404?page=com.atlassian.jira.plugin.... ]
RH Bugzilla Integration commented on ISPN-3404:
-----------------------------------------------
Tristan Tarrant <ttarrant(a)redhat.com> changed the Status of [bug 996045|https://bugzilla.redhat.com/show_bug.cgi?id=996045] from ASSIGNED to ON_QA
> ClusterCacheLoader NPE when ClusteredGetCommand response value is null
> ----------------------------------------------------------------------
>
> Key: ISPN-3404
> URL: https://issues.jboss.org/browse/ISPN-3404
> Project: Infinispan
> Issue Type: Bug
> Affects Versions: 6.0.0.Alpha2
> Reporter: Tomas Sykora
> Assignee: Pedro Ruivo
> Labels: 620
> Fix For: 6.0.0.Final
>
> Attachments: clusterCacheLoaderNullPointerResponseValueError.txt, server1DR2.log, server2DR2.log
>
>
> We found this issue in DR2 (Alpha2). In Alpha1 this was ok.
> Our internal test:
> {code:title=Bar.java|borderStyle=solid}
> @Test
> public void testLazyLoadingWhenStateTransferDisabled() throws Exception {
> controller.start(CONTAINER1);
> mc = new MemcachedHelper(server1.getMemcachedEndpoint().getInetAddress().getHostName(), server1.getMemcachedEndpoint()
> .getPort());
> mc.set("k1", "v1");
> mc.set("k2", "v2");
> assertEquals("v1", mc.get("k1"));
> assertEquals("v2", mc.get("k2"));
> assertEquals(2, server1.getDefaultCacheManager().getCache(CACHE_NAME).getNumberOfEntries());
> controller.start(CONTAINER2);
> mc2 = new MemcachedHelper(server2.getMemcachedEndpoint().getInetAddress().getHostName(), server2.getMemcachedEndpoint()
> .getPort());
> assertEquals(2, server2.getDefaultCacheManager().getClusterSize());
> //state-transfer = false -> no entries in the newly joined node
> assertEquals(0, server2.getDefaultCacheManager().getCache(CACHE_NAME).getNumberOfEntries());
> assertEquals("v1", mc2.get("k1")); //lazily load the entries
> assertEquals("v2", mc2.get("k2"));
> assertEquals(2, server2.getDefaultCacheManager().getCache(CACHE_NAME).getNumberOfEntries());
> mc2.set("k3", "v3"); // THIS IS FAILING HERE
> assertEquals(3, server2.getDefaultCacheManager().getCache(CACHE_NAME).getNumberOfEntries());
> assertEquals(3, server1.getDefaultCacheManager().getCache(CACHE_NAME).getNumberOfEntries());
> controller.kill(CONTAINER2);
> controller.kill(CONTAINER1);
> }
> {code}
> *mc2.set("k3", "v3"); = put into memcached cache, is throwing an exception: see in attachment.*
> You can see MemcachedHelper class here: https://code.engineering.redhat.com/gerrit/gitweb?p=jdg-functional-tests....
> I also include trace logs for both memcached jdg servers if they can be useful.
> ---------------
> I was able to replicate the same error in Infinispan test suite for Rest endpoint. It was enough to start 2 REST nodes and issue 1 put resulting into the same error.
> {code:title=Bar.java|borderStyle=solid}
> ConfigurationBuilder cfgBuilder = AbstractCacheTest.getDefaultClusteredCacheConfig(CacheMode.REPL_SYNC, false);
> cfgBuilder.clustering().hash().numOwners(1);
> cfgBuilder.clustering().stateTransfer().fetchInMemoryState(false);
> cfgBuilder.clustering().stateTransfer().timeout(20000);
> cfgBuilder.loaders().addClusterCacheLoader().remoteCallTimeout(60000);
> RestServerConfigurationBuilder restCfgBuilder = new RestServerConfigurationBuilder();
> addServer("1", 8890, TestCacheManagerFactory.createClusteredCacheManager(cfgBuilder), restCfgBuilder.build());
> addServer("2", 8891, TestCacheManagerFactory.createClusteredCacheManager(cfgBuilder), restCfgBuilder.build());
> startServers();
> ......
> PutMethod put = new PutMethod(PATH1 + "key1");
> put.setRequestEntity(new StringRequestEntity("value1", "application/text", null));
> call(put);
> assertEquals(put.getStatusCode(), HttpServletResponse.SC_OK);
> put.releaseConnection();
> {code}
> This scenario is different from aforementioned JDG internal test suite one but I expect this should be ok and should be performed without any problems.
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 1 month
[JBoss JIRA] (ISPN-3599) CommitCommand with replayed PrepareCommand executes rollback and then commit
by RH Bugzilla Integration (JIRA)
[ https://issues.jboss.org/browse/ISPN-3599?page=com.atlassian.jira.plugin.... ]
RH Bugzilla Integration commented on ISPN-3599:
-----------------------------------------------
Tristan Tarrant <ttarrant(a)redhat.com> changed the Status of [bug 1016054|https://bugzilla.redhat.com/show_bug.cgi?id=1016054] from POST to ON_QA
> CommitCommand with replayed PrepareCommand executes rollback and then commit
> ----------------------------------------------------------------------------
>
> Key: ISPN-3599
> URL: https://issues.jboss.org/browse/ISPN-3599
> Project: Infinispan
> Issue Type: Bug
> Components: State transfer, Transactions
> Affects Versions: 6.0.0.CR1
> Reporter: Radim Vansa
> Assignee: Pedro Ruivo
> Priority: Critical
> Labels: 620
> Fix For: 6.0.0.Final
>
>
> During state-transfer in tx cache, the node can receive {{CommitCommand}} from other node. After the node gets transaction data for affected segments, it creates the transaction with {{missingLookedUpEntries=true}} and the {{CommitCommand}} can be executed.
> In this command's {{perform(...)}} the transaction is *first* marked as completed, then it enters the interceptor chain. There, the {{PrepareCommand}} is created in {{StateTransferInterceptor.visitCommitCommand}} but after this is processed the {{TxInterceptor}} finds out that the transaction is already completed and executes {{RollbackCommand}}, clearing locks etc.
> Nevertheless, {{StateTransferInterceptor}} executes the initial {{CommitCommand}} afterwards. I suspect that this may be executed without the locks held.
> Anyway, it is not correct to execute both commit and rollback on the same transaction.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 1 month