DistributedSharedCacheFourNodesMapReduceTest.testInvokeMapReduceOnAllKeysWithCollator fails on IMB Java 6 --------------------------------------------------------------------------------------------------------- Key: ISPN-4101 URL: https://issues.jboss.org/browse/ISPN-4101 Project: Infinispan Issue Type: Bug Security Level: Public(Everyone can see) Components: Test Suite - Core Environment: RHEL5 x86_64, IMB JDK 6 Reporter: Vojtech Juranek Assignee: Vladimir Blagojevic Fix For: 7.0.0.Alpha5 {{org.infinispan.distexec.mapreduce.DistributedSharedCacheFourNodesMapReduceTest.testInvokeMapReduceOnAllKeysWithCollator}} fails with {noformat} org.infinispan.commons.CacheException: java.util.concurrent.ExecutionException: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from DistributedSharedCacheFourNodesMapReduceTest-NodeB-50126, see cause for remote stack trace at org.infinispan.distexec.mapreduce.MapReduceTask.execute(MapReduceTask.java:338) at org.infinispan.distexec.mapreduce.MapReduceTask.execute(MapReduceTask.java:624) at org.infinispan.distexec.mapreduce.BaseWordCountMapReduceTest.testInvokeMapReduceOnAllKeysWithCollator(BaseWordCountMapReduceTest.java:226) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:60) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37) at java.lang.reflect.Method.invoke(Method.java:611) at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:80) at org.testng.internal.Invoker.invokeMethod(Invoker.java:715) at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:907) at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1237) at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:127) at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:111) at org.testng.TestRunner.privateRun(TestRunner.java:767) at org.testng.TestRunner.run(TestRunner.java:617) at org.testng.SuiteRunner.runTest(SuiteRunner.java:334) at org.testng.SuiteRunner.access$000(SuiteRunner.java:37) at org.testng.SuiteRunner$SuiteWorker.run(SuiteRunner.java:368) at org.testng.internal.thread.ThreadUtil$2.call(ThreadUtil.java:64) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:314) at java.util.concurrent.FutureTask.run(FutureTask.java:149) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:906) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:929) at java.lang.Thread.run(Thread.java:761) Caused by: java.util.concurrent.ExecutionException: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from DistributedSharedCacheFourNodesMapReduceTest-NodeB-50126, see cause for remote stack trace at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:233) at java.util.concurrent.FutureTask.get(FutureTask.java:94) at org.infinispan.distexec.mapreduce.MapReduceTask$TaskPart.get(MapReduceTask.java:835) at org.infinispan.distexec.mapreduce.MapReduceTask.executeMapPhase(MapReduceTask.java:429) at org.infinispan.distexec.mapreduce.MapReduceTask.execute(MapReduceTask.java:332) ... 23 more Caused by: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from DistributedSharedCacheFourNodesMapReduceTest-NodeB-50126, see cause for remote stack trace at org.infinispan.remoting.transport.AbstractTransport.checkResponse(AbstractTransport.java:41) at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.processSingleCall(CommandAwareRpcDispatcher.java:358) at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:167) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:521) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:274) at org.infinispan.remoting.rpc.RpcManagerImpl$2.call(RpcManagerImpl.java:306) ... 5 more Caused by: org.infinispan.commons.CacheException: org.infinispan.commons.CacheException: Could not move intermediate keys/values for M/R task 5fcd5650-8b3e-49e0-b32b-7856a94d677e at org.infinispan.distexec.mapreduce.MapReduceManagerImpl.mapAndCombineForDistributedReduction(MapReduceManagerImpl.java:100) at org.infinispan.commands.read.MapCombineCommand.perform(MapCombineCommand.java:71) at org.infinispan.remoting.InboundInvocationHandlerImpl.handleInternal(InboundInvocationHandlerImpl.java:95) at org.infinispan.remoting.InboundInvocationHandlerImpl.access$000(InboundInvocationHandlerImpl.java:50) at org.infinispan.remoting.InboundInvocationHandlerImpl$2.run(InboundInvocationHandlerImpl.java:177) ... 3 more Caused by: org.infinispan.commons.CacheException: Could not move intermediate keys/values for M/R task 5fcd5650-8b3e-49e0-b32b-7856a94d677e at org.infinispan.distexec.mapreduce.MapReduceManagerImpl.migrateIntermediateKeys(MapReduceManagerImpl.java:287) at org.infinispan.distexec.mapreduce.MapReduceManagerImpl.combine(MapReduceManagerImpl.java:253) at org.infinispan.distexec.mapreduce.MapReduceManagerImpl.mapAndCombineForDistributedReduction(MapReduceManagerImpl.java:98) ... 7 more Caused by: org.infinispan.commons.CacheException: java.util.ConcurrentModificationException at org.infinispan.commons.util.Util.rewrapAsCacheException(Util.java:581) at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:176) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:521) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:274) at org.infinispan.interceptors.distribution.BaseDistributionInterceptor.handleNonTxWriteCommand(BaseDistributionInterceptor.java:238) at org.infinispan.interceptors.distribution.NonTxDistributionInterceptor.visitPutKeyValueCommand(NonTxDistributionInterceptor.java:72) at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98) at org.infinispan.interceptors.EntryWrappingInterceptor.invokeNextAndApplyChanges(EntryWrappingInterceptor.java:326) at org.infinispan.interceptors.EntryWrappingInterceptor.setSkipRemoteGetsAndInvokeNextForDataCommand(EntryWrappingInterceptor.java:407) at org.infinispan.interceptors.EntryWrappingInterceptor.visitPutKeyValueCommand(EntryWrappingInterceptor.java:164) at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98) at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitPutKeyValueCommand(AbstractLockingInterceptor.java:68) at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98) at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:112) at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:32) at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98) at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:112) at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:32) at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98) at org.infinispan.statetransfer.StateTransferInterceptor.handleNonTxWriteCommand(StateTransferInterceptor.java:219) at org.infinispan.statetransfer.StateTransferInterceptor.visitPutKeyValueCommand(StateTransferInterceptor.java:141) at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98) at org.infinispan.interceptors.CacheMgmtInterceptor.updateStoreStatistics(CacheMgmtInterceptor.java:148) at org.infinispan.interceptors.CacheMgmtInterceptor.visitPutKeyValueCommand(CacheMgmtInterceptor.java:134) at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98) at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:110) at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:73) at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:32) at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71) at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:333) at org.infinispan.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1404) at org.infinispan.CacheImpl.putInternal(CacheImpl.java:882) at org.infinispan.CacheImpl.put(CacheImpl.java:874) at org.infinispan.CacheImpl.put(CacheImpl.java:1449) at org.infinispan.CacheImpl.put(CacheImpl.java:219) at org.infinispan.distexec.mapreduce.MapReduceManagerImpl.migrateIntermediateKeys(MapReduceManagerImpl.java:279) ... 9 more Caused by: java.util.ConcurrentModificationException at java.util.LinkedList$LinkIterator.next(LinkedList.java:124) at java.util.LinkedList.writeObject(LinkedList.java:980) at sun.reflect.GeneratedMethodAccessor256.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37) at java.lang.reflect.Method.invoke(Method.java:611) at org.jboss.marshalling.reflect.SerializableClass.callWriteObject(SerializableClass.java:275) at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:1010) at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:1001) at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:888) at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:62) at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:115) at org.infinispan.remoting.responses.SuccessfulResponse$Externalizer.writeObject(SuccessfulResponse.java:71) at org.infinispan.remoting.responses.SuccessfulResponse$Externalizer.writeObject(SuccessfulResponse.java:64) at org.infinispan.marshall.core.ExternalizerTable$ExternalizerAdapter.writeObject(ExternalizerTable.java:395) at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:148) at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:62) at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:115) at org.infinispan.commons.marshall.jboss.AbstractJBossMarshaller.objectToObjectStream(AbstractJBossMarshaller.java:74) at org.infinispan.marshall.core.VersionAwareMarshaller.objectToBuffer(VersionAwareMarshaller.java:77) at org.infinispan.commons.marshall.AbstractMarshaller.objectToBuffer(AbstractMarshaller.java:41) at org.infinispan.commons.marshall.AbstractDelegatingMarshaller.objectToBuffer(AbstractDelegatingMarshaller.java:85) at org.infinispan.remoting.transport.jgroups.MarshallerAdapter.objectToBuffer(MarshallerAdapter.java:23) at org.jgroups.blocks.RequestCorrelator.sendReply(RequestCorrelator.java:486) at org.jgroups.blocks.RequestCorrelator$ResponseImpl.send(RequestCorrelator.java:540) at org.infinispan.remoting.InboundInvocationHandlerImpl.reply(InboundInvocationHandlerImpl.java:225) at org.infinispan.remoting.InboundInvocationHandlerImpl.access$200(InboundInvocationHandlerImpl.java:50) at org.infinispan.remoting.InboundInvocationHandlerImpl$2.run(InboundInvocationHandlerImpl.java:182) ... 3 more Caused by: an exception which occurred: in object org.infinispan.distexec.mapreduce.MapReduceManagerImpl$DeltaAwareList@7fc2 -> toString = [3, 1, 1] in object org.infinispan.remoting.responses.SuccessfulResponse@7fc2 -> toString = SuccessfulResponse{responseValue=[3, 1, 1]} {noformat}