Hey guys, Investigating why EmbeddedCacheManager#cacheRemove hiccups DistributedTwoNodesMapReduceTest. As you might recall upon end of MapReduceTask there is EmbeddedCacheManager#cacheRemove call to remove intermediate caches across the cluster. Very often, almost every test run, execution of cache remove is blocked by ongoing transactions that have not completed - more specifically TransactionTable shows pending remote transaction. This in turn prevents cache stop call, which in turn causes a timeout on cache remove thus failing the test. MapReduce uses txs in MapReduceManagerImpl#combine - it might well be that there is smth wrong there or it could be that somehow TxCompletionNotificationCommand is not cleaning up remote txs. Either way I would appreciate some help here - Mircea? Regards, Vladimir _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

Hey guys, Investigating why EmbeddedCacheManager#cacheRemove hiccups DistributedTwoNodesMapReduceTest. As you might recall upon end of MapReduceTask there is EmbeddedCacheManager#cacheRemove call to remove intermediate caches across the cluster. Very often, almost every test run, execution of cache remove is blocked by ongoing transactions that have not completed - more specifically TransactionTable shows pending remote transaction. This in turn prevents cache stop call, which in turn causes a timeout on cache remove thus failing the test. MapReduce uses txs in MapReduceManagerImpl#combine - it might well be that there is smth wrong there or it could be that somehow

TxCompletionNotificationCommand is not cleaning up remote txs. Either way I would appreciate some help here - Mircea? Regards, Vladimir _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

On 18 Oct 2012, at 16:41, Manik Surtani wrote: > > On 18 Oct 2012, at 17:45, Vladimir Blagojevic <vblagoje(a)redhat.com&gt; wrote: > >> Hey guys, >> >> Investigating why EmbeddedCacheManager#cacheRemove hiccups >> DistributedTwoNodesMapReduceTest. As you might recall upon end of >> MapReduceTask there is EmbeddedCacheManager#cacheRemove call to remove >> intermediate caches across the cluster. I think you meant ECM#removeCache? >> Very often, almost every test >> run, execution of cache remove is blocked by ongoing transactions that >> have not completed - more specifically TransactionTable shows pending >> remote transaction. This in turn prevents cache stop call, which in turn >> causes a timeout on cache remove thus failing the test. >> >> MapReduce uses txs in MapReduceManagerImpl#combine - it might well be >> that there is smth wrong there or it could be that somehow > > ^^^ Does this mean that to use Map/Reduce now, we *require* that a transaction manager is available, configured and present? Not really. It's only that the problem Vladimir describes only occurs when you run the MR against transactional caches (there is a tx leak which prevents caches to stop, that should be fixed by a pull request I sent).

On 18 Oct 2012, at 18:21, Manik Surtani wrote: > On 19 Oct 2012, at 02:19, Mircea Markus <mircea.markus(a)jboss.com&gt; wrote: > >> >> On 18 Oct 2012, at 16:41, Manik Surtani wrote: >> >>> >>> On 18 Oct 2012, at 17:45, Vladimir Blagojevic <vblagoje(a)redhat.com&gt; wrote: >>> >>>> Hey guys, >>>> >>>> Investigating why EmbeddedCacheManager#cacheRemove hiccups >>>> DistributedTwoNodesMapReduceTest. As you might recall upon end of >>>> MapReduceTask there is EmbeddedCacheManager#cacheRemove call to remove >>>> intermediate caches across the cluster. >> I think you meant ECM#removeCache? >>>> Very often, almost every test >>>> run, execution of cache remove is blocked by ongoing transactions that >>>> have not completed - more specifically TransactionTable shows pending >>>> remote transaction. This in turn prevents cache stop call, which in turn >>>> causes a timeout on cache remove thus failing the test. >>>> >>>> MapReduce uses txs in MapReduceManagerImpl#combine - it might well be >>>> that there is smth wrong there or it could be that somehow >>> >>> ^^^ Does this mean that to use Map/Reduce now, we *require* that a transaction manager is available, configured and present? >> Not really. It's only that the problem Vladimir describes only occurs when you run the MR against transactional caches (there is a tx leak which prevents caches to stop, that should be fixed by a pull request I sent). > > Ok. I was just wondering, since Vladimir did say MR uses txs in MapReduceManagerImpl#combine. Vladimir, could you pls clarify? You're right actually, the temporary cache created is transactional. it is built in the CreateCacheCommand and relies on the DummyTransactionManager, might be better to use batching perhaps? Or even not require for this cache to be transactional?

On 12-10-19 4:15 AM, Manik Surtani wrote: > > You're right actually, the temporary cache created is transactional. it is built in the CreateCacheCommand and relies on the DummyTransactionManager, might be better to use batching perhaps? Or even not require for this cache to be transactional? > > Yes, we should keep such internal caches as simple as possible. > Mircea/Manik, it could be batch I believe. If you recall Mircea, you and I talked about how to effectively move data across cluster deadlock free - we even agreed we should blog post about it :-) https://issues.jboss.org/browse/ISPN-2156 Mircea/Manik could I get some advice and code review for MapReduceManagerImpl#combine?

Thanks guys, Vladimir _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

On 19 Oct 2012, at 17:40, Mircea Markus <mircea.markus(a)jboss.com <mailto:mircea.markus@jboss.com>> wrote: > > On 19 Oct 2012, at 07:41, Vladimir Blagojevic wrote: > >> On 12-10-19 4:15 AM, Manik Surtani wrote: >>> >>> You're right actually, the temporary cache created is >>> transactional. it is built in the CreateCacheCommand and relies on >>> the DummyTransactionManager, might be better to use batching >>> perhaps? Or even not require for this cache to be transactional? >>> >>> Yes, we should keep such internal caches as simple as possible. >>> >> Mircea/Manik, it could be batch I believe. If you recall Mircea, you >> and I talked about how to effectively move data across cluster >> deadlock free - we even agreed we should blog post about it :-) >> https://issues.jboss.org/browse/ISPN-2156 >> >> Mircea/Manik could I get some advice and code review for >> MapReduceManagerImpl#combine? > Thanks Vladimir. The only change to make > is CreateCacheCommand.create, configure the cache to be batchable[1]. > Then in the MapReduceManagerImpl#combine, don't use the > TransactionManager.beggin/commit but the batch API. Is this in? Have we redone M/R not to rely on transactions and instead to use batching yet? Vladimir?