[JBoss JIRA] (ISPN-2836) org.jgroups.TimeoutException after invoking MapCombineCommand in Map/Reduce task with 2 nodes
by Pedro Ruivo (JIRA)
[ https://issues.jboss.org/browse/ISPN-2836?page=com.atlassian.jira.plugin.... ]
Pedro Ruivo commented on ISPN-2836:
-----------------------------------
[~afield] do you have any way to solve the decode problem? (to eliminate the chinese chars)
what I got so far running on my laptop:
* it is counting cache entries instead of words. The decode problem creates a 1MB cache entry with no spaces.
* a 580MB file is generating a lot of data (I don't know from where). I have 8GB (+ 8GB swap) and it took all the memory and half of the swap (with 4GB heap space per VM)
About the time spent, I have so far:
* Map phase: the time it takes to iterate over all keys and count the words
* Key migration phase: the time it takes to pick the results from the map phase and put them in a shared cache (for distributed reduction)
originator (the cache where radargun requested the map reduce task)
Map phase for task 7da673ca-9aaf-4e0b-a231-29325cfbae8a took 2356 milliseconds
/*TimeoutException, my computer has frozen for a while and a new view + merge view has generated*/
Migrating keys for task 7da673ca-9aaf-4e0b-a231-29325cfbae8a took 114773 milliseconds (Migrated 278 keys)
remote (the second cache)
Map phase for task 7da673ca-9aaf-4e0b-a231-29325cfbae8a took 2001 milliseconds
Migrating keys for task 7da673ca-9aaf-4e0b-a231-29325cfbae8a took 68121 milliseconds (Migrated 271 keys)
ps. Not all keys have been migrated.
> org.jgroups.TimeoutException after invoking MapCombineCommand in Map/Reduce task with 2 nodes
> ---------------------------------------------------------------------------------------------
>
> Key: ISPN-2836
> URL: https://issues.jboss.org/browse/ISPN-2836
> Project: Infinispan
> Issue Type: Bug
> Components: Distributed Execution and Map/Reduce
> Affects Versions: 5.2.1.Final
> Reporter: Alan Field
> Assignee: Pedro Ruivo
> Priority: Blocker
> Labels: onboard
> Fix For: 5.3.0.Final
>
> Attachments: afield-tcp-521-final.txt, benchmark-mapreduce-multifilesize.xml, dist-udp-no-tx.xml, jgroups-udp.xml, udp-edg-perf01.txt, udp-edg-perf02.txt
>
>
> Using RadarGun and two nodes to execute the example WordCount Map/Reduce job against a cache with ~550 keys with a value size of 1MB is producing a thread deadlock. The cache is distributed with transactions disabled.
> TCP transport deadlocks without throwing an exception. Disabling the send queue and setting UNICAST2.conn_expiry_timeout=0 prevents the deadlock, but the job does not complete. The nodes send "are-you-alive" messages back and forth, and I have seen the following exception:
> {noformat}
> 11:44:29,970 ERROR [org.jgroups.protocols.TCP] (OOB-98,default,edg-perf01-1907) failed sending message to edg-perf02-32536 (76 bytes): java.net.SocketException: Socket closed, cause: null
> at org.infinispan.distexec.mapreduce.MapReduceTask.execute(MapReduceTask.java:352)
> at org.radargun.cachewrappers.InfinispanMapReduceWrapper.executeMapReduceTask(InfinispanMapReduceWrapper.java:98)
> at org.radargun.stages.MapReduceStage.executeOnSlave(MapReduceStage.java:74)
> at org.radargun.Slave$2.run(Slave.java:103)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: java.util.concurrent.ExecutionException: org.infinispan.CacheException: org.jgroups.TimeoutException: timeout sending message to edg-perf02-32536
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
> at java.util.concurrent.FutureTask.get(FutureTask.java:83)
> at org.infinispan.distexec.mapreduce.MapReduceTask$TaskPart.get(MapReduceTask.java:832)
> at org.infinispan.distexec.mapreduce.MapReduceTask.executeMapPhaseWithLocalReduction(MapReduceTask.java:477)
> at org.infinispan.distexec.mapreduce.MapReduceTask.execute(MapReduceTask.java:350)
> ... 9 more
> Caused by: org.infinispan.CacheException: org.jgroups.TimeoutException: timeout sending message to edg-perf02-32536
> at org.infinispan.util.Util.rewrapAsCacheException(Util.java:541)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:186)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:515)
> 11:44:29,978 ERROR [org.jgroups.protocols.TCP] (Timer-3,default,edg-perf01-1907) failed sending message to edg-perf02-32536 (60 bytes): java.net.SocketException: Socket closed, cause: null
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:175)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:197)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:254)
> at org.infinispan.remoting.rpc.RpcManagerImpl.access$000(RpcManagerImpl.java:80)
> at org.infinispan.remoting.rpc.RpcManagerImpl$1.call(RpcManagerImpl.java:288)
> ... 5 more
> Caused by: org.jgroups.TimeoutException: timeout sending message to edg-perf02-32536
> at org.jgroups.blocks.MessageDispatcher.sendMessage(MessageDispatcher.java:390)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.processSingleCall(CommandAwareRpcDispatcher.java:301)
> 11:44:29,979 ERROR [org.jgroups.protocols.TCP] (Timer-4,default,edg-perf01-1907) failed sending message to edg-perf02-32536 (63 bytes): java.net.SocketException: Socket closed, cause: null
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:179)
> ... 11 more
> {noformat}
> With UDP transport, both threads are deadlocked. I will attach thread dumps from runs using TCP and UDP transport.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
10 years, 11 months
[JBoss JIRA] (ISPN-2990) L1ManagerImpl doesn't reliably invalidate with async caches
by Dan Berindei (JIRA)
[ https://issues.jboss.org/browse/ISPN-2990?page=com.atlassian.jira.plugin.... ]
Dan Berindei commented on ISPN-2990:
------------------------------------
The problem is not just with the ordering of the messages, the order of events on node B could be something like this:
1. Receive {{get(k1)}} command from A. Store the value v1 in the invocation context.
2. Receive {{put(k1, v2)}} command from A. Commit the value in the DataContainer.
3. Send an invalidate(k1) command to A
4. Send the response v1 for the {{get(k1)}} command to A.
5. Send the response v1 for the {{put(k1, v2)}} command to A.
The description of the bug implies that we don't do step 3 at the moment (i.e. we don't send an invalidation command to the originator of the put command). But even if we did, the {{get(k1)}} response is very likely to be processed on A after the {{invalidate(k1)}} command - meaning A will still keep v1 in its DataContainer.
> L1ManagerImpl doesn't reliably invalidate with async caches
> -----------------------------------------------------------
>
> Key: ISPN-2990
> URL: https://issues.jboss.org/browse/ISPN-2990
> Project: Infinispan
> Issue Type: Bug
> Components: Distributed Cache
> Affects Versions: 5.2.1.Final
> Reporter: Sebastian Tusk
> Assignee: William Burns
> Labels: onboard
>
> B is owner of k,v1
> A has k,v1 in L1
> 1. TX: A puts k,v2
> 2. TX: A sends async PrepareCommand k,v2 to B (one-phase-commit)
> 3. TX: A removes k,v1 from L1
> 4. A putForExternalRead k,v1 and has it in L1 again
> 5. TX: B executes PrepareCommand k,v2 but doesn't send invalidation to origin
> Result: A has k,v1 and B has k,v2
> Solution: For async caches send invalidation to origin too.
> The problem is that the owner updates the cache entry asynchronously. This gives the origin of the transaction time to request the entry. Here an outdated version is received and placed in L1. The owner never invalidates the entry and as result the cache is inconsistent.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
10 years, 11 months
[JBoss JIRA] (ISPN-3192) Concurrent TreeCache.move() calls with the same destination lose data
by Dan Berindei (JIRA)
Dan Berindei created ISPN-3192:
----------------------------------
Summary: Concurrent TreeCache.move() calls with the same destination lose data
Key: ISPN-3192
URL: https://issues.jboss.org/browse/ISPN-3192
Project: Infinispan
Issue Type: Bug
Components: Tree API
Affects Versions: 5.3.0.CR1
Reporter: Dan Berindei
Assignee: Dan Berindei
Fix For: 5.3.0.Final
The move method reads the contents of the destination node in the transaction/invocation context before locking it.
If there are multiple parallel movers with the same destination, some of the moved nodes might be lost. This sometimes happens in NodeMoveAPIPessimisticTest, causing random failures.
Note that even if the move() method locks the destination node, it will still be possible for the user to read the destination node in the same transaction and cause data loss. The move() method documentation should warn about this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
10 years, 11 months
[JBoss JIRA] (ISPN-3163) Replacing entry via HotRod which was initially stored via Memcached does not change CAS
by Galder Zamarreño (JIRA)
[ https://issues.jboss.org/browse/ISPN-3163?page=com.atlassian.jira.plugin.... ]
Work on ISPN-3163 started by Galder Zamarreño.
> Replacing entry via HotRod which was initially stored via Memcached does not change CAS
> ---------------------------------------------------------------------------------------
>
> Key: ISPN-3163
> URL: https://issues.jboss.org/browse/ISPN-3163
> Project: Infinispan
> Issue Type: Bug
> Affects Versions: 5.3.0.CR1
> Reporter: Martin Gencur
> Assignee: Galder Zamarreño
> Fix For: 5.3.0.Final
>
>
> Users might expect that CAS (check-and-set) operation will work even in compatibility mode which is currently not true in the following scenario:
> 1) store a key/value via Memcached
> 2) change the value via HotRod or Embedded
> 3) use Memcached's CAS operation
> In step #3 the memcached client will update the value even though the value was changed by another client in the meantime. The memcached client was supposed to change it only if it had not been changed in the meantime.
> The following test snippet shows the problem:
> {code:java}
> public void testMemcachedPutHotRodEmbbeddedReplaceMemcachedCASTest() throws Exception {
> final String key1 = "5";
> // 1. Put with Memcached
> Future<Boolean> f = cacheFactory.getMemcachedClient().set(key1, 0, "v1");
> assertTrue(f.get(60, TimeUnit.SECONDS));
> CASValue oldValue = cacheFactory.getMemcachedClient().gets(key1);
> // 2. Replace with Hot Rod
> VersionedValue versioned = cacheFactory.getHotRodCache().getVersioned(key1);
> assertTrue(cacheFactory.getHotRodCache().replaceWithVersion(key1, "v2", versioned.getVersion()));
> // 3. Replace with Embedded
> assertTrue(cacheFactory.getEmbeddedCache().replace(key1, "v2", "v3"));
> // 4. Get with Memcached and verify value/CAS
> CASValue newValue = cacheFactory.getMemcachedClient().gets(key1);
> assertEquals("v3", newValue.getValue());
> assertTrue("The version (CAS) should have changed", oldValue.getCas() != newValue.getCas());
> //<---- fails here
> }
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
10 years, 11 months
[JBoss JIRA] (ISPN-2990) L1ManagerImpl doesn't reliably invalidate with async caches
by Bela Ban (JIRA)
[ https://issues.jboss.org/browse/ISPN-2990?page=com.atlassian.jira.plugin.... ]
Bela Ban commented on ISPN-2990:
--------------------------------
If you're referring to the ordering between unicasts and multicasts then yes, there is no ordering between them
> L1ManagerImpl doesn't reliably invalidate with async caches
> -----------------------------------------------------------
>
> Key: ISPN-2990
> URL: https://issues.jboss.org/browse/ISPN-2990
> Project: Infinispan
> Issue Type: Bug
> Components: Distributed Cache
> Affects Versions: 5.2.1.Final
> Reporter: Sebastian Tusk
> Assignee: William Burns
> Labels: onboard
>
> B is owner of k,v1
> A has k,v1 in L1
> 1. TX: A puts k,v2
> 2. TX: A sends async PrepareCommand k,v2 to B (one-phase-commit)
> 3. TX: A removes k,v1 from L1
> 4. A putForExternalRead k,v1 and has it in L1 again
> 5. TX: B executes PrepareCommand k,v2 but doesn't send invalidation to origin
> Result: A has k,v1 and B has k,v2
> Solution: For async caches send invalidation to origin too.
> The problem is that the owner updates the cache entry asynchronously. This gives the origin of the transaction time to request the entry. Here an outdated version is received and placed in L1. The owner never invalidates the entry and as result the cache is inconsistent.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
10 years, 11 months
[JBoss JIRA] (ISPN-132) Distributed queue/list API
by Manik Surtani (JIRA)
[ https://issues.jboss.org/browse/ISPN-132?page=com.atlassian.jira.plugin.s... ]
Manik Surtani updated ISPN-132:
-------------------------------
Description:
Original request:
{quote}
Add support for retrieving entries from the cache (simple API and Tree API) via insertion (FIFO) order.
For example, cache.getNext(); would return the next entry in the queue.
{quote}
Updating this JIRA to provide an API for a collection that can be appended to without retrieving the entire collection. Essentially a list, with various concurrent access guarantees.
was:
Add support for retrieving entries from the cache (simple API and Tree API) via insertion (FIFO) order.
For example, cache.getNext(); would return the next entry in the queue.
> Distributed queue/list API
> --------------------------
>
> Key: ISPN-132
> URL: https://issues.jboss.org/browse/ISPN-132
> Project: Infinispan
> Issue Type: Feature Request
> Components: Core API, Fine-grained API
> Reporter: Bryan Grunow
> Assignee: Manik Surtani
> Fix For: 7.0.0.Final
>
>
> Original request:
> {quote}
> Add support for retrieving entries from the cache (simple API and Tree API) via insertion (FIFO) order.
> For example, cache.getNext(); would return the next entry in the queue.
> {quote}
> Updating this JIRA to provide an API for a collection that can be appended to without retrieving the entire collection. Essentially a list, with various concurrent access guarantees.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
10 years, 11 months
[JBoss JIRA] (ISPN-2815) Put to Transactional cache with enabled indexing to InfinispanDirectory doesn't work properly
by Anna Manukyan (JIRA)
[ https://issues.jboss.org/browse/ISPN-2815?page=com.atlassian.jira.plugin.... ]
Anna Manukyan commented on ISPN-2815:
-------------------------------------
Just a note, that the same issue appears if the default.indexmanager with value InfinispanIndexManager is removed.
> Put to Transactional cache with enabled indexing to InfinispanDirectory doesn't work properly
> ---------------------------------------------------------------------------------------------
>
> Key: ISPN-2815
> URL: https://issues.jboss.org/browse/ISPN-2815
> Project: Infinispan
> Issue Type: Bug
> Components: Querying
> Affects Versions: 5.2.0.Final
> Reporter: Anna Manukyan
> Assignee: Sanne Grinovero
> Labels: stable_embedded_query
> Fix For: 5.3.0.Final
>
> Attachments: InfinispanDirectoryNonIndexedValuesTest.java
>
>
> In case when using Index enabled cache (no-matter in which mode), with enabled InfinispanDirectory and enabled transactions, the cache put throws exception, although then the data appears in the cache.
> But then, when query is performed on that cache, it doesn't return any result.
> The cache config is:
> {code}
> ConfigurationBuilder c = getDefaultStandaloneCacheConfig(true);
> c.indexing()
> .enable()
> .indexLocalOnly(true)
> .addProperty("hibernate.search.default.indexmanager", "org.infinispan.query.indexmanager.InfinispanIndexManager")
> .addProperty("hibernate.search.default.directory_provider", "infinispan")
> .addProperty("hibernate.search.default.exclusive_index_use", "false")
> .addProperty("lucene_version", "LUCENE_36");
> TestCacheManagerFactory.createCacheManager(c);
> {code}
> While performing put to the created cache, the following exception appears in the logs, although the program flow continues properly:
> {code}
> 2013-02-12 13:03:00,733 ERROR [InvocationContextInterceptor] (Hibernate Search: Index updates queue processor for index indexA-1) ISPN000136: Execution error
> org.infinispan.util.concurrent.TimeoutException: Unable to acquire lock after [10 seconds] on key [write.lock|M|indexA] for requestor [GlobalTransaction:<null>:5:local]! Lock held by [GlobalTransaction:<null>:3:local]
> at org.infinispan.util.concurrent.locks.LockManagerImpl.lock(LockManagerImpl.java:213)
> at org.infinispan.util.concurrent.locks.LockManagerImpl.acquireLock(LockManagerImpl.java:186)
> at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.lockKeyAndCheckOwnership(AbstractTxLockingInterceptor.java:186)
> at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.lockAndRegisterBackupLock(AbstractTxLockingInterceptor.java:123)
> at org.infinispan.interceptors.locking.OptimisticLockingInterceptor$LockAcquisitionVisitor.lockAndRecord(OptimisticLockingInterceptor.java:254)
> at org.infinispan.interceptors.locking.OptimisticLockingInterceptor$LockAcquisitionVisitor.visitSingleKeyCommand(OptimisticLockingInterceptor.java:249)
> at org.infinispan.interceptors.locking.OptimisticLockingInterceptor$LockAcquisitionVisitor.visitPutKeyValueCommand(OptimisticLockingInterceptor.java:242)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77)
> at org.infinispan.interceptors.locking.OptimisticLockingInterceptor.acquireLocksVisitingCommands(OptimisticLockingInterceptor.java:335)
> at org.infinispan.interceptors.locking.OptimisticLockingInterceptor.visitPrepareCommand(OptimisticLockingInterceptor.java:123)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.NotificationInterceptor.visitPrepareCommand(NotificationInterceptor.java:58)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.TxInterceptor.invokeNextInterceptorAndVerifyTransaction(TxInterceptor.java:128)
> at org.infinispan.interceptors.TxInterceptor.visitPrepareCommand(TxInterceptor.java:115)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:128)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:92)
> at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:343)
> at org.infinispan.transaction.TransactionCoordinator.prepare(TransactionCoordinator.java:141)
> at org.infinispan.transaction.TransactionCoordinator.prepare(TransactionCoordinator.java:123)
> at org.infinispan.transaction.synchronization.SynchronizationAdapter.beforeCompletion(SynchronizationAdapter.java:68)
> at com.arjuna.ats.internal.jta.resources.arjunacore.SynchronizationImple.beforeCompletion(SynchronizationImple.java:76)
> at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.beforeCompletion(TwoPhaseCoordinator.java:273)
> at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:93)
> at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:164)
> at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1165)
> at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:117)
> at org.infinispan.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1172)
> at org.infinispan.CacheImpl.putIfAbsentInternal(CacheImpl.java:779)
> at org.infinispan.CacheImpl.putIfAbsent(CacheImpl.java:771)
> at org.infinispan.DecoratedCache.putIfAbsent(DecoratedCache.java:355)
> at org.infinispan.lucene.locking.BaseLuceneLock.obtain(BaseLuceneLock.java:63)
> at org.apache.lucene.store.Lock.obtain(Lock.java:72)
> at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:1098)
> at org.hibernate.search.backend.impl.lucene.IndexWriterHolder.createNewIndexWriter(IndexWriterHolder.java:148)
> at org.hibernate.search.backend.impl.lucene.IndexWriterHolder.getIndexWriter(IndexWriterHolder.java:115)
> at org.hibernate.search.backend.impl.lucene.AbstractWorkspaceImpl.getIndexWriter(AbstractWorkspaceImpl.java:117)
> at org.hibernate.search.backend.impl.lucene.SharedIndexWorkspaceImpl.getIndexWriter(SharedIndexWorkspaceImpl.java:78)
> at org.hibernate.search.backend.impl.lucene.LuceneBackendQueueTask.applyUpdates(LuceneBackendQueueTask.java:99)
> at org.hibernate.search.backend.impl.lucene.LuceneBackendQueueTask.run(LuceneBackendQueueTask.java:67)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)
> 2013-02-12 13:03:00,738 ERROR [TransactionCoordinator] (Hibernate Search: Index updates queue processor for index indexA-1) Error while processing prepare
> org.infinispan.util.concurrent.TimeoutException: Unable to acquire lock after [10 seconds] on key [write.lock|M|indexA] for requestor [GlobalTransaction:<null>:5:local]! Lock held by [GlobalTransaction:<null>:3:local]
> at org.infinispan.util.concurrent.locks.LockManagerImpl.lock(LockManagerImpl.java:213)
> at org.infinispan.util.concurrent.locks.LockManagerImpl.acquireLock(LockManagerImpl.java:186)
> at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.lockKeyAndCheckOwnership(AbstractTxLockingInterceptor.java:186)
> at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.lockAndRegisterBackupLock(AbstractTxLockingInterceptor.java:123)
> at org.infinispan.interceptors.locking.OptimisticLockingInterceptor$LockAcquisitionVisitor.lockAndRecord(OptimisticLockingInterceptor.java:254)
> at org.infinispan.interceptors.locking.OptimisticLockingInterceptor$LockAcquisitionVisitor.visitSingleKeyCommand(OptimisticLockingInterceptor.java:249)
> at org.infinispan.interceptors.locking.OptimisticLockingInterceptor$LockAcquisitionVisitor.visitPutKeyValueCommand(OptimisticLockingInterceptor.java:242)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77)
> at org.infinispan.interceptors.locking.OptimisticLockingInterceptor.acquireLocksVisitingCommands(OptimisticLockingInterceptor.java:335)
> at org.infinispan.interceptors.locking.OptimisticLockingInterceptor.visitPrepareCommand(OptimisticLockingInterceptor.java:123)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.NotificationInterceptor.visitPrepareCommand(NotificationInterceptor.java:58)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.TxInterceptor.invokeNextInterceptorAndVerifyTransaction(TxInterceptor.java:128)
> at org.infinispan.interceptors.TxInterceptor.visitPrepareCommand(TxInterceptor.java:115)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:128)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:92)
> at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:343)
> at org.infinispan.transaction.TransactionCoordinator.prepare(TransactionCoordinator.java:141)
> at org.infinispan.transaction.TransactionCoordinator.prepare(TransactionCoordinator.java:123)
> at org.infinispan.transaction.synchronization.SynchronizationAdapter.beforeCompletion(SynchronizationAdapter.java:68)
> at com.arjuna.ats.internal.jta.resources.arjunacore.SynchronizationImple.beforeCompletion(SynchronizationImple.java:76)
> at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.beforeCompletion(TwoPhaseCoordinator.java:273)
> at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:93)
> at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:164)
> at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1165)
> at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:117)
> at org.infinispan.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1172)
> at org.infinispan.CacheImpl.putIfAbsentInternal(CacheImpl.java:779)
> at org.infinispan.CacheImpl.putIfAbsent(CacheImpl.java:771)
> at org.infinispan.DecoratedCache.putIfAbsent(DecoratedCache.java:355)
> at org.infinispan.lucene.locking.BaseLuceneLock.obtain(BaseLuceneLock.java:63)
> at org.apache.lucene.store.Lock.obtain(Lock.java:72)
> at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:1098)
> at org.hibernate.search.backend.impl.lucene.IndexWriterHolder.createNewIndexWriter(IndexWriterHolder.java:148)
> at org.hibernate.search.backend.impl.lucene.IndexWriterHolder.getIndexWriter(IndexWriterHolder.java:115)
> at org.hibernate.search.backend.impl.lucene.AbstractWorkspaceImpl.getIndexWriter(AbstractWorkspaceImpl.java:117)
> at org.hibernate.search.backend.impl.lucene.SharedIndexWorkspaceImpl.getIndexWriter(SharedIndexWorkspaceImpl.java:78)
> at org.hibernate.search.backend.impl.lucene.LuceneBackendQueueTask.applyUpdates(LuceneBackendQueueTask.java:99)
> at org.hibernate.search.backend.impl.lucene.LuceneBackendQueueTask.run(LuceneBackendQueueTask.java:67)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)
> 2013-02-12 13:03:00,742 WARN [arjuna] (Hibernate Search: Index updates queue processor for index indexA-1) ARJUNA012125: TwoPhaseCoordinator.beforeCompletion - failed for SynchronizationImple< 0:ffff0a22814f:cd67:511a2f6a:9, SynchronizationAdapter{localTransaction=LocalTransaction{remoteLockedNodes=null, isMarkedForRollback=false, lockedKeys=null, backupKeyLocks=null, topologyId=-1, isFromStateTransfer=false} org.infinispan.transaction.synchronization.SyncLocalTransaction@5} org.infinispan.transaction.synchronization.SynchronizationAdapter@24 >
> org.infinispan.CacheException: Could not prepare.
> at org.infinispan.transaction.synchronization.SynchronizationAdapter.beforeCompletion(SynchronizationAdapter.java:70)
> at com.arjuna.ats.internal.jta.resources.arjunacore.SynchronizationImple.beforeCompletion(SynchronizationImple.java:76)
> at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.beforeCompletion(TwoPhaseCoordinator.java:273)
> at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:93)
> at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:164)
> at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1165)
> at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:117)
> at org.infinispan.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1172)
> at org.infinispan.CacheImpl.putIfAbsentInternal(CacheImpl.java:779)
> at org.infinispan.CacheImpl.putIfAbsent(CacheImpl.java:771)
> at org.infinispan.DecoratedCache.putIfAbsent(DecoratedCache.java:355)
> at org.infinispan.lucene.locking.BaseLuceneLock.obtain(BaseLuceneLock.java:63)
> at org.apache.lucene.store.Lock.obtain(Lock.java:72)
> at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:1098)
> at org.hibernate.search.backend.impl.lucene.IndexWriterHolder.createNewIndexWriter(IndexWriterHolder.java:148)
> at org.hibernate.search.backend.impl.lucene.IndexWriterHolder.getIndexWriter(IndexWriterHolder.java:115)
> at org.hibernate.search.backend.impl.lucene.AbstractWorkspaceImpl.getIndexWriter(AbstractWorkspaceImpl.java:117)
> at org.hibernate.search.backend.impl.lucene.SharedIndexWorkspaceImpl.getIndexWriter(SharedIndexWorkspaceImpl.java:78)
> at org.hibernate.search.backend.impl.lucene.LuceneBackendQueueTask.applyUpdates(LuceneBackendQueueTask.java:99)
> at org.hibernate.search.backend.impl.lucene.LuceneBackendQueueTask.run(LuceneBackendQueueTask.java:67)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)
> Caused by: javax.transaction.xa.XAException
> at org.infinispan.transaction.TransactionCoordinator.prepare(TransactionCoordinator.java:161)
> at org.infinispan.transaction.TransactionCoordinator.prepare(TransactionCoordinator.java:123)
> at org.infinispan.transaction.synchronization.SynchronizationAdapter.beforeCompletion(SynchronizationAdapter.java:68)
> ... 25 more
> 2013-02-12 13:03:00,744 WARN [CacheImpl] (Hibernate Search: Index updates queue processor for index indexA-1) ISPN000160: Could not complete injected transaction.
> javax.transaction.RollbackException: ARJUNA016053: Could not commit transaction.
> at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1177)
> at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:117)
> at org.infinispan.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1172)
> at org.infinispan.CacheImpl.putIfAbsentInternal(CacheImpl.java:779)
> at org.infinispan.CacheImpl.putIfAbsent(CacheImpl.java:771)
> at org.infinispan.DecoratedCache.putIfAbsent(DecoratedCache.java:355)
> at org.infinispan.lucene.locking.BaseLuceneLock.obtain(BaseLuceneLock.java:63)
> at org.apache.lucene.store.Lock.obtain(Lock.java:72)
> at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:1098)
> at org.hibernate.search.backend.impl.lucene.IndexWriterHolder.createNewIndexWriter(IndexWriterHolder.java:148)
> at org.hibernate.search.backend.impl.lucene.IndexWriterHolder.getIndexWriter(IndexWriterHolder.java:115)
> at org.hibernate.search.backend.impl.lucene.AbstractWorkspaceImpl.getIndexWriter(AbstractWorkspaceImpl.java:117)
> at org.hibernate.search.backend.impl.lucene.SharedIndexWorkspaceImpl.getIndexWriter(SharedIndexWorkspaceImpl.java:78)
> at org.hibernate.search.backend.impl.lucene.LuceneBackendQueueTask.applyUpdates(LuceneBackendQueueTask.java:99)
> at org.hibernate.search.backend.impl.lucene.LuceneBackendQueueTask.run(LuceneBackendQueueTask.java:67)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)
> Caused by: org.infinispan.CacheException: Could not prepare.
> at org.infinispan.transaction.synchronization.SynchronizationAdapter.beforeCompletion(SynchronizationAdapter.java:70)
> at com.arjuna.ats.internal.jta.resources.arjunacore.SynchronizationImple.beforeCompletion(SynchronizationImple.java:76)
> at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.beforeCompletion(TwoPhaseCoordinator.java:273)
> at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:93)
> at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:164)
> at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1165)
> ... 20 more
> Caused by: javax.transaction.xa.XAException
> at org.infinispan.transaction.TransactionCoordinator.prepare(TransactionCoordinator.java:161)
> at org.infinispan.transaction.TransactionCoordinator.prepare(TransactionCoordinator.java:123)
> at org.infinispan.transaction.synchronization.SynchronizationAdapter.beforeCompletion(SynchronizationAdapter.java:68)
> ... 25 more
> 2013-02-12 13:03:00,745 ERROR [LuceneBackendQueueTask] (Hibernate Search: Index updates queue processor for index indexA-1) HSEARCH000073: Error in backend
> org.infinispan.CacheException: Could not commit implicit transaction
> at org.infinispan.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1175)
> at org.infinispan.CacheImpl.putIfAbsentInternal(CacheImpl.java:779)
> at org.infinispan.CacheImpl.putIfAbsent(CacheImpl.java:771)
> at org.infinispan.DecoratedCache.putIfAbsent(DecoratedCache.java:355)
> at org.infinispan.lucene.locking.BaseLuceneLock.obtain(BaseLuceneLock.java:63)
> at org.apache.lucene.store.Lock.obtain(Lock.java:72)
> at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:1098)
> at org.hibernate.search.backend.impl.lucene.IndexWriterHolder.createNewIndexWriter(IndexWriterHolder.java:148)
> at org.hibernate.search.backend.impl.lucene.IndexWriterHolder.getIndexWriter(IndexWriterHolder.java:115)
> at org.hibernate.search.backend.impl.lucene.AbstractWorkspaceImpl.getIndexWriter(AbstractWorkspaceImpl.java:117)
> at org.hibernate.search.backend.impl.lucene.SharedIndexWorkspaceImpl.getIndexWriter(SharedIndexWorkspaceImpl.java:78)
> at org.hibernate.search.backend.impl.lucene.LuceneBackendQueueTask.applyUpdates(LuceneBackendQueueTask.java:99)
> at org.hibernate.search.backend.impl.lucene.LuceneBackendQueueTask.run(LuceneBackendQueueTask.java:67)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)
> Caused by: javax.transaction.RollbackException: ARJUNA016053: Could not commit transaction.
> at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1177)
> at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:117)
> at org.infinispan.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1172)
> ... 18 more
> Caused by: org.infinispan.CacheException: Could not prepare.
> at org.infinispan.transaction.synchronization.SynchronizationAdapter.beforeCompletion(SynchronizationAdapter.java:70)
> at com.arjuna.ats.internal.jta.resources.arjunacore.SynchronizationImple.beforeCompletion(SynchronizationImple.java:76)
> at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.beforeCompletion(TwoPhaseCoordinator.java:273)
> at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:93)
> at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:164)
> at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1165)
> ... 20 more
> Caused by: javax.transaction.xa.XAException
> at org.infinispan.transaction.TransactionCoordinator.prepare(TransactionCoordinator.java:161)
> at org.infinispan.transaction.TransactionCoordinator.prepare(TransactionCoordinator.java:123)
> at org.infinispan.transaction.synchronization.SynchronizationAdapter.beforeCompletion(SynchronizationAdapter.java:68)
> ... 25 more
> 2013-02-12 13:03:00,748 ERROR [LogErrorHandler] (Hibernate Search: Index updates queue processor for index indexA-1) HSEARCH000058: Exception occurred org.infinispan.CacheException: Could not commit implicit transaction
> Primary Failure:
> Entity org.infinispan.query.api.TestEntity Id L:10 Work Type org.hibernate.search.backend.UpdateLuceneWork
> org.infinispan.CacheException: Could not commit implicit transaction
> at org.infinispan.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1175)
> at org.infinispan.CacheImpl.putIfAbsentInternal(CacheImpl.java:779)
> at org.infinispan.CacheImpl.putIfAbsent(CacheImpl.java:771)
> at org.infinispan.DecoratedCache.putIfAbsent(DecoratedCache.java:355)
> at org.infinispan.lucene.locking.BaseLuceneLock.obtain(BaseLuceneLock.java:63)
> at org.apache.lucene.store.Lock.obtain(Lock.java:72)
> at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:1098)
> at org.hibernate.search.backend.impl.lucene.IndexWriterHolder.createNewIndexWriter(IndexWriterHolder.java:148)
> at org.hibernate.search.backend.impl.lucene.IndexWriterHolder.getIndexWriter(IndexWriterHolder.java:115)
> at org.hibernate.search.backend.impl.lucene.AbstractWorkspaceImpl.getIndexWriter(AbstractWorkspaceImpl.java:117)
> at org.hibernate.search.backend.impl.lucene.SharedIndexWorkspaceImpl.getIndexWriter(SharedIndexWorkspaceImpl.java:78)
> at org.hibernate.search.backend.impl.lucene.LuceneBackendQueueTask.applyUpdates(LuceneBackendQueueTask.java:99)
> at org.hibernate.search.backend.impl.lucene.LuceneBackendQueueTask.run(LuceneBackendQueueTask.java:67)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)
> Caused by: javax.transaction.RollbackException: ARJUNA016053: Could not commit transaction.
> at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1177)
> at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:117)
> at org.infinispan.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1172)
> ... 18 more
> Caused by: org.infinispan.CacheException: Could not prepare.
> at org.infinispan.transaction.synchronization.SynchronizationAdapter.beforeCompletion(SynchronizationAdapter.java:70)
> at com.arjuna.ats.internal.jta.resources.arjunacore.SynchronizationImple.beforeCompletion(SynchronizationImple.java:76)
> at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.beforeCompletion(TwoPhaseCoordinator.java:273)
> at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:93)
> at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:164)
> at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1165)
> ... 20 more
> Caused by: javax.transaction.xa.XAException
> at org.infinispan.transaction.TransactionCoordinator.prepare(TransactionCoordinator.java:161)
> at org.infinispan.transaction.TransactionCoordinator.prepare(TransactionCoordinator.java:123)
> at org.infinispan.transaction.synchronization.SynchronizationAdapter.beforeCompletion(SynchronizationAdapter.java:68)
> ... 25 more
> {code}
> After the put, the entries are in the cache, but the querying doesn't return results.
> You can find attached the failing test.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
10 years, 11 months