[JBoss JIRA] (ISPN-2781) NPE in AbstractTopologyAwareEncoder1x
by Michal Linhard (JIRA)
Michal Linhard created ISPN-2781:
------------------------------------
Summary: NPE in AbstractTopologyAwareEncoder1x
Key: ISPN-2781
URL: https://issues.jboss.org/browse/ISPN-2781
Project: Infinispan
Issue Type: Bug
Components: Server
Affects Versions: 5.2.0.CR3
Reporter: Michal Linhard
Assignee: Tristan Tarrant
got this in an elasticity test (32nodes)
{code}
06:23:20,660 ERROR [org.infinispan.server.hotrod.HotRodDecoder] (HotRodServerWorker-151) ISPN005009: Unexpected error before any request parameters read: java.lang.NullPointerException
at org.infinispan.server.hotrod.AbstractTopologyAwareEncoder1x$$anonfun$writeHashTopologyUpdate11$3.apply(AbstractTopologyAwareEncoder1x.scala:102) [infinispan-server-hotrod-5.2.0.CR3-redhat-1.jar:5.2.0.CR3-redhat-1]
at org.infinispan.server.hotrod.AbstractTopologyAwareEncoder1x$$anonfun$writeHashTopologyUpdate11$3.apply(AbstractTopologyAwareEncoder1x.scala:101) [infinispan-server-hotrod-5.2.0.CR3-redhat-1.jar:5.2.0.CR3-redhat-1]
at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
at org.infinispan.server.hotrod.AbstractTopologyAwareEncoder1x.writeHashTopologyUpdate11(AbstractTopologyAwareEncoder1x.scala:101) [infinispan-server-hotrod-5.2.0.CR3-redhat-1.jar:5.2.0.CR3-redhat-1]
at org.infinispan.server.hotrod.AbstractTopologyAwareEncoder1x.writeHashTopologyUpdate(AbstractTopologyAwareEncoder1x.scala:52) [infinispan-server-hotrod-5.2.0.CR3-redhat-1.jar:5.2.0.CR3-redhat-1]
at org.infinispan.server.hotrod.AbstractEncoder1x.writeHeader(AbstractEncoder1x.scala:63) [infinispan-server-hotrod-5.2.0.CR3-redhat-1.jar:5.2.0.CR3-redhat-1]
at org.infinispan.server.hotrod.HotRodEncoder.encode(HotRodEncoder.scala:63) [infinispan-server-hotrod-5.2.0.CR3-redhat-1.jar:5.2.0.CR3-redhat-1]
at org.jboss.netty.handler.codec.oneone.OneToOneEncoder.doEncode(OneToOneEncoder.java:66) [netty-3.6.1.Final.jar:]
at org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:59) [netty-3.6.1.Final.jar:]
at org.jboss.netty.channel.Channels.write(Channels.java:704) [netty-3.6.1.Final.jar:]
at org.jboss.netty.channel.Channels.write(Channels.java:671) [netty-3.6.1.Final.jar:]
at org.jboss.netty.channel.AbstractChannel.write(AbstractChannel.java:248) [netty-3.6.1.Final.jar:]
at org.infinispan.server.core.AbstractProtocolDecoder.writeResponse(AbstractProtocolDecoder.scala:179) [infinispan-server-core-5.2.0.CR3-redhat-1.jar:5.2.0.CR3-redhat-1]
at org.infinispan.server.hotrod.HotRodDecoder.customDecodeHeader(HotRodDecoder.scala:157) [infinispan-server-hotrod-5.2.0.CR3-redhat-1.jar:5.2.0.CR3-redhat-1]
at org.infinispan.server.core.AbstractProtocolDecoder.decodeHeader(AbstractProtocolDecoder.scala:105) [infinispan-server-core-5.2.0.CR3-redhat-1.jar:5.2.0.CR3-redhat-1]
at org.infinispan.server.core.AbstractProtocolDecoder.decode(AbstractProtocolDecoder.scala:70) [infinispan-server-core-5.2.0.CR3-redhat-1.jar:5.2.0.CR3-redhat-1]
at org.infinispan.server.core.AbstractProtocolDecoder.decode(AbstractProtocolDecoder.scala:47) [infinispan-server-core-5.2.0.CR3-redhat-1.jar:5.2.0.CR3-redhat-1]
at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:500) [netty-3.6.1.Final.jar:]
at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) [netty-3.6.1.Final.jar:]
at org.infinispan.server.core.AbstractProtocolDecoder.messageReceived(AbstractProtocolDecoder.scala:387) [infinispan-server-core-5.2.0.CR3-redhat-1.jar:5.2.0.CR3-redhat-1]
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) [netty-3.6.1.Final.jar:]
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) [netty-3.6.1.Final.jar:]
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) [netty-3.6.1.Final.jar:]
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:107) [netty-3.6.1.Final.jar:]
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:313) [netty-3.6.1.Final.jar:]
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88) [netty-3.6.1.Final.jar:]
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) [netty-3.6.1.Final.jar:]
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_33]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_33]
at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_33]
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 11 months
[JBoss JIRA] (ISPN-2561) RejectedExecutionException thrown from StaleTransactionCleanupService during cache shutdown
by Mircea Markus (JIRA)
[ https://issues.jboss.org/browse/ISPN-2561?page=com.atlassian.jira.plugin.... ]
Mircea Markus updated ISPN-2561:
--------------------------------
Fix Version/s: 5.2.0.Final
(was: 5.3.0.Final)
> RejectedExecutionException thrown from StaleTransactionCleanupService during cache shutdown
> -------------------------------------------------------------------------------------------
>
> Key: ISPN-2561
> URL: https://issues.jboss.org/browse/ISPN-2561
> Project: Infinispan
> Issue Type: Bug
> Components: Transactions
> Affects Versions: 5.2.0.Beta4
> Reporter: Adrian Nistor
> Assignee: Mircea Markus
> Priority: Minor
> Fix For: 5.2.0.Final
>
>
> We should not submit stale transaction cleanup tasks after we started shutdown. This exception is now caught and logged. It does not impact functionality but it clutters the logs a lot.
> {noformat}
> 2012-11-26 17:49:07,855 ERROR [org.infinispan.transaction.StaleTransactionCleanupService] (OOB-2,ISPN,NodeB-36237) Unable to submit task to executor
> java.util.concurrent.RejectedExecutionException
> at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:1768)
> at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
> at java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:215)
> at java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:397)
> at java.util.concurrent.ScheduledThreadPoolExecutor.submit(ScheduledThreadPoolExecutor.java:470)
> at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:600)
> at org.infinispan.transaction.StaleTransactionCleanupService.cleanTxForWhichTheOwnerLeft(StaleTransactionCleanupService.java:91)
> at org.infinispan.transaction.StaleTransactionCleanupService.onTopologyChange(StaleTransactionCleanupService.java:80)
> at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.infinispan.notifications.AbstractListenerImpl$ListenerInvocation$1.run(AbstractListenerImpl.java:200)
> at org.infinispan.util.concurrent.WithinThreadExecutor.execute(WithinThreadExecutor.java:44)
> at org.infinispan.notifications.AbstractListenerImpl$ListenerInvocation.invoke(AbstractListenerImpl.java:221)
> at org.infinispan.notifications.cachelistener.CacheNotifierImpl.notifyTopologyChanged(CacheNotifierImpl.java:356)
> at org.infinispan.statetransfer.StateTransferManagerImpl.doTopologyUpdate(StateTransferManagerImpl.java:191)
> at org.infinispan.statetransfer.StateTransferManagerImpl.access$000(StateTransferManagerImpl.java:60)
> at org.infinispan.statetransfer.StateTransferManagerImpl$1.updateConsistentHash(StateTransferManagerImpl.java:119)
> at org.infinispan.topology.LocalTopologyManagerImpl.handleConsistentHashUpdate(LocalTopologyManagerImpl.java:194)
> at org.infinispan.topology.CacheTopologyControlCommand.doPerform(CacheTopologyControlCommand.java:165)
> at org.infinispan.topology.CacheTopologyControlCommand.perform(CacheTopologyControlCommand.java:137)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommandFromLocalCluster(CommandAwareRpcDispatcher.java:252)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:219)
> at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:483)
> at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:390)
> at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:248)
> at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:604)
> at org.jgroups.JChannel.up(JChannel.java:688)
> at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1020)
> at org.jgroups.protocols.FRAG2.up(FRAG2.java:181)
> at org.jgroups.protocols.FC.up(FC.java:479)
> at org.jgroups.protocols.pbcast.GMS.up(GMS.java:896)
> at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:244)
> at org.jgroups.protocols.UNICAST2.up(UNICAST2.java:432)
> at org.jgroups.protocols.pbcast.NAKACK2.handleMessage(NAKACK2.java:721)
> at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:574)
> at org.jgroups.protocols.Discovery.up(Discovery.java:359)
> at org.jgroups.protocols.TP.passMessageUp(TP.java:1294)
> at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1857)
> at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1830)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 11 months
[JBoss JIRA] (ISPN-2406) StaleTransactionCleanupService should not attempt cleanup if the local cache is stopping
by Mircea Markus (JIRA)
[ https://issues.jboss.org/browse/ISPN-2406?page=com.atlassian.jira.plugin.... ]
Mircea Markus updated ISPN-2406:
--------------------------------
Fix Version/s: 5.2.0.Final
(was: 6.0.0.Final)
> StaleTransactionCleanupService should not attempt cleanup if the local cache is stopping
> ----------------------------------------------------------------------------------------
>
> Key: ISPN-2406
> URL: https://issues.jboss.org/browse/ISPN-2406
> Project: Infinispan
> Issue Type: Bug
> Components: State transfer, Transactions
> Affects Versions: 5.2.0.ALPHA1
> Reporter: Adrian Nistor
> Assignee: Mircea Markus
> Priority: Minor
> Fix For: 5.2.0.Final
>
>
> While a cache is stopping there is a small window of time in which the StaleTransactionCleanupService still runs until the component registry manages to stop the TransactionTable and StaleTransactionCleanupService. The service might try to rollback transactions but the cache can no longer accept the rollback command because of its current status. This leads to exceptions being logged. This has no negative impact but is not ideal.
> {noformat}
> 2012-10-15 19:23:19,740 TRACE [TransactionTable] (LockBreakingService,___defaultcache,NodeC-54695) Killing remote transaction originating on leaver GlobalTransaction:<NodeA-995>:1206:local
> 2012-10-15 19:23:19,740 TRACE [AbstractTransactionBoundaryCommand] (LockBreakingService,___defaultcache,NodeC-54695) About to execute tx command RollbackCommand {gtx=GlobalTransaction:<NodeA-995>:1206:remote, cacheName='___defaultcache', topologyId=-1}
> 2012-10-15 19:23:19,743 WARN [TransactionTable] (LockBreakingService,___defaultcache,NodeC-54695) ISPN000102: Unable to roll back global transaction GlobalTransaction:<NodeA-995>:1206:remote
> java.lang.IllegalStateException: Default cache is in 'TERMINATED' state and so it does not accept new invocations. Either restart it or recreate the cache container.
> at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:111)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:93)
> at org.infinispan.commands.AbstractVisitor.visitRollbackCommand(AbstractVisitor.java:132)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:55)
> at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:347)
> at org.infinispan.commands.tx.AbstractTransactionBoundaryCommand.perform(AbstractTransactionBoundaryCommand.java:118)
> at org.infinispan.transaction.TransactionTable.updateStateOnNodesLeaving(TransactionTable.java:243)
> at org.infinispan.transaction.StaleTransactionCleanupService$1.run(StaleTransactionCleanupService.java:150)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 11 months
[JBoss JIRA] (ISPN-2406) StaleTransactionCleanupService should not attempt cleanup if the local cache is stopping
by Mircea Markus (JIRA)
[ https://issues.jboss.org/browse/ISPN-2406?page=com.atlassian.jira.plugin.... ]
Mircea Markus resolved ISPN-2406.
---------------------------------
Resolution: Done
closed as part of ISPN-2483.
> StaleTransactionCleanupService should not attempt cleanup if the local cache is stopping
> ----------------------------------------------------------------------------------------
>
> Key: ISPN-2406
> URL: https://issues.jboss.org/browse/ISPN-2406
> Project: Infinispan
> Issue Type: Bug
> Components: State transfer, Transactions
> Affects Versions: 5.2.0.ALPHA1
> Reporter: Adrian Nistor
> Assignee: Mircea Markus
> Priority: Minor
> Fix For: 5.2.0.Final
>
>
> While a cache is stopping there is a small window of time in which the StaleTransactionCleanupService still runs until the component registry manages to stop the TransactionTable and StaleTransactionCleanupService. The service might try to rollback transactions but the cache can no longer accept the rollback command because of its current status. This leads to exceptions being logged. This has no negative impact but is not ideal.
> {noformat}
> 2012-10-15 19:23:19,740 TRACE [TransactionTable] (LockBreakingService,___defaultcache,NodeC-54695) Killing remote transaction originating on leaver GlobalTransaction:<NodeA-995>:1206:local
> 2012-10-15 19:23:19,740 TRACE [AbstractTransactionBoundaryCommand] (LockBreakingService,___defaultcache,NodeC-54695) About to execute tx command RollbackCommand {gtx=GlobalTransaction:<NodeA-995>:1206:remote, cacheName='___defaultcache', topologyId=-1}
> 2012-10-15 19:23:19,743 WARN [TransactionTable] (LockBreakingService,___defaultcache,NodeC-54695) ISPN000102: Unable to roll back global transaction GlobalTransaction:<NodeA-995>:1206:remote
> java.lang.IllegalStateException: Default cache is in 'TERMINATED' state and so it does not accept new invocations. Either restart it or recreate the cache container.
> at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:111)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:93)
> at org.infinispan.commands.AbstractVisitor.visitRollbackCommand(AbstractVisitor.java:132)
> at org.infinispan.commands.tx.RollbackCommand.acceptVisitor(RollbackCommand.java:55)
> at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:347)
> at org.infinispan.commands.tx.AbstractTransactionBoundaryCommand.perform(AbstractTransactionBoundaryCommand.java:118)
> at org.infinispan.transaction.TransactionTable.updateStateOnNodesLeaving(TransactionTable.java:243)
> at org.infinispan.transaction.StaleTransactionCleanupService$1.run(StaleTransactionCleanupService.java:150)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 11 months
[JBoss JIRA] (ISPN-2561) RejectedExecutionException thrown from StaleTransactionCleanupService during cache shutdown
by Mircea Markus (JIRA)
[ https://issues.jboss.org/browse/ISPN-2561?page=com.atlassian.jira.plugin.... ]
Mircea Markus resolved ISPN-2561.
---------------------------------
Resolution: Out of Date
closed as part of ISPN-2483.
> RejectedExecutionException thrown from StaleTransactionCleanupService during cache shutdown
> -------------------------------------------------------------------------------------------
>
> Key: ISPN-2561
> URL: https://issues.jboss.org/browse/ISPN-2561
> Project: Infinispan
> Issue Type: Bug
> Components: Transactions
> Affects Versions: 5.2.0.Beta4
> Reporter: Adrian Nistor
> Assignee: Mircea Markus
> Priority: Minor
> Fix For: 5.3.0.Final
>
>
> We should not submit stale transaction cleanup tasks after we started shutdown. This exception is now caught and logged. It does not impact functionality but it clutters the logs a lot.
> {noformat}
> 2012-11-26 17:49:07,855 ERROR [org.infinispan.transaction.StaleTransactionCleanupService] (OOB-2,ISPN,NodeB-36237) Unable to submit task to executor
> java.util.concurrent.RejectedExecutionException
> at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:1768)
> at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
> at java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:215)
> at java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:397)
> at java.util.concurrent.ScheduledThreadPoolExecutor.submit(ScheduledThreadPoolExecutor.java:470)
> at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:600)
> at org.infinispan.transaction.StaleTransactionCleanupService.cleanTxForWhichTheOwnerLeft(StaleTransactionCleanupService.java:91)
> at org.infinispan.transaction.StaleTransactionCleanupService.onTopologyChange(StaleTransactionCleanupService.java:80)
> at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.infinispan.notifications.AbstractListenerImpl$ListenerInvocation$1.run(AbstractListenerImpl.java:200)
> at org.infinispan.util.concurrent.WithinThreadExecutor.execute(WithinThreadExecutor.java:44)
> at org.infinispan.notifications.AbstractListenerImpl$ListenerInvocation.invoke(AbstractListenerImpl.java:221)
> at org.infinispan.notifications.cachelistener.CacheNotifierImpl.notifyTopologyChanged(CacheNotifierImpl.java:356)
> at org.infinispan.statetransfer.StateTransferManagerImpl.doTopologyUpdate(StateTransferManagerImpl.java:191)
> at org.infinispan.statetransfer.StateTransferManagerImpl.access$000(StateTransferManagerImpl.java:60)
> at org.infinispan.statetransfer.StateTransferManagerImpl$1.updateConsistentHash(StateTransferManagerImpl.java:119)
> at org.infinispan.topology.LocalTopologyManagerImpl.handleConsistentHashUpdate(LocalTopologyManagerImpl.java:194)
> at org.infinispan.topology.CacheTopologyControlCommand.doPerform(CacheTopologyControlCommand.java:165)
> at org.infinispan.topology.CacheTopologyControlCommand.perform(CacheTopologyControlCommand.java:137)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommandFromLocalCluster(CommandAwareRpcDispatcher.java:252)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:219)
> at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:483)
> at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:390)
> at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:248)
> at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:604)
> at org.jgroups.JChannel.up(JChannel.java:688)
> at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1020)
> at org.jgroups.protocols.FRAG2.up(FRAG2.java:181)
> at org.jgroups.protocols.FC.up(FC.java:479)
> at org.jgroups.protocols.pbcast.GMS.up(GMS.java:896)
> at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:244)
> at org.jgroups.protocols.UNICAST2.up(UNICAST2.java:432)
> at org.jgroups.protocols.pbcast.NAKACK2.handleMessage(NAKACK2.java:721)
> at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:574)
> at org.jgroups.protocols.Discovery.up(Discovery.java:359)
> at org.jgroups.protocols.TP.passMessageUp(TP.java:1294)
> at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1857)
> at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1830)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 11 months
[JBoss JIRA] (ISPN-2727) Transactional put failed after master node with enabled InfinispanIndexManager is killed
by Galder Zamarreño (JIRA)
[ https://issues.jboss.org/browse/ISPN-2727?page=com.atlassian.jira.plugin.... ]
Galder Zamarreño edited comment on ISPN-2727 at 1/31/13 10:50 AM:
------------------------------------------------------------------
The issue appears before any nodes have been killed. The problem seems related to ordering in which Hibernate and Infinispan synchronizations are executing. The actual exception is:
{code}2013-01-31 16:22:49,868 11797 DEBUG [org.infinispan.util.concurrent.locks.LockManagerImpl]
(Hibernate Search: Index updates queue processor for index person-1:LuceneIndexesLocking)
Failed to acquire lock write.lock|M|person, owner is GlobalTransaction:<null>:49:local{code}
So, when the first put is executed, org.infinispan.transaction.synchronization.SynchronizationAdapter is executed before org.hibernate.search.backend.impl.PostTransactionWorkQueueSynchronization, and that leads to "write.lock|M|person" being correctly unlocked. The processor kicked off by Hibernate Search's synchronization can then acquire the lock on "write.lock|M|person" without probs.
Second time though, org.hibernate.search.backend.impl.PostTransactionWorkQueueSynchronization is executed before org.infinispan.transaction.synchronization.SynchronizationAdapter, so the lock on "write.lock|M|person" is not released and the processing thread fails to acquire the lock as shown above.
Log snippets can be found in showing what I mentioned above:
https://gist.github.com/77207522649b79c814e0
IIRC, I've seen that before and AFAIK, JTA provides no guarantees in the order in which multiple synchronizations are executed. I tried disabling Infinispan's use of synchronization and the test then passes fine.
This needs some further debugging, but the solution could be here to not allow synchronization for indexed caches.
was (Author: galder.zamarreno):
The issue appears before any nodes have been killed. The problem seems related to ordering in which Hibernate and Infinispan synchronizations are executing. The actual exception is:
{code}2013-01-31 16:22:49,868 11797 DEBUG [org.infinispan.util.concurrent.locks.LockManagerImpl] (Hibernate Search: Index updates queue processor for index person-1:LuceneIndexesLocking) Failed to acquire lock write.lock|M|person, owner is GlobalTransaction:<null>:49:local{code}
So, when the first put is executed, org.infinispan.transaction.synchronization.SynchronizationAdapter is executed before org.hibernate.search.backend.impl.PostTransactionWorkQueueSynchronization, and that leads to "write.lock|M|person" being correctly unlocked. The processor kicked off by Hibernate Search's synchronization can then acquire the lock on "write.lock|M|person" without probs.
Second time though, org.hibernate.search.backend.impl.PostTransactionWorkQueueSynchronization is executed before org.infinispan.transaction.synchronization.SynchronizationAdapter, so the lock on "write.lock|M|person" is not released and the processing thread fails to acquire the lock as shown above.
Log snippets can be found in showing what I mentioned above:
https://gist.github.com/77207522649b79c814e0
IIRC, I've seen that before and AFAIK, JTA provides no guarantees in the order in which multiple synchronizations are executed. I tried disabling Infinispan's use of synchronization and the test then passes fine.
This needs some further debugging, but the solution could be here to not allow synchronization for indexed caches.
> Transactional put failed after master node with enabled InfinispanIndexManager is killed
> -----------------------------------------------------------------------------------------
>
> Key: ISPN-2727
> URL: https://issues.jboss.org/browse/ISPN-2727
> Project: Infinispan
> Issue Type: Bug
> Components: Querying
> Reporter: Anna Manukyan
> Assignee: Galder Zamarreño
> Fix For: 5.2.0.Final
>
> Attachments: MultiNodeDistributedTest.java, MultiNodeLocalTest.java, MultiNodeLocalTxTest.java, MultiNodeReplicatedTest.java, MultiNodeReplicatedTxTest.java
>
>
> I've added new test, which uses the following Infinispan configuration: REPL_SYNC clustering mode, with transaction enabled, as well as enabled indexing with InfinispanIndexManager.
> The test adds several nodes with the described configuration, adds entries to different nodes, and checks that the query for the entries returns the same result for all nodes.
> Then master node is killed, and then again new data is inserted and checked on all nodes. (Similiar test to https://github.com/infinispan/infinispan/blob/master/query/src/test/java/... but with enabled transaction).
> When the test tries to insert new entry to one of the caches, after the master node kill, the following exception appears:
> {code}
> testIndexingWorkDistribution(org.infinispan.query.distributed.MultiNodeReplicatedTxTest) Time elapsed: 1.656 sec <<< FAILURE!
> javax.transaction.RollbackException: ARJUNA016053: Could not commit transaction.
> at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1177)
> at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:117)
> at org.infinispan.query.distributed.MultiNodeDistributedTest.storeOn(MultiNodeDistributedTest.java:78)
> at org.infinispan.query.distributed.MultiNodeDistributedTest.testIndexingWorkDistribution(MultiNodeDistributedTest.java:102)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:601)
> at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:80)
> at org.testng.internal.Invoker.invokeMethod(Invoker.java:715)
> at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:907)
> at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1237)
> at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:127)
> at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:111)
> at org.testng.TestRunner.privateRun(TestRunner.java:767)
> at org.testng.TestRunner.run(TestRunner.java:617)
> at org.testng.SuiteRunner.runTest(SuiteRunner.java:334)
> at org.testng.SuiteRunner.access$000(SuiteRunner.java:37)
> at org.testng.SuiteRunner$SuiteWorker.run(SuiteRunner.java:368)
> at org.testng.internal.thread.ThreadUtil$2.call(ThreadUtil.java:64)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)
> Caused by: org.infinispan.CacheException: Could not prepare.
> at org.infinispan.transaction.synchronization.SynchronizationAdapter.beforeCompletion(SynchronizationAdapter.java:70)
> at com.arjuna.ats.internal.jta.resources.arjunacore.SynchronizationImple.beforeCompletion(SynchronizationImple.java:76)
> at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.beforeCompletion(TwoPhaseCoordinator.java:273)
> at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:93)
> at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:164)
> at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1165)
> ... 24 more
> Caused by: javax.transaction.xa.XAException
> at org.infinispan.transaction.TransactionCoordinator.prepare(TransactionCoordinator.java:161)
> at org.infinispan.transaction.TransactionCoordinator.prepare(TransactionCoordinator.java:123)
> at org.infinispan.transaction.synchronization.SynchronizationAdapter.beforeCompletion(SynchronizationAdapter.java:68)
> ... 29 more
> {code}
> You can find the test attached. The test which fails is MultiNodeReplicatedTxTest.java . The rest are the tests which are the parents and a bit modified compared to the version in infinispan repo.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 11 months
[JBoss JIRA] (ISPN-2727) Transactional put failed after master node with enabled InfinispanIndexManager is killed
by Galder Zamarreño (JIRA)
[ https://issues.jboss.org/browse/ISPN-2727?page=com.atlassian.jira.plugin.... ]
Galder Zamarreño commented on ISPN-2727:
----------------------------------------
The issue appears before any nodes have been killed. The problem seems related to ordering in which Hibernate and Infinispan synchronizations are executing. The actual exception is:
{code}2013-01-31 16:22:49,868 11797 DEBUG [org.infinispan.util.concurrent.locks.LockManagerImpl] (Hibernate Search: Index updates queue processor for index person-1:LuceneIndexesLocking) Failed to acquire lock write.lock|M|person, owner is GlobalTransaction:<null>:49:local{code}
So, when the first put is executed, org.infinispan.transaction.synchronization.SynchronizationAdapter is executed before org.hibernate.search.backend.impl.PostTransactionWorkQueueSynchronization, and that leads to "write.lock|M|person" being correctly unlocked. The processor kicked off by Hibernate Search's synchronization can then acquire the lock on "write.lock|M|person" without probs.
Second time though, org.hibernate.search.backend.impl.PostTransactionWorkQueueSynchronization is executed before org.infinispan.transaction.synchronization.SynchronizationAdapter, so the lock on "write.lock|M|person" is not released and the processing thread fails to acquire the lock as shown above.
Log snippets can be found in showing what I mentioned above:
https://gist.github.com/77207522649b79c814e0
IIRC, I've seen that before and AFAIK, JTA provides no guarantees in the order in which multiple synchronizations are executed. I tried disabling Infinispan's use of synchronization and the test then passes fine.
This needs some further debugging, but the solution could be here to not allow synchronization for indexed caches.
> Transactional put failed after master node with enabled InfinispanIndexManager is killed
> -----------------------------------------------------------------------------------------
>
> Key: ISPN-2727
> URL: https://issues.jboss.org/browse/ISPN-2727
> Project: Infinispan
> Issue Type: Bug
> Components: Querying
> Reporter: Anna Manukyan
> Assignee: Galder Zamarreño
> Fix For: 5.2.0.Final
>
> Attachments: MultiNodeDistributedTest.java, MultiNodeLocalTest.java, MultiNodeLocalTxTest.java, MultiNodeReplicatedTest.java, MultiNodeReplicatedTxTest.java
>
>
> I've added new test, which uses the following Infinispan configuration: REPL_SYNC clustering mode, with transaction enabled, as well as enabled indexing with InfinispanIndexManager.
> The test adds several nodes with the described configuration, adds entries to different nodes, and checks that the query for the entries returns the same result for all nodes.
> Then master node is killed, and then again new data is inserted and checked on all nodes. (Similiar test to https://github.com/infinispan/infinispan/blob/master/query/src/test/java/... but with enabled transaction).
> When the test tries to insert new entry to one of the caches, after the master node kill, the following exception appears:
> {code}
> testIndexingWorkDistribution(org.infinispan.query.distributed.MultiNodeReplicatedTxTest) Time elapsed: 1.656 sec <<< FAILURE!
> javax.transaction.RollbackException: ARJUNA016053: Could not commit transaction.
> at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1177)
> at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:117)
> at org.infinispan.query.distributed.MultiNodeDistributedTest.storeOn(MultiNodeDistributedTest.java:78)
> at org.infinispan.query.distributed.MultiNodeDistributedTest.testIndexingWorkDistribution(MultiNodeDistributedTest.java:102)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:601)
> at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:80)
> at org.testng.internal.Invoker.invokeMethod(Invoker.java:715)
> at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:907)
> at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1237)
> at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:127)
> at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:111)
> at org.testng.TestRunner.privateRun(TestRunner.java:767)
> at org.testng.TestRunner.run(TestRunner.java:617)
> at org.testng.SuiteRunner.runTest(SuiteRunner.java:334)
> at org.testng.SuiteRunner.access$000(SuiteRunner.java:37)
> at org.testng.SuiteRunner$SuiteWorker.run(SuiteRunner.java:368)
> at org.testng.internal.thread.ThreadUtil$2.call(ThreadUtil.java:64)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)
> Caused by: org.infinispan.CacheException: Could not prepare.
> at org.infinispan.transaction.synchronization.SynchronizationAdapter.beforeCompletion(SynchronizationAdapter.java:70)
> at com.arjuna.ats.internal.jta.resources.arjunacore.SynchronizationImple.beforeCompletion(SynchronizationImple.java:76)
> at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.beforeCompletion(TwoPhaseCoordinator.java:273)
> at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:93)
> at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:164)
> at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1165)
> ... 24 more
> Caused by: javax.transaction.xa.XAException
> at org.infinispan.transaction.TransactionCoordinator.prepare(TransactionCoordinator.java:161)
> at org.infinispan.transaction.TransactionCoordinator.prepare(TransactionCoordinator.java:123)
> at org.infinispan.transaction.synchronization.SynchronizationAdapter.beforeCompletion(SynchronizationAdapter.java:68)
> ... 29 more
> {code}
> You can find the test attached. The test which fails is MultiNodeReplicatedTxTest.java . The rest are the tests which are the parents and a bit modified compared to the version in infinispan repo.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 11 months
[JBoss JIRA] (ISPN-2727) Transactional put failed after master node with enabled InfinispanIndexManager is killed
by Galder Zamarreño (JIRA)
[ https://issues.jboss.org/browse/ISPN-2727?page=com.atlassian.jira.plugin.... ]
Work on ISPN-2727 started by Galder Zamarreño.
> Transactional put failed after master node with enabled InfinispanIndexManager is killed
> -----------------------------------------------------------------------------------------
>
> Key: ISPN-2727
> URL: https://issues.jboss.org/browse/ISPN-2727
> Project: Infinispan
> Issue Type: Bug
> Components: Querying
> Reporter: Anna Manukyan
> Assignee: Galder Zamarreño
> Fix For: 5.2.0.Final
>
> Attachments: MultiNodeDistributedTest.java, MultiNodeLocalTest.java, MultiNodeLocalTxTest.java, MultiNodeReplicatedTest.java, MultiNodeReplicatedTxTest.java
>
>
> I've added new test, which uses the following Infinispan configuration: REPL_SYNC clustering mode, with transaction enabled, as well as enabled indexing with InfinispanIndexManager.
> The test adds several nodes with the described configuration, adds entries to different nodes, and checks that the query for the entries returns the same result for all nodes.
> Then master node is killed, and then again new data is inserted and checked on all nodes. (Similiar test to https://github.com/infinispan/infinispan/blob/master/query/src/test/java/... but with enabled transaction).
> When the test tries to insert new entry to one of the caches, after the master node kill, the following exception appears:
> {code}
> testIndexingWorkDistribution(org.infinispan.query.distributed.MultiNodeReplicatedTxTest) Time elapsed: 1.656 sec <<< FAILURE!
> javax.transaction.RollbackException: ARJUNA016053: Could not commit transaction.
> at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1177)
> at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:117)
> at org.infinispan.query.distributed.MultiNodeDistributedTest.storeOn(MultiNodeDistributedTest.java:78)
> at org.infinispan.query.distributed.MultiNodeDistributedTest.testIndexingWorkDistribution(MultiNodeDistributedTest.java:102)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:601)
> at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:80)
> at org.testng.internal.Invoker.invokeMethod(Invoker.java:715)
> at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:907)
> at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1237)
> at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:127)
> at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:111)
> at org.testng.TestRunner.privateRun(TestRunner.java:767)
> at org.testng.TestRunner.run(TestRunner.java:617)
> at org.testng.SuiteRunner.runTest(SuiteRunner.java:334)
> at org.testng.SuiteRunner.access$000(SuiteRunner.java:37)
> at org.testng.SuiteRunner$SuiteWorker.run(SuiteRunner.java:368)
> at org.testng.internal.thread.ThreadUtil$2.call(ThreadUtil.java:64)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)
> Caused by: org.infinispan.CacheException: Could not prepare.
> at org.infinispan.transaction.synchronization.SynchronizationAdapter.beforeCompletion(SynchronizationAdapter.java:70)
> at com.arjuna.ats.internal.jta.resources.arjunacore.SynchronizationImple.beforeCompletion(SynchronizationImple.java:76)
> at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.beforeCompletion(TwoPhaseCoordinator.java:273)
> at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:93)
> at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:164)
> at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1165)
> ... 24 more
> Caused by: javax.transaction.xa.XAException
> at org.infinispan.transaction.TransactionCoordinator.prepare(TransactionCoordinator.java:161)
> at org.infinispan.transaction.TransactionCoordinator.prepare(TransactionCoordinator.java:123)
> at org.infinispan.transaction.synchronization.SynchronizationAdapter.beforeCompletion(SynchronizationAdapter.java:68)
> ... 29 more
> {code}
> You can find the test attached. The test which fails is MultiNodeReplicatedTxTest.java . The rest are the tests which are the parents and a bit modified compared to the version in infinispan repo.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 11 months