[JBoss JIRA] (ISPN-2958) Lucene Directory Read past EOF
by Mircea Markus (JIRA)
[ https://issues.jboss.org/browse/ISPN-2958?page=com.atlassian.jira.plugin.... ]
Mircea Markus updated ISPN-2958:
--------------------------------
Fix Version/s: 5.3.0.Final
> Lucene Directory Read past EOF
> ------------------------------
>
> Key: ISPN-2958
> URL: https://issues.jboss.org/browse/ISPN-2958
> Project: Infinispan
> Issue Type: Bug
> Components: Lucene Directory
> Affects Versions: 5.2.1.Final
> Reporter: Clement Pang
> Assignee: Sanne Grinovero
> Fix For: 5.3.0.Final
>
>
> This seems to be happening rather deterministically.
> Infinispan configuration (in JBoss EAP 6.1.0.Alpha):
> {code}
> <cache-container name="lucene">
> <local-cache name="dshell-index-data" start="EAGER">
> <eviction strategy="LIRS" max-entries="50000"/>
> <file-store path="lucene" passivation="true" purge="false"/>
> </local-cache>
> <local-cache name="dshell-index-metadata" start="EAGER">
> <file-store path="lucene" passivation="true" purge="false"/>
> </local-cache>
> <local-cache name="dshell-index-lock" start="EAGER">
> <file-store path="lucene" passivation="true" purge="false"/>
> </local-cache>
> </cache-container>
> {code}
> Upon shutting down the server and confirming that passivation did indeed write the data to disk, the subsequent start-up would fail right away with:
> {code}
> Caused by: org.hibernate.search.SearchException: Could not initialize index
> at org.hibernate.search.store.impl.DirectoryProviderHelper.initializeIndexIfNeeded(DirectoryProviderHelper.java:162)
> at org.hibernate.search.infinispan.impl.InfinispanDirectoryProvider.start(InfinispanDirectoryProvider.java:103)
> at org.hibernate.search.indexes.impl.DirectoryBasedIndexManager.initialize(DirectoryBasedIndexManager.java:104)
> at org.hibernate.search.indexes.impl.IndexManagerHolder.createIndexManager(IndexManagerHolder.java:227)
> ... 64 more
> Caused by: java.io.IOException: Read past EOF
> at org.infinispan.lucene.SingleChunkIndexInput.readByte(SingleChunkIndexInput.java:77)
> at org.apache.lucene.store.ChecksumIndexInput.readByte(ChecksumIndexInput.java:41)
> at org.apache.lucene.store.DataInput.readInt(DataInput.java:86)
> at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:272)
> at org.apache.lucene.index.IndexFileDeleter.<init>(IndexFileDeleter.java:182)
> at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:1168)
> at org.hibernate.search.store.impl.DirectoryProviderHelper.initializeIndexIfNeeded(DirectoryProviderHelper.java:157)
> ... 67 more
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 1 month
[JBoss JIRA] (ISPN-2960) '[ClusterTopologyManagerImpl] Failed to start rebalance: ... IllegalStateException: transport was closed' on cache stop
by Mircea Markus (JIRA)
[ https://issues.jboss.org/browse/ISPN-2960?page=com.atlassian.jira.plugin.... ]
Mircea Markus updated ISPN-2960:
--------------------------------
Assignee: Dan Berindei (was: Radoslav Husar)
> '[ClusterTopologyManagerImpl] Failed to start rebalance: ... IllegalStateException: transport was closed' on cache stop
> -----------------------------------------------------------------------------------------------------------------------
>
> Key: ISPN-2960
> URL: https://issues.jboss.org/browse/ISPN-2960
> Project: Infinispan
> Issue Type: Bug
> Components: State transfer
> Affects Versions: 5.2.5.Final
> Reporter: Radoslav Husar
> Assignee: Dan Berindei
>
> Components are still not keeping track whether the cache is stopping down and holf off stopping the channel. This is biting us in AS on undeloy/shutdown.
> {code}
> [JBossINF] [0m[31m07:07:58,044 ERROR [org.infinispan.topology.ClusterTopologyManagerImpl] (transport-thread-8) Failed to start rebalance: org.infinispan.CacheException: Remote (perf18/web) failed unexpectedly: java.util.concurrent.ExecutionException: org.infinispan.CacheException: Remote (perf18/web) failed unexpectedly
> [JBossINF] at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:232) [rt.jar:1.6.0_43]
> [JBossINF] at java.util.concurrent.FutureTask.get(FutureTask.java:91) [rt.jar:1.6.0_43]
> [JBossINF] at org.infinispan.topology.ClusterTopologyManagerImpl.executeOnClusterSync(ClusterTopologyManagerImpl.java:549)
> [JBossINF] at org.infinispan.topology.ClusterTopologyManagerImpl.broadcastRebalanceStart(ClusterTopologyManagerImpl.java:392)
> [JBossINF] at org.infinispan.topology.ClusterTopologyManagerImpl.startRebalance(ClusterTopologyManagerImpl.java:382)
> [JBossINF] at org.infinispan.topology.ClusterTopologyManagerImpl.access$000(ClusterTopologyManagerImpl.java:66)
> [JBossINF] at org.infinispan.topology.ClusterTopologyManagerImpl$1.call(ClusterTopologyManagerImpl.java:128)
> [JBossINF] at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) [rt.jar:1.6.0_43]
> [JBossINF] at java.util.concurrent.FutureTask.run(FutureTask.java:138) [rt.jar:1.6.0_43]
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) [rt.jar:1.6.0_43]
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) [rt.jar:1.6.0_43]
> [JBossINF] at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_43]
> [JBossINF] Caused by: org.infinispan.CacheException: Remote (perf18/web) failed unexpectedly
> [JBossINF] at org.infinispan.remoting.transport.AbstractTransport.parseResponseAndAddToResponseList(AbstractTransport.java:99)
> [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:541)
> [JBossINF] at org.infinispan.topology.ClusterTopologyManagerImpl$2.call(ClusterTopologyManagerImpl.java:531)
> [JBossINF] at org.infinispan.topology.ClusterTopologyManagerImpl$2.call(ClusterTopologyManagerImpl.java:528)
> [JBossINF] ... 5 more
> [JBossINF] Caused by: java.lang.IllegalStateException: transport was closed
> [JBossINF] at org.jgroups.blocks.GroupRequest.transportClosed(GroupRequest.java:273)
> [JBossINF] at org.jgroups.blocks.RequestCorrelator.stop(RequestCorrelator.java:269)
> [JBossINF] at org.jgroups.blocks.MessageDispatcher.stop(MessageDispatcher.java:152)
> [JBossINF] at org.jgroups.blocks.MessageDispatcher.channelDisconnected(MessageDispatcher.java:455)
> [JBossINF] at org.jgroups.Channel.notifyChannelDisconnected(Channel.java:507)
> [JBossINF] at org.jgroups.JChannel.disconnect(JChannel.java:363)
> [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.stop(JGroupsTransport.java:258)
> [JBossINF] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.6.0_43]
> [JBossINF] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) [rt.jar:1.6.0_43]
> [JBossINF] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) [rt.jar:1.6.0_43]
> [JBossINF] at java.lang.reflect.Method.invoke(Method.java:597) [rt.jar:1.6.0_43]
> [JBossINF] at org.infinispan.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:203)
> [JBossINF] at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:886)
> [JBossINF] at org.infinispan.factories.AbstractComponentRegistry.internalStop(AbstractComponentRegistry.java:693)
> [JBossINF] at org.infinispan.factories.AbstractComponentRegistry.stop(AbstractComponentRegistry.java:571)
> [JBossINF] at org.infinispan.factories.GlobalComponentRegistry.stop(GlobalComponentRegistry.java:260)
> [JBossINF] at org.infinispan.manager.DefaultCacheManager.stop(DefaultCacheManager.java:742)
> [JBossINF] at org.infinispan.manager.AbstractDelegatingEmbeddedCacheManager.stop(AbstractDelegatingEmbeddedCacheManager.java:179)
> [JBossINF] at org.jboss.as.clustering.infinispan.subsystem.EmbeddedCacheManagerService.stop(EmbeddedCacheManagerService.java:76)
> [JBossINF] at org.jboss.msc.service.ServiceControllerImpl$StopTask.stopService(ServiceControllerImpl.java:1911) [jboss-msc-1.0.4.GA-redhat-1.jar:1.0.4.GA-redhat-1]
> [JBossINF] at org.jboss.msc.service.ServiceControllerImpl$StopTask.run(ServiceControllerImpl.java:1874) [jboss-msc-1.0.4.GA-redhat-1.jar:1.0.4.GA-redhat-1]
> [JBossINF] ... 3 more
> {code}
> http://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-http-...
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 1 month
[JBoss JIRA] (ISPN-2960) '[ClusterTopologyManagerImpl] Failed to start rebalance: ... IllegalStateException: transport was closed' on cache stop
by Mircea Markus (JIRA)
[ https://issues.jboss.org/browse/ISPN-2960?page=com.atlassian.jira.plugin.... ]
Mircea Markus updated ISPN-2960:
--------------------------------
Assignee: Radoslav Husar (was: Mircea Markus)
> '[ClusterTopologyManagerImpl] Failed to start rebalance: ... IllegalStateException: transport was closed' on cache stop
> -----------------------------------------------------------------------------------------------------------------------
>
> Key: ISPN-2960
> URL: https://issues.jboss.org/browse/ISPN-2960
> Project: Infinispan
> Issue Type: Bug
> Components: State transfer
> Affects Versions: 5.2.5.Final
> Reporter: Radoslav Husar
> Assignee: Radoslav Husar
>
> Components are still not keeping track whether the cache is stopping down and holf off stopping the channel. This is biting us in AS on undeloy/shutdown.
> {code}
> [JBossINF] [0m[31m07:07:58,044 ERROR [org.infinispan.topology.ClusterTopologyManagerImpl] (transport-thread-8) Failed to start rebalance: org.infinispan.CacheException: Remote (perf18/web) failed unexpectedly: java.util.concurrent.ExecutionException: org.infinispan.CacheException: Remote (perf18/web) failed unexpectedly
> [JBossINF] at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:232) [rt.jar:1.6.0_43]
> [JBossINF] at java.util.concurrent.FutureTask.get(FutureTask.java:91) [rt.jar:1.6.0_43]
> [JBossINF] at org.infinispan.topology.ClusterTopologyManagerImpl.executeOnClusterSync(ClusterTopologyManagerImpl.java:549)
> [JBossINF] at org.infinispan.topology.ClusterTopologyManagerImpl.broadcastRebalanceStart(ClusterTopologyManagerImpl.java:392)
> [JBossINF] at org.infinispan.topology.ClusterTopologyManagerImpl.startRebalance(ClusterTopologyManagerImpl.java:382)
> [JBossINF] at org.infinispan.topology.ClusterTopologyManagerImpl.access$000(ClusterTopologyManagerImpl.java:66)
> [JBossINF] at org.infinispan.topology.ClusterTopologyManagerImpl$1.call(ClusterTopologyManagerImpl.java:128)
> [JBossINF] at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) [rt.jar:1.6.0_43]
> [JBossINF] at java.util.concurrent.FutureTask.run(FutureTask.java:138) [rt.jar:1.6.0_43]
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) [rt.jar:1.6.0_43]
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) [rt.jar:1.6.0_43]
> [JBossINF] at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_43]
> [JBossINF] Caused by: org.infinispan.CacheException: Remote (perf18/web) failed unexpectedly
> [JBossINF] at org.infinispan.remoting.transport.AbstractTransport.parseResponseAndAddToResponseList(AbstractTransport.java:99)
> [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:541)
> [JBossINF] at org.infinispan.topology.ClusterTopologyManagerImpl$2.call(ClusterTopologyManagerImpl.java:531)
> [JBossINF] at org.infinispan.topology.ClusterTopologyManagerImpl$2.call(ClusterTopologyManagerImpl.java:528)
> [JBossINF] ... 5 more
> [JBossINF] Caused by: java.lang.IllegalStateException: transport was closed
> [JBossINF] at org.jgroups.blocks.GroupRequest.transportClosed(GroupRequest.java:273)
> [JBossINF] at org.jgroups.blocks.RequestCorrelator.stop(RequestCorrelator.java:269)
> [JBossINF] at org.jgroups.blocks.MessageDispatcher.stop(MessageDispatcher.java:152)
> [JBossINF] at org.jgroups.blocks.MessageDispatcher.channelDisconnected(MessageDispatcher.java:455)
> [JBossINF] at org.jgroups.Channel.notifyChannelDisconnected(Channel.java:507)
> [JBossINF] at org.jgroups.JChannel.disconnect(JChannel.java:363)
> [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.stop(JGroupsTransport.java:258)
> [JBossINF] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.6.0_43]
> [JBossINF] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) [rt.jar:1.6.0_43]
> [JBossINF] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) [rt.jar:1.6.0_43]
> [JBossINF] at java.lang.reflect.Method.invoke(Method.java:597) [rt.jar:1.6.0_43]
> [JBossINF] at org.infinispan.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:203)
> [JBossINF] at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:886)
> [JBossINF] at org.infinispan.factories.AbstractComponentRegistry.internalStop(AbstractComponentRegistry.java:693)
> [JBossINF] at org.infinispan.factories.AbstractComponentRegistry.stop(AbstractComponentRegistry.java:571)
> [JBossINF] at org.infinispan.factories.GlobalComponentRegistry.stop(GlobalComponentRegistry.java:260)
> [JBossINF] at org.infinispan.manager.DefaultCacheManager.stop(DefaultCacheManager.java:742)
> [JBossINF] at org.infinispan.manager.AbstractDelegatingEmbeddedCacheManager.stop(AbstractDelegatingEmbeddedCacheManager.java:179)
> [JBossINF] at org.jboss.as.clustering.infinispan.subsystem.EmbeddedCacheManagerService.stop(EmbeddedCacheManagerService.java:76)
> [JBossINF] at org.jboss.msc.service.ServiceControllerImpl$StopTask.stopService(ServiceControllerImpl.java:1911) [jboss-msc-1.0.4.GA-redhat-1.jar:1.0.4.GA-redhat-1]
> [JBossINF] at org.jboss.msc.service.ServiceControllerImpl$StopTask.run(ServiceControllerImpl.java:1874) [jboss-msc-1.0.4.GA-redhat-1.jar:1.0.4.GA-redhat-1]
> [JBossINF] ... 3 more
> {code}
> http://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-http-...
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 1 month
[JBoss JIRA] (ISPN-2960) '[ClusterTopologyManagerImpl] Failed to start rebalance: ... IllegalStateException: transport was closed' on cache stop
by Mircea Markus (JIRA)
[ https://issues.jboss.org/browse/ISPN-2960?page=com.atlassian.jira.plugin.... ]
Mircea Markus commented on ISPN-2960:
-------------------------------------
[~dan.berindei] please confirm whether this is a duplicate of ISPN-2752 and close if so.
[~rhusar] does this have any impact on AS other that polluting the logs?
> '[ClusterTopologyManagerImpl] Failed to start rebalance: ... IllegalStateException: transport was closed' on cache stop
> -----------------------------------------------------------------------------------------------------------------------
>
> Key: ISPN-2960
> URL: https://issues.jboss.org/browse/ISPN-2960
> Project: Infinispan
> Issue Type: Bug
> Components: State transfer
> Affects Versions: 5.2.5.Final
> Reporter: Radoslav Husar
> Assignee: Mircea Markus
>
> Components are still not keeping track whether the cache is stopping down and holf off stopping the channel. This is biting us in AS on undeloy/shutdown.
> {code}
> [JBossINF] [0m[31m07:07:58,044 ERROR [org.infinispan.topology.ClusterTopologyManagerImpl] (transport-thread-8) Failed to start rebalance: org.infinispan.CacheException: Remote (perf18/web) failed unexpectedly: java.util.concurrent.ExecutionException: org.infinispan.CacheException: Remote (perf18/web) failed unexpectedly
> [JBossINF] at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:232) [rt.jar:1.6.0_43]
> [JBossINF] at java.util.concurrent.FutureTask.get(FutureTask.java:91) [rt.jar:1.6.0_43]
> [JBossINF] at org.infinispan.topology.ClusterTopologyManagerImpl.executeOnClusterSync(ClusterTopologyManagerImpl.java:549)
> [JBossINF] at org.infinispan.topology.ClusterTopologyManagerImpl.broadcastRebalanceStart(ClusterTopologyManagerImpl.java:392)
> [JBossINF] at org.infinispan.topology.ClusterTopologyManagerImpl.startRebalance(ClusterTopologyManagerImpl.java:382)
> [JBossINF] at org.infinispan.topology.ClusterTopologyManagerImpl.access$000(ClusterTopologyManagerImpl.java:66)
> [JBossINF] at org.infinispan.topology.ClusterTopologyManagerImpl$1.call(ClusterTopologyManagerImpl.java:128)
> [JBossINF] at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) [rt.jar:1.6.0_43]
> [JBossINF] at java.util.concurrent.FutureTask.run(FutureTask.java:138) [rt.jar:1.6.0_43]
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) [rt.jar:1.6.0_43]
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) [rt.jar:1.6.0_43]
> [JBossINF] at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_43]
> [JBossINF] Caused by: org.infinispan.CacheException: Remote (perf18/web) failed unexpectedly
> [JBossINF] at org.infinispan.remoting.transport.AbstractTransport.parseResponseAndAddToResponseList(AbstractTransport.java:99)
> [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:541)
> [JBossINF] at org.infinispan.topology.ClusterTopologyManagerImpl$2.call(ClusterTopologyManagerImpl.java:531)
> [JBossINF] at org.infinispan.topology.ClusterTopologyManagerImpl$2.call(ClusterTopologyManagerImpl.java:528)
> [JBossINF] ... 5 more
> [JBossINF] Caused by: java.lang.IllegalStateException: transport was closed
> [JBossINF] at org.jgroups.blocks.GroupRequest.transportClosed(GroupRequest.java:273)
> [JBossINF] at org.jgroups.blocks.RequestCorrelator.stop(RequestCorrelator.java:269)
> [JBossINF] at org.jgroups.blocks.MessageDispatcher.stop(MessageDispatcher.java:152)
> [JBossINF] at org.jgroups.blocks.MessageDispatcher.channelDisconnected(MessageDispatcher.java:455)
> [JBossINF] at org.jgroups.Channel.notifyChannelDisconnected(Channel.java:507)
> [JBossINF] at org.jgroups.JChannel.disconnect(JChannel.java:363)
> [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.stop(JGroupsTransport.java:258)
> [JBossINF] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.6.0_43]
> [JBossINF] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) [rt.jar:1.6.0_43]
> [JBossINF] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) [rt.jar:1.6.0_43]
> [JBossINF] at java.lang.reflect.Method.invoke(Method.java:597) [rt.jar:1.6.0_43]
> [JBossINF] at org.infinispan.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:203)
> [JBossINF] at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:886)
> [JBossINF] at org.infinispan.factories.AbstractComponentRegistry.internalStop(AbstractComponentRegistry.java:693)
> [JBossINF] at org.infinispan.factories.AbstractComponentRegistry.stop(AbstractComponentRegistry.java:571)
> [JBossINF] at org.infinispan.factories.GlobalComponentRegistry.stop(GlobalComponentRegistry.java:260)
> [JBossINF] at org.infinispan.manager.DefaultCacheManager.stop(DefaultCacheManager.java:742)
> [JBossINF] at org.infinispan.manager.AbstractDelegatingEmbeddedCacheManager.stop(AbstractDelegatingEmbeddedCacheManager.java:179)
> [JBossINF] at org.jboss.as.clustering.infinispan.subsystem.EmbeddedCacheManagerService.stop(EmbeddedCacheManagerService.java:76)
> [JBossINF] at org.jboss.msc.service.ServiceControllerImpl$StopTask.stopService(ServiceControllerImpl.java:1911) [jboss-msc-1.0.4.GA-redhat-1.jar:1.0.4.GA-redhat-1]
> [JBossINF] at org.jboss.msc.service.ServiceControllerImpl$StopTask.run(ServiceControllerImpl.java:1874) [jboss-msc-1.0.4.GA-redhat-1.jar:1.0.4.GA-redhat-1]
> [JBossINF] ... 3 more
> {code}
> http://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-http-...
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 1 month
[JBoss JIRA] (ISPN-2966) NBST: Concurrent leavers can lead to deadlock
by Dan Berindei (JIRA)
[ https://issues.jboss.org/browse/ISPN-2966?page=com.atlassian.jira.plugin.... ]
Dan Berindei commented on ISPN-2966:
------------------------------------
[~mircea.markus] I think this is pretty critical, because the transactions on NodeE are blocked waiting for NodeE to receive transaction data from NodeH.
[~pruivo] After node NodeH sends the LEAVE command to the coordinator, it may not receive any transactions any more. So it would be wrong for it to reply with a successful response to a GET_TRANSACTIONS command. Instead of accepting the REBALANCE_START command and replying to GET_TRANSACTIONS normally, we should wake up the GET_TRANSACTIONS thread and send an exception back to NodeE and NodeG.
There are some steps missing from the description:
{noformat}
8) NodeE delivers the LEAVE message from NodeH
9) NodeE broadcasts a CH_UPDATE message to NodeG and to itself (in the same thread)
10) While processing the CH_UPDATE message, NodeE tries to lock the LocalCacheStatus object. But it can't because the thread that's processing the REBALANCE_START command is already holding the lock.
11a) Eventually the GET_TRANSACTIONS commands time out, NodeE requests and receives transactions from NodeG, and then it can process the LEAVE command from NodeH.
11b) Alternatively, the LEAVE command times out on NodeH, and when the JGroups channel on NodeH shuts down the GET_TRANSACTIONS commands on NodeE and NodeG fail with a SuspectException.
{noformat}
If we change step 9) to process the CH_UPDATE message asynchronously, that will be enough to allow NodeH to stop and for the GET_TRANSACTIONS commands to fail.
> NBST: Concurrent leavers can lead to deadlock
> ---------------------------------------------
>
> Key: ISPN-2966
> URL: https://issues.jboss.org/browse/ISPN-2966
> Project: Infinispan
> Issue Type: Bug
> Reporter: Pedro Ruivo
> Assignee: Dan Berindei
> Labels: state_transfer
> Fix For: 5.3.0.Final
>
> Attachments: thread-dump.txt, trace.log
>
>
> This sequence of events, leads to a thread deadlock in the coordinator
> {code}
> 1) NodeF sends LEAVE message. new topologyId=8
> 2) NodeE delivers REBALANCE_START(8)
> 3) NodeF and NodeG delivers REBALANCE_START(8)
> 4) NodeH delivers GET_TRANSACTION(8) from NodeE ==> Transactions were requested by node ConcurrentNonOverlappingLeaveTest-NodeE-28744 with topology 8, greater than the local topology (7). Waiting for topology 8 to be installed locally.
> 5) NodeH sends LEAVE message. new topologyId=9
> 6) NodeH delivers REBALANCE_START(8) ==> Ignoring rebalance 8 for cache dist that doesn't exist locally
> 7) NodeH delivers GET_TRANSACTION(8) from NodeG ==> Transactions were requested by node ConcurrentNonOverlappingLeaveTest-NodeG-31669 with topology 8, greater than the local topology (7). Waiting for topology 8 to be installed locally.
> {code}
> Possible solutions are:
> - send the REBALANCE_START/CH_UPDATE async
> - throw an exception when a GET_TRANSACTION is received and the node is shutting down.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 1 month
[JBoss JIRA] (ISPN-2964) putForExternalRead to L1 not invalidated
by Mircea Markus (JIRA)
[ https://issues.jboss.org/browse/ISPN-2964?page=com.atlassian.jira.plugin.... ]
Mircea Markus updated ISPN-2964:
--------------------------------
Fix Version/s: 5.3.0.Alpha1
5.3.0.Final
> putForExternalRead to L1 not invalidated
> ----------------------------------------
>
> Key: ISPN-2964
> URL: https://issues.jboss.org/browse/ISPN-2964
> Project: Infinispan
> Issue Type: Bug
> Components: Distributed Cache
> Affects Versions: 5.2.1.Final
> Reporter: Sebastian Tusk
> Assignee: Mircea Markus
> Fix For: 5.3.0.Alpha1, 5.3.0.Final
>
>
> With transactional distributed caches it happens that Cache.putForExternalRead(k,v) places an entry into L1 that never gets invalidated. It seems to happen when the the owner of k doesn't have the entry. In this case the non owner puts k into his cache without having the owner registering this. Usually the owner stores all requesters in L1ManagerImpl.addRequester and sends out invalidations to the requesters. What should happen is that the entry is replicated to the owner.
> Cache config:
> <namedCache name="entity">
> <jmxStatistics enabled="true" />
> <clustering mode="dist">
> <stateTransfer fetchInMemoryState="false" timeout="20000" />
> <async />
> <l1 enabled="true" />
> <hash numOwners="1"/>
> </clustering>
> <locking isolationLevel="READ_COMMITTED"
> lockAcquisitionTimeout="15000" useLockStriping="false" />
> <eviction maxEntries="10000" strategy="LRU" />
> <expiration maxIdle="100000" wakeUpInterval="5000"/>
> <storeAsBinary storeKeysAsBinary="true" storeValuesAsBinary="false" enabled="false" />
> <transaction transactionMode="TRANSACTIONAL" autoCommit="false" lockingMode="OPTIMISTIC"/>
> </namedCache>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 1 month
[JBoss JIRA] (ISPN-2965) L1 and early invalidation leaves inconsistent state
by Mircea Markus (JIRA)
[ https://issues.jboss.org/browse/ISPN-2965?page=com.atlassian.jira.plugin.... ]
Mircea Markus updated ISPN-2965:
--------------------------------
Assignee: Adrian Nistor (was: Mircea Markus)
> L1 and early invalidation leaves inconsistent state
> ---------------------------------------------------
>
> Key: ISPN-2965
> URL: https://issues.jboss.org/browse/ISPN-2965
> Project: Infinispan
> Issue Type: Bug
> Components: Distributed Cache, Transactions
> Affects Versions: 5.2.1.Final
> Reporter: Sebastian Tusk
> Assignee: Adrian Nistor
> Labels: 5.2.x
> Fix For: 5.3.0.Alpha1, 5.3.0.Final
>
>
> In a distributed transactional cache with L1 enabled I can observe the following.
> Prepare cache by adding an entry with Cache.put( k, v1 ).
> 1. Node B starts with adding a changed value. Cache.put( k, v2 )
> 2. Node B TxDistributionInterceptor.visitPrepareCommand flushL1Caches sends invalidations.
> 3. Node A calls Cache.get( k ) retrieves v1 and stores this value in L1.
> 4. Node B proceeds with transaction.
> The result is that Node A answers subsequent Cache.get(k) with v1 and Node B answers with v2.
> It seems the invalidation is either send to early or must be synchronized in some way with the transaction.
> Cache config:
> <namedCache name="entity">
> <jmxStatistics enabled="true" />
> <clustering mode="dist">
> <stateTransfer fetchInMemoryState="false" timeout="20000" />
> <async />
> <l1 enabled="true" />
> <hash numOwners="1"/>
> </clustering>
> <locking isolationLevel="READ_COMMITTED"
> lockAcquisitionTimeout="15000" useLockStriping="false" />
> <eviction maxEntries="10000" strategy="LRU" />
> <expiration maxIdle="100000" wakeUpInterval="5000"/>
> <storeAsBinary storeKeysAsBinary="true" storeValuesAsBinary="false" enabled="false" />
> <transaction transactionMode="TRANSACTIONAL" autoCommit="false" lockingMode="OPTIMISTIC"/>
> </namedCache>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 1 month
[JBoss JIRA] (ISPN-2965) L1 and early invalidation leaves inconsistent state
by Mircea Markus (JIRA)
[ https://issues.jboss.org/browse/ISPN-2965?page=com.atlassian.jira.plugin.... ]
Mircea Markus commented on ISPN-2965:
-------------------------------------
Indeed a bug, well spotted!
In order to solve this the invalidation message should be sent *after* the new value was applied, i.e. the invalidation should happen after the EntryWrappingInterceptor applied the changes. For non-tx caches we already have an L1Interceptor so a solution might be to move it before the EntryWrapping interceptor. For tx caches the DistTxInterceptor would need to be refactored in a similar way.
> L1 and early invalidation leaves inconsistent state
> ---------------------------------------------------
>
> Key: ISPN-2965
> URL: https://issues.jboss.org/browse/ISPN-2965
> Project: Infinispan
> Issue Type: Bug
> Components: Distributed Cache, Transactions
> Affects Versions: 5.2.1.Final
> Reporter: Sebastian Tusk
> Assignee: Mircea Markus
> Labels: 5.2.x
> Fix For: 5.3.0.Alpha1, 5.3.0.Final
>
>
> In a distributed transactional cache with L1 enabled I can observe the following.
> Prepare cache by adding an entry with Cache.put( k, v1 ).
> 1. Node B starts with adding a changed value. Cache.put( k, v2 )
> 2. Node B TxDistributionInterceptor.visitPrepareCommand flushL1Caches sends invalidations.
> 3. Node A calls Cache.get( k ) retrieves v1 and stores this value in L1.
> 4. Node B proceeds with transaction.
> The result is that Node A answers subsequent Cache.get(k) with v1 and Node B answers with v2.
> It seems the invalidation is either send to early or must be synchronized in some way with the transaction.
> Cache config:
> <namedCache name="entity">
> <jmxStatistics enabled="true" />
> <clustering mode="dist">
> <stateTransfer fetchInMemoryState="false" timeout="20000" />
> <async />
> <l1 enabled="true" />
> <hash numOwners="1"/>
> </clustering>
> <locking isolationLevel="READ_COMMITTED"
> lockAcquisitionTimeout="15000" useLockStriping="false" />
> <eviction maxEntries="10000" strategy="LRU" />
> <expiration maxIdle="100000" wakeUpInterval="5000"/>
> <storeAsBinary storeKeysAsBinary="true" storeValuesAsBinary="false" enabled="false" />
> <transaction transactionMode="TRANSACTIONAL" autoCommit="false" lockingMode="OPTIMISTIC"/>
> </namedCache>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 1 month
[JBoss JIRA] (ISPN-2965) L1 and early invalidation leaves inconsistent state
by Mircea Markus (JIRA)
[ https://issues.jboss.org/browse/ISPN-2965?page=com.atlassian.jira.plugin.... ]
Mircea Markus updated ISPN-2965:
--------------------------------
Labels: 5.2.x (was: )
> L1 and early invalidation leaves inconsistent state
> ---------------------------------------------------
>
> Key: ISPN-2965
> URL: https://issues.jboss.org/browse/ISPN-2965
> Project: Infinispan
> Issue Type: Bug
> Components: Distributed Cache, Transactions
> Affects Versions: 5.2.1.Final
> Reporter: Sebastian Tusk
> Assignee: Mircea Markus
> Labels: 5.2.x
> Fix For: 5.3.0.Alpha1, 5.3.0.Final
>
>
> In a distributed transactional cache with L1 enabled I can observe the following.
> Prepare cache by adding an entry with Cache.put( k, v1 ).
> 1. Node B starts with adding a changed value. Cache.put( k, v2 )
> 2. Node B TxDistributionInterceptor.visitPrepareCommand flushL1Caches sends invalidations.
> 3. Node A calls Cache.get( k ) retrieves v1 and stores this value in L1.
> 4. Node B proceeds with transaction.
> The result is that Node A answers subsequent Cache.get(k) with v1 and Node B answers with v2.
> It seems the invalidation is either send to early or must be synchronized in some way with the transaction.
> Cache config:
> <namedCache name="entity">
> <jmxStatistics enabled="true" />
> <clustering mode="dist">
> <stateTransfer fetchInMemoryState="false" timeout="20000" />
> <async />
> <l1 enabled="true" />
> <hash numOwners="1"/>
> </clustering>
> <locking isolationLevel="READ_COMMITTED"
> lockAcquisitionTimeout="15000" useLockStriping="false" />
> <eviction maxEntries="10000" strategy="LRU" />
> <expiration maxIdle="100000" wakeUpInterval="5000"/>
> <storeAsBinary storeKeysAsBinary="true" storeValuesAsBinary="false" enabled="false" />
> <transaction transactionMode="TRANSACTIONAL" autoCommit="false" lockingMode="OPTIMISTIC"/>
> </namedCache>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 1 month
[JBoss JIRA] (ISPN-2965) L1 and early invalidation leaves inconsistent state
by Mircea Markus (JIRA)
[ https://issues.jboss.org/browse/ISPN-2965?page=com.atlassian.jira.plugin.... ]
Mircea Markus updated ISPN-2965:
--------------------------------
Fix Version/s: 5.3.0.Alpha1
5.3.0.Final
> L1 and early invalidation leaves inconsistent state
> ---------------------------------------------------
>
> Key: ISPN-2965
> URL: https://issues.jboss.org/browse/ISPN-2965
> Project: Infinispan
> Issue Type: Bug
> Components: Distributed Cache, Transactions
> Affects Versions: 5.2.1.Final
> Reporter: Sebastian Tusk
> Assignee: Mircea Markus
> Fix For: 5.3.0.Alpha1, 5.3.0.Final
>
>
> In a distributed transactional cache with L1 enabled I can observe the following.
> Prepare cache by adding an entry with Cache.put( k, v1 ).
> 1. Node B starts with adding a changed value. Cache.put( k, v2 )
> 2. Node B TxDistributionInterceptor.visitPrepareCommand flushL1Caches sends invalidations.
> 3. Node A calls Cache.get( k ) retrieves v1 and stores this value in L1.
> 4. Node B proceeds with transaction.
> The result is that Node A answers subsequent Cache.get(k) with v1 and Node B answers with v2.
> It seems the invalidation is either send to early or must be synchronized in some way with the transaction.
> Cache config:
> <namedCache name="entity">
> <jmxStatistics enabled="true" />
> <clustering mode="dist">
> <stateTransfer fetchInMemoryState="false" timeout="20000" />
> <async />
> <l1 enabled="true" />
> <hash numOwners="1"/>
> </clustering>
> <locking isolationLevel="READ_COMMITTED"
> lockAcquisitionTimeout="15000" useLockStriping="false" />
> <eviction maxEntries="10000" strategy="LRU" />
> <expiration maxIdle="100000" wakeUpInterval="5000"/>
> <storeAsBinary storeKeysAsBinary="true" storeValuesAsBinary="false" enabled="false" />
> <transaction transactionMode="TRANSACTIONAL" autoCommit="false" lockingMode="OPTIMISTIC"/>
> </namedCache>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 1 month