[JBoss JIRA] (ISPN-4599) InfinispanIndexManager locks held when primary node change
by Gustavo Fernandes (JIRA)
Gustavo Fernandes created ISPN-4599:
---------------------------------------
Summary: InfinispanIndexManager locks held when primary node change
Key: ISPN-4599
URL: https://issues.jboss.org/browse/ISPN-4599
Project: Infinispan
Issue Type: Bug
Security Level: Public (Everyone can see)
Components: Embedded Querying, Remote Querying
Affects Versions: 7.0.0.Alpha5
Reporter: Gustavo Fernandes
Assignee: Sanne Grinovero
Given the following configuration:
{code:xml}
<cache-container name="store" default-cache="default" statistics="true">
<transport cluster="Infinispan-Query-Cluster"/>
<!-- *************************************** -->
<!-- Default Cache, with indexing enabled. -->
<!-- *************************************** -->
<replicated-cache name="passivation" mode="SYNC" remote-timeout="20000" statistics="true">
<indexing index="LOCAL">
<property name="hibernate.search.default.indexmanager">
org.infinispan.query.indexmanager.InfinispanIndexManager
</property>
<property name="hibernate.search.default.directory_provider">infinispan</property>
</indexing>
</replicated-cache>
<replicated-cache name="LuceneIndexesMetadata" mode="SYNC" remote-timeout="25000">
<indexing index="NONE"/>
</replicated-cache>
<distributed-cache name="LuceneIndexesData" mode="SYNC" remote-timeout="25000">
<indexing index="NONE"/>
</distributed-cache>
<replicated-cache name="LuceneIndexesLocking" mode="SYNC" remote-timeout="25000">
<indexing index="NONE"/>
</replicated-cache>
</cache-container>
{code}
Scenario:
- Start Node 1
- Insert a few @Indexed objects in the cache
- Start Node 2
- Insert a few @Indexed objects in the cache
Node 2 cannot acquire the locks to write and throws errors:
{code}
org.hibernate.search.backend.impl.lucene.LuceneBackendQueueTask applyUpdates
ERROR: HSEARCH000072: Couldn't open the IndexWriter because of previous error: operation skipped, index ouf of sync!
Aug 01, 2014 4:48:23 PM org.hibernate.search.exception.impl.LogErrorHandler handleException
ERROR: HSEARCH000058: Exception occurred org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: org.infinispan.lucene.locking.BaseLuceneLock@2c1e889f
{code}
During the insertion on Node1, the InfinispanCommandsBackend elects Node 1 as the primary node (since it's the only one) and acquires the lock on the LuceneIndexesLocking.
When node 2 joins and the topology changes, the InfinispanCommandsBackend elects Node 2 as the primary node, but it fails to acquire the lock (held by Node 1)
--
This message was sent by Atlassian JIRA
(v6.2.6#6264)
9 years, 9 months
[JBoss JIRA] (ISPN-4598) ClientClusterEventsTest.testFilteringInCluster random failures
by Dan Berindei (JIRA)
Dan Berindei created ISPN-4598:
----------------------------------
Summary: ClientClusterEventsTest.testFilteringInCluster random failures
Key: ISPN-4598
URL: https://issues.jboss.org/browse/ISPN-4598
Project: Infinispan
Issue Type: Bug
Security Level: Public (Everyone can see)
Components: Test Suite - Server
Affects Versions: 7.0.0.Alpha5
Reporter: Dan Berindei
Assignee: Tristan Tarrant
Priority: Blocker
Fix For: 7.0.0.Beta1
The fix for ISPN-4378 introduced a random failure in {{ClientClusterEventsTest.testFilteringInCluster}}. It happened in a few PR builds in CI, and I'm getting it almost every time on my machine (with trace logging enabled):
{noformat}
2014-08-01 17:16:57,084 ERROR (remote-thread-ClientClusterEventsTest-NodeB-p757-t6) [InvocationContextInterceptor] ISPN000136: Execution error
java.lang.NullPointerException
at org.infinispan.server.hotrod.ClientListenerRegistry$BinaryFilter.accept(ClientListenerRegistry.scala:178)
at org.infinispan.server.hotrod.ClientListenerRegistry$BinaryFilter.accept(ClientListenerRegistry.scala:172)
at org.infinispan.notifications.cachelistener.CacheNotifierImpl$BaseCacheEntryListenerInvocation.shouldInvoke(CacheNotifierImpl.java:978)
at org.infinispan.notifications.cachelistener.CacheNotifierImpl$BaseCacheEntryListenerInvocation.invoke(CacheNotifierImpl.java:943)
at org.infinispan.notifications.cachelistener.CacheNotifierImpl.notifyCacheEntryCreated(CacheNotifierImpl.java:247)
at org.infinispan.container.EntryFactoryImpl.newMvccEntryForPut(EntryFactoryImpl.java:272)
at org.infinispan.container.EntryFactoryImpl.wrapEntryForPut(EntryFactoryImpl.java:204)
at org.infinispan.interceptors.EntryWrappingInterceptor.wrapEntryForPutIfNeeded(EntryWrappingInterceptor.java:182)
at org.infinispan.interceptors.EntryWrappingInterceptor.visitPutKeyValueCommand(EntryWrappingInterceptor.java:176)
{noformat}
Tristan realized that Scala doesn't execute field initializers when deserializing, so the marshaller is left {{null}} and we get a NPE.
--
This message was sent by Atlassian JIRA
(v6.2.6#6264)
9 years, 9 months
[JBoss JIRA] (ISPN-4597) Query interceptor should update index during commit
by Sanne Grinovero (JIRA)
[ https://issues.jboss.org/browse/ISPN-4597?page=com.atlassian.jira.plugin.... ]
Sanne Grinovero commented on ISPN-4597:
---------------------------------------
Dan, the {{visitCommitCommand}} has no access to the list of applied modifications, which really isn't optional to know what to do on the indexes.
Could core provide those details, or should we try stick this information somewhere during the prepare phase? Do you have suggestions to avoid leaks?
> Query interceptor should update index during commit
> ---------------------------------------------------
>
> Key: ISPN-4597
> URL: https://issues.jboss.org/browse/ISPN-4597
> Project: Infinispan
> Issue Type: Bug
> Security Level: Public(Everyone can see)
> Components: Embedded Querying
> Affects Versions: 7.0.0.Alpha5
> Reporter: Dan Berindei
> Assignee: Sanne Grinovero
> Fix For: 7.0.0.Beta1
>
>
> {{QueryInterceptor}} updates indexes during {{visitPrepareCommand}}. However, a tx that was prepared successfully is not guaranteed to be actually committed (e.g. if another node fails to prepare, or if another XA resource fails). The index should only be updated during {{visitPrepareCommand}} if the prepare is 1-phase, otherwise it should be updated during {{visitCommitCommand}}.
--
This message was sent by Atlassian JIRA
(v6.2.6#6264)
9 years, 9 months
[JBoss JIRA] (ISPN-4559) Replace fails with cache loader
by Pedro Ruivo (JIRA)
[ https://issues.jboss.org/browse/ISPN-4559?page=com.atlassian.jira.plugin.... ]
Work on ISPN-4559 started by Pedro Ruivo.
> Replace fails with cache loader
> -------------------------------
>
> Key: ISPN-4559
> URL: https://issues.jboss.org/browse/ISPN-4559
> Project: Infinispan
> Issue Type: Bug
> Security Level: Public(Everyone can see)
> Components: Loaders and Stores
> Affects Versions: 7.0.0.Alpha5
> Reporter: Dennis Reed
> Assignee: Pedro Ruivo
> Priority: Critical
> Fix For: 7.0.0.Beta1
>
>
> cache.replace(key, oldValue, newValue) compares the current value in the cache to oldValue, and if they differ it turns into a no-op.
> However, CacheLoaderInterceptor does not load entries for a ReplaceCommand.
> If the entry only exists in the loader and not in memory, this causes the replace to fail.
> CacheLoaderInterceptor must always load the value for a ReplaceCommand.
--
This message was sent by Atlassian JIRA
(v6.2.6#6264)
9 years, 9 months
[JBoss JIRA] (ISPN-4520) JdbcBinaryStoreTest.testLoadAndStoreWithLifespanAndIdle random failures
by Pedro Ruivo (JIRA)
[ https://issues.jboss.org/browse/ISPN-4520?page=com.atlassian.jira.plugin.... ]
Pedro Ruivo updated ISPN-4520:
------------------------------
Status: Resolved (was: Pull Request Sent)
Resolution: Done
> JdbcBinaryStoreTest.testLoadAndStoreWithLifespanAndIdle random failures
> -----------------------------------------------------------------------
>
> Key: ISPN-4520
> URL: https://issues.jboss.org/browse/ISPN-4520
> Project: Infinispan
> Issue Type: Bug
> Security Level: Public(Everyone can see)
> Components: Core, Test Suite - Core
> Affects Versions: 7.0.0.Alpha4
> Reporter: Dan Berindei
> Assignee: Pedro Ruivo
> Priority: Blocker
> Labels: testsuite_stability
> Fix For: 7.0.0.Beta1, 7.0.0.Alpha5
>
>
> I think the 1s timeout is a bit small for the CI machine:
> {noformat}
> java.lang.IllegalStateException: Purge has timed out
> at org.infinispan.persistence.BaseStoreTest.purgeExpired(BaseStoreTest.java:268)
> {noformat}
> The other methods using {{BaseStoreTest.purgeExpired}} and the other tests extending BaseStoreTest probably have the same problem.
--
This message was sent by Atlassian JIRA
(v6.2.6#6264)
9 years, 9 months
[JBoss JIRA] (ISPN-4580) Refactor BaseStoreTest
by Pedro Ruivo (JIRA)
[ https://issues.jboss.org/browse/ISPN-4580?page=com.atlassian.jira.plugin.... ]
Pedro Ruivo updated ISPN-4580:
------------------------------
Status: Resolved (was: Pull Request Sent)
Resolution: Done
> Refactor BaseStoreTest
> -----------------------
>
> Key: ISPN-4580
> URL: https://issues.jboss.org/browse/ISPN-4580
> Project: Infinispan
> Issue Type: Task
> Security Level: Public(Everyone can see)
> Affects Versions: 7.0.0.Alpha5
> Reporter: Pedro Ruivo
> Assignee: Pedro Ruivo
> Fix For: 7.0.0.Beta1
>
>
> Some refactor like:
> * remove duplicated code
> * use mock cache for the cache store (avoid creating cache manager and caches as they are not used)
> * prepare the code to use a custom TimeService
--
This message was sent by Atlassian JIRA
(v6.2.6#6264)
9 years, 9 months
[JBoss JIRA] (ISPN-4565) ReplTotalOrderVersionedStateTransferTest.testStateTransfer random failures
by Pedro Ruivo (JIRA)
[ https://issues.jboss.org/browse/ISPN-4565?page=com.atlassian.jira.plugin.... ]
Pedro Ruivo updated ISPN-4565:
------------------------------
Git Pull Request: https://github.com/infinispan/infinispan/pull/2766 (was: https://github.com/infinispan/infinispan/pull/2760)
> ReplTotalOrderVersionedStateTransferTest.testStateTransfer random failures
> --------------------------------------------------------------------------
>
> Key: ISPN-4565
> URL: https://issues.jboss.org/browse/ISPN-4565
> Project: Infinispan
> Issue Type: Bug
> Security Level: Public(Everyone can see)
> Components: Core, State Transfer, Test Suite - Core
> Affects Versions: 7.0.0.Alpha5
> Reporter: Dan Berindei
> Assignee: Pedro Ruivo
> Priority: Blocker
> Labels: testsuite_stability
> Fix For: 7.0.0.Beta1
>
>
> A NullPointerException appears while processing the 2nd tx:
> {noformat}
> 04:27:12,078 DEBUG (remote-thread-ReplTotalOrderVersionedStateTransferTest-NodeB-p12450-t4:) [TotalOrderInterceptor] Exception while rollback transaction ReplTotalOrderVersionedStateTransferTest-NodeC-12055:56786
> java.lang.NullPointerException
> at org.infinispan.transaction.impl.WriteSkewHelper.performTotalOrderWriteSkewCheckAndReturnNewVersions(WriteSkewHelper.java:76)
> at org.infinispan.interceptors.locking.ClusteringDependentLogic$AbstractClusteringDependentLogic.totalOrderCreateNewVersionsAndCheckForWriteSkews(ClusteringDependentLogic.java:133)
> at org.infinispan.interceptors.locking.ClusteringDependentLogic$AbstractClusteringDependentLogic.createNewVersionsAndCheckForWriteSkews(ClusteringDependentLogic.java:93)
> at org.infinispan.interceptors.totalorder.TotalOrderVersionedEntryWrappingInterceptor.visitPrepareCommand(TotalOrderVersionedEntryWrappingInterceptor.java:62)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.interceptors.NotificationInterceptor.visitPrepareCommand(NotificationInterceptor.java:36)
> at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.interceptors.TxInterceptor.invokeNextInterceptorAndVerifyTransaction(TxInterceptor.java:124)
> at org.infinispan.interceptors.TxInterceptor.visitPrepareCommand(TxInterceptor.java:111)
> at org.infinispan.interceptors.TxInterceptor.visitCommitCommand(TxInterceptor.java:184)
> at org.infinispan.commands.tx.CommitCommand.acceptVisitor(CommitCommand.java:32)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.interceptors.totalorder.TotalOrderInterceptor.visitSecondPhaseCommand(TotalOrderInterceptor.java:148)
> at org.infinispan.interceptors.totalorder.TotalOrderInterceptor.visitCommitCommand(TotalOrderInterceptor.java:125)
> {noformat}
> (The error message is misleading, this is a commit and not a rollback.)
> The 1st tx still fails with a WriteSkewException, but then the test fails because the 2nd tx didn't update the value:
> {noformat}
> 04:27:12,286 ERROR (testng-ReplTotalOrderVersionedStateTransferTest:) [UnitTestTestNGListener] Test testStateTransfer(org.infinispan.tx.totalorder.statetransfer.ReplTotalOrderVersionedStateTransferTest) failed.
> java.lang.AssertionError: expected:<new world> but was:<world>
> at org.testng.AssertJUnit.fail(AssertJUnit.java:59)
> at org.testng.AssertJUnit.failNotEquals(AssertJUnit.java:364)
> at org.testng.AssertJUnit.assertEquals(AssertJUnit.java:80)
> at org.testng.AssertJUnit.assertEquals(AssertJUnit.java:88)
> at org.infinispan.container.versioning.VersionedReplStateTransferTest.testStateTransfer(VersionedReplStateTransferTest.java:89)
> {noformat}
> Full log here: http://ci.infinispan.org/viewLog.html?buildId=9816&buildTypeId=Infinispan...
--
This message was sent by Atlassian JIRA
(v6.2.6#6264)
9 years, 9 months
[JBoss JIRA] (ISPN-4574) PartitionHandling: consider less than numOwners partitions
by Dan Berindei (JIRA)
[ https://issues.jboss.org/browse/ISPN-4574?page=com.atlassian.jira.plugin.... ]
Dan Berindei updated ISPN-4574:
-------------------------------
Priority: Critical (was: Major)
> PartitionHandling: consider less than numOwners partitions
> ----------------------------------------------------------
>
> Key: ISPN-4574
> URL: https://issues.jboss.org/browse/ISPN-4574
> Project: Infinispan
> Issue Type: Bug
> Security Level: Public(Everyone can see)
> Reporter: Mircea Markus
> Assignee: Mircea Markus
> Priority: Critical
>
> If the following two conditions are simultaneously met during a split brain:
> a) less than numOwners nodes leave
> b) two clusters are formed having that size
> Then both partitions would be considered as available. E.g. a cluster of 6 nodes with numOwners=4, with a split brain resulting in 3 and 3 nodes partitions.
> We should add some logic to prevent this situation, e.g. by requiring the remaining partition to have at least numOwners members.
--
This message was sent by Atlassian JIRA
(v6.2.6#6264)
9 years, 9 months
[JBoss JIRA] (ISPN-4574) PartitionHandling: consider less than numOwners partitions
by Dan Berindei (JIRA)
[ https://issues.jboss.org/browse/ISPN-4574?page=com.atlassian.jira.plugin.... ]
Dan Berindei reassigned ISPN-4574:
----------------------------------
Assignee: Dan Berindei (was: Mircea Markus)
> PartitionHandling: consider less than numOwners partitions
> ----------------------------------------------------------
>
> Key: ISPN-4574
> URL: https://issues.jboss.org/browse/ISPN-4574
> Project: Infinispan
> Issue Type: Bug
> Security Level: Public(Everyone can see)
> Reporter: Mircea Markus
> Assignee: Dan Berindei
> Priority: Critical
>
> If the following two conditions are simultaneously met during a split brain:
> a) less than numOwners nodes leave
> b) two clusters are formed having that size
> Then both partitions would be considered as available. E.g. a cluster of 6 nodes with numOwners=4, with a split brain resulting in 3 and 3 nodes partitions.
> We should add some logic to prevent this situation, e.g. by requiring the remaining partition to have at least numOwners members.
--
This message was sent by Atlassian JIRA
(v6.2.6#6264)
9 years, 9 months
[JBoss JIRA] (ISPN-4574) PartitionHandling: consider less than numOwners partitions
by Dan Berindei (JIRA)
[ https://issues.jboss.org/browse/ISPN-4574?page=com.atlassian.jira.plugin.... ]
Dan Berindei commented on ISPN-4574:
------------------------------------
I've increased the priority, since this scenario would be pretty common with 2 servers and numOwners = 2.
> PartitionHandling: consider less than numOwners partitions
> ----------------------------------------------------------
>
> Key: ISPN-4574
> URL: https://issues.jboss.org/browse/ISPN-4574
> Project: Infinispan
> Issue Type: Bug
> Security Level: Public(Everyone can see)
> Reporter: Mircea Markus
> Assignee: Dan Berindei
> Priority: Critical
>
> If the following two conditions are simultaneously met during a split brain:
> a) less than numOwners nodes leave
> b) two clusters are formed having that size
> Then both partitions would be considered as available. E.g. a cluster of 6 nodes with numOwners=4, with a split brain resulting in 3 and 3 nodes partitions.
> We should add some logic to prevent this situation, e.g. by requiring the remaining partition to have at least numOwners members.
--
This message was sent by Atlassian JIRA
(v6.2.6#6264)
9 years, 9 months