[JBoss JIRA] (ISPN-3737) L1 requestor registered after value read
by Radim Vansa (JIRA)
[ https://issues.jboss.org/browse/ISPN-3737?page=com.atlassian.jira.plugin.... ]
Radim Vansa commented on ISPN-3737:
-----------------------------------
I have experienced that with RC. But you can check out as soon as you get the unit test reproducing this with RR.
> L1 requestor registered after value read
> ----------------------------------------
>
> Key: ISPN-3737
> URL: https://issues.jboss.org/browse/ISPN-3737
> Project: Infinispan
> Issue Type: Bug
> Components: Distributed Cache
> Affects Versions: 6.0.0.Final
> Reporter: Radim Vansa
> Assignee: William Burns
> Priority: Critical
> Labels: 620
>
> As the L1 requestor is registered only after the value is retrieved from data container, the (transactional) update of the value may not invalide the entry after write and the cache gets inconsistent.
> Consider this interleaving of operations (G=get request from other node, C=commit)
> R: read value -> old value
> C: update old -> new
> C: notify requestors for key
> R: add requestor for key
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
10 years, 5 months
[JBoss JIRA] (ISPN-3704) StateTransfer's PutKeyValueCommand may trigger remote get
by Radim Vansa (JIRA)
[ https://issues.jboss.org/browse/ISPN-3704?page=com.atlassian.jira.plugin.... ]
Radim Vansa commented on ISPN-3704:
-----------------------------------
Not directly, but the un-needed work executes RPC, which is intrinsically fragile. And the RPC may eventually cause transaction failure which leads to inconsistency (entry loss) - that's why I've increased the priority.
> StateTransfer's PutKeyValueCommand may trigger remote get
> ---------------------------------------------------------
>
> Key: ISPN-3704
> URL: https://issues.jboss.org/browse/ISPN-3704
> Project: Infinispan
> Issue Type: Bug
> Components: Distributed Cache
> Affects Versions: 6.0.0.CR1
> Reporter: Radim Vansa
> Assignee: Mircea Markus
> Priority: Critical
>
> In TX mode with write skew check on, ST executing puts may trigger a remote get.
> The condition in TxDistributionInterceptor.handleTxWriteCommand should probably be switched from
> {code}
> if (ctx.isOriginLocal() && !skipRemoteGet || command.isConditional() || shouldFetchRemoteValuesForWriteSkewCheck(ctx, command))
> remoteGetBeforeWrite(ctx, command, recipientGenerator);
> {code}
> to
> {code}
> if (!skipRemoteGet && (ctx.isOriginLocal() || command.isConditional() || shouldFetchRemoteValuesForWriteSkewCheck(ctx, command)))
> remoteGetBeforeWrite(ctx, command, recipientGenerator);
> {code}
> EDIT:
> I have also registered a situation where the Prepare/Commit command was executed remotely from within the ST because the topology has changed during the remote get. This should be avoided as well.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
10 years, 5 months
[JBoss JIRA] (ISPN-3432) Data put to index enabled cache with Infinispan Directory provider using Async. JDBC StringBased CacheStore fails
by Adrian Nistor (JIRA)
[ https://issues.jboss.org/browse/ISPN-3432?page=com.atlassian.jira.plugin.... ]
Adrian Nistor commented on ISPN-3432:
-------------------------------------
There are two problems here:
#1 I re-analyzed the bug and it seems the myriad of "org.infinispan.util.concurrent.TimeoutException: Unable to acquire lock after..." were fixed some time ago by ISPN-2815. This can be confirmed by reverting the commit for ISPN-2815 from master; then we get lots of lock acquire exceptions. When running on latest master they no longer appear.
#2 The attached config contains a major problem with the jdbc config. Each of the 4 nodes has 3 cachestores that all write in the _same_ h2 database. This leads to lots of:
{noformat}
2013-12-02 10:01:58,094 ERROR [JdbcStringBasedStore] (AsyncStoreProcessor-LuceneIndexesData_custom-6) ISPN008024: Error while storing string key to database; key: 'segments_3|0|16384|person'
org.h2.jdbc.JdbcSQLException: Unique index or primary key violation: "PRIMARY_KEY_7 ON PUBLIC.""ISPN_STRING_TABLE_LuceneIndexesData_custom""(ID_COLUMN)"; SQL statement:
INSERT INTO "ISPN_STRING_TABLE_LuceneIndexesData_custom" (DATA_COLUMN, TIMESTAMP_COLUMN, ID_COLUMN) VALUES(?,?,?) [23505-166]
at org.h2.message.DbException.getJdbcSQLException(DbException.java:329)
at org.h2.message.DbException.get(DbException.java:169)
at org.h2.message.DbException.get(DbException.java:146)
at org.h2.index.BaseIndex.getDuplicateKeyException(BaseIndex.java:81)
at org.h2.index.TreeIndex.add(TreeIndex.java:62)
at org.h2.table.RegularTable.addRow(RegularTable.java:121)
at org.h2.command.dml.Insert.insertRows(Insert.java:124)
at org.h2.command.dml.Insert.update(Insert.java:84)
at org.h2.command.CommandContainer.update(CommandContainer.java:73)
at org.h2.command.Command.executeUpdate(Command.java:226)
at org.h2.jdbc.JdbcPreparedStatement.executeUpdateInternal(JdbcPreparedStatement.java:143)
at org.h2.jdbc.JdbcPreparedStatement.executeUpdate(JdbcPreparedStatement.java:129)
at com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.executeUpdate(NewProxyPreparedStatement.java:105)
at org.infinispan.persistence.jdbc.stringbased.JdbcStringBasedStore.write(JdbcStringBasedStore.java:166)
at org.infinispan.persistence.async.AsyncCacheWriter.applyModificationsSync(AsyncCacheWriter.java:154)
at org.infinispan.persistence.async.AsyncCacheWriter$AsyncStoreProcessor.retryWork(AsyncCacheWriter.java:329)
at org.infinispan.persistence.async.AsyncCacheWriter$AsyncStoreProcessor.run(AsyncCacheWriter.java:313)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
{noformat}
and
{noformat}
2013-12-02 10:02:01,890 ERROR [JdbcStringBasedStore] (persistence-thread-0,NodeJ) ISPN008007: SQL error while fetching all StoredEntries
org.h2.jdbc.JdbcSQLException: Table "ISPN_STRING_TABLE_LuceneIndexesMetadata_custom" not found; SQL statement:
SELECT DATA_COLUMN,ID_COLUMN, TIMESTAMP_COLUMN FROM "ISPN_STRING_TABLE_LuceneIndexesMetadata_custom" WHERE TIMESTAMP_COLUMN > ? OR TIMESTAMP_COLUMN < 0 [42102-166]
at org.h2.message.DbException.getJdbcSQLException(DbException.java:329)
at org.h2.message.DbException.get(DbException.java:169)
at org.h2.message.DbException.get(DbException.java:146)
at org.h2.command.Parser.readTableOrView(Parser.java:4757)
at org.h2.command.Parser.readTableFilter(Parser.java:1084)
at org.h2.command.Parser.parseSelectSimpleFromPart(Parser.java:1690)
at org.h2.command.Parser.parseSelectSimple(Parser.java:1797)
at org.h2.command.Parser.parseSelectSub(Parser.java:1684)
at org.h2.command.Parser.parseSelectUnion(Parser.java:1527)
at org.h2.command.Parser.parseSelect(Parser.java:1515)
at org.h2.command.Parser.parsePrepared(Parser.java:405)
at org.h2.command.Parser.parse(Parser.java:279)
at org.h2.command.Parser.parse(Parser.java:251)
at org.h2.command.Parser.prepareCommand(Parser.java:217)
at org.h2.engine.Session.prepareLocal(Session.java:415)
at org.h2.engine.Session.prepareCommand(Session.java:364)
at org.h2.jdbc.JdbcConnection.prepareCommand(JdbcConnection.java:1111)
at org.h2.jdbc.JdbcPreparedStatement.<init>(JdbcPreparedStatement.java:71)
at org.h2.jdbc.JdbcConnection.prepareStatement(JdbcConnection.java:266)
at com.mchange.v2.c3p0.impl.NewProxyConnection.prepareStatement(NewProxyConnection.java:213)
at org.infinispan.persistence.jdbc.stringbased.JdbcStringBasedStore$2.call(JdbcStringBasedStore.java:321)
at org.infinispan.persistence.jdbc.stringbased.JdbcStringBasedStore$2.call(JdbcStringBasedStore.java:309)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
...
{noformat}
The 4 nodes should not share the database in this case. By giving each node's database a unique name (instead of all being connectionUrl="jdbc:h2:mem:infinispan_string_based") the above jdbc errors disappear.
Since Anna supplied a config file by node java code, I've tested this using a variation of ClusteredCacheWithAsyncDirTest and then DistributedMassIndexingTest with the supplied config.
Please note these tests do not use multiple threads to cause the errors. Just one thread is enough to trigger the exceptions I mentioned.
I can no longer access the log file mentioned: https://jenkins.mw.lab.eng.bos.redhat.com/hudson/view/JDG/view/PERF-LIB/j...
I think this issue can be closed as a consequence of ISPN-2815, but before doing so it would be good to run the radargun "performance tests for Index enabled Infinispan cache" Anna mentioned just to make sure the no problems surface under bigger load.
> Data put to index enabled cache with Infinispan Directory provider using Async. JDBC StringBased CacheStore fails
> -----------------------------------------------------------------------------------------------------------------
>
> Key: ISPN-3432
> URL: https://issues.jboss.org/browse/ISPN-3432
> Project: Infinispan
> Issue Type: Bug
> Components: Querying
> Affects Versions: 6.0.0.Alpha1
> Reporter: Anna Manukyan
> Assignee: Adrian Nistor
> Priority: Critical
> Labels: 620
> Fix For: 6.1.0.Final
>
> Attachments: async-config.xml
>
>
> Hi,
> this issue is related to the ISPN-3090, but I thought to specify this case separately for bringing detailed explanation for the configuration and thrown exceptions.
> The issue relates to the performance tests for Index enabled Infinispan cache, with configured Infinispan directory and Async JDBC. String Based Cache store.
> The tests are running on 4 nodes and performing puts/gets on all nodes with many threads.
> The problem is that, during data put, the following exceptions are thrown continuously:
> {code}
> 04:04:05,633 ERROR [org.hibernate.search.exception.impl.LogErrorHandler] (Hibernate Search: Index updates queue processor for index query-1) HSEARCH000058: Exception occurred org.apache.lucene.index.IndexNotFoundException: no segments* file found in InfinispanDirectory{indexName='query'}: files: []
> Primary Failure:
> Entity org.radargun.cachewrappers.InfinispanQueryWrapper$QueryableData Id S:_InstallBenchmarkStage_0 Work Type org.hibernate.search.backend.UpdateLuceneWork
> org.apache.lucene.index.IndexNotFoundException: no segments* file found in InfinispanDirectory{indexName='query'}: files: []
> at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:667)
> at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:554)
> at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:359)
> at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:1138)
> at org.hibernate.search.backend.impl.lucene.IndexWriterHolder.createNewIndexWriter(IndexWriterHolder.java:148)
> at org.hibernate.search.backend.impl.lucene.IndexWriterHolder.getIndexWriter(IndexWriterHolder.java:115)
> at org.hibernate.search.backend.impl.lucene.AbstractWorkspaceImpl.getIndexWriter(AbstractWorkspaceImpl.java:117)
> at org.hibernate.search.backend.impl.lucene.LuceneBackendQueueTask.applyUpdates(LuceneBackendQueueTask.java:101)
> at org.hibernate.search.backend.impl.lucene.LuceneBackendQueueTask.run(LuceneBackendQueueTask.java:67)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:724)
> ......
> 04:14:21,605 ERROR [org.hibernate.search.exception.impl.LogErrorHandler] (Hibernate Search: Index updates queue processor for index query-1) HSEARCH000058: Exception occurred org.apache.lucene.index.IndexNotFoundException: no segments* file found in InfinispanDirectory{indexName='query'}: files: []
> Primary Failure:
> Entity org.radargun.cachewrappers.InfinispanQueryWrapper$QueryableData Id S:key_0_0_0000000000000017 Work Type org.hibernate.search.backend.UpdateLuceneWork
> org.apache.lucene.index.IndexNotFoundException: no segments* file found in InfinispanDirectory{indexName='query'}: files: []
> at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:667)
> at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:554)
> at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:359)
> at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:1138)
> at org.hibernate.search.backend.impl.lucene.IndexWriterHolder.createNewIndexWriter(IndexWriterHolder.java:148)
> at org.hibernate.search.backend.impl.lucene.IndexWriterHolder.getIndexWriter(IndexWriterHolder.java:115)
> at org.hibernate.search.backend.impl.lucene.AbstractWorkspaceImpl.getIndexWriter(AbstractWorkspaceImpl.java:117)
> at org.hibernate.search.backend.impl.lucene.LuceneBackendQueueTask.applyUpdates(LuceneBackendQueueTask.java:101)
> at org.hibernate.search.backend.impl.lucene.LuceneBackendQueueTask.run(LuceneBackendQueueTask.java:67)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:724)
> {code}
> You can find the cache configuration attached.
> Yet another thing to mention:
> if the following line is added to the cache configuration:
> {code}
> <property name="default.indexmanager" value="org.infinispan.query.indexmanager.InfinispanIndexManager" />
> {code}
> then the issue is gone - no lock issue appears then.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
10 years, 5 months
[JBoss JIRA] (ISPN-3130) Cancelling an InboundTransferTask should be idempotent
by RH Bugzilla Integration (JIRA)
[ https://issues.jboss.org/browse/ISPN-3130?page=com.atlassian.jira.plugin.... ]
RH Bugzilla Integration updated ISPN-3130:
------------------------------------------
Bugzilla Update: Perform
Bugzilla References: https://bugzilla.redhat.com/show_bug.cgi?id=1036447
> Cancelling an InboundTransferTask should be idempotent
> ------------------------------------------------------
>
> Key: ISPN-3130
> URL: https://issues.jboss.org/browse/ISPN-3130
> Project: Infinispan
> Issue Type: Bug
> Components: State transfer
> Affects Versions: 5.2.6.Final
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Fix For: 5.2.7.Final, 5.3.0.CR1, 5.3.0.Final
>
>
> When installing a new topology, it's possible for the {{StateConsumerImpl.cancelSegments}} method to fail with an {{IllegalArgumentException}}:
> {code}
> 05:43:56,513 WARN [org.infinispan.topology.CacheTopologyControlCommand] (notification-thread-0) ISPN000071: Caught exception when handling command CacheTopologyControlCommand{cache=default-host/clusterbench, type=CH_UPDATE, sender=perf21/web, joinInfo=null, topologyId=23, currentCH=DefaultConsistentHash{numSegments=80, numOwners=2, members=[perf21/web, perf18/web, perf19/web]}, pendingCH=null, throwable=null, viewId=8}: java.lang.IllegalArgumentException: The task is already cancelled.
> at org.infinispan.statetransfer.InboundTransferTask.cancelSegments(InboundTransferTask.java:167)
> at org.infinispan.statetransfer.StateConsumerImpl.cancelTransfers(StateConsumerImpl.java:774)
> at org.infinispan.statetransfer.StateConsumerImpl.onTopologyUpdate(StateConsumerImpl.java:314)
> at org.infinispan.statetransfer.StateTransferManagerImpl.doTopologyUpdate(StateTransferManagerImpl.java:195)
> at org.infinispan.statetransfer.StateTransferManagerImpl.access$000(StateTransferManagerImpl.java:61)
> at org.infinispan.statetransfer.StateTransferManagerImpl$1.updateConsistentHash(StateTransferManagerImpl.java:121)
> at org.infinispan.topology.LocalTopologyManagerImpl.handleConsistentHashUpdate(LocalTopologyManagerImpl.java:202)
> at org.infinispan.topology.CacheTopologyControlCommand.doPerform(CacheTopologyControlCommand.java:165)
> at org.infinispan.topology.CacheTopologyControlCommand.perform(CacheTopologyControlCommand.java:137)
> at org.infinispan.topology.ClusterTopologyManagerImpl.executeOnClusterSync(ClusterTopologyManagerImpl.java:540)
> at org.infinispan.topology.ClusterTopologyManagerImpl.broadcastConsistentHashUpdate(ClusterTopologyManagerImpl.java:332)
> at org.infinispan.topology.ClusterTopologyManagerImpl.updateCacheStatusAfterMerge(ClusterTopologyManagerImpl.java:319)
> at org.infinispan.topology.ClusterTopologyManagerImpl.handleNewView(ClusterTopologyManagerImpl.java:236)
> at org.infinispan.topology.ClusterTopologyManagerImpl$ClusterViewListener.handleViewChange(ClusterTopologyManagerImpl.java:579)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.6.0_43]
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) [rt.jar:1.6.0_43]
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) [rt.jar:1.6.0_43]
> at java.lang.reflect.Method.invoke(Method.java:597) [rt.jar:1.6.0_43]
> at org.infinispan.notifications.AbstractListenerImpl$ListenerInvocation$1.run(AbstractListenerImpl.java:212)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) [rt.jar:1.6.0_43]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) [rt.jar:1.6.0_43]
> at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_43]
> {code}
> Instead, it should just log a message and ignore the cancel request.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
10 years, 5 months