[JBoss JIRA] (ISPN-3842) Inconsistent L1 in non-tx distributed cache in certain circumstances
by William Burns (JIRA)
[ https://issues.jboss.org/browse/ISPN-3842?page=com.atlassian.jira.plugin.... ]
William Burns updated ISPN-3842:
--------------------------------
Fix Version/s: 7.0.0.Final
> Inconsistent L1 in non-tx distributed cache in certain circumstances
> --------------------------------------------------------------------
>
> Key: ISPN-3842
> URL: https://issues.jboss.org/browse/ISPN-3842
> Project: Infinispan
> Issue Type: Bug
> Affects Versions: 6.0.0.Final
> Reporter: Mikolaj Gierulski
> Assignee: William Burns
> Fix For: 7.0.0.Final
>
>
> In my poc environment there are two nodes in dist non-tx sync cluster with L1 enabled and numOwners=1.
> Node A, in a loop, reads one key (K), which is stored on node B (in test case it performs about 1 000 000 reads per second).
> From time to time K is updated on node B. This causes an L1 invalidation message sent to A, and K is fetched from B upon next read attempt.
> But whenever I run my test, I come to a situation, where updates of K no longer invalidate it on A, and A sees old value of K.
> When this happens, I can see in logs of node A:
> {noformat}
> 18:21:33.296 [remote-thread-0] TRACE o.i.i.d.L1NonTxInterceptor - L1 invalidation found a pending update for key K - need to block until finished
> 18:21:33.296 [remote-thread-0] TRACE o.i.i.d.L1NonTxInterceptor - Pending L1 update completed successfully: true - L1 invalidation can occur for key K
> 18:21:33.296 [remote-thread-0] TRACE o.i.c.write.InvalidateL1Command - Preparing to invalidate keys [K]
> 18:21:33.296 [remote-thread-0] TRACE o.i.c.write.InvalidateL1Command - Invalidating key K.
> 18:21:33.296 [remote-thread-0] TRACE o.i.commands.write.RemoveCommand - Nothing to remove since the entry is null or we have a null entry
> {noformat}
> While logs of node B show:
> {noformat}
> 18:21:33.200 [OOB-1,AGST-2012000591-33853] TRACE o.i.distribution.L1ManagerImpl - Registering requestor AGST-2012000591-25400 for key 'K'
> 18:21:33.266 [remote-thread-1] TRACE o.i.distribution.L1ManagerImpl - Invalidating keys [K] on nodes [AGST-2012000591-25400]. Use multicast? false
> 18:21:33.269 [remote-thread-1] TRACE o.i.i.d.L1NonTxInterceptor - Allowing entry to commit as local node is owner
> 18:21:33.269 [OOB-2,AGST-2012000591-33853] TRACE o.i.distribution.L1ManagerImpl - Registering requestor AGST-2012000591-25400 for key 'K'
> 18:21:33.269 [remote-thread-1] TRACE o.i.i.d.L1NonTxInterceptor - Sending additional invalidation for requestors if necessary.
> 18:21:33.269 [remote-thread-1] TRACE o.i.distribution.L1ManagerImpl - Invalidating keys [K] on nodes [AGST-2012000591-25400]. Use multicast? false
> 18:21:33.270 [remote-thread-1] INFO p.c.a.ispn.WriteTask - Update task runtime millis 3
> 18:21:33.271 [OOB-1,AGST-2012000591-33853] TRACE o.i.distribution.L1ManagerImpl - Registering requestor AGST-2012000591-25400 for key 'K'
> 18:21:33.293 [remote-thread-1] TRACE o.i.distribution.L1ManagerImpl - Invalidating keys [K] on nodes [AGST-2012000591-25400]. Use multicast? false
> 18:21:33.295 [remote-thread-1] TRACE o.i.i.d.L1NonTxInterceptor - Allowing entry to commit as local node is owner
> 18:21:33.295 [OOB-2,AGST-2012000591-33853] TRACE o.i.distribution.L1ManagerImpl - Registering requestor AGST-2012000591-25400 for key 'K'
> 18:21:33.295 [remote-thread-1] TRACE o.i.i.d.L1NonTxInterceptor - Sending additional invalidation for requestors if necessary.
> 18:21:33.295 [remote-thread-1] TRACE o.i.distribution.L1ManagerImpl - Invalidating keys [K] on nodes [AGST-2012000591-25400]. Use multicast? false
> 18:21:33.295 [remote-thread-1] INFO p.c.a.ispn.WriteTask - Update task runtime millis 2
> 18:21:33.476 [remote-thread-1] TRACE o.i.distribution.L1ManagerImpl - No L1 caches to invalidate for keys [K]
> 18:21:33.476 [remote-thread-1] TRACE o.i.i.d.L1NonTxInterceptor - Allowing entry to commit as local node is owner
> 18:21:33.476 [remote-thread-1] TRACE o.i.i.d.L1NonTxInterceptor - Sending additional invalidation for requestors if necessary.
> 18:21:33.476 [remote-thread-1] TRACE o.i.distribution.L1ManagerImpl - No L1 caches to invalidate for keys [K]
> {noformat}
> So it seems, that after this:
> bq. 'L1 invalidation found a pending update for key K - need to block until finished'
> B no longer knows A holds K in L1, and no longer sends invalidation commands after updates.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years
[JBoss JIRA] (ISPN-2034) Add backwards serialization compatibility tests
by Dan Berindei (JIRA)
[ https://issues.jboss.org/browse/ISPN-2034?page=com.atlassian.jira.plugin.... ]
Dan Berindei updated ISPN-2034:
-------------------------------
Fix Version/s: 7.0.0.Alpha1
(was: 6.0.1.Final)
> Add backwards serialization compatibility tests
> -----------------------------------------------
>
> Key: ISPN-2034
> URL: https://issues.jboss.org/browse/ISPN-2034
> Project: Infinispan
> Issue Type: Enhancement
> Components: Marshalling
> Reporter: Galder Zamarreño
> Assignee: Galder Zamarreño
> Labels: compatibility
> Fix For: 7.0.0.Alpha1
>
>
> Take each existing minor branch, i.e. 4.1.x, 4.0.x,...etc and modify VersionAwareMarshallerTest so that payloads are stored in a data file.
> Then, take data files for each branch and store in git (master and 5.1.x), and try reading them with the existing externalizer/marshalling set up.
> It should work...
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years
[JBoss JIRA] (ISPN-2958) Lucene Directory Read past EOF
by Dan Berindei (JIRA)
[ https://issues.jboss.org/browse/ISPN-2958?page=com.atlassian.jira.plugin.... ]
Dan Berindei updated ISPN-2958:
-------------------------------
Fix Version/s: 7.0.0.Alpha1
(was: 6.0.1.Final)
> Lucene Directory Read past EOF
> ------------------------------
>
> Key: ISPN-2958
> URL: https://issues.jboss.org/browse/ISPN-2958
> Project: Infinispan
> Issue Type: Bug
> Components: Lucene Directory
> Affects Versions: 5.2.1.Final
> Reporter: Clement Pang
> Assignee: Pedro Ruivo
> Labels: retest, stable_embedded_query
> Fix For: 7.0.0.Alpha1
>
>
> This seems to be happening rather deterministically.
> Infinispan configuration (in JBoss EAP 6.1.0.Alpha):
> {code}
> <cache-container name="lucene">
> <local-cache name="dshell-index-data" start="EAGER">
> <eviction strategy="LIRS" max-entries="50000"/>
> <file-store path="lucene" passivation="true" purge="false"/>
> </local-cache>
> <local-cache name="dshell-index-metadata" start="EAGER">
> <file-store path="lucene" passivation="true" purge="false"/>
> </local-cache>
> <local-cache name="dshell-index-lock" start="EAGER">
> <file-store path="lucene" passivation="true" purge="false"/>
> </local-cache>
> </cache-container>
> {code}
> Upon shutting down the server and confirming that passivation did indeed write the data to disk, the subsequent start-up would fail right away with:
> {code}
> Caused by: org.hibernate.search.SearchException: Could not initialize index
> at org.hibernate.search.store.impl.DirectoryProviderHelper.initializeIndexIfNeeded(DirectoryProviderHelper.java:162)
> at org.hibernate.search.infinispan.impl.InfinispanDirectoryProvider.start(InfinispanDirectoryProvider.java:103)
> at org.hibernate.search.indexes.impl.DirectoryBasedIndexManager.initialize(DirectoryBasedIndexManager.java:104)
> at org.hibernate.search.indexes.impl.IndexManagerHolder.createIndexManager(IndexManagerHolder.java:227)
> ... 64 more
> Caused by: java.io.IOException: Read past EOF
> at org.infinispan.lucene.SingleChunkIndexInput.readByte(SingleChunkIndexInput.java:77)
> at org.apache.lucene.store.ChecksumIndexInput.readByte(ChecksumIndexInput.java:41)
> at org.apache.lucene.store.DataInput.readInt(DataInput.java:86)
> at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:272)
> at org.apache.lucene.index.IndexFileDeleter.<init>(IndexFileDeleter.java:182)
> at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:1168)
> at org.hibernate.search.store.impl.DirectoryProviderHelper.initializeIndexIfNeeded(DirectoryProviderHelper.java:157)
> ... 67 more
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years
[JBoss JIRA] (ISPN-2956) putIfAbsent on Hot Rod Java client doesn't reliably fulfil contract
by Dan Berindei (JIRA)
[ https://issues.jboss.org/browse/ISPN-2956?page=com.atlassian.jira.plugin.... ]
Dan Berindei updated ISPN-2956:
-------------------------------
Fix Version/s: 7.0.0.Alpha1
(was: 6.0.1.Final)
> putIfAbsent on Hot Rod Java client doesn't reliably fulfil contract
> -------------------------------------------------------------------
>
> Key: ISPN-2956
> URL: https://issues.jboss.org/browse/ISPN-2956
> Project: Infinispan
> Issue Type: Bug
> Reporter: Galder Zamarreño
> Assignee: Galder Zamarreño
> Priority: Minor
> Labels: hotrod-java-client, remote-clients
> Fix For: 7.0.0.Alpha1
>
>
> Hot Rod's putIfAbsent might have issues on some edge cases:
> {quote}I want to know whether the putting entry already exists in the remote
> cache cluster, or not.
> I thought that RemoteCache.putIfAbsent() would be useful for that
> purpose, i.e.,
> {code}
> if (remoteCache.putIfAbsent(k,v) == null) {
> // new entry.
> } else {
> // k already exists.
> }
> {code}
> But no.
> The putIfAbsent() for new entry may return non-null value, if one of the
> server crushed while putting.
> The behavior is like the following:
> 1. Client do putIfAbsent(k,v).
> 2. The server receives the request and sends replication requests to
> other servers. If the server crushed before completing replication, some
> servers own that (k,v), but others not.
> 3. Client receives the error. The putIfAbsent() internally retries the
> same request to the next server in the cluster server list.
> 4. If the next server owns the (k,v), the putIfAbsent() returns the
> replicated (k,v) at the step 2, without any error.
> So, putIfAbsent() is not reliable for knowing whether the putting entry
> is *exactly* new or not.
> Does anyone have any idea/workaround for this purpose?{quote}
> A workaround is to do this:
> {quote}We got a simple solution, which can be applied to our customer's application.
> If each value part of putting (k,v) is unique or contains unique value,
> the client can do *double check* wether the entry is new.
> {code}
> val = System.nanoTime(); // or uuid is also useful.
> if ((ret = cache.putIfAbsent(key, val)) == null
> || ret.equals(val)) {
> // new entry, if the return value is just the same.
> } else {
> // key already exists.
> }
> {code}
> We are proposing this workaround which almost works fine.{quote}
> However, this is a bit of a cludge.
> Hot Rod should be improved with an operation that allows a version to be passed when entry is created, instead of relying on the client generating it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years