[JBoss JIRA] (ISPN-8719) KeySet.(iterator|spliterator|stream) not compatible with versions before 9.1
by Rostislav Svoboda (JIRA)
[ https://issues.jboss.org/browse/ISPN-8719?page=com.atlassian.jira.plugin.... ]
Rostislav Svoboda commented on ISPN-8719:
-----------------------------------------
This always depends on the point of view ... from technical perspective, from product feature/compatibility perspective, from LP perspective, etc. Anyway, thank you for looking into this issue!
> KeySet.(iterator|spliterator|stream) not compatible with versions before 9.1
> ----------------------------------------------------------------------------
>
> Key: ISPN-8719
> URL: https://issues.jboss.org/browse/ISPN-8719
> Project: Infinispan
> Issue Type: Bug
> Affects Versions: 9.1.0.Final, 9.2.0.Final
> Reporter: Marek Posolda
> Assignee: William Burns
> Priority: Critical
> Fix For: 9.1.8.Final, 9.2.2.Final, 9.3.0.Alpha1
>
> Attachments: InfinispanRemote.java
>
>
> Steps to reproduce:
> 1) Use infinispan server version 8.2.6 (or JDG server 7.1.0) and start it.
> {code}
> cd JDG_HOME/bin
> ./standalone.sh
> {code}
> 2) Create sample java project having dependency on latest dependency 9.2.0.CR1 in pom.xml:
> {code}
> <dependencies>
> <dependency>
> <groupId>org.infinispan</groupId>
> <artifactId>infinispan-core</artifactId>
> <version>9.2.0.CR1</version>
> </dependency>
> <dependency>
> <groupId>org.infinispan</groupId>
> <artifactId>infinispan-cachestore-remote</artifactId>
> <version>9.2.0.CR1</version>
> </dependency>
> </dependencies>
> {code}
> 3) Add one simple java class based on the tutorial: http://infinispan.org/tutorials/simple/remote/ . The only difference is that I use hotRod protocolVersion 2.5 and calling:
> {code}
> remoteCache.keySet().iterator().hasNext()
> {code}. I am attaching the class in attachement.
> 4) Run the class. Seeing exception in both server log and on client-side.
> Server exception
> {code}
> 10:44:20,365 ERROR [org.infinispan.server.hotrod.CacheDecodeContext] (HotRodServerWorker-6-1) ISPN005003: Exception reported: java.lang.IllegalStateException: ISPN006016: Factory 'org.infinispan.server.hotrod.HotRodServer$ToEmptyBytesKeyValueFilterConverter' not found in server
> at org.infinispan.server.hotrod.iteration.DefaultIterationManager.getFactory(DefaultIterationManager.java:148)
> at org.infinispan.server.hotrod.iteration.DefaultIterationManager.start(DefaultIterationManager.java:131)
> at org.infinispan.server.hotrod.ContextHandler.realRead(ContextHandler.java:175)
> at org.infinispan.server.hotrod.ContextHandler.lambda$channelRead0$1(ContextHandler.java:57)
> at org.infinispan.server.hotrod.ContextHandler$$Lambda$86/1492247987.run(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> Client exception:
> {code}
> Jan 24, 2018 10:44:20 AM org.infinispan.client.hotrod.impl.protocol.Codec20 checkForErrorsInResponseStatus
> WARN: ISPN004005: Error received from the server: java.lang.IllegalStateException: ISPN006016: Factory 'org.infinispan.server.hotrod.HotRodServer$ToEmptyBytesKeyValueFilterConverter' not found in server
> Exception in thread "main" org.infinispan.client.hotrod.exceptions.HotRodClientException:Request for messageId=5 returned server error (status=0x85): java.lang.IllegalStateException: ISPN006016: Factory 'org.infinispan.server.hotrod.HotRodServer$ToEmptyBytesKeyValueFilterConverter' not found in server
> at org.infinispan.client.hotrod.impl.protocol.Codec20.checkForErrorsInResponseStatus(Codec20.java:408)
> at org.infinispan.client.hotrod.impl.protocol.Codec20.readPartialHeader(Codec20.java:162)
> at org.infinispan.client.hotrod.impl.protocol.Codec20.readHeader(Codec20.java:148)
> at org.infinispan.client.hotrod.impl.operations.HotRodOperation.readHeaderAndValidate(HotRodOperation.java:60)
> at org.infinispan.client.hotrod.impl.operations.IterationStartOperation.executeOperation(IterationStartOperation.java:72)
> at org.infinispan.client.hotrod.impl.operations.IterationStartOperation.executeOperation(IterationStartOperation.java:21)
> at org.infinispan.client.hotrod.impl.operations.RetryOnFailureOperation.execute(RetryOnFailureOperation.java:56)
> at org.infinispan.client.hotrod.impl.iteration.RemoteCloseableIterator.startInternal(RemoteCloseableIterator.java:127)
> at org.infinispan.client.hotrod.impl.iteration.RemoteCloseableIterator.start(RemoteCloseableIterator.java:140)
> at org.infinispan.client.hotrod.impl.RemoteCacheImpl.retrieveEntries(RemoteCacheImpl.java:162)
> at org.infinispan.client.hotrod.impl.RemoteCacheImpl.retrieveEntries(RemoteCacheImpl.java:168)
> at org.infinispan.client.hotrod.impl.RemoteCacheImpl.retrieveEntries(RemoteCacheImpl.java:173)
> at org.infinispan.client.hotrod.impl.RemoteCacheImpl$KeySet.iterator(RemoteCacheImpl.java:553)
> at org.mposolda.ispn.InfinispanRemote.main(InfinispanRemote.java:34)
> {code}
> Indeed, When looking at this line of class RemoteCacheImpl, I see that it references the class, which seem that it was added in HotRodServer version 9. This looks like the cause of the error: https://github.com/infinispan/infinispan/blob/master/client/hotrod-client...
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
6 years, 9 months
[JBoss JIRA] (ISPN-9014) Conflict resolution consistent hash should not include nodes that are not in the merged cluster view
by Ryan Emerson (JIRA)
[ https://issues.jboss.org/browse/ISPN-9014?page=com.atlassian.jira.plugin.... ]
Ryan Emerson updated ISPN-9014:
-------------------------------
Status: Pull Request Sent (was: Coding In Progress)
Git Pull Request: https://github.com/infinispan/infinispan/pull/5902
> Conflict resolution consistent hash should not include nodes that are not in the merged cluster view
> ----------------------------------------------------------------------------------------------------
>
> Key: ISPN-9014
> URL: https://issues.jboss.org/browse/ISPN-9014
> Project: Infinispan
> Issue Type: Bug
> Components: Test Suite - Core
> Affects Versions: 9.2.1.Final
> Reporter: Dan Berindei
> Assignee: Ryan Emerson
> Labels: testsuite_stability
> Fix For: 9.3.0.Alpha1
>
>
> Conflict resolution fails when trying to read entries from nodes that are not in the JGroups cluster view, and this causes random failures in {{ClusterListenerDistTest.testClusterListenerNodeGoesDown random}}.
> # NodeA leaves the cluster, but still manages to start a rebalance with [NodeB, NodeC] (topology id 11)
> # One node doesn't receive topology 11, so NodeB becomes coordinator and starts conflict resolution with all 3 nodes in the pending CH (topology 12)
> # Conflict resolution fails because NodeB and NodeC can't read the entries from NodeA
> # {{onPartitionMerge}} also queued a rebalance, so NodeB starts a new rebalance without canceling the previous rebalance first (topology 13)
> # Because there is no reset topology, NodeB thinks it already requested all the segments for NodeC's in topology 11, so it doesn't add any new inbound transfer
> # NodeC's state response arrives on NodeB with topology 11, NodeB discards it, and state transfer hangs.
> {noformat}
> 14:52:52,426 INFO (testng-Test:[cluster-listener]) [CLUSTER] ISPN000310: Starting cluster-wide rebalance for cache cluster-listener, topology CacheTopology{id=11, phase=READ_OLD_WRITE_ALL, rebalanceId=4, currentCH=DefaultConsistentHash{ns=256, owners = (2)[Test-NodeB-45145: 128+50, Test-NodeC-20831: 128+49]}, pendingCH=DefaultConsistentHash{ns=256, owners = (2)[Test-NodeB-45145: 131+125, Test-NodeC-20831: 125+131]}, unionCH=null, actualMembers=[Test-NodeB-45145, Test-NodeC-20831], persistentUUIDs=[301597c4-a4e4-46a6-8983-53e698ef70f7, ae95a681-2ba1-4e04-bfe5-05aa59425149]}
> 14:52:52,479 DEBUG (stateTransferExecutor-thread-Test-NodeB-p23774-t4:[Merge-3]) [ClusterCacheStatus] Recovered 2 partition(s) for cache cluster-listener: [CacheTopology{id=11, phase=READ_OLD_WRITE_ALL, rebalanceId=4, currentCH=DefaultConsistentHash{ns=256, owners = (2)[Test-NodeB-45145: 128+50, Test-NodeC-20831: 128+49]}, pendingCH=DefaultConsistentHash{ns=256, owners = (2)[Test-NodeB-45145: 131+125, Test-NodeC-20831: 125+131]}, unionCH=null, actualMembers=[Test-NodeB-45145, Test-NodeC-20831], persistentUUIDs=[301597c4-a4e4-46a6-8983-53e698ef70f7, ae95a681-2ba1-4e04-bfe5-05aa59425149]}, CacheTopology{id=9, phase=NO_REBALANCE, rebalanceId=3, currentCH=DefaultConsistentHash{ns=256, owners = (3)[Test-NodeA-57087: 78+79, Test-NodeB-45145: 90+88, Test-NodeC-20831: 88+89]}, pendingCH=null, unionCH=null, actualMembers=[Test-NodeA-57087, Test-NodeB-45145, Test-NodeC-20831], persistentUUIDs=[48e3ddc7-ee97-42d8-a57d-283e8d28ec25, 301597c4-a4e4-46a6-8983-53e698ef70f7, ae95a681-2ba1-4e04-bfe5-05aa59425149]}]
> 14:52:52,484 DEBUG (stateTransferExecutor-thread-Test-NodeB-p23774-t4:[Merge-3]) [ClusterTopologyManagerImpl] Updating cluster-wide current topology for cache cluster-listener, topology = CacheTopology{id=12, phase=CONFLICT_RESOLUTION, rebalanceId=5, currentCH=DefaultConsistentHash{ns=256, owners = (2)[Test-NodeB-45145: 128+50, Test-NodeC-20831: 128+49]}, pendingCH=DefaultConsistentHash{ns=256, owners = (3)[Test-NodeB-45145: 128+50, Test-NodeC-20831: 128+49, Test-NodeA-57087: 0+157]}, unionCH=DefaultConsistentHash{ns=256, owners = (3)[Test-NodeB-45145: 128+50, Test-NodeC-20831: 128+49, Test-NodeA-57087: 0+157]}, actualMembers=[Test-NodeB-45145, Test-NodeA-57087, Test-NodeC-20831], persistentUUIDs=[301597c4-a4e4-46a6-8983-53e698ef70f7, 48e3ddc7-ee97-42d8-a57d-283e8d28ec25, ae95a681-2ba1-4e04-bfe5-05aa59425149]}, availability mode = null
> 14:52:52,488 ERROR (stateTransferExecutor-thread-Test-NodeB-p23774-t4:[Merge-3]) [DefaultConflictManager] Cache cluster-listener encountered exception whilst trying to resolve conflicts on merge: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node Test-NodeA-57087 was suspected
> 14:52:52,532 INFO (stateTransferExecutor-thread-Test-NodeB-p23774-t4:[Merge-3]) [CLUSTER] ISPN000310: Starting cluster-wide rebalance for cache cluster-listener, topology CacheTopology{id=13, phase=READ_OLD_WRITE_ALL, rebalanceId=6, currentCH=DefaultConsistentHash{ns=256, owners = (2)[Test-NodeB-45145: 128+50, Test-NodeC-20831: 128+49]}, pendingCH=DefaultConsistentHash{ns=256, owners = (2)[Test-NodeB-45145: 131+125, Test-NodeC-20831: 125+131]}, unionCH=null, actualMembers=[Test-NodeB-45145, Test-NodeC-20831], persistentUUIDs=[301597c4-a4e4-46a6-8983-53e698ef70f7, ae95a681-2ba1-4e04-bfe5-05aa59425149]}
> 14:52:52,577 TRACE (stateTransferExecutor-thread-Test-NodeB-p23774-t3:[StateRequest-cluster-listener]) [StateConsumerImpl] Waiting for inbound transfer to finish: InboundTransferTask{segments={19-21 28-33 38-44 50-55 60-62 72 77-79 86-91 101 104-107 113 116-126 168-169 172 181-182 188-189 195-197 200-202 223-226 235 242 245 249-254}, finishedSegments={}, unfinishedSegments={19-21 28-33 38-44 50-55 60-62 72 77-79 86-91 101 104-107 113 116-126 168-169 172 181-182 188-189 195-197 200-202 223-226 235 242 245 249-254}, source=Test-NodeC-20831, isCancelled=false, completionFuture=java.util.concurrent.CompletableFuture@110952e8[Not completed], topologyId=11, timeout=240000, cacheName=cluster-listener}
> 14:52:52,584 DEBUG (remote-thread-Test-NodeB-p23771-t4:[cluster-listener]) [StateConsumerImpl] Discarding state response with old topology id 11 for cache cluster-listener, state transfer request topology was true
> 14:52:52,584 TRACE (remote-thread-Test-NodeB-p23771-t4:[]) [JGroupsTransport] Test-NodeB-45145 sending response for request 13 to Test-NodeC-20831: null
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
6 years, 9 months
[JBoss JIRA] (ISPN-9032) OperationsDuringMergeConflictTest.testPartitionMergePolicy random failures
by Ryan Emerson (JIRA)
[ https://issues.jboss.org/browse/ISPN-9032?page=com.atlassian.jira.plugin.... ]
Ryan Emerson updated ISPN-9032:
-------------------------------
Status: Pull Request Sent (was: Coding In Progress)
Git Pull Request: https://github.com/infinispan/infinispan/pull/5902
> OperationsDuringMergeConflictTest.testPartitionMergePolicy random failures
> --------------------------------------------------------------------------
>
> Key: ISPN-9032
> URL: https://issues.jboss.org/browse/ISPN-9032
> Project: Infinispan
> Issue Type: Bug
> Components: Test Suite - Core
> Affects Versions: 9.2.1.Final
> Reporter: Dan Berindei
> Assignee: Ryan Emerson
> Labels: testsuite_stability
> Fix For: 9.3.0.Alpha1
>
>
> The test modifies a key in both partitions and checks that each partition still sees its own value for a while after receiving the [merged cluster view|https://github.com/infinispan/infinispan/blob/ee87349f0b134b53b3090f...].
> It installs a {{BlockStateResponseCommandHandler}} to block the conflict resolution responses and to keep the pre-merge values in the non-preferred topology owners. However, it does not block the installation of the {{CONFLICT_RESOLUTION}} cache topology. And during conflict resolution the read CH is the preferred topology's read CH, so non-preferred topology owners will ask the primary owner for the value:
> {noformat}
> 17:12:07,088 INFO (testng-Test:[]) [JGroupsTransport] ISPN000093: Received new, MERGED cluster view for channel ISPN: MergeView::[Test-NodeI-48950|10] (4) [Test-NodeI-48950, Test-NodeJ-6577, Test-NodeK-2872, Test-NodeL-23686], 2 subgroups: [Test-NodeI-48950|8] (2) [Test-NodeI-48950, Test-NodeJ-6577], [Test-NodeK-2872|9] (2) [Test-NodeK-2872, Test-NodeL-23686]
> 17:12:07,134 TRACE (stateTransferExecutor-thread-Test-NodeI-p50536-t2:[Merge-10]) [DefaultConflictManager] Cache ___defaultcache attempting to receive all replicas for segment 0 with topology CacheTopology{id=20, rebalanceId=7, currentCH=DefaultConsistentHash{ns=256, owners = (2)[Test-NodeK-2872: 134+122, Test-NodeL-23686: 122+134]}, pendingCH=DefaultConsistentHash{ns=256, owners = (4)[Test-NodeK-2872: 134+122, Test-NodeL-23686: 122+134, Test-NodeI-48950: 0+256, Test-NodeJ-6577: 0+256]}, unionCH=DefaultConsistentHash{ns=256, owners = (4)[Test-NodeK-2872: 134+122, Test-NodeL-23686: 122+134, Test-NodeI-48950: 0+256, Test-NodeJ-6577: 0+256]}, phase=CONFLICT_RESOLUTION, actualMembers=[Test-NodeK-2872, Test-NodeL-23686, Test-NodeI-48950, Test-NodeJ-6577], persistentUUIDs=[40b58191-37f1-4fa4-8e4b-f1a7c17bfa59, 208d742d-ecc9-4792-9dd5-f74d5666f5a2, d8914096-a5a4-4e1e-95a0-27c50fee9999, 7464e58b-5e23-4c1b-909e-4e53055f12ca]}
> 17:12:07,212 TRACE (testng-Test:[]) [BaseDistributionInterceptor] Perform remote get for key MagicKey{13A4/DDFB0E77/56@Test-NodeI-48950}. currentTopologyId=20, owners=[Test-NodeK-2872, Test-NodeL-23686]
> 17:12:07,212 TRACE (testng-Test:[]) [RpcManagerImpl] Test-NodeI-48950 invoking ClusteredGetCommand{key=MagicKey{13A4/DDFB0E77/56@Test-NodeI-48950}, flags=[]} to recipient list [Test-NodeK-2872, Test-NodeL-23686] with options RpcOptions{timeout=15000, unit=MILLISECONDS, deliverOrder=NONE, responseFilter=org.infinispan.interceptors.impl.BaseRpcInterceptor$1@241e783a, responseMode=WAIT_FOR_VALID_RESPONSE}
> 17:12:07,225 TRACE (jgroups-4,Test-NodeI-48950:[]) [CommandAwareRpcDispatcher] Got acceptable response: Responses{
> Test-NodeK-2872: value=SuccessfulResponse{responseValue=ImmortalCacheValue {value=B}} , received=true, suspected=false
> 17:12:07,226 ERROR (testng-Test:[]) [TestSuiteProgress] Test failed: org.infinispan.conflict.impl.OperationsDuringMergeConflictTest.testPartitionMergePolicy
> java.lang.AssertionError: Key=MagicKey{13A4/DDFB0E77/56@Test-NodeI-48950}, Value=A, Cache Index=0, Topology=CacheTopology{id=20, rebalanceId=7, currentCH=PartitionerConsistentHash:DefaultConsistentHash{ns=256, owners = (2)[Test-NodeK-2872: 134+122, Test-NodeL-23686: 122+134]}, pendingCH=PartitionerConsistentHash:DefaultConsistentHash{ns=256, owners = (4)[Test-NodeK-2872: 134+122, Test-NodeL-23686: 122+134, Test-NodeI-48950: 0+256, Test-NodeJ-6577: 0+256]}, unionCH=PartitionerConsistentHash:DefaultConsistentHash{ns=256, owners = (4)[Test-NodeK-2872: 134+122, Test-NodeL-23686: 122+134, Test-NodeI-48950: 0+256, Test-NodeJ-6577: 0+256]}, phase=CONFLICT_RESOLUTION, actualMembers=[Test-NodeK-2872, Test-NodeL-23686, Test-NodeI-48950, Test-NodeJ-6577], persistentUUIDs=[40b58191-37f1-4fa4-8e4b-f1a7c17bfa59, 208d742d-ecc9-4792-9dd5-f74d5666f5a2, d8914096-a5a4-4e1e-95a0-27c50fee9999, 7464e58b-5e23-4c1b-909e-4e53055f12ca]} expected:<A> but was:<B>
> at org.testng.AssertJUnit.fail(AssertJUnit.java:59) ~[testng-6.8.8.jar:?]
> at org.testng.AssertJUnit.failNotEquals(AssertJUnit.java:364) ~[testng-6.8.8.jar:?]
> at org.testng.AssertJUnit.assertEquals(AssertJUnit.java:80) ~[testng-6.8.8.jar:?]
> at org.infinispan.conflict.impl.BaseMergePolicyTest.assertCacheGet(BaseMergePolicyTest.java:160) ~[test-classes/:?]
> at org.infinispan.conflict.impl.OperationsDuringMergeConflictTest.performMerge(OperationsDuringMergeConflictTest.java:121) ~[test-classes/:?]
> at org.infinispan.conflict.impl.BaseMergePolicyTest.testPartitionMergePolicy(BaseMergePolicyTest.java:116) ~[test-classes/:?]
> {noformat}
> https://ci.infinispan.org/job/Infinispan/job/master/543/testReport/org.in...
> Maybe we should use the union CH for reading during conflict resolution instead?
> Speaking of which, {{PreferAvailabilityStrategy}} sets the {{unionCH}} field in the conflict resolution {{CacheTopology}}, but it is ignored because {{CacheTopologyControlCommand}} doesn't have a field for it. We should set {{unionCH=null}} in {{PreferAvailabilityStrategy}} to make things clearer. We should also try to reduce the number of times we log the cache topology during conflict resolution (even if it's only at trace level).
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
6 years, 9 months
[JBoss JIRA] (ISPN-9044) In Cluster - Infinispan - SingleFileStore - fetchPersistentState/StateTransfer not transferring complete data to Joining Node
by Debashish Bharali (JIRA)
[ https://issues.jboss.org/browse/ISPN-9044?page=com.atlassian.jira.plugin.... ]
Debashish Bharali edited comment on ISPN-9044 at 4/6/18 8:50 AM:
-----------------------------------------------------------------
[~dan.berindei] We are working on using ReplStateTransferCacheLoaderTest shared by you.
Meanwhile, do you have any other earlier reported case/issue similar to the ours reported case.
And is there any possible way/tool to *see the content of a '.DAT'* file of *'SingleFileStore'*?
was (Author: debashish.bharali):
[~dan.berindei] We are working on using ReplStateTransferCacheLoaderTest shared by you.
Meanwhile, do you have any other earlier reported case/issue similar to the same.
And is there any possible way/tool to *see the content of a '.DAT'* file of *'SingleFileStore'*?
> In Cluster - Infinispan - SingleFileStore - fetchPersistentState/StateTransfer not transferring complete data to Joining Node
> -----------------------------------------------------------------------------------------------------------------------------
>
> Key: ISPN-9044
> URL: https://issues.jboss.org/browse/ISPN-9044
> Project: Infinispan
> Issue Type: Bug
> Components: Lucene Directory
> Affects Versions: 8.2.5.Final
> Reporter: Debashish Bharali
> Priority: Critical
> Attachments: neutrino-hibernatesearch-infinispan.xml
>
>
> Infinispan - SingleFileStore - fetchPersistentState/StateTransfer not transferring complete data to Joining Node.
> Related to ISPN-8980 (https://issues.jboss.org/browse/ISPN-8980).
> We are using Hibernate Search Indexes - Lucene indexes being stored on Infinispan with SingleFileStore.
> In case of more than 1 node. For example 4 nodes. We are observing below behaviour.
> Below are the steps:
> # We startup the first node *'N1'* in maintenance mode - with MassIndexer - creating initial indexes.
> # Now after all the MassIndexer/EntityLoader threads ends (after 1-2 Hrs). I.e. MassIndexing has been completed. We startup all other 3 nodes *'N2' , 'N3' and 'N4'*. Without MassIndexer.
> # Now on moderate to heavy application usage (concurrency), we are again getting the same exception of *Exception occurred java.io.FileNotFoundException: Error loading metadata for index file. Which indicates, {color:red}Some entries are not present in cache.{color}*
> # *But this exception comes only on the other 3 nodes (N2, N3 and N4). Not on the first node N1.*
> # On checking the sizes of the Cache stores in all the Nodes, the 3 Nodes (N2,N3 and N4) are having almost equal size (600 MB), which is 50%-70% of the size of Cache Stores of N1 (1.2 GB).
> # We have repeated these steps multiple times. Even switched MassIndexing node to other 3 nodes too. We have even reduced the number of nodes to 2.
> # *But the behaviour is exactly same. I.e. Exception on all the nodes except the initial node doing MassIndexing.*
> # {color:red} It seems like, *'N1's* cache-store's persistent state is not getting fetched by *'N2' 'N3' and 'N4'*, when these node joins joins.{color}
> # This is indicated by the fact that, FileNotFoundException doesn't comes in 'N1'. It comes in other nodes only (who joined later -- like N2, N3 & N4). And size of cache store's *'.DAT'* files are smaller then *'N1's*.
> Require urgent support.
> Attaching the corresponding Infinispan config file (neutrino-hibernatesearch-infinispan.xml)
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
6 years, 9 months
[JBoss JIRA] (ISPN-9044) In Cluster - Infinispan - SingleFileStore - fetchPersistentState/StateTransfer not transferring complete data to Joining Node
by Debashish Bharali (JIRA)
[ https://issues.jboss.org/browse/ISPN-9044?page=com.atlassian.jira.plugin.... ]
Debashish Bharali commented on ISPN-9044:
-----------------------------------------
[~dan.berindei] We are working on using ReplStateTransferCacheLoaderTest shared by you.
Meanwhile, do you have any other earlier reported case/issue similar to the same.
And is there any possible way/tool to *see the content of a '.DAT'* file of *'SingleFileStore'*?
> In Cluster - Infinispan - SingleFileStore - fetchPersistentState/StateTransfer not transferring complete data to Joining Node
> -----------------------------------------------------------------------------------------------------------------------------
>
> Key: ISPN-9044
> URL: https://issues.jboss.org/browse/ISPN-9044
> Project: Infinispan
> Issue Type: Bug
> Components: Lucene Directory
> Affects Versions: 8.2.5.Final
> Reporter: Debashish Bharali
> Priority: Critical
> Attachments: neutrino-hibernatesearch-infinispan.xml
>
>
> Infinispan - SingleFileStore - fetchPersistentState/StateTransfer not transferring complete data to Joining Node.
> Related to ISPN-8980 (https://issues.jboss.org/browse/ISPN-8980).
> We are using Hibernate Search Indexes - Lucene indexes being stored on Infinispan with SingleFileStore.
> In case of more than 1 node. For example 4 nodes. We are observing below behaviour.
> Below are the steps:
> # We startup the first node *'N1'* in maintenance mode - with MassIndexer - creating initial indexes.
> # Now after all the MassIndexer/EntityLoader threads ends (after 1-2 Hrs). I.e. MassIndexing has been completed. We startup all other 3 nodes *'N2' , 'N3' and 'N4'*. Without MassIndexer.
> # Now on moderate to heavy application usage (concurrency), we are again getting the same exception of *Exception occurred java.io.FileNotFoundException: Error loading metadata for index file. Which indicates, {color:red}Some entries are not present in cache.{color}*
> # *But this exception comes only on the other 3 nodes (N2, N3 and N4). Not on the first node N1.*
> # On checking the sizes of the Cache stores in all the Nodes, the 3 Nodes (N2,N3 and N4) are having almost equal size (600 MB), which is 50%-70% of the size of Cache Stores of N1 (1.2 GB).
> # We have repeated these steps multiple times. Even switched MassIndexing node to other 3 nodes too. We have even reduced the number of nodes to 2.
> # *But the behaviour is exactly same. I.e. Exception on all the nodes except the initial node doing MassIndexing.*
> # {color:red} It seems like, *'N1's* cache-store's persistent state is not getting fetched by *'N2' 'N3' and 'N4'*, when these node joins joins.{color}
> # This is indicated by the fact that, FileNotFoundException doesn't comes in 'N1'. It comes in other nodes only (who joined later -- like N2, N3 & N4). And size of cache store's *'.DAT'* files are smaller then *'N1's*.
> Require urgent support.
> Attaching the corresponding Infinispan config file (neutrino-hibernatesearch-infinispan.xml)
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
6 years, 9 months