[JBoss JIRA] (ISPN-2741) VersionedConditionalOperationsTest.testRemoveIfPresent failing randomly
by Galder Zamarreño (JIRA)
[ https://issues.jboss.org/browse/ISPN-2741?page=com.atlassian.jira.plugin.... ]
Work on ISPN-2741 started by Galder Zamarreño.
> VersionedConditionalOperationsTest.testRemoveIfPresent failing randomly
> -----------------------------------------------------------------------
>
> Key: ISPN-2741
> URL: https://issues.jboss.org/browse/ISPN-2741
> Project: Infinispan
> Issue Type: Bug
> Reporter: Galder Zamarreño
> Assignee: Galder Zamarreño
> Labels: testsuite_stability
> Fix For: 5.2.0.Final
>
> Attachments: testRemoveIfPresent-0.log
>
>
> {code}testRemoveIfPresent(org.infinispan.container.versioning.VersionedConditionalOperationsTest) Time elapsed: 0.133 sec <<< FAILURE!
> java.lang.AssertionError
> at org.infinispan.container.versioning.VersionedConditionalOperationsTest.assertEquals(VersionedConditionalOperationsTest.java:155)
> at org.infinispan.container.versioning.VersionedConditionalOperationsTest.assertCacheValue(VersionedConditionalOperationsTest.java:146)
> at org.infinispan.container.versioning.VersionedConditionalOperationsTest.testRemoveIfPresent(VersionedConditionalOperationsTest.java:104)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 11 months
[JBoss JIRA] (ISPN-2581) StateTransferManagerImpl.waitForInitialStateTransferToComplete() returns too soon
by Adrian Nistor (JIRA)
[ https://issues.jboss.org/browse/ISPN-2581?page=com.atlassian.jira.plugin.... ]
Adrian Nistor updated ISPN-2581:
--------------------------------
Status: Resolved (was: Pull Request Sent)
Resolution: Done
Integrated. Thanks!
> StateTransferManagerImpl.waitForInitialStateTransferToComplete() returns too soon
> ---------------------------------------------------------------------------------
>
> Key: ISPN-2581
> URL: https://issues.jboss.org/browse/ISPN-2581
> Project: Infinispan
> Issue Type: Bug
> Components: State transfer
> Affects Versions: 5.2.0.Beta5
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Fix For: 5.2.0.Final
>
>
> StateTransferManagerImpl.waitForInitialStateTransferToComplete() returns as soon as a joining node confirmed to the coordinator that it received all the data it needed (see STMI.notifyEndOfTopologyUpdate()).
> It should return only after the coordinator has confirmed the end of the rebalance with a new topology update (see STMI.doTopologyUpdate()).
> This should make it more likely for the tests suite clusters to be in a stable state by the time the test starts, and should help with the random state transfer-related failures in non-state transfer tests.
> Instead we should make sure that we do have tests that check forwarding behaviour explicitly.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 11 months
[JBoss JIRA] (ISPN-2712) Initial state transfer doesn't appear to all be persisted when using eviction in a replicated cluster
by Dan Berindei (JIRA)
[ https://issues.jboss.org/browse/ISPN-2712?page=com.atlassian.jira.plugin.... ]
Dan Berindei commented on ISPN-2712:
------------------------------------
When eviction is enabled, the data container is split in {{concurrencyLevel}} segments, and eviction is done individually on each segment. So instead of starting to evict entries when the total number of entries in the cache is {{maxEntries}}, we start to evict entries when the total number of entries in each segment is {{maxEntries/concurrencyLevel}}.
This obviously breaks down when {{concurrencyLevel >= maxEntries}}, so {{concurrencyLevel}} is limited to {{maxEntries/2}}. Still, that means just 2 entries in each segment. It's very easy for keys to get clustered in particular segments and to get evicted too soon.
I'm not aware of any changes in this area since 5.1, but I suggest lowering {{concurrencyLevel}} to 50 to avoid the problem.
> Initial state transfer doesn't appear to all be persisted when using eviction in a replicated cluster
> ------------------------------------------------------------------------------------------------------
>
> Key: ISPN-2712
> URL: https://issues.jboss.org/browse/ISPN-2712
> Project: Infinispan
> Issue Type: Bug
> Affects Versions: 5.2.0.CR1
> Reporter: Randall Hauch
> Assignee: Adrian Nistor
> Priority: Critical
> Fix For: 5.2.0.Final
>
> Attachments: ispn_eviction_log.txt, spectrum-repository-infinispan.xml
>
>
> Using a clustered cache with 2 nodes, where the cache in each node is configured identically with replication, eviction and (non-shared) file cache store. (See attached configuration.)
> The first (coordinator) process in the cluster is started and populated with 293 entries. Then the first process continually adds a few entries every 5 seconds. After a short delay, the second process is started, at which point it joins the cluster and starts the state-transfer process; logging shows in the first process that all 293 entries are transferred to the new cluster member, and the second log shows that they are all received. The second process then attempts to look for a specific entry that was created during initial population in the first process. This fails to find the existing entry.
> By enabling trace logging and "IDE breakpoint output messages" around state transfer, it's visible that from the 293 keys, only 218 are placed into the cache, the others being lost.
> (This problem was originally discovered when clustering ModeShape, which behaves roughly in the manner described above. The initial entries that are populated upon initialization are content created when a new repository is started. The second process looks for this content, and if it finds the content it knows not to create all of this initial content. However, if it doesn't find it, it thinks the repository has not yet been initialized and that it should create the initial content. The problem described by this bug then manifests itself in ModeShape through dozens of exceptions because the repository has been corrupted. See MODE-1745 for details on this problem. ModeShape's corresponding known issue for this issue, ISPN-2712, is MODE-1754.)
> The eviction is configured like this:
> {code:xml}
> <eviction strategy="LIRS" maxEntries="1000"/>
> {code}
> The attached log file is from the second process (the "receiver" node) and it contains the following key points:
> * line 40 - the total number of keys & entries to be transferred = 293
> * line 1352 and from there onwards 1358 / 1364 / i + 6 - the data container's size stops growing at 218, while the other entries are being sent. This means that in effect, they are ignored.
> * line 1797 - the loop from {{org.infinispan.statetransfer.StateConsumerImpl#doApplyState}} finishes
> Disabling eviction fixes the problem and all 293 nodes are placed in Node2's cache.
> (I initially marked this as CRITICAL priority, though it is a blocker for our use of Infinispan 5.2.)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 11 months
[JBoss JIRA] (ISPN-2697) HotRodServer startup fails when its record cannot be inserted into topology cache
by Bela Ban (JIRA)
[ https://issues.jboss.org/browse/ISPN-2697?page=com.atlassian.jira.plugin.... ]
Bela Ban commented on ISPN-2697:
--------------------------------
Re Dan's comment at 15/Jan/13 2:07 PM:
Why is tagging a message as RSVP hacky ? Infinispan does tag certain RPCs with RSVP, so why is that not hacky, and marking this RPC *is* hacky ?
The suggestion to increase repl-timeout to be greater than desired_avg_gossip * cluster-size won't work IMO, because this might be a long timeout.
With desired_avg_gossip * cluster-size, we're getting roughly the same number of STABLE messages in the cluster, regardless of the cluster size, but at the expense of STABILITY messages being sent less frequently with increasing cluster size.
The reason I did it that way is that the STABLE protocol is mainly used to garbage collect messages received by everyone. When many messages are sent, the size based STABLE kicks in. When only few messages are sent, I wanted to keep the STABLE induced messages to a minimum.
Using STABLE to learn of the highest message seqnos received is a functionality that's piggy backed on STABLE messages, but that's not the main priority.
If you guys think that the way desired_avg_gossip * cluster-size is computed is bad, I can always change it to enforce an upper bound on *STABILITY* rather then *STABLE* messages, but that means more STABLE-related messages with increasing cluster size.
WDYT ?
> HotRodServer startup fails when its record cannot be inserted into topology cache
> ---------------------------------------------------------------------------------
>
> Key: ISPN-2697
> URL: https://issues.jboss.org/browse/ISPN-2697
> Project: Infinispan
> Issue Type: Bug
> Components: Remote protocols
> Affects Versions: 5.2.0.Beta6
> Reporter: Radim Vansa
> Assignee: Galder Zamarreño
> Priority: Critical
> Fix For: 5.2.0.Final
>
>
> When the HotRodServer starts it inserts its record to __hotRodTopologyCache ({{HotRodServer.addSelfToTopologyView(...)}}).
> However, this put may very easily fail - as the command is broadcasted using NAKACK2 protocol, if the message gets lost and there's no following broadcasted message, the message will be not retransmitted and the put operation times out (Replication timeout), which fails the whole HotRodServer startup, all because of one lost UDP message.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 11 months
[JBoss JIRA] (ISPN-2581) StateTransferManagerImpl.waitForInitialStateTransferToComplete() returns too soon
by Dan Berindei (JIRA)
[ https://issues.jboss.org/browse/ISPN-2581?page=com.atlassian.jira.plugin.... ]
Dan Berindei updated ISPN-2581:
-------------------------------
Status: Pull Request Sent (was: Coding In Progress)
Git Pull Request: https://github.com/infinispan/infinispan/pull/1605
Don't flag the initial state transfer as complete as soon as we send the
rebalance confirmation to the coordinator. It will be flagged while
processing the next topology update.
> StateTransferManagerImpl.waitForInitialStateTransferToComplete() returns too soon
> ---------------------------------------------------------------------------------
>
> Key: ISPN-2581
> URL: https://issues.jboss.org/browse/ISPN-2581
> Project: Infinispan
> Issue Type: Bug
> Components: State transfer
> Affects Versions: 5.2.0.Beta5
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Fix For: 5.2.0.Final
>
>
> StateTransferManagerImpl.waitForInitialStateTransferToComplete() returns as soon as a joining node confirmed to the coordinator that it received all the data it needed (see STMI.notifyEndOfTopologyUpdate()).
> It should return only after the coordinator has confirmed the end of the rebalance with a new topology update (see STMI.doTopologyUpdate()).
> This should make it more likely for the tests suite clusters to be in a stable state by the time the test starts, and should help with the random state transfer-related failures in non-state transfer tests.
> Instead we should make sure that we do have tests that check forwarding behaviour explicitly.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 11 months