<div class="gmail_extra"><br><div class="gmail_quote">On Wed, Dec 5, 2012 at 4:20 PM, Sanne Grinovero <span dir="ltr"><<a href="mailto:sanne@infinispan.org" target="_blank">sanne@infinispan.org</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="HOEnZb"><div class="h5">On 5 December 2012 14:01, Bela Ban <<a href="mailto:bban@redhat.com">bban@redhat.com</a>> wrote:<br>
><br>
> On 12/5/12 1:23 PM, Sanne Grinovero wrote:<br>
>> On 5 December 2012 11:02, Galder Zamarreņo <<a href="mailto:galder@redhat.com">galder@redhat.com</a>> wrote:<br>
>>> On Dec 4, 2012, at 10:22 AM, Sanne Grinovero <<a href="mailto:sanne@infinispan.org">sanne@infinispan.org</a>> wrote:<br>
>>><br>
>>>> On 4 December 2012 09:14, Galder Zamarreņo <<a href="mailto:galder@redhat.com">galder@redhat.com</a>> wrote:<br>
>>>>> Hey Dan/Adrian,<br>
>>>>><br>
>>>>> Re: <a href="https://issues.jboss.org/browse/ISPN-2541" target="_blank">https://issues.jboss.org/browse/ISPN-2541</a><br>
>>>>><br>
>>>>> I'm looking at this intermittent failure, and it seems to be caused by the fact that the test does not wait for the cluster to be formed when the new node is started, which can lead a replication timeout failure from the new joining node.<br>
>>>>><br>
>>>>> The test can easily be fixed by waiting for cluster to form, and then do the call.<br>
>>>>><br>
>>>> [...]<br>
>>>><br>
>>>> I don't think the cache should ever be in an illegal state to be used<br>
>>>> after being started. So Infinispan should not require tests to wait<br>
>>>> for a "cluster to be formed", I'd rather guarantee that after a cache<br>
>>>> is started it's usable.<br>
>>> Precisely, which is why I raised the flag instead of going down the easy path.<br>
>>><br>
>>>> If this is not possible, then any application would also need to wait<br>
>>>> for that "cluster formed" event, and we should expose an API for that.<br>
>>> The problem is considering when a cluster is formed. How many nodes should you wait for?<br>
>>><br>
>> Why can't we rely on JGroups Discovery to know that, as a user I<br>
>> already specified the expected initial group size with<br>
>> num_initial_members<br>
>> Don't want to repeat that configuration ;-)<br>
><br>
><br>
> I don't understand this discussion: when a new node join, it'll return<br>
> from JChannel.connect() when it received a JOIN response from the<br>
> coordinator, with the current view... or are you guys talking about<br>
> Infinispan's 'service views' ?<br>
<br>
</div></div>+1<br>
<br>
That's why I'm confused too, and not understanding how it is possible<br>
that a Cache is returned to the application - which doesn't have a<br>
clue about number of expected nodes - in a state for which the<br>
"cluster is not formed yet". That should never happen!?<br>
<br></blockquote><div><br>It's simple: getCache() returns once the joiner has received ownership of some segments (in distributed mode) and once it received all the data it owner (dist and repl). This does not guarantee that the other nodes see the joiner as a full member at the time getCache() has returned.<br>
<br>This doesn't mean that the cache is not functional, on the contrary we could return even before the joiner had received the data and the cache would still work. But because some nodes think state transfer is still in progress, the tests do run into state transfer corner cases that aren't handled properly (they're getting rarer, but we still have them).<br>
<br> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
I never understood why the test framework in Infinispan requires this<br>
to happen in all tests - even in the cases listed by Mircea that the<br>
testsuite is looking for something very specific, I would expect the<br>
wait to be unnecessary. (or more precisely, to have been blocked<br>
already for long enough)<br>
<div class="HOEnZb"><div class="h5"><br></div></div></blockquote><div><br>getCache() only waits enough for the cache to "work", it doesn't wait (and I don't think it should wait) for all the other nodes to acknowledge the joiner as a full member (i.e. in the "read" consistent hash). Because of this, assertions made on nodes other than the joiner can fail (in addition to the aforementioned corner cases in state transfer).<br>
<br>It's also possible (and it was quite likely with older JGroups versions) that a joiner would actually form a new cluster by itself instead of joining the existing nodes in a single cluster. When that happens, getCache() definitely returns without the cluster being formed, and we have to wait for the separate clusters to find each other and merge before running our test.<br>
<br>Cheers<br>Dan<br><br></div></div></div>