<div dir="ltr"><br><div class="gmail_extra"><br><br><div class="gmail_quote">On Sat, Apr 13, 2013 at 3:02 AM, Sanne Grinovero <span dir="ltr"><<a href="mailto:sanne@hibernate.org" target="_blank">sanne@hibernate.org</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">that's right, as suggested by Emmanuel I plan to separate the JGroups<br>
Sync/Async options from the worker.execution property so you can play<br>
with the two independently.<br>
I think the JGroups option's default could depend on the backend - if<br>
not otherwise specified, and if we all agree it doesn't make it too<br>
confusing.<br>
<br>
@All, the performance problem seemed to be caused by a problem in<br>
JGroups, which I've logged here:<br>
<a href="https://issues.jboss.org/browse/JGRP-1617" target="_blank">https://issues.jboss.org/browse/JGRP-1617</a><br>
<br>
For the record, the first operation was indeed triggering some lazy<br>
initialization of indexes, which in turn would trigger a Lucene<br>
Directory being started, triggering 3 Cache starts which in turn would<br>
trigger 6 state transfer processes: so indeed the first operation<br>
would not be exactly "cheap" performance wise, still this would<br>
complete in about 120 milliseconds.<br>
The same cost is paid again when the second node is hit the first<br>
time, after that index write operations block the writer for <1ms (not<br>
investigated further on potential throughput).<br>
<br>
Not being sure about the options of depending to a newer JGroups<br>
release or the complexity of a fix, I'll implement a workaround in<br>
HSearch in the scope of HSEARCH-1296.<br>
<br>
As a lesson learned, I think we need to polish some of our TRACE level<br>
messaged to include the cache name: </blockquote><div><br></div><div><br>Sanne, we already push the cache name in the NDC if trace is enabled for the "entry point" of the thread. So for your purpose, I think enabling trace for org.infinispan.interceptors.InvocationContextInterceptor and including %x in your pattern layout should work.<br>
<br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">to resolve this we had not just<br>
many threads and components but also 4 of them where using JGroups<br>
(interleaving messages of all sorts) and 9 different caches where<br>
involved for each simple write operation in CD: made it interesting to<br>
figure what was going on! Also I'm wondering how hard it would be to<br>
have a log parser which converts my 10GB of text log from today in a<br>
graphical sequence diagram.<br>
Big thanks to Mircea who helped me figuring this out.<br>
<span class=""><font color="#888888"><br>
Sanne<br>
</font></span><div class=""><div class="h5"><br>
On 12 April 2013 21:10, Ales Justin <<a href="mailto:ales.justin@gmail.com">ales.justin@gmail.com</a>> wrote:<br>
> I think we need more fine-grained config for this new JGroups sync feature.<br>
><br>
> I added this to our cache config<br>
><br>
> <property name="hibernate.search.default.worker.execution">async</property><br>
><br>
> and it broke our tests.<br>
><br>
> Where previous (old / non JGroups sync) behavior worked.<br>
><br>
> It of course also works without this async config,<br>
> but in this case we don't need sync / ACK JGroups message.<br>
> (we didn't have one before and it worked ;-)<br>
><br>
> -Ales<br>
><br>
> On Apr 11, 2013, at 11:51 PM, Sanne Grinovero <<a href="mailto:sanne@hibernate.org">sanne@hibernate.org</a>> wrote:<br>
><br>
>> There is a "blackhole" indexing backend, which pipes all indexing<br>
>> requests > /dev/null<br>
>><br>
>> Set this as an Infinispan Query configuration property:<br>
>><br>
>> default.worker.backend = blackhole<br>
>><br>
>> Of course that means that the index will not be updated: you might<br>
>> need to adapt your test to tolerate that, but the point is not<br>
>> functional testing but to verify how much the SYNC option on the<br>
>> JGroups backend is actually slowing you down. I suspect the<br>
>> performance penalty is not in the network but in the fact you're now<br>
>> waiting for the index operations, while in async you where not waiting<br>
>> for them to be flushed.<br>
>><br>
>> If you can identify which part is slow, then we can help you with<br>
>> better configuration options.<br>
>><br>
>><br>
>> On 11 April 2013 20:47, Ales Justin <<a href="mailto:ales.justin@gmail.com">ales.justin@gmail.com</a>> wrote:<br>
>>> What do you mean?<br>
>>><br>
>>> On Apr 11, 2013, at 21:41, Sanne Grinovero <<a href="mailto:sanne@hibernate.org">sanne@hibernate.org</a>> wrote:<br>
>>><br>
>>> You could try the new sync version but setting the blackhole backend on the<br>
>>> master node to remove the indexing overhead from the picture.<br>
>>><br>
>>> On Apr 11, 2013 8:39 PM, "Sanne Grinovero" <<a href="mailto:sanne@hibernate.org">sanne@hibernate.org</a>> wrote:<br>
>>>><br>
>>>> Are you sure that the async version actually had applied all writes to the<br>
>>>> index in the measured interval?<br>
>>>><br>
>>>> On Apr 11, 2013 8:13 PM, "Ales Justin" <<a href="mailto:ales.justin@gmail.com">ales.justin@gmail.com</a>> wrote:<br>
>>>>><br>
>>>>> Although this change fixes query lookup,<br>
>>>>> it adds horrible performance:<br>
>>>>><br>
>>>>> Running CapeDwarf cluster QueryTest:<br>
>>>>><br>
>>>>> with HSEARCH-1296<br>
>>>>><br>
>>>>> 21:00:27,188 INFO<br>
>>>>> [org.hibernate.search.indexes.impl.DirectoryBasedIndexManager]<br>
>>>>> (http-/192.168.1.102:8080-1) HSEARCH000168: Serialization service Avro<br>
>>>>> SerializationProvider v1.0 being used for index<br>
>>>>> 'default_capedwarf-test__com.google.appengine.api.datastore.Entity'<br>
>>>>> 21:01:17,911 INFO [org.jboss.web] (ServerService Thread Pool -- 49)<br>
>>>>> JBAS018224: Unregister web context: /capedwarf-tests<br>
>>>>><br>
>>>>> 50sec<br>
>>>>><br>
>>>>> old 4.2.0.Final HS<br>
>>>>><br>
>>>>> 21:08:19,988 INFO<br>
>>>>> [org.hibernate.search.indexes.impl.DirectoryBasedIndexManager]<br>
>>>>> (http-/192.168.1.102:8080-2) HSEARCH000168: Serialization service Avro<br>
>>>>> SerializationProvider v1.0 being used for index<br>
>>>>> 'default_capedwarf-test__com.google.appengine.api.datastore.Entity'<br>
>>>>> 21:08:20,829 INFO [org.jboss.web] (ServerService Thread Pool -- 49)<br>
>>>>> JBAS018224: Unregister web context: /capedwarf-tests<br>
>>>>><br>
>>>>> 841ms<br>
>>>>><br>
>>>>> ---<br>
>>>>><br>
>>>>> I added<br>
>>>>><br>
>>>>> <property name="enable_bundling">true</property><br>
>>>>><br>
>>>>> to AS jgroups transport config, but no improvement.<br>
>>>>><br>
>>>>> Any (other) idea?<br>
>>>>><br>
>>>>> -Ales<br>
>>>>><br>
>>>>><br>
>>>>> _______________________________________________<br>
>>>>> hibernate-dev mailing list<br>
>>>>> <a href="mailto:hibernate-dev@lists.jboss.org">hibernate-dev@lists.jboss.org</a><br>
>>>>> <a href="https://lists.jboss.org/mailman/listinfo/hibernate-dev" target="_blank">https://lists.jboss.org/mailman/listinfo/hibernate-dev</a><br>
><br>
><br>
> _______________________________________________<br>
> hibernate-dev mailing list<br>
> <a href="mailto:hibernate-dev@lists.jboss.org">hibernate-dev@lists.jboss.org</a><br>
> <a href="https://lists.jboss.org/mailman/listinfo/hibernate-dev" target="_blank">https://lists.jboss.org/mailman/listinfo/hibernate-dev</a><br>
_______________________________________________<br>
infinispan-dev mailing list<br>
<a href="mailto:infinispan-dev@lists.jboss.org">infinispan-dev@lists.jboss.org</a><br>
<a href="https://lists.jboss.org/mailman/listinfo/infinispan-dev" target="_blank">https://lists.jboss.org/mailman/listinfo/infinispan-dev</a><br>
</div></div></blockquote></div><br></div></div>