<html><head><meta http-equiv="Content-Type" content="text/html charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">Shouldn't this "synchronous" flag still be used?<div><br></div><div><a href="https://github.com/Sanne/hibernate-search/blob/077f29c245d2d6e960cd6ab59ff58752320d5658/hibernate-search-engine/src/main/java/org/hibernate/search/backend/impl/jgroups/DispatcherMessageSender.java#L57">https://github.com/Sanne/hibernate-search/blob/077f29c245d2d6e960cd6ab59ff58752320d5658/hibernate-search-engine/src/main/java/org/hibernate/search/backend/impl/jgroups/DispatcherMessageSender.java#L57</a></div><div><br></div><div>e.g.</div><div>if (synchronous) {</div><div><span style="white-space: pre-wrap; "> int size = dispatcher.getChannel().getView().getMembers().size();</span></div><div><span style="white-space: pre-wrap; "> RequestOptions options = RequestOptions.SYNC();</span></div><div><span style="white-space: pre-wrap; "> options.setRspFilter( new WaitAllFilter( size ) );</span></div><div>} else {</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>options = <span style="white-space: pre-wrap; ">RequestOptions.ASYNC();</span></div><div>}</div><div><br></div><div>-Ales</div><div><br><div><div>On Apr 13, 2013, at 11:25 AM, Ales Justin <<a href="mailto:ales.justin@gmail.com">ales.justin@gmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite">Hmmm, did you try our QueryTest with this fix?<br><br>With HS update (your jgroupsWorkaround branch), my current run:<br><br>Running org.jboss.test.capedwarf.cluster.test.QueryTest<br>Tests run: 9, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 14.287 sec <<< FAILURE!<br><br>Results :<br><br>Failed tests: deleteAndQueryInA(org.jboss.test.capedwarf.cluster.test.QueryTest): Should not be here: null<br> deleteAndQueryInA_2(org.jboss.test.capedwarf.cluster.test.QueryTest): Should not be here: null<br><br>-Ales<br><br>On Apr 13, 2013, at 2:02 AM, Sanne Grinovero <<a href="mailto:sanne@hibernate.org">sanne@hibernate.org</a>> wrote:<br><br><blockquote type="cite">that's right, as suggested by Emmanuel I plan to separate the JGroups<br>Sync/Async options from the worker.execution property so you can play<br>with the two independently.<br>I think the JGroups option's default could depend on the backend - if<br>not otherwise specified, and if we all agree it doesn't make it too<br>confusing.<br><br>@All, the performance problem seemed to be caused by a problem in<br>JGroups, which I've logged here:<br><a href="https://issues.jboss.org/browse/JGRP-1617">https://issues.jboss.org/browse/JGRP-1617</a><br><br>For the record, the first operation was indeed triggering some lazy<br>initialization of indexes, which in turn would trigger a Lucene<br>Directory being started, triggering 3 Cache starts which in turn would<br>trigger 6 state transfer processes: so indeed the first operation<br>would not be exactly "cheap" performance wise, still this would<br>complete in about 120 milliseconds.<br>The same cost is paid again when the second node is hit the first<br>time, after that index write operations block the writer for <1ms (not<br>investigated further on potential throughput).<br><br>Not being sure about the options of depending to a newer JGroups<br>release or the complexity of a fix, I'll implement a workaround in<br>HSearch in the scope of HSEARCH-1296.<br><br>As a lesson learned, I think we need to polish some of our TRACE level<br>messaged to include the cache name: to resolve this we had not just<br>many threads and components but also 4 of them where using JGroups<br>(interleaving messages of all sorts) and 9 different caches where<br>involved for each simple write operation in CD: made it interesting to<br>figure what was going on! Also I'm wondering how hard it would be to<br>have a log parser which converts my 10GB of text log from today in a<br>graphical sequence diagram.<br>Big thanks to Mircea who helped me figuring this out.<br><br>Sanne<br><br>On 12 April 2013 21:10, Ales Justin <ales.justin@gmail.com> wrote:<br><blockquote type="cite">I think we need more fine-grained config for this new JGroups sync feature.<br><br>I added this to our cache config<br><br> <property name="hibernate.search.default.worker.execution">async</property><br><br>and it broke our tests.<br><br>Where previous (old / non JGroups sync) behavior worked.<br><br>It of course also works without this async config,<br>but in this case we don't need sync / ACK JGroups message.<br>(we didn't have one before and it worked ;-)<br><br>-Ales<br><br>On Apr 11, 2013, at 11:51 PM, Sanne Grinovero <sanne@hibernate.org> wrote:<br><br><blockquote type="cite">There is a "blackhole" indexing backend, which pipes all indexing<br>requests > /dev/null<br><br>Set this as an Infinispan Query configuration property:<br><br> default.worker.backend = blackhole<br><br>Of course that means that the index will not be updated: you might<br>need to adapt your test to tolerate that, but the point is not<br>functional testing but to verify how much the SYNC option on the<br>JGroups backend is actually slowing you down. I suspect the<br>performance penalty is not in the network but in the fact you're now<br>waiting for the index operations, while in async you where not waiting<br>for them to be flushed.<br><br>If you can identify which part is slow, then we can help you with<br>better configuration options.<br><br><br>On 11 April 2013 20:47, Ales Justin <ales.justin@gmail.com> wrote:<br><blockquote type="cite">What do you mean?<br><br>On Apr 11, 2013, at 21:41, Sanne Grinovero <sanne@hibernate.org> wrote:<br><br>You could try the new sync version but setting the blackhole backend on the<br>master node to remove the indexing overhead from the picture.<br><br>On Apr 11, 2013 8:39 PM, "Sanne Grinovero" <sanne@hibernate.org> wrote:<br><blockquote type="cite"><br>Are you sure that the async version actually had applied all writes to the<br>index in the measured interval?<br><br>On Apr 11, 2013 8:13 PM, "Ales Justin" <ales.justin@gmail.com> wrote:<br><blockquote type="cite"><br>Although this change fixes query lookup,<br>it adds horrible performance:<br><br>Running CapeDwarf cluster QueryTest:<br><br>with HSEARCH-1296<br><br>21:00:27,188 INFO<br>[org.hibernate.search.indexes.impl.DirectoryBasedIndexManager]<br>(http-/192.168.1.102:8080-1) HSEARCH000168: Serialization service Avro<br>SerializationProvider v1.0 being used for index<br>'default_capedwarf-test__com.google.appengine.api.datastore.Entity'<br>21:01:17,911 INFO [org.jboss.web] (ServerService Thread Pool -- 49)<br>JBAS018224: Unregister web context: /capedwarf-tests<br><br>50sec<br><br>old 4.2.0.Final HS<br><br>21:08:19,988 INFO<br>[org.hibernate.search.indexes.impl.DirectoryBasedIndexManager]<br>(http-/192.168.1.102:8080-2) HSEARCH000168: Serialization service Avro<br>SerializationProvider v1.0 being used for index<br>'default_capedwarf-test__com.google.appengine.api.datastore.Entity'<br>21:08:20,829 INFO [org.jboss.web] (ServerService Thread Pool -- 49)<br>JBAS018224: Unregister web context: /capedwarf-tests<br><br>841ms<br><br>---<br><br>I added<br><br> <property name="enable_bundling">true</property><br><br>to AS jgroups transport config, but no improvement.<br><br>Any (other) idea?<br><br>-Ales<br><br><br>_______________________________________________<br>hibernate-dev mailing list<br>hibernate-dev@lists.jboss.org<br>https://lists.jboss.org/mailman/listinfo/hibernate-dev<br></blockquote></blockquote></blockquote></blockquote><br><br>_______________________________________________<br>hibernate-dev mailing list<br>hibernate-dev@lists.jboss.org<br>https://lists.jboss.org/mailman/listinfo/hibernate-dev<br></blockquote></blockquote><br></blockquote></div><br></div></body></html>