Are you sure that the async version actually had applied all writes to the index in the measured interval? On Apr 11, 2013 8:13 PM, "Ales Justin" <ales.justin(a)gmail.com&gt; wrote: > Although this change fixes query lookup, > it adds horrible performance: > > Running CapeDwarf cluster QueryTest: > > with HSEARCH-1296 > > 21:00:27,188 INFO > [org.hibernate.search.indexes.impl.DirectoryBasedIndexManager] > (http-/192.168.1.102:8080-1) HSEARCH000168: Serialization service Avro > SerializationProvider v1.0 being used for index > 'default_capedwarf-test__com.google.appengine.api.datastore.Entity' > 21:01:17,911 INFO [org.jboss.web] (ServerService Thread Pool -- 49) > JBAS018224: Unregister web context: /capedwarf-tests > > 50sec > > old 4.2.0.Final HS > > 21:08:19,988 INFO > [org.hibernate.search.indexes.impl.DirectoryBasedIndexManager] > (http-/192.168.1.102:8080-2) HSEARCH000168: Serialization service Avro > SerializationProvider v1.0 being used for index > 'default_capedwarf-test__com.google.appengine.api.datastore.Entity' > 21:08:20,829 INFO [org.jboss.web] (ServerService Thread Pool -- 49) > JBAS018224: Unregister web context: /capedwarf-tests > > 841ms > > --- > > I added > > <property name="enable_bundling">true</property> > > to AS jgroups transport config, but no improvement. > > Any (other) idea? > > -Ales > > > _______________________________________________ > hibernate-dev mailing list > hibernate-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/hibernate-dev >

What do you mean? On Apr 11, 2013, at 21:41, Sanne Grinovero <sanne(a)hibernate.org&gt; wrote: You could try the new sync version but setting the blackhole backend on the master node to remove the indexing overhead from the picture. On Apr 11, 2013 8:39 PM, "Sanne Grinovero" <sanne(a)hibernate.org&gt; wrote: > > Are you sure that the async version actually had applied all writes to the > index in the measured interval? > > On Apr 11, 2013 8:13 PM, "Ales Justin" <ales.justin(a)gmail.com&gt; wrote: >> >> Although this change fixes query lookup, >> it adds horrible performance: >> >> Running CapeDwarf cluster QueryTest: >> >> with HSEARCH-1296 >> >> 21:00:27,188 INFO >> [org.hibernate.search.indexes.impl.DirectoryBasedIndexManager] >> (http-/192.168.1.102:8080-1) HSEARCH000168: Serialization service Avro >> SerializationProvider v1.0 being used for index >> 'default_capedwarf-test__com.google.appengine.api.datastore.Entity' >> 21:01:17,911 INFO [org.jboss.web] (ServerService Thread Pool -- 49) >> JBAS018224: Unregister web context: /capedwarf-tests >> >> 50sec >> >> old 4.2.0.Final HS >> >> 21:08:19,988 INFO >> [org.hibernate.search.indexes.impl.DirectoryBasedIndexManager] >> (http-/192.168.1.102:8080-2) HSEARCH000168: Serialization service Avro >> SerializationProvider v1.0 being used for index >> 'default_capedwarf-test__com.google.appengine.api.datastore.Entity' >> 21:08:20,829 INFO [org.jboss.web] (ServerService Thread Pool -- 49) >> JBAS018224: Unregister web context: /capedwarf-tests >> >> 841ms >> >> --- >> >> I added >> >> <property name="enable_bundling">true</property> >> >> to AS jgroups transport config, but no improvement. >> >> Any (other) idea? >> >> -Ales >> >> >> _______________________________________________ >> hibernate-dev mailing list >> hibernate-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/hibernate-dev

I think we need more fine-grained config for this new JGroups sync feature. I added this to our cache config <property name="hibernate.search.default.worker.execution">async</property> and it broke our tests. Where previous (old / non JGroups sync) behavior worked. It of course also works without this async config, but in this case we don't need sync / ACK JGroups message. (we didn't have one before and it worked ;-) -Ales On Apr 11, 2013, at 11:51 PM, Sanne Grinovero <sanne(a)hibernate.org&gt; wrote: > There is a "blackhole" indexing backend, which pipes all indexing > requests > /dev/null > > Set this as an Infinispan Query configuration property: > > default.worker.backend = blackhole > > Of course that means that the index will not be updated: you might > need to adapt your test to tolerate that, but the point is not > functional testing but to verify how much the SYNC option on the > JGroups backend is actually slowing you down. I suspect the > performance penalty is not in the network but in the fact you're now > waiting for the index operations, while in async you where not waiting > for them to be flushed. > > If you can identify which part is slow, then we can help you with > better configuration options. > > > On 11 April 2013 20:47, Ales Justin <ales.justin(a)gmail.com&gt; wrote: >> What do you mean? >> >> On Apr 11, 2013, at 21:41, Sanne Grinovero <sanne(a)hibernate.org&gt; wrote: >> >> You could try the new sync version but setting the blackhole backend on the >> master node to remove the indexing overhead from the picture. >> >> On Apr 11, 2013 8:39 PM, "Sanne Grinovero" <sanne(a)hibernate.org&gt; wrote: >>> >>> Are you sure that the async version actually had applied all writes to the >>> index in the measured interval? >>> >>> On Apr 11, 2013 8:13 PM, "Ales Justin" <ales.justin(a)gmail.com&gt; wrote: >>>> >>>> Although this change fixes query lookup, >>>> it adds horrible performance: >>>> >>>> Running CapeDwarf cluster QueryTest: >>>> >>>> with HSEARCH-1296 >>>> >>>> 21:00:27,188 INFO >>>> [org.hibernate.search.indexes.impl.DirectoryBasedIndexManager] >>>> (http-/192.168.1.102:8080-1) HSEARCH000168: Serialization service Avro >>>> SerializationProvider v1.0 being used for index >>>> 'default_capedwarf-test__com.google.appengine.api.datastore.Entity' >>>> 21:01:17,911 INFO [org.jboss.web] (ServerService Thread Pool -- 49) >>>> JBAS018224: Unregister web context: /capedwarf-tests >>>> >>>> 50sec >>>> >>>> old 4.2.0.Final HS >>>> >>>> 21:08:19,988 INFO >>>> [org.hibernate.search.indexes.impl.DirectoryBasedIndexManager] >>>> (http-/192.168.1.102:8080-2) HSEARCH000168: Serialization service Avro >>>> SerializationProvider v1.0 being used for index >>>> 'default_capedwarf-test__com.google.appengine.api.datastore.Entity' >>>> 21:08:20,829 INFO [org.jboss.web] (ServerService Thread Pool -- 49) >>>> JBAS018224: Unregister web context: /capedwarf-tests >>>> >>>> 841ms >>>> >>>> --- >>>> >>>> I added >>>> >>>> <property name="enable_bundling">true</property> >>>> >>>> to AS jgroups transport config, but no improvement. >>>> >>>> Any (other) idea? >>>> >>>> -Ales >>>> >>>> >>>> _______________________________________________ >>>> hibernate-dev mailing list >>>> hibernate-dev(a)lists.jboss.org >>>> https://lists.jboss.org/mailman/listinfo/hibernate-dev _______________________________________________ hibernate-dev mailing list hibernate-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/hibernate-dev

Hmmm, did you try our QueryTest with this fix? With HS update (your jgroupsWorkaround branch), my current run: Running org.jboss.test.capedwarf.cluster.test.QueryTest Tests run: 9, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 14.287 sec <<< FAILURE! Results : Failed tests: deleteAndQueryInA(org.jboss.test.capedwarf.cluster.test.QueryTest): Should not be here: null deleteAndQueryInA_2(org.jboss.test.capedwarf.cluster.test.QueryTest): Should not be here: null -Ales On Apr 13, 2013, at 2:02 AM, Sanne Grinovero <sanne(a)hibernate.org&gt; wrote: > that's right, as suggested by Emmanuel I plan to separate the JGroups > Sync/Async options from the worker.execution property so you can play > with the two independently. > I think the JGroups option's default could depend on the backend - if > not otherwise specified, and if we all agree it doesn't make it too > confusing. > > @All, the performance problem seemed to be caused by a problem in > JGroups, which I've logged here: > https://issues.jboss.org/browse/JGRP-1617 > > For the record, the first operation was indeed triggering some lazy > initialization of indexes, which in turn would trigger a Lucene > Directory being started, triggering 3 Cache starts which in turn would > trigger 6 state transfer processes: so indeed the first operation > would not be exactly "cheap" performance wise, still this would > complete in about 120 milliseconds. > The same cost is paid again when the second node is hit the first > time, after that index write operations block the writer for <1ms (not > investigated further on potential throughput). > > Not being sure about the options of depending to a newer JGroups > release or the complexity of a fix, I'll implement a workaround in > HSearch in the scope of HSEARCH-1296. > > As a lesson learned, I think we need to polish some of our TRACE level > messaged to include the cache name: to resolve this we had not just > many threads and components but also 4 of them where using JGroups > (interleaving messages of all sorts) and 9 different caches where > involved for each simple write operation in CD: made it interesting to > figure what was going on! Also I'm wondering how hard it would be to > have a log parser which converts my 10GB of text log from today in a > graphical sequence diagram. > Big thanks to Mircea who helped me figuring this out. > > Sanne > > On 12 April 2013 21:10, Ales Justin <ales.justin(a)gmail.com&gt; wrote: >> I think we need more fine-grained config for this new JGroups sync feature. >> >> I added this to our cache config >> >> <property name="hibernate.search.default.worker.execution">async</property> >> >> and it broke our tests. >> >> Where previous (old / non JGroups sync) behavior worked. >> >> It of course also works without this async config, >> but in this case we don't need sync / ACK JGroups message. >> (we didn't have one before and it worked ;-) >> >> -Ales >> >> On Apr 11, 2013, at 11:51 PM, Sanne Grinovero <sanne(a)hibernate.org&gt; wrote: >> >>> There is a "blackhole" indexing backend, which pipes all indexing >>> requests > /dev/null >>> >>> Set this as an Infinispan Query configuration property: >>> >>> default.worker.backend = blackhole >>> >>> Of course that means that the index will not be updated: you might >>> need to adapt your test to tolerate that, but the point is not >>> functional testing but to verify how much the SYNC option on the >>> JGroups backend is actually slowing you down. I suspect the >>> performance penalty is not in the network but in the fact you're now >>> waiting for the index operations, while in async you where not waiting >>> for them to be flushed. >>> >>> If you can identify which part is slow, then we can help you with >>> better configuration options. >>> >>> >>> On 11 April 2013 20:47, Ales Justin <ales.justin(a)gmail.com&gt; wrote: >>>> What do you mean? >>>> >>>> On Apr 11, 2013, at 21:41, Sanne Grinovero <sanne(a)hibernate.org&gt; wrote: >>>> >>>> You could try the new sync version but setting the blackhole backend on the >>>> master node to remove the indexing overhead from the picture. >>>> >>>> On Apr 11, 2013 8:39 PM, "Sanne Grinovero" <sanne(a)hibernate.org&gt; wrote: >>>>> >>>>> Are you sure that the async version actually had applied all writes to the >>>>> index in the measured interval? >>>>> >>>>> On Apr 11, 2013 8:13 PM, "Ales Justin" <ales.justin(a)gmail.com&gt; wrote: >>>>>> >>>>>> Although this change fixes query lookup, >>>>>> it adds horrible performance: >>>>>> >>>>>> Running CapeDwarf cluster QueryTest: >>>>>> >>>>>> with HSEARCH-1296 >>>>>> >>>>>> 21:00:27,188 INFO >>>>>> [org.hibernate.search.indexes.impl.DirectoryBasedIndexManager] >>>>>> (http-/192.168.1.102:8080-1) HSEARCH000168: Serialization service Avro >>>>>> SerializationProvider v1.0 being used for index >>>>>> 'default_capedwarf-test__com.google.appengine.api.datastore.Entity' >>>>>> 21:01:17,911 INFO [org.jboss.web] (ServerService Thread Pool -- 49) >>>>>> JBAS018224: Unregister web context: /capedwarf-tests >>>>>> >>>>>> 50sec >>>>>> >>>>>> old 4.2.0.Final HS >>>>>> >>>>>> 21:08:19,988 INFO >>>>>> [org.hibernate.search.indexes.impl.DirectoryBasedIndexManager] >>>>>> (http-/192.168.1.102:8080-2) HSEARCH000168: Serialization service Avro >>>>>> SerializationProvider v1.0 being used for index >>>>>> 'default_capedwarf-test__com.google.appengine.api.datastore.Entity' >>>>>> 21:08:20,829 INFO [org.jboss.web] (ServerService Thread Pool -- 49) >>>>>> JBAS018224: Unregister web context: /capedwarf-tests >>>>>> >>>>>> 841ms >>>>>> >>>>>> --- >>>>>> >>>>>> I added >>>>>> >>>>>> <property name="enable_bundling">true</property> >>>>>> >>>>>> to AS jgroups transport config, but no improvement. >>>>>> >>>>>> Any (other) idea? >>>>>> >>>>>> -Ales >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> hibernate-dev mailing list >>>>>> hibernate-dev(a)lists.jboss.org >>>>>> https://lists.jboss.org/mailman/listinfo/hibernate-dev >> >> >> _______________________________________________ >> hibernate-dev mailing list >> hibernate-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/hibernate-dev

Are you sure that the async version actually had applied all writes to the index in the measured interval? On Apr 11, 2013 8:13 PM, "Ales Justin" <ales.justin(a)gmail.com&gt; wrote: > Although this change fixes query lookup, > it adds horrible performance: > > Running CapeDwarf cluster QueryTest: > > with HSEARCH-1296 > > 21:00:27,188 INFO [org.hibernate.search.indexes.impl.DirectoryBasedIndexManager] (http-/192.168.1.102:8080-1) HSEARCH000168: Serialization service Avro SerializationProvider v1.0 being used for index 'default_capedwarf-test__com.google.appengine.api.datastore.Entity' > 21:01:17,911 INFO [org.jboss.web] (ServerService Thread Pool -- 49) JBAS018224: Unregister web context: /capedwarf-tests > > 50sec > > old 4.2.0.Final HS > > 21:08:19,988 INFO [org.hibernate.search.indexes.impl.DirectoryBasedIndexManager] (http-/192.168.1.102:8080-2) HSEARCH000168: Serialization service Avro SerializationProvider v1.0 being used for index 'default_capedwarf-test__com.google.appengine.api.datastore.Entity' > 21:08:20,829 INFO [org.jboss.web] (ServerService Thread Pool -- 49) JBAS018224: Unregister web context: /capedwarf-tests > > 841ms > > --- > > I added > > <property name="enable_bundling">true</property> > > to AS jgroups transport config, but no improvement. > > Any (other) idea? > > -Ales > > > _______________________________________________ > hibernate-dev mailing list > hibernate-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/hibernate-dev