[infinispan-dev] [hibernate-dev] HSEARCH-1296

Mon Apr 15 04:50:46 EDT 2013

On Sat, Apr 13, 2013 at 3:02 AM, Sanne Grinovero <sanne at hibernate.org>wrote:

> that's right, as suggested by Emmanuel I plan to separate the JGroups
> Sync/Async options from the worker.execution property so you can play
> with the two independently.
> I think the JGroups option's default could depend on the backend - if
> not otherwise specified, and if we all agree it doesn't make it too
> confusing.
>
> @All, the performance problem seemed to be caused by a problem in
> JGroups, which I've logged here:
> https://issues.jboss.org/browse/JGRP-1617
>
> For the record, the first operation was indeed triggering some lazy
> initialization of indexes, which in turn would trigger a Lucene
> Directory being started, triggering 3 Cache starts which in turn would
> trigger 6 state transfer processes: so indeed the first operation
> would not be exactly "cheap" performance wise, still this would
> complete in about 120 milliseconds.
> The same cost is paid again when the second node is hit the first
> time, after that index write operations block the writer for <1ms (not
> investigated further on potential throughput).
>
> Not being sure about the options of depending to a newer JGroups
> release or the complexity of a fix, I'll implement a workaround in
> HSearch in the scope of HSEARCH-1296.
>
> As a lesson learned, I think we need to polish some of our TRACE level
> messaged to include the cache name:

Sanne, we already push the cache name in the NDC if trace is enabled for
the "entry point" of the thread. So for your purpose, I think enabling
trace for org.infinispan.interceptors.InvocationContextInterceptor and
including %x in your pattern layout should work.

> to resolve this we had not just
> many threads and components but also 4 of them where using JGroups
> (interleaving messages of all sorts) and 9 different caches where
> involved for each simple write operation in CD: made it interesting to
> figure what was going on! Also I'm wondering how hard it would be to
> have a log parser which converts my 10GB of text log from today in a
> graphical sequence diagram.
> Big thanks to Mircea who helped me figuring this out.
>
> Sanne
>
> On 12 April 2013 21:10, Ales Justin <ales.justin at gmail.com> wrote:
> > I think we need more fine-grained config for this new JGroups sync
> feature.
> >
> > I added this to our cache config
> >
> >                         <property
> name="hibernate.search.default.worker.execution">async</property>
> >
> > and it broke our tests.
> >
> > Where previous (old / non JGroups sync) behavior worked.
> >
> > It of course also works  without this async config,
> > but in this case we don't need sync / ACK JGroups message.
> > (we didn't have one before and it worked ;-)
> >
> > -Ales
> >
> > On Apr 11, 2013, at 11:51 PM, Sanne Grinovero <sanne at hibernate.org>
> wrote:
> >
> >> There is a "blackhole" indexing backend, which pipes all indexing
> >> requests > /dev/null
> >>
> >> Set this as an Infinispan Query configuration property:
> >>
> >>    default.worker.backend = blackhole
> >>
> >> Of course that means that the index will not be updated: you might
> >> need to adapt your test to tolerate that, but the point is not
> >> functional testing but to verify how much the SYNC option on the
> >> JGroups backend is actually slowing you down. I suspect the
> >> performance penalty is not in the network but in the fact you're now
> >> waiting for the index operations, while in async you where not waiting
> >> for them to be flushed.
> >>
> >> If you can identify which part is slow, then we can help you with
> >> better configuration options.
> >>
> >>
> >> On 11 April 2013 20:47, Ales Justin <ales.justin at gmail.com> wrote:
> >>> What do you mean?
> >>>
> >>> On Apr 11, 2013, at 21:41, Sanne Grinovero <sanne at hibernate.org>
> wrote:
> >>>
> >>> You could try the new sync version but setting the blackhole backend
> on the
> >>> master node to remove the indexing overhead from the picture.
> >>>
> >>> On Apr 11, 2013 8:39 PM, "Sanne Grinovero" <sanne at hibernate.org>
> wrote:
> >>>>
> >>>> Are you sure that the async version actually had applied all writes
> to the
> >>>> index in the measured interval?
> >>>>
> >>>> On Apr 11, 2013 8:13 PM, "Ales Justin" <ales.justin at gmail.com> wrote:
> >>>>>
> >>>>> Although this change fixes query lookup,
> >>>>> it adds horrible performance:
> >>>>>
> >>>>> Running CapeDwarf cluster QueryTest:
> >>>>>
> >>>>> with HSEARCH-1296
> >>>>>
> >>>>> 21:00:27,188 INFO
> >>>>> [org.hibernate.search.indexes.impl.DirectoryBasedIndexManager]
> >>>>> (http-/192.168.1.102:8080-1) HSEARCH000168: Serialization service
> Avro
> >>>>> SerializationProvider v1.0 being used for index
> >>>>> 'default_capedwarf-test__com.google.appengine.api.datastore.Entity'
> >>>>> 21:01:17,911 INFO  [org.jboss.web] (ServerService Thread Pool -- 49)
> >>>>> JBAS018224: Unregister web context: /capedwarf-tests
> >>>>>
> >>>>> 50sec
> >>>>>
> >>>>> old 4.2.0.Final HS
> >>>>>
> >>>>> 21:08:19,988 INFO
> >>>>> [org.hibernate.search.indexes.impl.DirectoryBasedIndexManager]
> >>>>> (http-/192.168.1.102:8080-2) HSEARCH000168: Serialization service
> Avro
> >>>>> SerializationProvider v1.0 being used for index
> >>>>> 'default_capedwarf-test__com.google.appengine.api.datastore.Entity'
> >>>>> 21:08:20,829 INFO  [org.jboss.web] (ServerService Thread Pool -- 49)
> >>>>> JBAS018224: Unregister web context: /capedwarf-tests
> >>>>>
> >>>>> 841ms
> >>>>>
> >>>>> ---
> >>>>>
> >>>>> I added
> >>>>>
> >>>>>                    <property name="enable_bundling">true</property>
> >>>>>
> >>>>> to AS jgroups transport config, but no improvement.
> >>>>>
> >>>>> Any (other) idea?
> >>>>>
> >>>>> -Ales
> >>>>>
> >>>>>
> >>>>> _______________________________________________
> >>>>> hibernate-dev mailing list
> >>>>> hibernate-dev at lists.jboss.org
> >>>>> https://lists.jboss.org/mailman/listinfo/hibernate-dev
> >
> >
> > _______________________________________________
> > hibernate-dev mailing list
> > hibernate-dev at lists.jboss.org
> > https://lists.jboss.org/mailman/listinfo/hibernate-dev
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/infinispan-dev/attachments/20130415/21e80804/attachment.html 