[hibernate-dev] HSEARCH-1296

[Sanne Grinovero]

that's right, as suggested by Emmanuel I plan to separate the JGroups
Sync/Async options from the worker.execution property so you can play
with the two independently.
I think the JGroups option's default could depend on the backend - if
not otherwise specified, and if we all agree it doesn't make it too
confusing.

@All, the performance problem seemed to be caused by a problem in
JGroups, which I've logged here:
https://issues.jboss.org/browse/JGRP-1617

For the record, the first operation was indeed triggering some lazy
initialization of indexes, which in turn would trigger a Lucene
Directory being started, triggering 3 Cache starts which in turn would
trigger 6 state transfer processes: so indeed the first operation
would not be exactly "cheap" performance wise, still this would
complete in about 120 milliseconds.
The same cost is paid again when the second node is hit the first
time, after that index write operations block the writer for <1ms (not
investigated further on potential throughput).

Not being sure about the options of depending to a newer JGroups
release or the complexity of a fix, I'll implement a workaround in
HSearch in the scope of HSEARCH-1296.

As a lesson learned, I think we need to polish some of our TRACE level
messaged to include the cache name: to resolve this we had not just
many threads and components but also 4 of them where using JGroups
(interleaving messages of all sorts) and 9 different caches where
involved for each simple write operation in CD: made it interesting to
figure what was going on! Also I'm wondering how hard it would be to
have a log parser which converts my 10GB of text log from today in a
graphical sequence diagram.
Big thanks to Mircea who helped me figuring this out.

Sanne

On 12 April 2013 21:10, Ales Justin <ales.justin at gmail.com> wrote:
> I think we need more fine-grained config for this new JGroups sync feature.
>
> I added this to our cache config
>
>                         <property name="hibernate.search.default.worker.execution">async</property>
>
> and it broke our tests.
>
> Where previous (old / non JGroups sync) behavior worked.
>
> It of course also works  without this async config,
> but in this case we don't need sync / ACK JGroups message.
> (we didn't have one before and it worked ;-)
>
> -Ales
>
> On Apr 11, 2013, at 11:51 PM, Sanne Grinovero <sanne at hibernate.org> wrote:
>
>> There is a "blackhole" indexing backend, which pipes all indexing
>> requests > /dev/null
>>
>> Set this as an Infinispan Query configuration property:
>>
>>    default.worker.backend = blackhole
>>
>> Of course that means that the index will not be updated: you might
>> need to adapt your test to tolerate that, but the point is not
>> functional testing but to verify how much the SYNC option on the
>> JGroups backend is actually slowing you down. I suspect the
>> performance penalty is not in the network but in the fact you're now
>> waiting for the index operations, while in async you where not waiting
>> for them to be flushed.
>>
>> If you can identify which part is slow, then we can help you with
>> better configuration options.
>>
>>
>> On 11 April 2013 20:47, Ales Justin <ales.justin at gmail.com> wrote:
>>> What do you mean?
>>>
>>> On Apr 11, 2013, at 21:41, Sanne Grinovero <sanne at hibernate.org> wrote:
>>>
>>> You could try the new sync version but setting the blackhole backend on the
>>> master node to remove the indexing overhead from the picture.
>>>
>>> On Apr 11, 2013 8:39 PM, "Sanne Grinovero" <sanne at hibernate.org> wrote:
>>>>
>>>> Are you sure that the async version actually had applied all writes to the
>>>> index in the measured interval?
>>>>
>>>> On Apr 11, 2013 8:13 PM, "Ales Justin" <ales.justin at gmail.com> wrote:
>>>>>
>>>>> Although this change fixes query lookup,
>>>>> it adds horrible performance:
>>>>>
>>>>> Running CapeDwarf cluster QueryTest:
>>>>>
>>>>> with HSEARCH-1296
>>>>>
>>>>> 21:00:27,188 INFO
>>>>> [org.hibernate.search.indexes.impl.DirectoryBasedIndexManager]
>>>>> (http-/192.168.1.102:8080-1) HSEARCH000168: Serialization service Avro
>>>>> SerializationProvider v1.0 being used for index
>>>>> 'default_capedwarf-test__com.google.appengine.api.datastore.Entity'
>>>>> 21:01:17,911 INFO  [org.jboss.web] (ServerService Thread Pool -- 49)
>>>>> JBAS018224: Unregister web context: /capedwarf-tests
>>>>>
>>>>> 50sec
>>>>>
>>>>> old 4.2.0.Final HS
>>>>>
>>>>> 21:08:19,988 INFO
>>>>> [org.hibernate.search.indexes.impl.DirectoryBasedIndexManager]
>>>>> (http-/192.168.1.102:8080-2) HSEARCH000168: Serialization service Avro
>>>>> SerializationProvider v1.0 being used for index
>>>>> 'default_capedwarf-test__com.google.appengine.api.datastore.Entity'
>>>>> 21:08:20,829 INFO  [org.jboss.web] (ServerService Thread Pool -- 49)
>>>>> JBAS018224: Unregister web context: /capedwarf-tests
>>>>>
>>>>> 841ms
>>>>>
>>>>> ---
>>>>>
>>>>> I added
>>>>>
>>>>>                    <property name="enable_bundling">true</property>
>>>>>
>>>>> to AS jgroups transport config, but no improvement.
>>>>>
>>>>> Any (other) idea?
>>>>>
>>>>> -Ales
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> hibernate-dev mailing list
>>>>> hibernate-dev at lists.jboss.org
>>>>> https://lists.jboss.org/mailman/listinfo/hibernate-dev
>
>
> _______________________________________________
> hibernate-dev mailing list
> hibernate-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/hibernate-dev