[infinispan-dev] HSEARCH-1296

Ales Justin ales.justin at gmail.com
Fri Apr 12 09:03:06 EDT 2013 

In this particular case, there shouldn't be any other indexing going on.

Our 5 indexable caches:

"default" - Datastore
"logs" -- logging
"search" -- text search
"prospective_search" -- some weird search :-)
"tasks" -- JMS-like stuff

We disable logging for this test, and there is no Search, PSearch or Queue taking place.

Hence the only ops should be those 4 ops to "default" cache.

We did see huge perf drop when trace logging is enabled,
and that wouldn't surprise me then, but that's not the case here.

But 4 ops, for 50sec ...

On Apr 12, 2013, at 2:41 PM, Emmanuel Bernard <emmanuel at hibernate.org> wrote:

> The index master is essentially a queue + the queue processor.
> So if many people as the index master to index data (even as async).
> the sync call will have to wait for all the other ones to be done before
> carrying on.
> But are the other put involving indexed entities? If not, then that's
> not what is going on.
> 
> I can't explain one index operation taking 50s if nothing is happening
> in parallel.
> As Sanne said, use the blackhole backend on your index master node. We
> will know if the index is really taking all that time.
> 
> Emmanuel
> 
> On Fri 2013-04-12 14:19, Ales Justin wrote:
>> To our "default" cache - which is where we store Datastore entries,
>> the test only does 1 put, 1 get, 1 delete, 1 query.
>> 
>> https://github.com/capedwarf/capedwarf-blue/blob/master/cluster-tests/src/test/java/org/jboss/test/capedwarf/cluster/test/QueryTest.java
>> 
>> There are other indexes that participate in this test as well, "logging" cache, etc.
>> But only "default" is configured with new "sync" stuff,
>> others all have this:
>> 
>>                    <indexing index="LOCAL">
>>                        <property name="hibernate.search.default.worker.execution">async</property>
>> 
>> So I fail to see how this is "bunch of them".
>> 
>> -Ales
>> 
>> On Apr 12, 2013, at 2:12 PM, Emmanuel Bernard <emmanuel at hibernate.org> wrote:
>> 
>>> No, it is one RPC call for every put or delete in the grid. I'm sure the
>>> test makes a bunch of them.
>>> 
>>> On Fri 2013-04-12 12:40, Bela Ban wrote:
>>>> I might have misunderstood something: if this is *one* sync RPC, then 
>>>> enabling or disabling of bundling won't have an impact; as 60ms added to 
>>>> 50 secs doesn't make a diff.
>>>> 
>>>> On 4/12/13 12:20 PM, Sanne Grinovero wrote:
>>>>> +1
>>>>> I'm going to verify that. Some index tuning options won't hurt either,
>>>>> but first I'll set the blackhole to confirm that indexing is the cause.
>>>>> 
>>>>> On Apr 12, 2013 11:12 AM, "Bela Ban" <bban at redhat.com
>>>>> <mailto:bban at redhat.com>> wrote:
>>>>> 
>>>>>   OK, then it isn't JGroups related. Probably some indexing work done by
>>>>>   HS is on the critical path now, and wasn't before. The test probably
>>>>>   returned before that work was done.
>>>>> 
>>>>>   On 4/12/13 11:21 AM, Ales Justin wrote:
>>>>>> Still the same -- after changing this to false.
>>>>>> 
>>>>>> 11:20:04,013 INFO
>>>>>     [org.hibernate.search.indexes.impl.DirectoryBasedIndexManager]
>>>>>   (http-/192.168.30.235:8080-2) HSEARCH000168: Serialization service
>>>>>   Avro SerializationProvider v1.0 being used for index
>>>>>   'default_capedwarf-test__com.google.appengine.api.datastore.Entity'
>>>>>> 11:20:54,730 INFO  [org.jboss.web] (ServerService Thread Pool --
>>>>>   59) JBAS018224: Unregister web context: /capedwarf-tests
>>>>>> 
>>>>>> 
>>>>>> On Apr 12, 2013, at 9:43 AM, Bela Ban <bban at redhat.com
>>>>>   <mailto:bban at redhat.com>> wrote:
>>>>>> 
>>>>>>> You need to set enable_bundling to *false*, not true !
>>>>>>> 
>>>>>>> On 4/11/13 9:13 PM, Ales Justin wrote:
>>>>>>>> Although this change fixes query lookup,
>>>>>>>> it adds horrible performance:
>>>>>>>> 
>>>>>>>> Running CapeDwarf cluster QueryTest:
>>>>>>>> 
>>>>>>>> with HSEARCH-1296
>>>>>>>> 
>>>>>>>> 21:00:27,188 INFO
>>>>>     [org.hibernate.search.indexes.impl.DirectoryBasedIndexManager]
>>>>>   (http-/192.168.1.102:8080-1) HSEARCH000168: Serialization service
>>>>>   Avro SerializationProvider v1.0 being used for index
>>>>>   'default_capedwarf-test__com.google.appengine.api.datastore.Entity'
>>>>>>>> 21:01:17,911 INFO  [org.jboss.web] (ServerService Thread Pool
>>>>>   -- 49) JBAS018224: Unregister web context: /capedwarf-tests
>>>>>>>> 
>>>>>>>> 50sec
>>>>>>>> 
>>>>>>>> old 4.2.0.Final HS
>>>>>>>> 
>>>>>>>> 21:08:19,988 INFO
>>>>>     [org.hibernate.search.indexes.impl.DirectoryBasedIndexManager]
>>>>>   (http-/192.168.1.102:8080-2) HSEARCH000168: Serialization service
>>>>>   Avro SerializationProvider v1.0 being used for index
>>>>>   'default_capedwarf-test__com.google.appengine.api.datastore.Entity'
>>>>>>>> 21:08:20,829 INFO  [org.jboss.web] (ServerService Thread Pool
>>>>>   -- 49) JBAS018224: Unregister web context: /capedwarf-tests
>>>>>>>> 
>>>>>>>> 841ms
>>>>>>>> 
>>>>>>>> ---
>>>>>>>> 
>>>>>>>> I added
>>>>>>>> 
>>>>>>>>                    <property
>>>>>   name="enable_bundling">true</property>
>>>>>>>> 
>>>>>>>> to AS jgroups transport config, but no improvement.
>>>>>>>> 
>>>>>>>> Any (other) idea?
>>>>>>>> 
>>>>>>>> -Ales
>>>>>>>> 
>>>>>>>> 
>>>>>>>> _______________________________________________
>>>>>>>> infinispan-dev mailing list
>>>>>>>> infinispan-dev at lists.jboss.org
>>>>>   <mailto:infinispan-dev at lists.jboss.org>
>>>>>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>>>>>>> 
>>>>>>> 
>>>>>>> --
>>>>>>> Bela Ban, JGroups lead (http://www.jgroups.org)
>>>>>>> _______________________________________________
>>>>>>> infinispan-dev mailing list
>>>>>>> infinispan-dev at lists.jboss.org
>>>>>   <mailto:infinispan-dev at lists.jboss.org>
>>>>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>>>>> 
>>>>>> 
>>>>>> _______________________________________________
>>>>>> infinispan-dev mailing list
>>>>>> infinispan-dev at lists.jboss.org
>>>>>   <mailto:infinispan-dev at lists.jboss.org>
>>>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>>>>> 
>>>>> 
>>>>>   --
>>>>>   Bela Ban, JGroups lead (http://www.jgroups.org)
>>>>>   _______________________________________________
>>>>>   infinispan-dev mailing list
>>>>>   infinispan-dev at lists.jboss.org <mailto:infinispan-dev at lists.jboss.org>
>>>>>   https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>>>> 
>>>>> 
>>>>> 
>>>>> _______________________________________________
>>>>> infinispan-dev mailing list
>>>>> infinispan-dev at lists.jboss.org
>>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>>>> 
>>>> 
>>>> -- 
>>>> Bela Ban, JGroups lead (http://www.jgroups.org)
>>>> _______________________________________________
>>>> infinispan-dev mailing list
>>>> infinispan-dev at lists.jboss.org
>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>> _______________________________________________
>>> infinispan-dev mailing list
>>> infinispan-dev at lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>> 
> 
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> 
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev

More information about the infinispan-dev mailing list