[infinispan-issues] [JBoss JIRA] (ISPN-5452) Query Execution using Hibernate Search slow for large volume data

[Prashant Thakur (JIRA)]

    [ https://issues.jboss.org/browse/ISPN-5452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13075320#comment-13075320 ] 

Prashant Thakur edited comment on ISPN-5452 at 6/8/15 7:26 AM:
---------------------------------------------------------------

Thanks Sanne,

We will do some profiling at application end and update this thread. Most of your points make lot of sense. The strategy we follow is to start with single client thread and then proceed with increasing the number of threads to get to optimal figure on number of clients. Similarily increasing number of worker threads at Infinispan level also helps. We have not seen extreme CPU utilization at infinispan side even when we saw 80k/sec kind of performance the cpu utilization was within 20-25%, Hence we are a bit optimistic on getting to a higher value if somehow we can decrease latency by increasing CPU utilization.

We are doing two levels of marshalling and also bringing in extension to data container in between which might slow down things a bit which we can confirm from profiling each in isolation. 



was (Author: prashant.thakur):
Thanks Sanne,

We will do some profiling at application end and update this thread. Most of your points make lot of sense. The strategy we follow is to start with single client thread and then proceed with increasing the number of threads to get to optimal figure on number of clients. Similarily increasing number of worker threads at Infinispan level also helps. We have not seen extreme CPU utilization end at infinispan side even when we saw 80k/sec kind of performance the cpu utilization was within 20-25%, Hence we are a bit optimistic on getting to a higher value if somehow we can decrease latency by increasing CPU utilization.

We are doing two levels of marshalling and also bringing in extension to data container in between which might slow down things a bit which we can confirm from profiling each in isolation. 


> Query Execution using Hibernate Search slow for large volume data
> -----------------------------------------------------------------
>
>                 Key: ISPN-5452
>                 URL: https://issues.jboss.org/browse/ISPN-5452
>             Project: Infinispan
>          Issue Type: Bug
>          Components: Configuration, Remote Querying
>    Affects Versions: 7.2.1.Final
>         Environment: Linux
>            Reporter: Prashant Thakur
>
> While benchmarking Infinispan we found that Querying is very slow when compared with Hibernate Search in Isolation
> Single node of Infinispan  
> Memory allocated 230GB. No GC seen throughout query operation.
> Total required after full GC was 122GB.
> Setup 240 million records each of avg size 330 bytes .
> System has 16 cores and 40 worker threads were allocated at server side.
> With Single Client thread throughput was 900 req/sec in remote and 3k per sec in embedded more same request with Hibernate Search in Isolation gives throughput of 14000 req/sec.
> For 50 threads of clients the throughput was limited to 15k req/sec while hibernate search gives 80k req/sec for 10 threads.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)