[JBoss JIRA] (ISPN-5452) Query Execution using Hibernate Search slow for large volume data
by Sanne Grinovero (JIRA)
[ https://issues.jboss.org/browse/ISPN-5452?page=com.atlassian.jira.plugin.... ]
Sanne Grinovero commented on ISPN-5452:
---------------------------------------
Thanks [~prashant.thakur]!
I've formatted the examples a bit, could you confirm which performance results you get for each of cases 2, 3 and 4?
The difference between 2 and 3 is that - of course - it goes over Hot Rod.
The difference between example 3 and 4 is that the query string is parsed, but it's also possible that we generate a slightly different query.
> Query Execution using Hibernate Search slow for large volume data
> -----------------------------------------------------------------
>
> Key: ISPN-5452
> URL: https://issues.jboss.org/browse/ISPN-5452
> Project: Infinispan
> Issue Type: Bug
> Components: Configuration, Remote Querying
> Affects Versions: 7.2.1.Final
> Environment: Linux
> Reporter: Prashant Thakur
>
> While benchmarking Infinispan we found that Querying is very slow when compared with Hibernate Search in Isolation
> Single node of Infinispan
> Memory allocated 230GB. No GC seen throughout query operation.
> Total required after full GC was 122GB.
> Setup 240 million records each of avg size 330 bytes .
> System has 16 cores and 40 worker threads were allocated at server side.
> With Single Client thread throughput was 900 req/sec in remote and 3k per sec in embedded more same request with Hibernate Search in Isolation gives throughput of 14000 req/sec.
> For 50 threads of clients the throughput was limited to 15k req/sec while hibernate search gives 80k req/sec for 10 threads.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
10 years, 11 months
[JBoss JIRA] (ISPN-5452) Query Execution using Hibernate Search slow for large volume data
by Sanne Grinovero (JIRA)
[ https://issues.jboss.org/browse/ISPN-5452?page=com.atlassian.jira.plugin.... ]
Sanne Grinovero updated ISPN-5452:
----------------------------------
Steps to Reproduce:
1. Create a Batch job to Create 240 million entries in Infinispan with all entries Indexed
2. Execute the following query through hotrod remote client
{code:java}String queryString= "SELECT ACCOUNT_ID,ID,PRODUCT_TYPE,FIRST_CALL,ID,IMEI from com.subex.spark.common.distributedcaching.data.Subscriber where PHONE_NUMBER='" +prefix+ i+"' AND STATUS in (1,2) and SUBSCRIBER_TYPE = 0";
QueryFactory qf = Search.getQueryFactory(distcacheclient.getRemoteCache());
RemoteQuery remoteQuery = new RemoteQuery(qf,(RemoteCacheImpl)distcacheclient.getRemoteCache(), distcacheclient.getSerializationContext(), queryString, 0, (int)numOfRecords);
List<Subscriber> list = remoteQuery.list();{code}
3. Following Query was executed in Embedded mode
{code:java}String queryString= "SELECT ACCOUNT_ID,ID,PRODUCT_TYPE,FIRST_CALL,ID,IMEI from com.subex.spark.common.distributedcaching.data.Subscriber where " + "PHONE_NUMBER='" +prefix+ i+"' AND STATUS in (1,2) and SUBSCRIBER_TYPE = 0";
QueryFactory qf = org.infinispan.query.Search.getQueryFactory(ispnCacheServer.getCache("SUBSCRIBER"));
Query query = qf.from("com.subex.spark.common.distributedcaching.data.Subscriber").setProjection("ACCOUNT_ID","ID","PRODUCT_TYPE","FIRST_CALL","ID","IMEI").
having("PHONE_NUMBER").eq(prefix+ i).and().having("STATUS").in(1,2).and().having("SUBSCRIBER_TYPE").eq(0).toBuilder().build();
List<Subscriber> list = query.list();
rs = query.list().iterator();{code}
4. Hibernate Search Query
{code}QueryContextBuilder queryBuilder = searchSession.getSearchFactory().buildQueryBuilder();
Query query = queryBuilder.forEntity(Subscriber.class).get().keyword().onField("ID").matching(Long.parseLong(prefix+i)).createQuery();
org.hibernate.Query hibernateQuery=searchSession.createFullTextQuery(query, Subscriber.class).setProjection("ACCOUNT_ID","ACCOUNT_NAME","CONNECTION_TYPE","CONTACT_PHONE_NUMBER","CURRENT_BALANCE","CUST_ALERT_CONTACT_NUMBER","CUST_ALERT_EMAIL_ID","DATE_OF_BIRTH","DEALER_NAME","DS_NAME","FILE_NAME","FIRST_CALL","GROUPS","PVN","HOME_PHONE_NUMBER","ID","ID_NUMBER","IMEI","IMSI","IS_UPDATE","MCN1","MCN2","MODIFIED_DATE","NETWORK_ID","NOTIFICATION_GROUPS","OFFICE_PHONE_NUMBER","OPTIONAL_FIELD_1","OPTIONAL_FIELD_10","OPTIONAL_FIELD_11","OPTIONAL_FIELD_12","OPTIONAL_FIELD_13","OPTIONAL_FIELD_14","OPTIONAL_FIELD_15","OPTIONAL_FIELD_2","OPTIONAL_FIELD_3","OPTIONAL_FIELD_4","OPTIONAL_FIELD_5","OPTIONAL_FIELD_6","OPTIONAL_FIELD_7","OPTIONAL_FIELD_8","OPTIONAL_FIELD_9","PHONE_NUMBER","PRODUCT_TYPE","QOS","SERVICES","SERVICE_NUMBER_TYPE","SSID","STATUS","SUBSCRIBER_DOA","SUBSCRIBER_TYPE","SUBSCRIBER_UID","SUBSCRIBER_UID_DOA"); {code}
This contains the list of all fields in the Cache all of which are indexed.
was:
1. Create a Batch job to Create 240 million entries in Infinispan with all entries Indexed
2. Execute the following query through hotrod remote client
{code:java}String queryString= "SELECT ACCOUNT_ID,ID,PRODUCT_TYPE,FIRST_CALL,ID,IMEI from com.subex.spark.common.distributedcaching.data.Subscriber where PHONE_NUMBER='" +prefix+ i+"' AND STATUS in (1,2) and SUBSCRIBER_TYPE = 0";
QueryFactory qf = Search.getQueryFactory(distcacheclient.getRemoteCache());
RemoteQuery remoteQuery = new RemoteQuery(qf,(RemoteCacheImpl)distcacheclient.getRemoteCache(), distcacheclient.getSerializationContext(), queryString, 0, (int)numOfRecords);
List<Subscriber> list = remoteQuery.list();{code}
3. Following Query was executed in Embedded mode
{code:java}String queryString= "SELECT ACCOUNT_ID,ID,PRODUCT_TYPE,FIRST_CALL,ID,IMEI from com.subex.spark.common.distributedcaching.data.Subscriber where " + "PHONE_NUMBER='" +prefix+ i+"' AND STATUS in (1,2) and SUBSCRIBER_TYPE = 0";
QueryFactory qf = org.infinispan.query.Search.getQueryFactory(ispnCacheServer.getCache("SUBSCRIBER"));
Query query = qf.from("com.subex.spark.common.distributedcaching.data.Subscriber").setProjection("ACCOUNT_ID","ID","PRODUCT_TYPE","FIRST_CALL","ID","IMEI").
having("PHONE_NUMBER").eq(prefix+ i).and().having("STATUS").in(1,2).and().having("SUBSCRIBER_TYPE").eq(0).toBuilder().build();
List<Subscriber> list = query.list();
rs = query.list().iterator();{code}
4. Hibernate Search Query
{code}QueryContextBuilder queryBuilder = searchSession.getSearchFactory().buildQueryBuilder();
Query query = queryBuilder.forEntity(Subscriber.class).get().keyword().onField("ID").matching(Long.parseLong(prefix+i)).createQuery();{code}
org.hibernate.Query hibernateQuery=searchSession.createFullTextQuery(query, Subscriber.class).setProjection("ACCOUNT_ID","ACCOUNT_NAME","CONNECTION_TYPE","CONTACT_PHONE_NUMBER","CURRENT_BALANCE","CUST_ALERT_CONTACT_NUMBER","CUST_ALERT_EMAIL_ID","DATE_OF_BIRTH","DEALER_NAME","DS_NAME","FILE_NAME","FIRST_CALL","GROUPS","PVN","HOME_PHONE_NUMBER","ID","ID_NUMBER","IMEI","IMSI","IS_UPDATE","MCN1","MCN2","MODIFIED_DATE","NETWORK_ID","NOTIFICATION_GROUPS","OFFICE_PHONE_NUMBER","OPTIONAL_FIELD_1","OPTIONAL_FIELD_10","OPTIONAL_FIELD_11","OPTIONAL_FIELD_12","OPTIONAL_FIELD_13","OPTIONAL_FIELD_14","OPTIONAL_FIELD_15","OPTIONAL_FIELD_2","OPTIONAL_FIELD_3","OPTIONAL_FIELD_4","OPTIONAL_FIELD_5","OPTIONAL_FIELD_6","OPTIONAL_FIELD_7","OPTIONAL_FIELD_8","OPTIONAL_FIELD_9","PHONE_NUMBER","PRODUCT_TYPE","QOS","SERVICES","SERVICE_NUMBER_TYPE","SSID","STATUS","SUBSCRIBER_DOA","SUBSCRIBER_TYPE","SUBSCRIBER_UID","SUBSCRIBER_UID_DOA"); {code}
This contains the list of all fields in the Cache all of which are indexed.
> Query Execution using Hibernate Search slow for large volume data
> -----------------------------------------------------------------
>
> Key: ISPN-5452
> URL: https://issues.jboss.org/browse/ISPN-5452
> Project: Infinispan
> Issue Type: Bug
> Components: Configuration, Remote Querying
> Affects Versions: 7.2.1.Final
> Environment: Linux
> Reporter: Prashant Thakur
>
> While benchmarking Infinispan we found that Querying is very slow when compared with Hibernate Search in Isolation
> Single node of Infinispan
> Memory allocated 230GB. No GC seen throughout query operation.
> Total required after full GC was 122GB.
> Setup 240 million records each of avg size 330 bytes .
> System has 16 cores and 40 worker threads were allocated at server side.
> With Single Client thread throughput was 900 req/sec in remote and 3k per sec in embedded more same request with Hibernate Search in Isolation gives throughput of 14000 req/sec.
> For 50 threads of clients the throughput was limited to 15k req/sec while hibernate search gives 80k req/sec for 10 threads.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
10 years, 11 months
[JBoss JIRA] (ISPN-5452) Query Execution using Hibernate Search slow for large volume data
by Sanne Grinovero (JIRA)
[ https://issues.jboss.org/browse/ISPN-5452?page=com.atlassian.jira.plugin.... ]
Sanne Grinovero updated ISPN-5452:
----------------------------------
Steps to Reproduce:
1. Create a Batch job to Create 240 million entries in Infinispan with all entries Indexed
2. Execute the following query through hotrod remote client
{code:java}String queryString= "SELECT ACCOUNT_ID,ID,PRODUCT_TYPE,FIRST_CALL,ID,IMEI from com.subex.spark.common.distributedcaching.data.Subscriber where PHONE_NUMBER='" +prefix+ i+"' AND STATUS in (1,2) and SUBSCRIBER_TYPE = 0";
QueryFactory qf = Search.getQueryFactory(distcacheclient.getRemoteCache());
RemoteQuery remoteQuery = new RemoteQuery(qf,(RemoteCacheImpl)distcacheclient.getRemoteCache(), distcacheclient.getSerializationContext(), queryString, 0, (int)numOfRecords);
List<Subscriber> list = remoteQuery.list();{code}
3. Following Query was executed in Embedded mode
{code:java}String queryString= "SELECT ACCOUNT_ID,ID,PRODUCT_TYPE,FIRST_CALL,ID,IMEI from com.subex.spark.common.distributedcaching.data.Subscriber where " + "PHONE_NUMBER='" +prefix+ i+"' AND STATUS in (1,2) and SUBSCRIBER_TYPE = 0";
QueryFactory qf = org.infinispan.query.Search.getQueryFactory(ispnCacheServer.getCache("SUBSCRIBER"));
Query query = qf.from("com.subex.spark.common.distributedcaching.data.Subscriber").setProjection("ACCOUNT_ID","ID","PRODUCT_TYPE","FIRST_CALL","ID","IMEI").
having("PHONE_NUMBER").eq(prefix+ i).and().having("STATUS").in(1,2).and().having("SUBSCRIBER_TYPE").eq(0).toBuilder().build();
List<Subscriber> list = query.list();
rs = query.list().iterator();{code}
4. Hibernate Search Query
{code}QueryContextBuilder queryBuilder = searchSession.getSearchFactory().buildQueryBuilder();
Query query = queryBuilder.forEntity(Subscriber.class).get().keyword().onField("ID").matching(Long.parseLong(prefix+i)).createQuery();{code}
org.hibernate.Query hibernateQuery=searchSession.createFullTextQuery(query, Subscriber.class).setProjection("ACCOUNT_ID","ACCOUNT_NAME","CONNECTION_TYPE","CONTACT_PHONE_NUMBER","CURRENT_BALANCE","CUST_ALERT_CONTACT_NUMBER","CUST_ALERT_EMAIL_ID","DATE_OF_BIRTH","DEALER_NAME","DS_NAME","FILE_NAME","FIRST_CALL","GROUPS","PVN","HOME_PHONE_NUMBER","ID","ID_NUMBER","IMEI","IMSI","IS_UPDATE","MCN1","MCN2","MODIFIED_DATE","NETWORK_ID","NOTIFICATION_GROUPS","OFFICE_PHONE_NUMBER","OPTIONAL_FIELD_1","OPTIONAL_FIELD_10","OPTIONAL_FIELD_11","OPTIONAL_FIELD_12","OPTIONAL_FIELD_13","OPTIONAL_FIELD_14","OPTIONAL_FIELD_15","OPTIONAL_FIELD_2","OPTIONAL_FIELD_3","OPTIONAL_FIELD_4","OPTIONAL_FIELD_5","OPTIONAL_FIELD_6","OPTIONAL_FIELD_7","OPTIONAL_FIELD_8","OPTIONAL_FIELD_9","PHONE_NUMBER","PRODUCT_TYPE","QOS","SERVICES","SERVICE_NUMBER_TYPE","SSID","STATUS","SUBSCRIBER_DOA","SUBSCRIBER_TYPE","SUBSCRIBER_UID","SUBSCRIBER_UID_DOA"); {code}
This contains the list of all fields in the Cache all of which are indexed.
was:
1. Create a Batch job to Create 240 million entries in Infinispan with all entries Indexed
2. Execute the following query through hotrod remote client
{code:java}String queryString= "SELECT ACCOUNT_ID,ID,PRODUCT_TYPE,FIRST_CALL,ID,IMEI from com.subex.spark.common.distributedcaching.data.Subscriber where PHONE_NUMBER='" +prefix+ i+"' AND STATUS in (1,2) and SUBSCRIBER_TYPE = 0";
QueryFactory qf = Search.getQueryFactory(distcacheclient.getRemoteCache());
RemoteQuery remoteQuery = new RemoteQuery(qf,(RemoteCacheImpl)distcacheclient.getRemoteCache(), distcacheclient.getSerializationContext(), queryString, 0, (int)numOfRecords);
List<Subscriber> list = remoteQuery.list();{code}
3. Following Query was executed in Embedded mode
{code:java}String queryString= "SELECT ACCOUNT_ID,ID,PRODUCT_TYPE,FIRST_CALL,ID,IMEI from com.subex.spark.common.distributedcaching.data.Subscriber where " + "PHONE_NUMBER='" +prefix+ i+"' AND STATUS in (1,2) and SUBSCRIBER_TYPE = 0";
QueryFactory qf = org.infinispan.query.Search.getQueryFactory(ispnCacheServer.getCache("SUBSCRIBER"));
Query query = qf.from("com.subex.spark.common.distributedcaching.data.Subscriber").setProjection("ACCOUNT_ID","ID","PRODUCT_TYPE","FIRST_CALL","ID","IMEI").
having("PHONE_NUMBER").eq(prefix+ i).and().having("STATUS").in(1,2).and().having("SUBSCRIBER_TYPE").eq(0).toBuilder().build();
List<Subscriber> list = query.list();
rs = query.list().iterator();{code}
4. Hibernate Search Query
{code}QueryContextBuilder queryBuilder = searchSession.getSearchFactory().buildQueryBuilder();
Query query = queryBuilder.forEntity(Subscriber.class).get().keyword().onField("ID").matching(Long.parseLong(prefix+i)).createQuery();{code}
5.
{code}org.hibernate.Query hibernateQuery=searchSession.createFullTextQuery(query, Subscriber.class).setProjection("ACCOUNT_ID","ACCOUNT_NAME","CONNECTION_TYPE","CONTACT_PHONE_NUMBER","CURRENT_BALANCE","CUST_ALERT_CONTACT_NUMBER","CUST_ALERT_EMAIL_ID","DATE_OF_BIRTH","DEALER_NAME","DS_NAME","FILE_NAME","FIRST_CALL","GROUPS","PVN","HOME_PHONE_NUMBER","ID","ID_NUMBER","IMEI","IMSI","IS_UPDATE","MCN1","MCN2","MODIFIED_DATE","NETWORK_ID","NOTIFICATION_GROUPS","OFFICE_PHONE_NUMBER","OPTIONAL_FIELD_1","OPTIONAL_FIELD_10","OPTIONAL_FIELD_11","OPTIONAL_FIELD_12","OPTIONAL_FIELD_13","OPTIONAL_FIELD_14","OPTIONAL_FIELD_15","OPTIONAL_FIELD_2","OPTIONAL_FIELD_3","OPTIONAL_FIELD_4","OPTIONAL_FIELD_5","OPTIONAL_FIELD_6","OPTIONAL_FIELD_7","OPTIONAL_FIELD_8","OPTIONAL_FIELD_9","PHONE_NUMBER","PRODUCT_TYPE","QOS","SERVICES","SERVICE_NUMBER_TYPE","SSID","STATUS","SUBSCRIBER_DOA","SUBSCRIBER_TYPE","SUBSCRIBER_UID","SUBSCRIBER_UID_DOA"); {code}
This contains the list of all fields in the Cache all of which are indexed.
> Query Execution using Hibernate Search slow for large volume data
> -----------------------------------------------------------------
>
> Key: ISPN-5452
> URL: https://issues.jboss.org/browse/ISPN-5452
> Project: Infinispan
> Issue Type: Bug
> Components: Configuration, Remote Querying
> Affects Versions: 7.2.1.Final
> Environment: Linux
> Reporter: Prashant Thakur
>
> While benchmarking Infinispan we found that Querying is very slow when compared with Hibernate Search in Isolation
> Single node of Infinispan
> Memory allocated 230GB. No GC seen throughout query operation.
> Total required after full GC was 122GB.
> Setup 240 million records each of avg size 330 bytes .
> System has 16 cores and 40 worker threads were allocated at server side.
> With Single Client thread throughput was 900 req/sec in remote and 3k per sec in embedded more same request with Hibernate Search in Isolation gives throughput of 14000 req/sec.
> For 50 threads of clients the throughput was limited to 15k req/sec while hibernate search gives 80k req/sec for 10 threads.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
10 years, 11 months
[JBoss JIRA] (ISPN-5452) Query Execution using Hibernate Search slow for large volume data
by Sanne Grinovero (JIRA)
[ https://issues.jboss.org/browse/ISPN-5452?page=com.atlassian.jira.plugin.... ]
Sanne Grinovero updated ISPN-5452:
----------------------------------
Status: Open (was: New)
> Query Execution using Hibernate Search slow for large volume data
> -----------------------------------------------------------------
>
> Key: ISPN-5452
> URL: https://issues.jboss.org/browse/ISPN-5452
> Project: Infinispan
> Issue Type: Bug
> Components: Configuration, Remote Querying
> Affects Versions: 7.2.1.Final
> Environment: Linux
> Reporter: Prashant Thakur
>
> While benchmarking Infinispan we found that Querying is very slow when compared with Hibernate Search in Isolation
> Single node of Infinispan
> Memory allocated 230GB. No GC seen throughout query operation.
> Total required after full GC was 122GB.
> Setup 240 million records each of avg size 330 bytes .
> System has 16 cores and 40 worker threads were allocated at server side.
> With Single Client thread throughput was 900 req/sec in remote and 3k per sec in embedded more same request with Hibernate Search in Isolation gives throughput of 14000 req/sec.
> For 50 threads of clients the throughput was limited to 15k req/sec while hibernate search gives 80k req/sec for 10 threads.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
10 years, 11 months
[JBoss JIRA] (ISPN-5452) Query Execution using Hibernate Search slow for large volume data
by Sanne Grinovero (JIRA)
[ https://issues.jboss.org/browse/ISPN-5452?page=com.atlassian.jira.plugin.... ]
Sanne Grinovero updated ISPN-5452:
----------------------------------
Steps to Reproduce:
1. Create a Batch job to Create 240 million entries in Infinispan with all entries Indexed
2. Execute the following query through hotrod remote client
{code:java}String queryString= "SELECT ACCOUNT_ID,ID,PRODUCT_TYPE,FIRST_CALL,ID,IMEI from com.subex.spark.common.distributedcaching.data.Subscriber where PHONE_NUMBER='" +prefix+ i+"' AND STATUS in (1,2) and SUBSCRIBER_TYPE = 0";
QueryFactory qf = Search.getQueryFactory(distcacheclient.getRemoteCache());
RemoteQuery remoteQuery = new RemoteQuery(qf,(RemoteCacheImpl)distcacheclient.getRemoteCache(), distcacheclient.getSerializationContext(), queryString, 0, (int)numOfRecords);
List<Subscriber> list = remoteQuery.list();{code}
3. Following Query was executed in Embedded mode
{code:java}String queryString= "SELECT ACCOUNT_ID,ID,PRODUCT_TYPE,FIRST_CALL,ID,IMEI from com.subex.spark.common.distributedcaching.data.Subscriber where " + "PHONE_NUMBER='" +prefix+ i+"' AND STATUS in (1,2) and SUBSCRIBER_TYPE = 0";
QueryFactory qf = org.infinispan.query.Search.getQueryFactory(ispnCacheServer.getCache("SUBSCRIBER"));
Query query = qf.from("com.subex.spark.common.distributedcaching.data.Subscriber").setProjection("ACCOUNT_ID","ID","PRODUCT_TYPE","FIRST_CALL","ID","IMEI").
having("PHONE_NUMBER").eq(prefix+ i).and().having("STATUS").in(1,2).and().having("SUBSCRIBER_TYPE").eq(0).toBuilder().build();
List<Subscriber> list = query.list();
rs = query.list().iterator();{code}
4. Hibernate Search Query
{code}QueryContextBuilder queryBuilder = searchSession.getSearchFactory().buildQueryBuilder();
Query query = queryBuilder.forEntity(Subscriber.class).get().keyword().onField("ID").matching(Long.parseLong(prefix+i)).createQuery();{code}
5.
{code}org.hibernate.Query hibernateQuery=searchSession.createFullTextQuery(query, Subscriber.class).setProjection("ACCOUNT_ID","ACCOUNT_NAME","CONNECTION_TYPE","CONTACT_PHONE_NUMBER","CURRENT_BALANCE","CUST_ALERT_CONTACT_NUMBER","CUST_ALERT_EMAIL_ID","DATE_OF_BIRTH","DEALER_NAME","DS_NAME","FILE_NAME","FIRST_CALL","GROUPS","PVN","HOME_PHONE_NUMBER","ID","ID_NUMBER","IMEI","IMSI","IS_UPDATE","MCN1","MCN2","MODIFIED_DATE","NETWORK_ID","NOTIFICATION_GROUPS","OFFICE_PHONE_NUMBER","OPTIONAL_FIELD_1","OPTIONAL_FIELD_10","OPTIONAL_FIELD_11","OPTIONAL_FIELD_12","OPTIONAL_FIELD_13","OPTIONAL_FIELD_14","OPTIONAL_FIELD_15","OPTIONAL_FIELD_2","OPTIONAL_FIELD_3","OPTIONAL_FIELD_4","OPTIONAL_FIELD_5","OPTIONAL_FIELD_6","OPTIONAL_FIELD_7","OPTIONAL_FIELD_8","OPTIONAL_FIELD_9","PHONE_NUMBER","PRODUCT_TYPE","QOS","SERVICES","SERVICE_NUMBER_TYPE","SSID","STATUS","SUBSCRIBER_DOA","SUBSCRIBER_TYPE","SUBSCRIBER_UID","SUBSCRIBER_UID_DOA"); {code}
This contains the list of all fields in the Cache all of which are indexed.
was:
1. Create a Batch job to Create 240 million entries in Infinispan with all entries Indexed
2. Execute the following query through hotrod remote client
String queryString= "SELECT ACCOUNT_ID,ID,PRODUCT_TYPE,FIRST_CALL,ID,IMEI from com.subex.spark.common.distributedcaching.data.Subscriber where PHONE_NUMBER='" +prefix+ i+"' AND STATUS in (1,2) and SUBSCRIBER_TYPE = 0";
QueryFactory qf = Search.getQueryFactory(distcacheclient.getRemoteCache());
RemoteQuery remoteQuery = new RemoteQuery(qf,(RemoteCacheImpl)distcacheclient.getRemoteCache(), distcacheclient.getSerializationContext(), queryString, 0, (int)numOfRecords);
List<Subscriber> list = remoteQuery.list();
3. Following Query was executed in Embedded mode
StringBuilder sb = new StringBuilder();
String queryString= "SELECT ACCOUNT_ID,ID,PRODUCT_TYPE,FIRST_CALL,ID,IMEI from com.subex.spark.common.distributedcaching.data.Subscriber where "
+ "PHONE_NUMBER='" +prefix+ i+"' AND STATUS in (1,2) and SUBSCRIBER_TYPE = 0";
log.debug("Performing Query"+queryString);
QueryFactory qf = org.infinispan.query.Search.getQueryFactory(ispnCacheServer.getCache("SUBSCRIBER"));
Query query = qf.from("com.subex.spark.common.distributedcaching.data.Subscriber").setProjection("ACCOUNT_ID","ID","PRODUCT_TYPE","FIRST_CALL","ID","IMEI").
having("PHONE_NUMBER").eq(prefix+ i).and().having("STATUS").in(1,2).and().having("SUBSCRIBER_TYPE").eq(0).toBuilder().build();
List<Subscriber> list = query.list();
rs = query.list().iterator();
Hibernate Search Query
QueryContextBuilder queryBuilder = searchSession.getSearchFactory().buildQueryBuilder();
Query query = queryBuilder.forEntity(Subscriber.class).get().keyword().onField("ID").matching(Long.parseLong(prefix+i)).createQuery();
5) org.hibernate.Query hibernateQuery=searchSession.createFullTextQuery(query, Subscriber.class).setProjection("ACCOUNT_ID","ACCOUNT_NAME","CONNECTION_TYPE","CONTACT_PHONE_NUMBER","CURRENT_BALANCE","CUST_ALERT_CONTACT_NUMBER","CUST_ALERT_EMAIL_ID","DATE_OF_BIRTH","DEALER_NAME","DS_NAME","FILE_NAME","FIRST_CALL","GROUPS","PVN","HOME_PHONE_NUMBER","ID","ID_NUMBER","IMEI","IMSI","IS_UPDATE","MCN1","MCN2","MODIFIED_DATE","NETWORK_ID","NOTIFICATION_GROUPS","OFFICE_PHONE_NUMBER","OPTIONAL_FIELD_1","OPTIONAL_FIELD_10","OPTIONAL_FIELD_11","OPTIONAL_FIELD_12","OPTIONAL_FIELD_13","OPTIONAL_FIELD_14","OPTIONAL_FIELD_15","OPTIONAL_FIELD_2","OPTIONAL_FIELD_3","OPTIONAL_FIELD_4","OPTIONAL_FIELD_5","OPTIONAL_FIELD_6","OPTIONAL_FIELD_7","OPTIONAL_FIELD_8","OPTIONAL_FIELD_9","PHONE_NUMBER","PRODUCT_TYPE","QOS","SERVICES","SERVICE_NUMBER_TYPE","SSID","STATUS","SUBSCRIBER_DOA","SUBSCRIBER_TYPE","SUBSCRIBER_UID","SUBSCRIBER_UID_DOA");
This contains the list of all fields in the Cache all of which are indexed.
> Query Execution using Hibernate Search slow for large volume data
> -----------------------------------------------------------------
>
> Key: ISPN-5452
> URL: https://issues.jboss.org/browse/ISPN-5452
> Project: Infinispan
> Issue Type: Bug
> Components: Configuration, Remote Querying
> Affects Versions: 7.2.1.Final
> Environment: Linux
> Reporter: Prashant Thakur
>
> While benchmarking Infinispan we found that Querying is very slow when compared with Hibernate Search in Isolation
> Single node of Infinispan
> Memory allocated 230GB. No GC seen throughout query operation.
> Total required after full GC was 122GB.
> Setup 240 million records each of avg size 330 bytes .
> System has 16 cores and 40 worker threads were allocated at server side.
> With Single Client thread throughput was 900 req/sec in remote and 3k per sec in embedded more same request with Hibernate Search in Isolation gives throughput of 14000 req/sec.
> For 50 threads of clients the throughput was limited to 15k req/sec while hibernate search gives 80k req/sec for 10 threads.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
10 years, 11 months
[JBoss JIRA] (ISPN-5452) Query Execution using Hibernate Search slow for large volume data
by Prashant Thakur (JIRA)
[ https://issues.jboss.org/browse/ISPN-5452?page=com.atlassian.jira.plugin.... ]
Prashant Thakur updated ISPN-5452:
----------------------------------
Summary: Query Execution using Hibernate Search slow for large volume data (was: Query Execution using Hibernate Query slow for large volume data)
> Query Execution using Hibernate Search slow for large volume data
> -----------------------------------------------------------------
>
> Key: ISPN-5452
> URL: https://issues.jboss.org/browse/ISPN-5452
> Project: Infinispan
> Issue Type: Bug
> Components: Configuration, Remote Querying
> Affects Versions: 7.2.1.Final
> Environment: Linux
> Reporter: Prashant Thakur
>
> While benchmarking Infinispan we found that Querying is very slow when compared with Hibernate Search in Isolation
> Single node of Infinispan
> Memory allocated 230GB. No GC seen throughout query operation.
> Total required after full GC was 122GB.
> Setup 240 million records each of avg size 330 bytes .
> System has 16 cores and 40 worker threads were allocated at server side.
> With Single Client thread throughput was 900 req/sec in remote and 3k per sec in embedded more same request with Hibernate Search in Isolation gives throughput of 14000 req/sec.
> For 50 threads of clients the throughput was limited to 15k req/sec while hibernate search gives 80k req/sec for 10 threads.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
10 years, 11 months
[JBoss JIRA] (ISPN-5449) HotRod ISPN000136: Execution error java.lang.ClassCastException: [B cannot be cast to org.infinispan.container.entries.CacheEntry - See more at: https://developer.jboss.org/thread/257699#sthash.ibv3Nflq.dpuf
by Adrian Nistor (JIRA)
[ https://issues.jboss.org/browse/ISPN-5449?page=com.atlassian.jira.plugin.... ]
Adrian Nistor updated ISPN-5449:
--------------------------------
Status: Open (was: New)
> HotRod ISPN000136: Execution error java.lang.ClassCastException: [B cannot be cast to org.infinispan.container.entries.CacheEntry - See more at: https://developer.jboss.org/thread/257699#sthash.ibv3Nflq.dpuf
> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: ISPN-5449
> URL: https://issues.jboss.org/browse/ISPN-5449
> Project: Infinispan
> Issue Type: Bug
> Reporter: Enrico Olivelli
>
> Hi,
> I'm going to upgrade to Infinspan 7.2.1final but I got into a showstopper. This error occours very frequently and the cluster becomes not usable.
>
> In dev enviroment I'm using only Java HotRod clients and 3 Java HotRod Servers. When I put a "byte[]" in a cache then the following error occurs.
> Using only one Hotrod server the error does not occur.
>
>
> ERROR interceptors.InvocationContextInterceptor: ISPN000136: Execution error
> java.lang.ClassCastException: [B cannot be cast to org.infinispan.container.entries.CacheEntry
> at org.infinispan.interceptors.compat.BaseTypeConverterInterceptor.visitGetCacheEntryCommand(BaseTypeConverterInterceptor.java:118)
> at org.infinispan.commands.read.GetCacheEntryCommand.acceptVisitor(GetCacheEntryCommand.java:39)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:102)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:71)
> at org.infinispan.commands.AbstractVisitor.visitGetCacheEntryCommand(AbstractVisitor.java:91)
> at org.infinispan.commands.read.GetCacheEntryCommand.acceptVisitor(GetCacheEntryCommand.java:39)
> at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:336)
> at org.infinispan.cache.impl.CacheImpl.getCacheEntry(CacheImpl.java:437)
> at org.infinispan.cache.impl.DecoratedCache.getCacheEntry(DecoratedCache.java:547)
> at org.infinispan.server.hotrod.CacheDecodeContext.get(CacheDecodeContext.scala:180)
> at org.infinispan.server.hotrod.HotRodDecoder.org$infinispan$server$hotrod$HotRodDecoder$$decodeKey(HotRodDecoder.scala:103)
> at org.infinispan.server.hotrod.HotRodDecoder$$anonfun$decode$1.apply$mcV$sp(HotRodDecoder.scala:48)
> at org.infinispan.server.hotrod.HotRodDecoder$$anon$1.run(HotRodDecoder.scala:206)
> at org.infinispan.server.hotrod.HotRodDecoder$$anon$1.run(HotRodDecoder.scala:205)
> at org.infinispan.security.Security.doAs(Security.java:143)
> at org.infinispan.server.hotrod.HotRodDecoder.wrapSecurity(HotRodDecoder.scala:205)
> at org.infinispan.server.hotrod.HotRodDecoder.decode(HotRodDecoder.scala:45)
> at io.netty.handler.codec.ReplayingDecoder.callDecode(ReplayingDecoder.java:370)
> at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:168)
> at org.infinispan.server.hotrod.HotRodDecoder.org$infinispan$server$core$transport$StatsChannelHandler$$super$channelRead(HotRodDecoder.scala:31)
> at org.infinispan.server.core.transport.StatsChannelHandler$class.channelRead(StatsChannelHandler.scala:32)
> at org.infinispan.server.hotrod.HotRodDecoder.channelRead(HotRodDecoder.scala:31)
> ....
>
>
> Cache Server configuration:
> Configuration wildcard = new ConfigurationBuilder()
> .locking().lockAcquisitionTimeout(lockAcquisitionTimeout)
> .concurrencyLevel(10000).isolationLevel(IsolationLevel.READ_COMMITTED).useLockStriping(true)
> .clustering()
> .cacheMode(CacheMode.DIST_SYNC)
> .l1().lifespan(l1ttl).enable()
> .hash().numOwners(numOwners).capacityFactor(capacityFactor)
> .partitionHandling().enabled(false)
> .stateTransfer().awaitInitialTransfer(false).timeout(initialTransferTimeout).fetchInMemoryState(false)
> .storeAsBinary().enabled(true).storeKeysAsBinary(false).storeValuesAsBinary(false)
> .jmxStatistics().enable()
> .unsafe().unreliableReturnValues(true) // put e remove non ritornano il valore precedente, tanto a noi non interessa
> .compatibility().enable()
> .build();
>
> Hotrod Endpoint:
> HotRodServerConfigurationBuilder configBuilder = new HotRodServerConfigurationBuilder()
> .host(address)
> .port(port);
> hotrodserver.start(configBuilder
> .build(),
> manager);
>
> Client side config:
> org.infinispan.client.hotrod.configuration.ConfigurationBuilder clientBuilder = new org.infinispan.client.hotrod.configuration.ConfigurationBuilder().forceReturnValues(false);
> clientBuilder.nearCache().mode(NearCacheMode.LAZY).maxEntries(10000);
> (for each server....)
> clientBuilder.addServer().host(serveraddress).port(serverport);
> RemoteCacheManager remoteCacheManager = new RemoteCacheManager(clientBuilder.build());
> - See more at: https://developer.jboss.org/thread/257699#sthash.ibv3Nflq.dpuf
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
10 years, 11 months
[JBoss JIRA] (ISPN-5452) Query Execution using Hibernate Query slow for large volume data
by Prashant Thakur (JIRA)
[ https://issues.jboss.org/browse/ISPN-5452?page=com.atlassian.jira.plugin.... ]
Prashant Thakur commented on ISPN-5452:
---------------------------------------
Its in Compatibility mode
> Query Execution using Hibernate Query slow for large volume data
> ----------------------------------------------------------------
>
> Key: ISPN-5452
> URL: https://issues.jboss.org/browse/ISPN-5452
> Project: Infinispan
> Issue Type: Bug
> Components: Configuration, Remote Querying
> Affects Versions: 7.2.1.Final
> Environment: Linux
> Reporter: Prashant Thakur
>
> While benchmarking Infinispan we found that Querying is very slow when compared with Hibernate Search in Isolation
> Single node of Infinispan
> Memory allocated 230GB. No GC seen throughout query operation.
> Total required after full GC was 122GB.
> Setup 240 million records each of avg size 330 bytes .
> System has 16 cores and 40 worker threads were allocated at server side.
> With Single Client thread throughput was 900 req/sec in remote and 3k per sec in embedded more same request with Hibernate Search in Isolation gives throughput of 14000 req/sec.
> For 50 threads of clients the throughput was limited to 15k req/sec while hibernate search gives 80k req/sec for 10 threads.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
10 years, 11 months
[JBoss JIRA] (ISPN-5452) Query Execution using Hibernate Query slow for large volume data
by Adrian Nistor (JIRA)
[ https://issues.jboss.org/browse/ISPN-5452?page=com.atlassian.jira.plugin.... ]
Adrian Nistor commented on ISPN-5452:
-------------------------------------
Prashant,
thanks for logging this!
How do you store the data? Protobuf or compat mode?
> Query Execution using Hibernate Query slow for large volume data
> ----------------------------------------------------------------
>
> Key: ISPN-5452
> URL: https://issues.jboss.org/browse/ISPN-5452
> Project: Infinispan
> Issue Type: Bug
> Components: Configuration, Remote Querying
> Affects Versions: 7.2.1.Final
> Environment: Linux
> Reporter: Prashant Thakur
>
> While benchmarking Infinispan we found that Querying is very slow when compared with Hibernate Search in Isolation
> Single node of Infinispan
> Memory allocated 230GB. No GC seen throughout query operation.
> Total required after full GC was 122GB.
> Setup 240 million records each of avg size 330 bytes .
> System has 16 cores and 40 worker threads were allocated at server side.
> With Single Client thread throughput was 900 req/sec in remote and 3k per sec in embedded more same request with Hibernate Search in Isolation gives throughput of 14000 req/sec.
> For 50 threads of clients the throughput was limited to 15k req/sec while hibernate search gives 80k req/sec for 10 threads.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
10 years, 11 months
[JBoss JIRA] (ISPN-5452) Query Execution using Hibernate Query slow for large volume data
by Prashant Thakur (JIRA)
[ https://issues.jboss.org/browse/ISPN-5452?page=com.atlassian.jira.plugin.... ]
Prashant Thakur commented on ISPN-5452:
---------------------------------------
https://developer.jboss.org/message/928982?et=watches.email.thread#928982
> Query Execution using Hibernate Query slow for large volume data
> ----------------------------------------------------------------
>
> Key: ISPN-5452
> URL: https://issues.jboss.org/browse/ISPN-5452
> Project: Infinispan
> Issue Type: Bug
> Components: Configuration, Remote Querying
> Affects Versions: 7.2.1.Final
> Environment: Linux
> Reporter: Prashant Thakur
>
> While benchmarking Infinispan we found that Querying is very slow when compared with Hibernate Search in Isolation
> Single node of Infinispan
> Memory allocated 230GB. No GC seen throughout query operation.
> Total required after full GC was 122GB.
> Setup 240 million records each of avg size 330 bytes .
> System has 16 cores and 40 worker threads were allocated at server side.
> With Single Client thread throughput was 900 req/sec in remote and 3k per sec in embedded more same request with Hibernate Search in Isolation gives throughput of 14000 req/sec.
> For 50 threads of clients the throughput was limited to 15k req/sec while hibernate search gives 80k req/sec for 10 threads.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
10 years, 11 months