[JBoss JIRA] (ISPN-5524) Race condition in SemaphoreCompletionService.executeFront()
by RH Bugzilla Integration (JIRA)
[ https://issues.jboss.org/browse/ISPN-5524?page=com.atlassian.jira.plugin.... ]
RH Bugzilla Integration commented on ISPN-5524:
-----------------------------------------------
Dan Berindei <dberinde(a)redhat.com> changed the Status of [bug 1229181|https://bugzilla.redhat.com/show_bug.cgi?id=1229181] from ASSIGNED to POST
> Race condition in SemaphoreCompletionService.executeFront()
> -----------------------------------------------------------
>
> Key: ISPN-5524
> URL: https://issues.jboss.org/browse/ISPN-5524
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Affects Versions: 7.2.2.Final, 8.0.0.Alpha1
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Priority: Blocker
> Labels: testsuite_stability
> Fix For: 8.0.0.Alpha2, 7.2.3.Final
>
>
> If two threads call {{executeFront()}} at the same time, it's possible that none of them executes any task, although there are permits available, and one of the threads just added a task in the queue. This scenario causes random initial state transfer timeouts in the testsuite:
> # T1: continueTaskInBackground(): queue = empty, permits = 0
> # T1: backgroundTaskFinished()
> # T1: = semaphore.release(): queue = empty, permits = 1
> # T1: = executeFront()
> # T1: == semaphore.acquire() -> true: queue = empty, permits = 0
> # T2: == queue.poll() -> null
> # T2: submit(task)
> # T2: = executeFront()
> # T2: == semaphore.acquire() -> false
> # T2: return without executing any task
> # T1: return without executing any task
> Failure example: http://ci.infinispan.org/viewLog.html?buildId=27153&tab=buildResultsDiv&b...
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
9 years, 7 months
[JBoss JIRA] (ISPN-5529) ClusterListenerReplTest.testPrimaryOwnerGoesDownAfterBackupRaisesEvent random failures
by Dan Berindei (JIRA)
[ https://issues.jboss.org/browse/ISPN-5529?page=com.atlassian.jira.plugin.... ]
Dan Berindei updated ISPN-5529:
-------------------------------
Status: Open (was: New)
> ClusterListenerReplTest.testPrimaryOwnerGoesDownAfterBackupRaisesEvent random failures
> --------------------------------------------------------------------------------------
>
> Key: ISPN-5529
> URL: https://issues.jboss.org/browse/ISPN-5529
> Project: Infinispan
> Issue Type: Bug
> Components: Core, Test Suite - Core
> Affects Versions: 7.2.2.Final, 8.0.0.Alpha1
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Priority: Blocker
> Labels: testsuite_stability
> Fix For: 8.0.0.Alpha2
>
>
> Because the cache is replicated, both cache 0 or cache 2 are backup owners of the tested key, and both of them can become primary owners after cache 1 is killed. If cache 0 becomes the primary owner, {{cache.put(key, FIRST_VALUE)}} may return {{null}} and/or the cluster listener may receive only one event:
> {noformat}
> java.lang.AssertionError: expected:<null> but was:<first-value>
> at org.testng.AssertJUnit.assertEquals(AssertJUnit.java:101)
> at org.testng.AssertJUnit.assertEquals(AssertJUnit.java:108)
> at org.infinispan.notifications.cachelistener.cluster.ClusterListenerReplTest.testPrimaryOwnerGoesDownAfterBackupRaisesEvent(ClusterListenerReplTest.java:119)
> {noformat}
> {noformat}
> 12:23:53,438 ERROR (testng-ClusterListenerReplTest:) [UnitTestTestNGListener] Test testPrimaryOwnerGoesDownAfterBackupRaisesEvent(org.infinispan.notifications.cachelistener.cluster.ClusterListenerReplTest) failed.java.lang.AssertionError: expected:<1> but was:<2>
> at org.testng.AssertJUnit.fail(AssertJUnit.java:59)
> at org.testng.AssertJUnit.failNotEquals(AssertJUnit.java:364)
> at org.testng.AssertJUnit.assertEquals(AssertJUnit.java:80)
> at org.testng.AssertJUnit.assertEquals(AssertJUnit.java:245)
> at org.testng.AssertJUnit.assertEquals(AssertJUnit.java:252)
> at org.infinispan.notifications.cachelistener.cluster.ClusterListenerReplTest.testPrimaryOwnerGoesDownAfterBackupRaisesEvent(ClusterListenerReplTest.java:121)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
9 years, 7 months
[JBoss JIRA] (ISPN-5529) ClusterListenerReplTest.testPrimaryOwnerGoesDownAfterBackupRaisesEvent random failures
by Dan Berindei (JIRA)
[ https://issues.jboss.org/browse/ISPN-5529?page=com.atlassian.jira.plugin.... ]
Dan Berindei updated ISPN-5529:
-------------------------------
Labels: testsuite_stability (was: )
> ClusterListenerReplTest.testPrimaryOwnerGoesDownAfterBackupRaisesEvent random failures
> --------------------------------------------------------------------------------------
>
> Key: ISPN-5529
> URL: https://issues.jboss.org/browse/ISPN-5529
> Project: Infinispan
> Issue Type: Bug
> Components: Core, Test Suite - Core
> Affects Versions: 7.2.2.Final, 8.0.0.Alpha1
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Priority: Blocker
> Labels: testsuite_stability
> Fix For: 8.0.0.Alpha2
>
>
> Because the cache is replicated, both cache 0 or cache 2 are backup owners of the tested key, and both of them can become primary owners after cache 1 is killed. If cache 0 becomes the primary owner, {{cache.put(key, FIRST_VALUE)}} may return {{null}} and/or the cluster listener may receive only one event:
> {noformat}
> java.lang.AssertionError: expected:<null> but was:<first-value>
> at org.testng.AssertJUnit.assertEquals(AssertJUnit.java:101)
> at org.testng.AssertJUnit.assertEquals(AssertJUnit.java:108)
> at org.infinispan.notifications.cachelistener.cluster.ClusterListenerReplTest.testPrimaryOwnerGoesDownAfterBackupRaisesEvent(ClusterListenerReplTest.java:119)
> {noformat}
> {noformat}
> 12:23:53,438 ERROR (testng-ClusterListenerReplTest:) [UnitTestTestNGListener] Test testPrimaryOwnerGoesDownAfterBackupRaisesEvent(org.infinispan.notifications.cachelistener.cluster.ClusterListenerReplTest) failed.java.lang.AssertionError: expected:<1> but was:<2>
> at org.testng.AssertJUnit.fail(AssertJUnit.java:59)
> at org.testng.AssertJUnit.failNotEquals(AssertJUnit.java:364)
> at org.testng.AssertJUnit.assertEquals(AssertJUnit.java:80)
> at org.testng.AssertJUnit.assertEquals(AssertJUnit.java:245)
> at org.testng.AssertJUnit.assertEquals(AssertJUnit.java:252)
> at org.infinispan.notifications.cachelistener.cluster.ClusterListenerReplTest.testPrimaryOwnerGoesDownAfterBackupRaisesEvent(ClusterListenerReplTest.java:121)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
9 years, 7 months
[JBoss JIRA] (ISPN-5529) ClusterListenerReplTest.testPrimaryOwnerGoesDownAfterBackupRaisesEvent random failures
by Dan Berindei (JIRA)
Dan Berindei created ISPN-5529:
----------------------------------
Summary: ClusterListenerReplTest.testPrimaryOwnerGoesDownAfterBackupRaisesEvent random failures
Key: ISPN-5529
URL: https://issues.jboss.org/browse/ISPN-5529
Project: Infinispan
Issue Type: Bug
Components: Core, Test Suite - Core
Affects Versions: 8.0.0.Alpha1, 7.2.2.Final
Reporter: Dan Berindei
Assignee: Dan Berindei
Priority: Blocker
Fix For: 8.0.0.Alpha2
Because the cache is replicated, both cache 0 or cache 2 are backup owners of the tested key, and both of them can become primary owners after cache 1 is killed. If cache 0 becomes the primary owner, {{cache.put(key, FIRST_VALUE)}} may return {{null}} and/or the cluster listener may receive only one event:
{noformat}
java.lang.AssertionError: expected:<null> but was:<first-value>
at org.testng.AssertJUnit.assertEquals(AssertJUnit.java:101)
at org.testng.AssertJUnit.assertEquals(AssertJUnit.java:108)
at org.infinispan.notifications.cachelistener.cluster.ClusterListenerReplTest.testPrimaryOwnerGoesDownAfterBackupRaisesEvent(ClusterListenerReplTest.java:119)
{noformat}
{noformat}
12:23:53,438 ERROR (testng-ClusterListenerReplTest:) [UnitTestTestNGListener] Test testPrimaryOwnerGoesDownAfterBackupRaisesEvent(org.infinispan.notifications.cachelistener.cluster.ClusterListenerReplTest) failed.java.lang.AssertionError: expected:<1> but was:<2>
at org.testng.AssertJUnit.fail(AssertJUnit.java:59)
at org.testng.AssertJUnit.failNotEquals(AssertJUnit.java:364)
at org.testng.AssertJUnit.assertEquals(AssertJUnit.java:80)
at org.testng.AssertJUnit.assertEquals(AssertJUnit.java:245)
at org.testng.AssertJUnit.assertEquals(AssertJUnit.java:252)
at org.infinispan.notifications.cachelistener.cluster.ClusterListenerReplTest.testPrimaryOwnerGoesDownAfterBackupRaisesEvent(ClusterListenerReplTest.java:121)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
9 years, 7 months
[JBoss JIRA] (ISPN-5452) Query Execution using Hibernate Search slow for large volume data
by Prashant Thakur (JIRA)
[ https://issues.jboss.org/browse/ISPN-5452?page=com.atlassian.jira.plugin.... ]
Prashant Thakur edited comment on ISPN-5452 at 6/8/15 7:26 AM:
---------------------------------------------------------------
Thanks Sanne,
We will do some profiling at application end and update this thread. Most of your points make lot of sense. The strategy we follow is to start with single client thread and then proceed with increasing the number of threads to get to optimal figure on number of clients. Similarily increasing number of worker threads at Infinispan level also helps. We have not seen extreme CPU utilization at infinispan side even when we saw 80k/sec kind of performance the cpu utilization was within 20-25%, Hence we are a bit optimistic on getting to a higher value if somehow we can decrease latency by increasing CPU utilization.
We are doing two levels of marshalling and also bringing in extension to data container in between which might slow down things a bit which we can confirm from profiling each in isolation.
was (Author: prashant.thakur):
Thanks Sanne,
We will do some profiling at application end and update this thread. Most of your points make lot of sense. The strategy we follow is to start with single client thread and then proceed with increasing the number of threads to get to optimal figure on number of clients. Similarily increasing number of worker threads at Infinispan level also helps. We have not seen extreme CPU utilization end at infinispan side even when we saw 80k/sec kind of performance the cpu utilization was within 20-25%, Hence we are a bit optimistic on getting to a higher value if somehow we can decrease latency by increasing CPU utilization.
We are doing two levels of marshalling and also bringing in extension to data container in between which might slow down things a bit which we can confirm from profiling each in isolation.
> Query Execution using Hibernate Search slow for large volume data
> -----------------------------------------------------------------
>
> Key: ISPN-5452
> URL: https://issues.jboss.org/browse/ISPN-5452
> Project: Infinispan
> Issue Type: Bug
> Components: Configuration, Remote Querying
> Affects Versions: 7.2.1.Final
> Environment: Linux
> Reporter: Prashant Thakur
>
> While benchmarking Infinispan we found that Querying is very slow when compared with Hibernate Search in Isolation
> Single node of Infinispan
> Memory allocated 230GB. No GC seen throughout query operation.
> Total required after full GC was 122GB.
> Setup 240 million records each of avg size 330 bytes .
> System has 16 cores and 40 worker threads were allocated at server side.
> With Single Client thread throughput was 900 req/sec in remote and 3k per sec in embedded more same request with Hibernate Search in Isolation gives throughput of 14000 req/sec.
> For 50 threads of clients the throughput was limited to 15k req/sec while hibernate search gives 80k req/sec for 10 threads.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
9 years, 7 months
[JBoss JIRA] (ISPN-5452) Query Execution using Hibernate Search slow for large volume data
by Prashant Thakur (JIRA)
[ https://issues.jboss.org/browse/ISPN-5452?page=com.atlassian.jira.plugin.... ]
Prashant Thakur commented on ISPN-5452:
---------------------------------------
Thanks Sanne,
We will do some profiling at application end and update this thread. Most of your points make lot of sense. The strategy we follow is to start with single client thread and then proceed with increasing the number of threads to get to optimal figure on number of clients. Similarily increasing number of worker threads at Infinispan level also helps. We have not seen extreme CPU utilization end at infinispan side even when we saw 80k/sec kind of performance the cpu utilization was within 20-25%, Hence we are a bit optimistic on getting to a higher value if somehow we can decrease latency by increasing CPU utilization.
We are doing two levels of marshalling and also bringing in extension to data container in between which might slow down things a bit which we can confirm from profiling each in isolation.
> Query Execution using Hibernate Search slow for large volume data
> -----------------------------------------------------------------
>
> Key: ISPN-5452
> URL: https://issues.jboss.org/browse/ISPN-5452
> Project: Infinispan
> Issue Type: Bug
> Components: Configuration, Remote Querying
> Affects Versions: 7.2.1.Final
> Environment: Linux
> Reporter: Prashant Thakur
>
> While benchmarking Infinispan we found that Querying is very slow when compared with Hibernate Search in Isolation
> Single node of Infinispan
> Memory allocated 230GB. No GC seen throughout query operation.
> Total required after full GC was 122GB.
> Setup 240 million records each of avg size 330 bytes .
> System has 16 cores and 40 worker threads were allocated at server side.
> With Single Client thread throughput was 900 req/sec in remote and 3k per sec in embedded more same request with Hibernate Search in Isolation gives throughput of 14000 req/sec.
> For 50 threads of clients the throughput was limited to 15k req/sec while hibernate search gives 80k req/sec for 10 threads.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
9 years, 7 months
[JBoss JIRA] (ISPN-5528) Expose segment ownership in the Hot Rod client
by Gustavo Fernandes (JIRA)
[ https://issues.jboss.org/browse/ISPN-5528?page=com.atlassian.jira.plugin.... ]
Gustavo Fernandes updated ISPN-5528:
------------------------------------
Description:
ISPN-5219 exposes entry iteration via Hot Rod and allows to filter based on segments.
In order to be able to eventually calculate the segments to be filtered on, ownership information for a cache needs to be exposed as well.
API suggestion by @wburns:
{code}
Set<Integer> getSegments()
Map<Address, Set<Integer>> getSegmentsByServer()
{code}
was:
ISPN-5219 exposes entry iteration via Hot Rod and allows to filter based on segments.
In order to be able to eventually calculate the segments to be filtered on, ownership information for a cache needs to be exposed. API suggestion by @wburns:
{code}
Set<Integer> getSegments()
Map<Address, Set<Integer>> getSegmentsByServer()
{code}
> Expose segment ownership in the Hot Rod client
> ----------------------------------------------
>
> Key: ISPN-5528
> URL: https://issues.jboss.org/browse/ISPN-5528
> Project: Infinispan
> Issue Type: Feature Request
> Components: Hadoop Integration
> Affects Versions: 8.0.0.Alpha1
> Reporter: Gustavo Fernandes
> Assignee: Gustavo Fernandes
>
> ISPN-5219 exposes entry iteration via Hot Rod and allows to filter based on segments.
> In order to be able to eventually calculate the segments to be filtered on, ownership information for a cache needs to be exposed as well.
> API suggestion by @wburns:
> {code}
> Set<Integer> getSegments()
> Map<Address, Set<Integer>> getSegmentsByServer()
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
9 years, 7 months
[JBoss JIRA] (ISPN-5528) Expose segment ownership via Hot Rod
by Gustavo Fernandes (JIRA)
Gustavo Fernandes created ISPN-5528:
---------------------------------------
Summary: Expose segment ownership via Hot Rod
Key: ISPN-5528
URL: https://issues.jboss.org/browse/ISPN-5528
Project: Infinispan
Issue Type: Feature Request
Components: Hadoop Integration
Affects Versions: 8.0.0.Alpha1
Reporter: Gustavo Fernandes
Assignee: Gustavo Fernandes
ISPN-5219 exposes entry iteration via Hot Rod and allows to filter based on segments.
In order to be able to eventually calculate the segments to be filtered on, ownership information for a cache needs to be exposed. API suggestion by @wburns:
{code}
Set<Integer> getSegments()
Map<Address, Set<Integer>> getSegmentsByServer()
{code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
9 years, 7 months
[JBoss JIRA] (ISPN-5528) Expose segment ownership in the Hot Rod client
by Gustavo Fernandes (JIRA)
[ https://issues.jboss.org/browse/ISPN-5528?page=com.atlassian.jira.plugin.... ]
Gustavo Fernandes updated ISPN-5528:
------------------------------------
Summary: Expose segment ownership in the Hot Rod client (was: Expose segment ownership via Hot Rod)
> Expose segment ownership in the Hot Rod client
> ----------------------------------------------
>
> Key: ISPN-5528
> URL: https://issues.jboss.org/browse/ISPN-5528
> Project: Infinispan
> Issue Type: Feature Request
> Components: Hadoop Integration
> Affects Versions: 8.0.0.Alpha1
> Reporter: Gustavo Fernandes
> Assignee: Gustavo Fernandes
>
> ISPN-5219 exposes entry iteration via Hot Rod and allows to filter based on segments.
> In order to be able to eventually calculate the segments to be filtered on, ownership information for a cache needs to be exposed. API suggestion by @wburns:
> {code}
> Set<Integer> getSegments()
> Map<Address, Set<Integer>> getSegmentsByServer()
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
9 years, 7 months
[JBoss JIRA] (ISPN-5452) Query Execution using Hibernate Search slow for large volume data
by Sanne Grinovero (JIRA)
[ https://issues.jboss.org/browse/ISPN-5452?page=com.atlassian.jira.plugin.... ]
Sanne Grinovero commented on ISPN-5452:
---------------------------------------
Hi Prashant,
there are two aspects at play: latency (response time) for a single query, and how many queries you can get done per unit of time (often measures as queries / second).
The response time of a single query is expected to be longer for a remote query (of course! since it goes over your network) but the execution strategy is identical among all three approaches. The amount of queries per second handled by a single server is approximately the same for each of the three as well: Remote Query adds a little overhead toll as it's processing remote calls, but that being only a minor aspect it should only provide a small deviation.
In other words, even though the latency would be higher, I don't expect significant changes in the amount of queries throughput the system can handle. RMI is not as efficient as Hot Rod so that makes your tests even more surprising.
I can't comment on the 50-100 us: that seems a reasonable figure but it highly depends on the index size, the kind of queries, the kind of hardware and general OS tuning. What is important to take away is that wathever figure you can get on remote query, you should be able to better that when using embedded query.
{quote}... then we are sure its an application issue{quote}
Don't forget it might be a problem of tests design. I wouldn't be surprised if some of your clients get in locking problems because of "too good" performance from the server. Similarly you could have some race in data collection on the testing clients.
> Query Execution using Hibernate Search slow for large volume data
> -----------------------------------------------------------------
>
> Key: ISPN-5452
> URL: https://issues.jboss.org/browse/ISPN-5452
> Project: Infinispan
> Issue Type: Bug
> Components: Configuration, Remote Querying
> Affects Versions: 7.2.1.Final
> Environment: Linux
> Reporter: Prashant Thakur
>
> While benchmarking Infinispan we found that Querying is very slow when compared with Hibernate Search in Isolation
> Single node of Infinispan
> Memory allocated 230GB. No GC seen throughout query operation.
> Total required after full GC was 122GB.
> Setup 240 million records each of avg size 330 bytes .
> System has 16 cores and 40 worker threads were allocated at server side.
> With Single Client thread throughput was 900 req/sec in remote and 3k per sec in embedded more same request with Hibernate Search in Isolation gives throughput of 14000 req/sec.
> For 50 threads of clients the throughput was limited to 15k req/sec while hibernate search gives 80k req/sec for 10 threads.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
9 years, 7 months