[JBoss JIRA] (ISPN-5126) DistributedExecutorService ignores unsuccessful responses
by Dan Berindei (JIRA)
[ https://issues.jboss.org/browse/ISPN-5126?page=com.atlassian.jira.plugin.... ]
Dan Berindei reassigned ISPN-5126:
----------------------------------
Assignee: Vladimir Blagojevic
> DistributedExecutorService ignores unsuccessful responses
> ---------------------------------------------------------
>
> Key: ISPN-5126
> URL: https://issues.jboss.org/browse/ISPN-5126
> Project: Infinispan
> Issue Type: Bug
> Components: Core, Distributed Execution and Map/Reduce
> Affects Versions: 7.1.0.Alpha1, 7.0.3.Final
> Reporter: Dan Berindei
> Assignee: Vladimir Blagojevic
> Priority: Critical
> Labels: testsuite_stability
>
> I got a failure in {{DistributedExecutorFailureTest}} (master only) on my machine because the distributed executor ignores {{CacheNotFoundResponse}} responses:
> {noformat}
> 15:18:45,516 TRACE (transport-thread-NodeA-p30129-t4:) [CommandAwareRpcDispatcher] Response: CacheNotFoundResponse
> 15:18:45,516 TRACE (transport-thread-NodeA-p30129-t4:) [RpcManagerImpl] Response(s) to DistributedExecuteCommand [cache=null, keys=[], callable=org.infinispan.distexec.DistributedExecutorFailoverTest$SleepingSimpleCallable@23a9b62a] is {NodeB-4305=CacheNotFoundResponse}
> 15:18:45,517 ERROR (testng-DistributedExecutorFailoverTest:) [UnitTestTestNGListener] Test testBasicTargetRemoteDistributedCallable(org.infinispan.distexec.DistributedExecutorFailoverTest) failed.
> java.lang.AssertionError: expected:<1> but was:<null>
> at org.testng.AssertJUnit.fail(AssertJUnit.java:59)
> at org.testng.AssertJUnit.failNotEquals(AssertJUnit.java:364)
> at org.testng.AssertJUnit.assertEquals(AssertJUnit.java:80)
> at org.testng.AssertJUnit.assertEquals(AssertJUnit.java:88)
> at org.infinispan.distexec.DistributedExecutorFailoverTest.testBasicTargetRemoteDistributedCallable(DistributedExecutorFailoverTest.java:74)
> {noformat}
> {{RemoteDistributedTaskPart.retrieveResult()}} ignores any response that's not a {{SuccessfulResponse}} and returns {{null}}. It should throw an exception instead, so that the failover policy can retry it on another node.
--
This message was sent by Atlassian JIRA
(v6.3.11#6341)
9 years, 4 months
[JBoss JIRA] (ISPN-5126) DistributedExecutorService ignores unsuccessful responses
by Dan Berindei (JIRA)
Dan Berindei created ISPN-5126:
----------------------------------
Summary: DistributedExecutorService ignores unsuccessful responses
Key: ISPN-5126
URL: https://issues.jboss.org/browse/ISPN-5126
Project: Infinispan
Issue Type: Bug
Components: Core, Distributed Execution and Map/Reduce
Affects Versions: 7.0.3.Final, 7.1.0.Alpha1
Reporter: Dan Berindei
Priority: Critical
I got a failure in {{DistributedExecutorFailureTest}} (master only) on my machine because the distributed executor ignores {{CacheNotFoundResponse}} responses:
{noformat}
15:18:45,516 TRACE (transport-thread-NodeA-p30129-t4:) [CommandAwareRpcDispatcher] Response: CacheNotFoundResponse
15:18:45,516 TRACE (transport-thread-NodeA-p30129-t4:) [RpcManagerImpl] Response(s) to DistributedExecuteCommand [cache=null, keys=[], callable=org.infinispan.distexec.DistributedExecutorFailoverTest$SleepingSimpleCallable@23a9b62a] is {NodeB-4305=CacheNotFoundResponse}
15:18:45,517 ERROR (testng-DistributedExecutorFailoverTest:) [UnitTestTestNGListener] Test testBasicTargetRemoteDistributedCallable(org.infinispan.distexec.DistributedExecutorFailoverTest) failed.
java.lang.AssertionError: expected:<1> but was:<null>
at org.testng.AssertJUnit.fail(AssertJUnit.java:59)
at org.testng.AssertJUnit.failNotEquals(AssertJUnit.java:364)
at org.testng.AssertJUnit.assertEquals(AssertJUnit.java:80)
at org.testng.AssertJUnit.assertEquals(AssertJUnit.java:88)
at org.infinispan.distexec.DistributedExecutorFailoverTest.testBasicTargetRemoteDistributedCallable(DistributedExecutorFailoverTest.java:74)
{noformat}
{{RemoteDistributedTaskPart.retrieveResult()}} ignores any response that's not a {{SuccessfulResponse}} and returns {{null}}. It should throw an exception instead, so that the failover policy can retry it on another node.
--
This message was sent by Atlassian JIRA
(v6.3.11#6341)
9 years, 4 months
[JBoss JIRA] (ISPN-5126) DistributedExecutorService ignores unsuccessful responses
by Dan Berindei (JIRA)
[ https://issues.jboss.org/browse/ISPN-5126?page=com.atlassian.jira.plugin.... ]
Dan Berindei updated ISPN-5126:
-------------------------------
Status: Open (was: New)
> DistributedExecutorService ignores unsuccessful responses
> ---------------------------------------------------------
>
> Key: ISPN-5126
> URL: https://issues.jboss.org/browse/ISPN-5126
> Project: Infinispan
> Issue Type: Bug
> Components: Core, Distributed Execution and Map/Reduce
> Affects Versions: 7.1.0.Alpha1, 7.0.3.Final
> Reporter: Dan Berindei
> Priority: Critical
> Labels: testsuite_stability
>
> I got a failure in {{DistributedExecutorFailureTest}} (master only) on my machine because the distributed executor ignores {{CacheNotFoundResponse}} responses:
> {noformat}
> 15:18:45,516 TRACE (transport-thread-NodeA-p30129-t4:) [CommandAwareRpcDispatcher] Response: CacheNotFoundResponse
> 15:18:45,516 TRACE (transport-thread-NodeA-p30129-t4:) [RpcManagerImpl] Response(s) to DistributedExecuteCommand [cache=null, keys=[], callable=org.infinispan.distexec.DistributedExecutorFailoverTest$SleepingSimpleCallable@23a9b62a] is {NodeB-4305=CacheNotFoundResponse}
> 15:18:45,517 ERROR (testng-DistributedExecutorFailoverTest:) [UnitTestTestNGListener] Test testBasicTargetRemoteDistributedCallable(org.infinispan.distexec.DistributedExecutorFailoverTest) failed.
> java.lang.AssertionError: expected:<1> but was:<null>
> at org.testng.AssertJUnit.fail(AssertJUnit.java:59)
> at org.testng.AssertJUnit.failNotEquals(AssertJUnit.java:364)
> at org.testng.AssertJUnit.assertEquals(AssertJUnit.java:80)
> at org.testng.AssertJUnit.assertEquals(AssertJUnit.java:88)
> at org.infinispan.distexec.DistributedExecutorFailoverTest.testBasicTargetRemoteDistributedCallable(DistributedExecutorFailoverTest.java:74)
> {noformat}
> {{RemoteDistributedTaskPart.retrieveResult()}} ignores any response that's not a {{SuccessfulResponse}} and returns {{null}}. It should throw an exception instead, so that the failover policy can retry it on another node.
--
This message was sent by Atlassian JIRA
(v6.3.11#6341)
9 years, 4 months
[JBoss JIRA] (ISPN-5103) Inefficient index updates cause high cost merges and increase overall latency
by Sanne Grinovero (JIRA)
[ https://issues.jboss.org/browse/ISPN-5103?page=com.atlassian.jira.plugin.... ]
Sanne Grinovero edited comment on ISPN-5103 at 1/7/15 12:43 PM:
----------------------------------------------------------------
As a reminder of our IRC chat:
it seems this would be safe as Infinispan Query handles the identifiers in a slightly different way than Hibernate Search / ORM : the {{org.infinispan.query.Transformer}} guarantees a unique relation between each encoded id and makes it possible to reconstruct from it not only the ID instance but also the {{Transformer}} implementation to be used for the transformation itself.
A similar optimisation wouldn't be always safe in ORM world as the {{org.hibernate.search.bridge.TwoWayFieldBridge}} needs to be known, or we'd need to be sure that different {{FieldBridge}} implementations would encode values in some unique way.
For example we'd have a problem when mapping Integer(5) and Long(5) using keyword encoding as they would both map to "5", which could be back-mapped to either the Integer or the Long version, hence deleting the other entry from the index as well.
was (Author: sannegrinovero):
As a reminder of our IRC chat:
it seems this would be safe as Infinispan Query handles the identifiers in a slightly different way than Hibernate Search / ORM : the {{org.infinispan.query.Transformer}} guarantees a unique relation between each encoded id and makes it possible to reconstruct from it not only the ID instance but also the {{Transformer}} implementation to be used for the transformation itself.
A similar optimisation wouldn't be always safe in ORM world as the {{org.hibernate.search.bridge.TwoWayFieldBridge}} needs to be known, or we'd need to be sure that different {{FieldBridge}} implementations would encode values in some unique way.
For example we'd have a problem when mapping Integer(5) and Long(5) using keyword encoding as they would both map to "5".
> Inefficient index updates cause high cost merges and increase overall latency
> -----------------------------------------------------------------------------
>
> Key: ISPN-5103
> URL: https://issues.jboss.org/browse/ISPN-5103
> Project: Infinispan
> Issue Type: Enhancement
> Components: Embedded Querying
> Affects Versions: 7.0.2.Final, 7.1.0.Alpha1
> Reporter: Gustavo Fernandes
> Assignee: Gustavo Fernandes
>
> Currently every change to the index is done Lucene-wise combining two operations:
> * Delete by query, using a boolean query on the id plus the entity class
> * Add
>
> Under high load, specially during merges those numerous deletes provoke very long delays causing high latency.
> We should instead use a simple Lucene Update to add/change documents, since internally it translates to a Delete by term plus an Add operation, and delete by terms are extremely efficient in Lucene.
> Some local tests showed average latency of updating the index using this strategy to drop 4 times, both for the SYNC and ASYNC backends
> With relation to sharing the index between entities, which was the original motivation of the Delete by query plus add strategy, we have two scenarios:
> * Same cache with multiple entity types: that's a non-issue, since obviously there's no id collision in this case
> * Different caches with the same index: this scenario happens when different caches shares the same index, for ex:
> {code}
> @Indexed(indexName=common)
> public class Country { ... }
> @Indexed(indexName=common)
> public class Currency { ... }
> cm.getCache("currencies").put(1, new Currency(...))
> cm.getCache("countries").put(1, new Country(...))
> {code}
> This would require a delete by query in order to persist both a Country and a Currency with id=1.
> It would also require setting "default.exclusive_index_use", "false", with the associated cost of having to reopen the IndexWriter on every operation.
> Given the performance gain of doing a simple Update is considerable, we should make the corner case supported by extra configuration or alternatively, generate a unique @ProvidedId, including the entity class or the cache name that work for all cases described above.
--
This message was sent by Atlassian JIRA
(v6.3.11#6341)
9 years, 4 months
[JBoss JIRA] (ISPN-5103) Inefficient index updates cause high cost merges and increase overall latency
by Sanne Grinovero (JIRA)
[ https://issues.jboss.org/browse/ISPN-5103?page=com.atlassian.jira.plugin.... ]
Sanne Grinovero commented on ISPN-5103:
---------------------------------------
right, but it uses the {{Stored}} flag so it's quite more expensive (both time and space) than most other fields so I'd keep it in mind for future polishing.
> Inefficient index updates cause high cost merges and increase overall latency
> -----------------------------------------------------------------------------
>
> Key: ISPN-5103
> URL: https://issues.jboss.org/browse/ISPN-5103
> Project: Infinispan
> Issue Type: Enhancement
> Components: Embedded Querying
> Affects Versions: 7.0.2.Final, 7.1.0.Alpha1
> Reporter: Gustavo Fernandes
> Assignee: Gustavo Fernandes
>
> Currently every change to the index is done Lucene-wise combining two operations:
> * Delete by query, using a boolean query on the id plus the entity class
> * Add
>
> Under high load, specially during merges those numerous deletes provoke very long delays causing high latency.
> We should instead use a simple Lucene Update to add/change documents, since internally it translates to a Delete by term plus an Add operation, and delete by terms are extremely efficient in Lucene.
> Some local tests showed average latency of updating the index using this strategy to drop 4 times, both for the SYNC and ASYNC backends
> With relation to sharing the index between entities, which was the original motivation of the Delete by query plus add strategy, we have two scenarios:
> * Same cache with multiple entity types: that's a non-issue, since obviously there's no id collision in this case
> * Different caches with the same index: this scenario happens when different caches shares the same index, for ex:
> {code}
> @Indexed(indexName=common)
> public class Country { ... }
> @Indexed(indexName=common)
> public class Currency { ... }
> cm.getCache("currencies").put(1, new Currency(...))
> cm.getCache("countries").put(1, new Country(...))
> {code}
> This would require a delete by query in order to persist both a Country and a Currency with id=1.
> It would also require setting "default.exclusive_index_use", "false", with the associated cost of having to reopen the IndexWriter on every operation.
> Given the performance gain of doing a simple Update is considerable, we should make the corner case supported by extra configuration or alternatively, generate a unique @ProvidedId, including the entity class or the cache name that work for all cases described above.
--
This message was sent by Atlassian JIRA
(v6.3.11#6341)
9 years, 4 months