[JBoss JIRA] (ISPN-6981) AffinityIndexManager fails to index documents in async mode
by Gustavo Fernandes (JIRA)
[ https://issues.jboss.org/browse/ISPN-6981?page=com.atlassian.jira.plugin.... ]
Gustavo Fernandes updated ISPN-6981:
------------------------------------
Status: Pull Request Sent (was: Open)
Git Pull Request: https://github.com/infinispan/infinispan/pull/4553
> AffinityIndexManager fails to index documents in async mode
> -----------------------------------------------------------
>
> Key: ISPN-6981
> URL: https://issues.jboss.org/browse/ISPN-6981
> Project: Infinispan
> Issue Type: Bug
> Components: Embedded Querying
> Affects Versions: 9.0.0.Alpha4
> Reporter: Gustavo Fernandes
> Assignee: Gustavo Fernandes
>
> During topology changes in async mode ("<index>.worker.execution" set to "async"), the {{AfinityIndexManager}} sometimes fails to index entries. Some of errors thrown:
> {noformat}
> 19:06:04,638 ERROR (Hibernate Search: Index updates queue processor for index entity.5-1) [LogErrorHandler] HSEARCH000058: Exception occurred
> org.apache.lucene.store.LockObtainFailedException: lock instance already assigned
> {noformat}
> {noformat}
> 18:53:59,553 ERROR (Hibernate Search: Index updates queue processor for index entity.2-1) [LuceneBackendQueueTask] HSEARCH000073: Error in
> backend
> org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
> at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:720)
> at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:734)
> {noformat}
> {noformat}
> 18:55:31,328 ERROR (Hibernate Search: Commit Scheduler for index entity.143-1) [AffinityErrorHandler] HSEARCH000117: IOException
> on the IndexWriter
> java.io.IOException: This lock is no longer being held
> at org.infinispan.lucene.impl.BaseLuceneLock.ensureValid(BaseLuceneLock.java:89)
> at org.apache.lucene.store.LockValidatingDirectoryWrapper.createOutput(LockValidatingDirectoryWrapper.java:43)
> at org.apache.lucene.store.TrackingDirectoryWrapper.createOutput(TrackingDirectoryWrapper.java:43)
> at org.apache.lucene.codecs.lucene50.Lucene50PostingsWriter.<init>(Lucene50PostingsWriter.java:105)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
8 years
[JBoss JIRA] (ISPN-6981) AffinityIndexManager fails to index documents in async mode
by Gustavo Fernandes (JIRA)
[ https://issues.jboss.org/browse/ISPN-6981?page=com.atlassian.jira.plugin.... ]
Gustavo Fernandes updated ISPN-6981:
------------------------------------
Summary: AffinityIndexManager fails to index documents in async mode (was: AffinityIndexManager sometimes fails to index document in async mode)
> AffinityIndexManager fails to index documents in async mode
> -----------------------------------------------------------
>
> Key: ISPN-6981
> URL: https://issues.jboss.org/browse/ISPN-6981
> Project: Infinispan
> Issue Type: Bug
> Components: Embedded Querying
> Affects Versions: 9.0.0.Alpha4
> Reporter: Gustavo Fernandes
> Assignee: Gustavo Fernandes
>
> During topology changes in async mode ("<index>.worker.execution" set to "async"), the {{AfinityIndexManager}} sometimes fails to index entries. Some of errors thrown:
> {noformat}
> 19:06:04,638 ERROR (Hibernate Search: Index updates queue processor for index entity.5-1) [LogErrorHandler] HSEARCH000058: Exception occurred
> org.apache.lucene.store.LockObtainFailedException: lock instance already assigned
> {noformat}
> {noformat}
> 18:53:59,553 ERROR (Hibernate Search: Index updates queue processor for index entity.2-1) [LuceneBackendQueueTask] HSEARCH000073: Error in
> backend
> org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
> at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:720)
> at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:734)
> {noformat}
> {noformat}
> 18:55:31,328 ERROR (Hibernate Search: Commit Scheduler for index entity.143-1) [AffinityErrorHandler] HSEARCH000117: IOException
> on the IndexWriter
> java.io.IOException: This lock is no longer being held
> at org.infinispan.lucene.impl.BaseLuceneLock.ensureValid(BaseLuceneLock.java:89)
> at org.apache.lucene.store.LockValidatingDirectoryWrapper.createOutput(LockValidatingDirectoryWrapper.java:43)
> at org.apache.lucene.store.TrackingDirectoryWrapper.createOutput(TrackingDirectoryWrapper.java:43)
> at org.apache.lucene.codecs.lucene50.Lucene50PostingsWriter.<init>(Lucene50PostingsWriter.java:105)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
8 years
[JBoss JIRA] (ISPN-6919) Improve non-tx writes (triangle)
by Dan Berindei (JIRA)
[ https://issues.jboss.org/browse/ISPN-6919?page=com.atlassian.jira.plugin.... ]
Dan Berindei commented on ISPN-6919:
------------------------------------
[~ksobolew] I'm afraid the functional API's commands can't run without exclusive access to the entries they modify, either.
[~pruivo] numOwners == 1 should really be described separately. If we exclude that, the first case becomes "originator != primary"
What happens when the backups install a new topology in which they're no longer owners, or if the backups crash?
Also, why do we need the replication to the backups to be ordered? That won't help if the primary owner crashes, and another node becomes the primary owner.
More general, we should really have some diagrams to discuss how the triangle is supposed to handle the various corner cases that appear when a node joins/leaves/crashes, or when there's a split+merge. I'm not saying this approach won't work with topology changes, it's just that I have no idea whether how it's supposed to work.
> Improve non-tx writes (triangle)
> --------------------------------
>
> Key: ISPN-6919
> URL: https://issues.jboss.org/browse/ISPN-6919
> Project: Infinispan
> Issue Type: Feature Request
> Components: Core
> Reporter: Pedro Ruivo
> Assignee: Pedro Ruivo
>
> The current algorithm has 4 messages in the network (worst case) happended sequentially:
> 1: originator => primary owner
> 2: primary owner => backups
> 3: backups => primary owner (ack)
> 4: primary owner => originator (reply)
> The algorithm can be improved to the following:
> 1: originator => primary owner
> 2: primary owner => backups & primary owner => originator (parallel)
> 3: backups => originator & backups => primary owners (acks)
> The main flow would be: originator => primary => backups => originator (<= there is the triangle :) )
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
8 years
[JBoss JIRA] (ISPN-6919) Improve non-tx writes (triangle)
by Dan Berindei (JIRA)
[ https://issues.jboss.org/browse/ISPN-6919?page=com.atlassian.jira.plugin.... ]
Dan Berindei edited comment on ISPN-6919 at 9/7/16 10:58 AM:
-------------------------------------------------------------
[~ksobolew] I'm afraid the functional API's commands can't run without exclusive access to the entries they modify, either.
[~pruivo] numOwners == 1 should really be described separately. If we exclude that, the first case becomes "originator != primary"
What happens when the backups install a new topology in which they're no longer owners, or if the backups crash?
Also, why do we need the replication to the backups to be ordered? That won't help if the primary owner crashes, and another node becomes the primary owner.
More general, we should really have some diagrams to discuss how the triangle is supposed to handle the various corner cases that appear when a node joins/leaves/crashes, or when there's a split+merge. I'm not saying this approach won't work with topology changes, it's just that I have no idea how it's supposed to work.
was (Author: dan.berindei):
[~ksobolew] I'm afraid the functional API's commands can't run without exclusive access to the entries they modify, either.
[~pruivo] numOwners == 1 should really be described separately. If we exclude that, the first case becomes "originator != primary"
What happens when the backups install a new topology in which they're no longer owners, or if the backups crash?
Also, why do we need the replication to the backups to be ordered? That won't help if the primary owner crashes, and another node becomes the primary owner.
More general, we should really have some diagrams to discuss how the triangle is supposed to handle the various corner cases that appear when a node joins/leaves/crashes, or when there's a split+merge. I'm not saying this approach won't work with topology changes, it's just that I have no idea whether how it's supposed to work.
> Improve non-tx writes (triangle)
> --------------------------------
>
> Key: ISPN-6919
> URL: https://issues.jboss.org/browse/ISPN-6919
> Project: Infinispan
> Issue Type: Feature Request
> Components: Core
> Reporter: Pedro Ruivo
> Assignee: Pedro Ruivo
>
> The current algorithm has 4 messages in the network (worst case) happended sequentially:
> 1: originator => primary owner
> 2: primary owner => backups
> 3: backups => primary owner (ack)
> 4: primary owner => originator (reply)
> The algorithm can be improved to the following:
> 1: originator => primary owner
> 2: primary owner => backups & primary owner => originator (parallel)
> 3: backups => originator & backups => primary owners (acks)
> The main flow would be: originator => primary => backups => originator (<= there is the triangle :) )
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
8 years
[JBoss JIRA] (ISPN-6919) Improve non-tx writes (triangle)
by Pedro Ruivo (JIRA)
[ https://issues.jboss.org/browse/ISPN-6919?page=com.atlassian.jira.plugin.... ]
Pedro Ruivo updated ISPN-6919:
------------------------------
Status: Open (was: New)
> Improve non-tx writes (triangle)
> --------------------------------
>
> Key: ISPN-6919
> URL: https://issues.jboss.org/browse/ISPN-6919
> Project: Infinispan
> Issue Type: Feature Request
> Components: Core
> Reporter: Pedro Ruivo
> Assignee: Pedro Ruivo
>
> The current algorithm has 4 messages in the network (worst case) happended sequentially:
> 1: originator => primary owner
> 2: primary owner => backups
> 3: backups => primary owner (ack)
> 4: primary owner => originator (reply)
> The algorithm can be improved to the following:
> 1: originator => primary owner
> 2: primary owner => backups & primary owner => originator (parallel)
> 3: backups => originator & backups => primary owners (acks)
> The main flow would be: originator => primary => backups => originator (<= there is the triangle :) )
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
8 years
[JBoss JIRA] (ISPN-6919) Improve non-tx writes (triangle)
by Pedro Ruivo (JIRA)
[ https://issues.jboss.org/browse/ISPN-6919?page=com.atlassian.jira.plugin.... ]
Work on ISPN-6919 started by Pedro Ruivo.
-----------------------------------------
> Improve non-tx writes (triangle)
> --------------------------------
>
> Key: ISPN-6919
> URL: https://issues.jboss.org/browse/ISPN-6919
> Project: Infinispan
> Issue Type: Feature Request
> Components: Core
> Reporter: Pedro Ruivo
> Assignee: Pedro Ruivo
>
> The current algorithm has 4 messages in the network (worst case) happended sequentially:
> 1: originator => primary owner
> 2: primary owner => backups
> 3: backups => primary owner (ack)
> 4: primary owner => originator (reply)
> The algorithm can be improved to the following:
> 1: originator => primary owner
> 2: primary owner => backups & primary owner => originator (parallel)
> 3: backups => originator & backups => primary owners (acks)
> The main flow would be: originator => primary => backups => originator (<= there is the triangle :) )
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
8 years