[JBoss JIRA] (ISPN-9003) Clustered maxIdle expiration
by William Burns (JIRA)
[ https://issues.jboss.org/browse/ISPN-9003?page=com.atlassian.jira.plugin.... ]
William Burns edited comment on ISPN-9003 at 4/3/18 10:22 AM:
--------------------------------------------------------------
> You didn't mention it, so I wasn't 100% sure you'd want to keep running it, and whether it would do the same thing on all the owners.
Yeah reaper is always ran on all nodes for all expired entries. Its behavior wouldn't change.
> I was thinking about the node where the entries are written. And I trust you about the iterator, but if there's a store on the source node, the state provider will call DataContainer.containsKey(), which may or may not refresh the entry (javadoc doesn't say anything about it).
The method containsKey also doesn't update it. Only gets and write methods :)
> maxIdle works based on reads, not just writes, and the difference between reads on the primary and the backups can be just as big as maxIdle is. That's why I said we'd need to update the last-access timestamp on the primary after running the algorithm (and on the backups as well, if the backups can also trigger clustered maxIdle expiration).
Yes, this algorithm would update all nodes last access times after it found that there was even one node that wasn't expired. I forgot to mention that in the algorithm. I originally had it on an earlier line, but then realized that line isn't doable because it is still possible that the entry is expired and we don't want a node to temporarily allow the read and then find it expired immediately after. Adding back in the line again but at the right spot.
was (Author: william.burns):
> You didn't mention it, so I wasn't 100% sure you'd want to keep running it, and whether it would do the same thing on all the owners.
Yeah reaper is always on all nodes for all expired entries. Its behavior wouldn't change.
> I was thinking about the node where the entries are written. And I trust you about the iterator, but if there's a store on the source node, the state provider will call DataContainer.containsKey(), which may or may not refresh the entry (javadoc doesn't say anything about it).
The method containsKey also doesn't update it. Only gets and write methods :)
> maxIdle works based on reads, not just writes, and the difference between reads on the primary and the backups can be just as big as maxIdle is. That's why I said we'd need to update the last-access timestamp on the primary after running the algorithm (and on the backups as well, if the backups can also trigger clustered maxIdle expiration).
Yes, this algorithm would update all nodes last access times after it found that there was even one node that wasn't expired. I forgot to mention that in the algorithm. I originally had it on an earlier line, but then realized that line isn't doable because it is still possible that the entry is expired and we don't want a node to temporarily allow the read and then find it expired immediately after. Adding back in the line again but at the right spot.
> Clustered maxIdle expiration
> ----------------------------
>
> Key: ISPN-9003
> URL: https://issues.jboss.org/browse/ISPN-9003
> Project: Infinispan
> Issue Type: Enhancement
> Reporter: Tristan Tarrant
> Assignee: William Burns
> Fix For: 9.3.0.Beta1, 9.3.0.Final
>
>
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
6 years
[JBoss JIRA] (ISPN-9034) InfinispanExtensionIT should depend on "org.infinispan"
by Ryan Emerson (JIRA)
Ryan Emerson created ISPN-9034:
----------------------------------
Summary: InfinispanExtensionIT should depend on "org.infinispan"
Key: ISPN-9034
URL: https://issues.jboss.org/browse/ISPN-9034
Project: Infinispan
Issue Type: Bug
Components: Test Suite - Server, WildFly modules
Affects Versions: 9.2.1.Final
Reporter: Ryan Emerson
Assignee: Ryan Emerson
Fix For: 9.2.2.Final
Currently the deployment in InfinispanExtensionIT does not have an explicit dependency on "org.infinispan" which means that a ClassNotFoundException will be thrown if certain methods are called on the container. The test should be updated so that the deployment has the appropriate manifest dependency, but if said dependency is removed the test will fail.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
6 years
[JBoss JIRA] (ISPN-9003) Clustered maxIdle expiration
by William Burns (JIRA)
[ https://issues.jboss.org/browse/ISPN-9003?page=com.atlassian.jira.plugin.... ]
William Burns commented on ISPN-9003:
-------------------------------------
> You didn't mention it, so I wasn't 100% sure you'd want to keep running it, and whether it would do the same thing on all the owners.
Yeah reaper is always on all nodes for all expired entries. Its behavior wouldn't change.
> I was thinking about the node where the entries are written. And I trust you about the iterator, but if there's a store on the source node, the state provider will call DataContainer.containsKey(), which may or may not refresh the entry (javadoc doesn't say anything about it).
The method containsKey also doesn't update it. Only gets and write methods :)
> maxIdle works based on reads, not just writes, and the difference between reads on the primary and the backups can be just as big as maxIdle is. That's why I said we'd need to update the last-access timestamp on the primary after running the algorithm (and on the backups as well, if the backups can also trigger clustered maxIdle expiration).
Yes, this algorithm would update all nodes last access times after it found that there was even one node that wasn't expired. I forgot to mention that in the algorithm. I originally had it on an earlier line, but then realized that line isn't doable because it is still possible that the entry is expired and we don't want a node to temporarily allow the read and then find it expired immediately after. Adding back in the line again but at the right spot.
> Clustered maxIdle expiration
> ----------------------------
>
> Key: ISPN-9003
> URL: https://issues.jboss.org/browse/ISPN-9003
> Project: Infinispan
> Issue Type: Enhancement
> Reporter: Tristan Tarrant
> Assignee: William Burns
> Fix For: 9.3.0.Beta1, 9.3.0.Final
>
>
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
6 years
[JBoss JIRA] (ISPN-9003) Clustered maxIdle expiration
by William Burns (JIRA)
[ https://issues.jboss.org/browse/ISPN-9003?page=com.atlassian.jira.plugin.... ]
William Burns edited comment on ISPN-9003 at 4/3/18 10:17 AM:
--------------------------------------------------------------
Talking with [~NadirX] we believe the best approach may be a consensus based removal of maxIdle entries. That is it will do the following things
1. Read occurs on node where maxIdle is expired and sends request to primary owner (blocks the read)
2. Primary owner locks the key
3. Primary sends message to all nodes asking if entry is expired due to maxIdle -(also updates the updated time on all keys - just in case it isn't expired)-
4. Primary receives responses to decide if it is expired
5a. Primary responds to read node and tells them if it was expired
5b. Primary removes from all nodes if it was expired or sends request to all nodes to update last access time.
6. Cluster wide expiration occurs if necessary allowing for listeners to be invoked (including cluster listeners - which we didn't do before)
The above approach gives no overhead for reads when the entry is not present and is not expired. The only overhead is if a node finds that the entry was expired via maxIdle and then must block to confirm if it should be removed. However the chance of this occurring seems a bit low depending on what maxIdle was set to. If the expiration reaper finds the expired entry first it will properly update its timestamps or remove as needed.
Another side effect of this is that maxIdle may be refreshed for all entries that are transferred via state transfer since I don't think the access time is currently replicated when state transfer occurs.
was (Author: william.burns):
Talking with [~NadirX] we believe the best approach may be a consensus based removal of maxIdle entries. That is it will do the following things
1. Read occurs on node where maxIdle is expired and sends request to primary owner (blocks the read)
2. Primary owner locks the key
3. Primary sends message to all nodes asking if entry is expired due to maxIdle -(also updates the updated time on all keys - just in case it isn't expired)-
4. Primary receives responses to decide if it is expired
5a. Primary responds to read node and tells them if it was expired
5b. Primary removes from all nodes if it was expired
6. Cluster wide expiration occurs if necessary allowing for listeners to be invoked (including cluster listeners - which we didn't do before)
The above approach gives no overhead for reads when the entry is not present and is not expired. The only overhead is if a node finds that the entry was expired via maxIdle and then must block to confirm if it should be removed. However the chance of this occurring seems a bit low depending on what maxIdle was set to. If the expiration reaper finds the expired entry first it will properly update its timestamps or remove as needed.
Another side effect of this is that maxIdle may be refreshed for all entries that are transferred via state transfer since I don't think the access time is currently replicated when state transfer occurs.
> Clustered maxIdle expiration
> ----------------------------
>
> Key: ISPN-9003
> URL: https://issues.jboss.org/browse/ISPN-9003
> Project: Infinispan
> Issue Type: Enhancement
> Reporter: Tristan Tarrant
> Assignee: William Burns
> Fix For: 9.3.0.Beta1, 9.3.0.Final
>
>
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
6 years
[JBoss JIRA] (ISPN-9003) Clustered maxIdle expiration
by Dan Berindei (JIRA)
[ https://issues.jboss.org/browse/ISPN-9003?page=com.atlassian.jira.plugin.... ]
Dan Berindei commented on ISPN-9003:
------------------------------------
> We already have a timer task, the expiration reaper
You didn't mention it, so I wasn't 100% sure you'd want to keep running it, and whether it would do the same thing on all the owners.
> Also state transfer entities are not refreshed from an iterator invocation (they only refresh on a get or write operation).
I was thinking about the node where the entries are written. And I trust you about the iterator, but if there's a store on the source node, the state provider will call {{DataContainer.containsKey()}}, which may or may not refresh the entry (javadoc doesn't say anything about it).
> In regards to divergent clocks, this is not an issue as long as we don't send timestampes across nodes (which we don't). Every node uses the provided offset of its own current time to detect expiration. Thus since writes should occur somewhat close to each other in absolute time the expiration of various nodes would be done with only a time differential of just about the latency time between primary and backup. So this could cause an entry to be resurrected if someone read it between this time, but tbh this seems more than okay to me. Lifespan works in a similar fashion, but it actually errs on the other side in that if it expired on any node it assumes all are expired (which is much less costly network wise).
{{maxIdle}} works based on reads, not just writes, and the difference between reads on the primary and the backups can be just as big as {{maxIdle}} is. That's why I said we'd need to update the last-access timestamp on the primary after running the algorithm (and on the backups as well, if the backups can also trigger clustered {{maxIdle}} expiration).
> Clustered maxIdle expiration
> ----------------------------
>
> Key: ISPN-9003
> URL: https://issues.jboss.org/browse/ISPN-9003
> Project: Infinispan
> Issue Type: Enhancement
> Reporter: Tristan Tarrant
> Assignee: William Burns
> Fix For: 9.3.0.Beta1, 9.3.0.Final
>
>
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
6 years
[JBoss JIRA] (ISPN-8903) Conflict resolution not initiated if node rejoins with same topology
by Dan Berindei (JIRA)
[ https://issues.jboss.org/browse/ISPN-8903?page=com.atlassian.jira.plugin.... ]
Dan Berindei edited comment on ISPN-8903 at 4/3/18 9:46 AM:
------------------------------------------------------------
B couldn't update any entries, because it still uses a cache topology where A co-owns all the segments, and even if accessible A would have rejected the updates because of the lower topology id. In my view that means no conflict, and no reason to run the merge policy.
was (Author: dan.berindei):
A couldn't update any entries, because it still uses a cache topology where B co-owns all the segments, and even if accessible B would have rejected the updates because of the lower topology id. In my view that means no conflict, and no reason to run the merge policy.
> Conflict resolution not initiated if node rejoins with same topology
> --------------------------------------------------------------------
>
> Key: ISPN-8903
> URL: https://issues.jboss.org/browse/ISPN-8903
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Affects Versions: 9.2.0.Final
> Reporter: Ryan Emerson
> Assignee: Ryan Emerson
> Fix For: 9.2.1.Final, 9.3.0.Final
>
>
> The current logic in PreferAvailabilityStrategy and PreferConsistencyStrategy assumes that when a split brain occurs, the two partitions will continue to operate independently before a merge occurs.
> Consider a cluster \{A,B\} which partitions into P1 \{A\} and P2 \{B\}. P1 continues to operate and update cache entries, however P2 makes no progress (possibly down to a long GC pause). When P2 merges into P1, no conflict resolution occurs as the maxTopology contains all of the possible owners. Conflict resolution should be attempted in this scenario, as it's possible that entries have been put to P1 during the partition and therefore P2 will have stale values.
> This can be reproduced by creating two nodes, pausing one process, wait for split and then resuming the process. No CR will occur.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
6 years
[JBoss JIRA] (ISPN-8903) Conflict resolution not initiated if node rejoins with same topology
by Dan Berindei (JIRA)
[ https://issues.jboss.org/browse/ISPN-8903?page=com.atlassian.jira.plugin.... ]
Dan Berindei commented on ISPN-8903:
------------------------------------
A couldn't update any entries, because it still uses a cache topology where B co-owns all the segments, and even if accessible B would have rejected the updates because of the lower topology id. In my view that means no conflict, and no reason to run the merge policy.
> Conflict resolution not initiated if node rejoins with same topology
> --------------------------------------------------------------------
>
> Key: ISPN-8903
> URL: https://issues.jboss.org/browse/ISPN-8903
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Affects Versions: 9.2.0.Final
> Reporter: Ryan Emerson
> Assignee: Ryan Emerson
> Fix For: 9.2.1.Final, 9.3.0.Final
>
>
> The current logic in PreferAvailabilityStrategy and PreferConsistencyStrategy assumes that when a split brain occurs, the two partitions will continue to operate independently before a merge occurs.
> Consider a cluster \{A,B\} which partitions into P1 \{A\} and P2 \{B\}. P1 continues to operate and update cache entries, however P2 makes no progress (possibly down to a long GC pause). When P2 merges into P1, no conflict resolution occurs as the maxTopology contains all of the possible owners. Conflict resolution should be attempted in this scenario, as it's possible that entries have been put to P1 during the partition and therefore P2 will have stale values.
> This can be reproduced by creating two nodes, pausing one process, wait for split and then resuming the process. No CR will occur.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
6 years
[JBoss JIRA] (ISPN-8689) Wildfly-modules should be created using wildfly feature packs
by Ryan Emerson (JIRA)
[ https://issues.jboss.org/browse/ISPN-8689?page=com.atlassian.jira.plugin.... ]
Ryan Emerson commented on ISPN-8689:
------------------------------------
[~fercoli] +1 that makes sense. Tbh I was thinking of creating several feature packs so that we can make the loaded dependencies as fine grained as possible, i.e. basic embedded/counters/query etc. Once I start to look at this issue I will know more about what is possible and will update the ticket accordingly.
> Wildfly-modules should be created using wildfly feature packs
> -------------------------------------------------------------
>
> Key: ISPN-8689
> URL: https://issues.jboss.org/browse/ISPN-8689
> Project: Infinispan
> Issue Type: Enhancement
> Components: WildFly modules
> Reporter: Ryan Emerson
> Assignee: Ryan Emerson
> Fix For: 9.3.0.Final
>
>
> Currently we utilise an ant script to create the wildfly-module zips, instead we should utilise the `wildfly-feature-pack-build-maven-plugin` to generate them.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
6 years