[JBoss JIRA] (ISPN-11176) XSite Max Idle
by Dan Berindei (Jira)
[ https://issues.redhat.com/browse/ISPN-11176?page=com.atlassian.jira.plugi... ]
Dan Berindei commented on ISPN-11176:
-------------------------------------
I have an alternative proposal that I've been mulling over the last few days.
*TLDR;* Pre-expire entries at time {{last access + max-idle timeout}}, actually expire them at time {{last access + max-idle timeout + max-idle delay}}, or as soon as we know all other sites have also pre-expired the entry. Reads of pre-expired entries do not keep the entries alive.
How about, instead of sending touch commands based on the time the entry was last read, we send them based on the time the entry is supposed to expire?
The purge expired entries task would scan not just for entries that are already expired, but also for entries that are expired based on the last positive-touch command sent to the remote sites. Then it's going to send batches of touch+ commands to the remote sites for all the entries in the 2nd set (positive-touch for the ones that have newer accesses locally and negative-touch for the ones that don't).
When a read wants to remove an expired entry, it will first check whether the other sites sent a touch+ command for that entry. If there is a negative touch from all backups, or if the entry should have expired more than a {{max-idle delay}} (e.g. 1 min) ago, then the entry is considered truly expired and removed in the local cluster.
Since there is no synchronous communication with the remote sites, reads that happen after the entry should have expired (or maybe just after the local site sent the negative-touch command) must not extend the lifespan of the entry.
In case site 1's positive-touch command did take more than {{max-idle delay}} to reach site 2, and site 2 has already removed the entry, site 2 will send back (through IRAC) a remove-expired command to force site 1 to remove the entry.
In order to remove the likelihood of this happening, we can send the positive-touch {{max-idle delay}} before the time the entry would expire based on the last sent positive-touch command. This requires the max-idle timeout to be at least twice as big as {{max-idle delay}}, or it wouldn't be very efficient, but it seems like a reasonable limitation.
The advantage over option 2 is that for entries that are read often, we only send a remote touch command once per {{max-idle timeout - max-idle delay}}.
The advantage over option 1 is that all the x-site RPCs are in the asynchronous, so all read operations are fast.
The main disadvantage over option 1 is that we can have reads that see a value but don't keep it alive. But IMO it matches how IRAC would handle the application updating a value in site 1 (in order to "touch" it) and removing the same value in site 2 (because the application considers it expired).
Another disadvantage over option 1 is that we would need to send *-touch commands from all owners, if we rely on the backup owners sending positive-touch commands when they become primary owner that might be too late.
The disadvantage over option 2 is that, just like option 1, we can't have a purely active-backup relationship between sites any more. Here, each site has to know which other sites are active and keep per-entry metadata about which sites sent *-touch commands, so the take-offline behaviour may need tweaking.
Just like option 2, when a remote site is inaccessible but not yet offline, *-touch commands may be lost and max-idle entries may expire prematurely. Option 1 would make reads on entries that should expire time out instead.
When a remote site is brought back online, it would need x-site state transfer to include the timestamp of the last access and the timestamp of the last sent *-touch command timestamp so that the lifetime of the transferred entries is kept in sync. This also requires clock skew between all machines in all sites to be smaller than {{max-idle delay}}, but I think it's a reasonable assumption, and we could warn when not satisfied. I think this is a middle ground between option 1 (more tolerant, at most delaying the expiration with the clock skew) and option 2 (less tolerant, possibly expiring entries on just one site).
> XSite Max Idle
> --------------
>
> Key: ISPN-11176
> URL: https://issues.redhat.com/browse/ISPN-11176
> Project: Infinispan
> Issue Type: Enhancement
> Components: Cross-Site Replication, Expiration
> Reporter: Will Burns
> Assignee: Will Burns
> Priority: Major
> Fix For: 12.0.0.Final
>
>
> Max idle expiration currently doesn't work with xsite. That is if an entry was written and replicated to both sites but one site never reads the value, but the other does. If they then need to read the value from the other site it will be expired (assuming the max idle time has elapsed).
> There are a few ways we can do this.
> 1. Keep access times local to every site. When a site finds an entry is expired it asks the other site(s) if it has a more recent access. If a site is known to have gone down we should touch all entries, since they may not have updated access times. Requires very little additional xsite communication.
> 2. Batch touch commands and only send every so often. Has window of loss, but should be small. Requires more site usage. Wouldn't work for really low max idle times as an entry could expire before the touch command is replicated.
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
5 years, 8 months
[JBoss JIRA] (ISPN-11176) XSite Max Idle
by Dan Berindei (Jira)
[ https://issues.redhat.com/browse/ISPN-11176?page=com.atlassian.jira.plugi... ]
Dan Berindei commented on ISPN-11176:
-------------------------------------
{quote}At the last F2F, while discussing locking w/x-site, I brought up the possibility of partitioning entries with a primary site (in the similar way that we partition them within a single cluster). With this in place, max-idle processing would be initiated only on the primary site for a given entry and cascaded to backup sites accordingly. It would, however, require that reads from the backup sites "touch" the primary site. Ideally, reads from a backup site would be rare - and only happen on failure of the primary site. For normal operation, this strategy has the advantage of minimal x-site traffic and simplifies max-idle processing by using a single authority for determining when to expire a given entry thereby preventing premature expirations.{quote}
Thanks for this [~pferraro], I meant to mention partitioning entries w/ a meta-consistent-hash, but I forgot about it.
Unfortunately, I see 2 problems with this approach:
1. We don't have a proper meta-cluster-view. Each site only knows which other sites still have a site master in the bridge cluster view, and which sites are "online" as backups of caches from the local cluster. None of them are good enough as a source of truth to determine the primary site of a key IMO, so we'd need something new.
2. We don't know 100% what the users want from x-site max-idle, but one of the requirements seems to be that it has to work with a "dumb" load balancer, directing requests to different sites randomly.
> XSite Max Idle
> --------------
>
> Key: ISPN-11176
> URL: https://issues.redhat.com/browse/ISPN-11176
> Project: Infinispan
> Issue Type: Enhancement
> Components: Cross-Site Replication, Expiration
> Reporter: Will Burns
> Assignee: Will Burns
> Priority: Major
> Fix For: 12.0.0.Final
>
>
> Max idle expiration currently doesn't work with xsite. That is if an entry was written and replicated to both sites but one site never reads the value, but the other does. If they then need to read the value from the other site it will be expired (assuming the max idle time has elapsed).
> There are a few ways we can do this.
> 1. Keep access times local to every site. When a site finds an entry is expired it asks the other site(s) if it has a more recent access. If a site is known to have gone down we should touch all entries, since they may not have updated access times. Requires very little additional xsite communication.
> 2. Batch touch commands and only send every so often. Has window of loss, but should be small. Requires more site usage. Wouldn't work for really low max idle times as an entry could expire before the touch command is replicated.
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
5 years, 8 months
[JBoss JIRA] (ISPN-11176) XSite Max Idle
by Dan Berindei (Jira)
[ https://issues.redhat.com/browse/ISPN-11176?page=com.atlassian.jira.plugi... ]
Dan Berindei commented on ISPN-11176:
-------------------------------------
{quote}
The problem if it is async is you have a window of consistency loss if a node is taken down. We can mitigate this issue by performing a touch all when it occurs. I was told this is not acceptable, and I agree, for clustered max idle where it would occur for a single node. However, I believe that it should be okay when it would occur only when an entire site is lost.
{quote}
Does the consistency loss mean a write could be undone, or is it something else that's maybe more palatable to users?
I agree sending a touch command for all non-expired entries would be very expensive, especially for x-site where the user would make some estimations before deployment of how much bandwidth Infinispan would require, and having to send+process lots of max-idle commands would increase the latency of all the other x-site commands going through the same site masters.
One thing that's not clear to me is when you say "an entire site is lost", are you talking about a site being taken offline? Because a site can easily disappear from the bridge cluster's view for a while just because its site master crashed.
OTOH the take offline policies are maybe too lenient (although I haven't checked if Pedro made any recent changes w/ IRAC), so a sync x-site RPC is likely to time out before the site is taken offline.
> XSite Max Idle
> --------------
>
> Key: ISPN-11176
> URL: https://issues.redhat.com/browse/ISPN-11176
> Project: Infinispan
> Issue Type: Enhancement
> Components: Cross-Site Replication, Expiration
> Reporter: Will Burns
> Assignee: Will Burns
> Priority: Major
> Fix For: 12.0.0.Final
>
>
> Max idle expiration currently doesn't work with xsite. That is if an entry was written and replicated to both sites but one site never reads the value, but the other does. If they then need to read the value from the other site it will be expired (assuming the max idle time has elapsed).
> There are a few ways we can do this.
> 1. Keep access times local to every site. When a site finds an entry is expired it asks the other site(s) if it has a more recent access. If a site is known to have gone down we should touch all entries, since they may not have updated access times. Requires very little additional xsite communication.
> 2. Batch touch commands and only send every so often. Has window of loss, but should be small. Requires more site usage. Wouldn't work for really low max idle times as an entry could expire before the touch command is replicated.
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
5 years, 8 months
[JBoss JIRA] (ISPN-11942) JCache CacheEntryCreatedListener got TimeoutException ISPN000299 when listener calling code which accessing the created cache value
by Sammy Chu (Jira)
[ https://issues.redhat.com/browse/ISPN-11942?page=com.atlassian.jira.plugi... ]
Sammy Chu commented on ISPN-11942:
----------------------------------
Debugging found 2 issues:
# Before refactoring of ISPN-8651, the JCache#get() operation will NOT involve locking if no JCache expiry policy set. After refactoring, due to using ReadWriteMap, it will always do locking even no JCache expriy policy set, and always produce this issue.
# If JCache expiry policy is set, even before refactoring of ISPN-8651, this issue is reproducible.
This is strange that event listener should be run in the same thread, right? So theoretically even the cache entry is locked, the lock owner should be the same and should not have this issue happened, right?
> JCache CacheEntryCreatedListener got TimeoutException ISPN000299 when listener calling code which accessing the created cache value
> -----------------------------------------------------------------------------------------------------------------------------------
>
> Key: ISPN-11942
> URL: https://issues.redhat.com/browse/ISPN-11942
> Project: Infinispan
> Issue Type: Bug
> Components: API, Core, JCache
> Affects Versions: 11.0.3.Final
> Reporter: Sammy Chu
> Priority: Major
> Attachments: ISPN-11942-fail.log, ISPN-11942-success.log, ISPN-11942.zip
>
>
> We are running into issue when upgrading infinispan from 9.1.7.Final to 9.2.0.Final and got the TimeoutException: ISPN000299: Unable to acquire lock after X seconds for key Y...
> We reduced our settings and finally reproduce it in a simple setup:
> # configured a local cache (i.e. TestCache) with default settings
> # a simple service bean ServiceA with a dummy method foo() annotated with @CacheResult(cacheName = 'TestCache'), which just return string "THIS_IS_RESULT_OF_FOO"
> # a simple service bean ServiceB with a dummy method bar() which just return (ServiceA.foo() + "_PROCESSED_BY_BAR")
> # a JCache CacheEntryCreatedListener which calling ServiceB.bar() and register to TestCache
> When calling ServiceA.foo(), expecting ServiceA will generate a random string and got cached, then trigger the cache entry created listener to notify ServiceB to regenerate its value.
> We also tested on latest stable version (10.1.8.Final) and still have such issue. We checked versions one by one and confirmed this bug introduced between change of 9.1.7 and 9.2.0.
> [^ISPN-11942.zip] is a simple standalone application to reproduce this issue, change infinispan version from 10.1.8.Final to 9.1.7.Final will work correctly. Attached the trace log [^ISPN-11942-fail.log] when stay on 10.1.8.Final and [^ISPN-11942-success.log] when changed to 9.1.7.Final.
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
5 years, 8 months
[JBoss JIRA] (ISPN-12207) Functional Map APIs not functional properly with invalidation cache
by Sammy Chu (Jira)
[ https://issues.redhat.com/browse/ISPN-12207?page=com.atlassian.jira.plugi... ]
Sammy Chu edited comment on ISPN-12207 at 8/10/20 4:59 AM:
-----------------------------------------------------------
This issue also affecting infinispan-jcache module functionality as since 9.1.7+ (ISPN-8651) it is refactored to use FunctionalMap APIs, which is not working when dealing with invalidation cache.
was (Author: sammychu):
This issue also affecting infinispan-jcache module functionality as since 9.1.7+ it is refactored to use FunctionalMap APIs, which is not working when dealing with invalidation cache.
> Functional Map APIs not functional properly with invalidation cache
> -------------------------------------------------------------------
>
> Key: ISPN-12207
> URL: https://issues.redhat.com/browse/ISPN-12207
> Project: Infinispan
> Issue Type: Bug
> Components: API, Core
> Affects Versions: 11.0.3.Final
> Reporter: Sammy Chu
> Priority: Critical
> Attachments: ISPN-12207.source.zip, ISPN-12207.war
>
>
> Functional Map API not working properly when working with invalidation cache. When in a clustered environment, expecting cache put and cache remove will notify cluster members to do cache eviction when dealing when invalidation cache. But observed no cache eviction occurred in cluster members when using Functional Map APIs to deal with invalidation cache.
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
5 years, 8 months
[JBoss JIRA] (ISPN-11942) JCache CacheEntryCreatedListener got TimeoutException ISPN000299 when listener calling code which accessing the created cache value
by Sammy Chu (Jira)
[ https://issues.redhat.com/browse/ISPN-11942?page=com.atlassian.jira.plugi... ]
Sammy Chu updated ISPN-11942:
-----------------------------
Component/s: API
Core
> JCache CacheEntryCreatedListener got TimeoutException ISPN000299 when listener calling code which accessing the created cache value
> -----------------------------------------------------------------------------------------------------------------------------------
>
> Key: ISPN-11942
> URL: https://issues.redhat.com/browse/ISPN-11942
> Project: Infinispan
> Issue Type: Bug
> Components: API, Core, JCache
> Affects Versions: 11.0.3.Final
> Reporter: Sammy Chu
> Priority: Major
> Attachments: ISPN-11942-fail.log, ISPN-11942-success.log, ISPN-11942.zip
>
>
> We are running into issue when upgrading infinispan from 9.1.7.Final to 9.2.0.Final and got the TimeoutException: ISPN000299: Unable to acquire lock after X seconds for key Y...
> We reduced our settings and finally reproduce it in a simple setup:
> # configured a local cache (i.e. TestCache) with default settings
> # a simple service bean ServiceA with a dummy method foo() annotated with @CacheResult(cacheName = 'TestCache'), which just return string "THIS_IS_RESULT_OF_FOO"
> # a simple service bean ServiceB with a dummy method bar() which just return (ServiceA.foo() + "_PROCESSED_BY_BAR")
> # a JCache CacheEntryCreatedListener which calling ServiceB.bar() and register to TestCache
> When calling ServiceA.foo(), expecting ServiceA will generate a random string and got cached, then trigger the cache entry created listener to notify ServiceB to regenerate its value.
> We also tested on latest stable version (10.1.8.Final) and still have such issue. We checked versions one by one and confirmed this bug introduced between change of 9.1.7 and 9.2.0.
> [^ISPN-11942.zip] is a simple standalone application to reproduce this issue, change infinispan version from 10.1.8.Final to 9.1.7.Final will work correctly. Attached the trace log [^ISPN-11942-fail.log] when stay on 10.1.8.Final and [^ISPN-11942-success.log] when changed to 9.1.7.Final.
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
5 years, 8 months
[JBoss JIRA] (ISPN-12207) Functional Map APIs not functional properly with invalidation cache
by Sammy Chu (Jira)
[ https://issues.redhat.com/browse/ISPN-12207?page=com.atlassian.jira.plugi... ]
Sammy Chu commented on ISPN-12207:
----------------------------------
This issue also affecting infinispan-jcache module functionality as since 9.1.7+ it is refactored to use FunctionalMap APIs, which is not working when dealing with invalidation cache.
> Functional Map APIs not functional properly with invalidation cache
> -------------------------------------------------------------------
>
> Key: ISPN-12207
> URL: https://issues.redhat.com/browse/ISPN-12207
> Project: Infinispan
> Issue Type: Bug
> Components: API, Core
> Affects Versions: 11.0.3.Final
> Reporter: Sammy Chu
> Priority: Critical
> Attachments: ISPN-12207.source.zip, ISPN-12207.war
>
>
> Functional Map API not working properly when working with invalidation cache. When in a clustered environment, expecting cache put and cache remove will notify cluster members to do cache eviction when dealing when invalidation cache. But observed no cache eviction occurred in cluster members when using Functional Map APIs to deal with invalidation cache.
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
5 years, 8 months
[JBoss JIRA] (ISPN-12207) Functional Map APIs not functional properly with invalidation cache
by Sammy Chu (Jira)
[ https://issues.redhat.com/browse/ISPN-12207?page=com.atlassian.jira.plugi... ]
Sammy Chu updated ISPN-12207:
-----------------------------
Steps to Reproduce:
# Deploy [^ISPN-12207.war] to any Servlet container (e.g. Wildfly) with at least 2 nodes
# Navigate to http://<domain>/ISPN-12207/get.jsp in node 1 (this will generate the initial data and put to the cache, note invalidation cache will call `putForExternalRead()` instead
# Navigate to http://<domain>/ISPN-12207/get.jsp in node 2 (similarly, generate initial data)
# Navigate to http://<domain>/ISPN-12207/remove.jsp in node 1, to evict the cache data
# Navigate to http://<domain>/ISPN-12207/get.jsp in node 2, expecting invalidation cache data got regenerated, but actually still keeping the previous data
# Navigate to http://<domain>/ISPN-12207/update.jsp in node 1, to force regenerate data and put to the cache (invalidation cache also will call `put()` instead of `putForExternalRead()`)
# Navigate to http://<domain>/ISPN-12207/get.jsp in node 2, expecting invalidation cache data got regenerated, but actually still keeping the previous data
was:
# Deploy [^ISPN-12207.war] to any Servlet container (e.g. Wildfly) with at least 2 nodes
# Navigate to http://<domain>/ISPN-12207/get.jsp in node 1 (this will generate the initial data and put to the cache, note invalidation cache will call `putForExternalRead()` instead
# Navigate to http://<domain>/ISPN-1227/get.jsp in node 2 (similarly, generate initial data)
# Navigate to http://<domain>/ISPN-12207/remove.jsp in node 1, to evict the cache data
# Navigate to http://<domain>/ISPN-12207/get.jsp in node 2, expecting invalidation cache data got regenerated, but actually still keeping the previous data
# Navigate to http://<domain>/ISPN-12207/update.jsp in node 1, to force regenerate data and put to the cache (invalidation cache also will call `put()` instead of `putForExternalRead()`)
# Navigate to http://<domain>/ISPN-12207/get.jsp in node 2, expecting invalidation cache data got regenerated, but actually still keeping the previous data
> Functional Map APIs not functional properly with invalidation cache
> -------------------------------------------------------------------
>
> Key: ISPN-12207
> URL: https://issues.redhat.com/browse/ISPN-12207
> Project: Infinispan
> Issue Type: Bug
> Components: API, Core
> Affects Versions: 11.0.3.Final
> Reporter: Sammy Chu
> Priority: Critical
> Attachments: ISPN-12207.source.zip, ISPN-12207.war
>
>
> Functional Map API not working properly when working with invalidation cache. When in a clustered environment, expecting cache put and cache remove will notify cluster members to do cache eviction when dealing when invalidation cache. But observed no cache eviction occurred in cluster members when using Functional Map APIs to deal with invalidation cache.
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
5 years, 8 months
[JBoss JIRA] (ISPN-12207) Functional Map APIs not functional properly with invalidation cache
by Sammy Chu (Jira)
[ https://issues.redhat.com/browse/ISPN-12207?page=com.atlassian.jira.plugi... ]
Sammy Chu updated ISPN-12207:
-----------------------------
Steps to Reproduce:
# Deploy [^ISPN-12207.war] to any Servlet container (e.g. Wildfly) with at least 2 nodes
# Navigate to http://<domain>/ISPN-12207/get.jsp in node 1 (this will generate the initial data and put to the cache, note invalidation cache will call `putForExternalRead()` instead
# Navigate to http://<domain>/ISPN-1227/get.jsp in node 2 (similarly, generate initial data)
# Navigate to http://<domain>/ISPN-12207/remove.jsp in node 1, to evict the cache data
# Navigate to http://<domain>/ISPN-12207/get.jsp in node 2, expecting invalidation cache data got regenerated, but actually still keeping the previous data
# Navigate to http://<domain>/ISPN-12207/update.jsp in node 1, to force regenerate data and put to the cache (invalidation cache also will call `put()` instead of `putForExternalRead()`)
# Navigate to http://<domain>/ISPN-12207/get.jsp in node 2, expecting invalidation cache data got regenerated, but actually still keeping the previous data
was:Reproducer will be attached later
> Functional Map APIs not functional properly with invalidation cache
> -------------------------------------------------------------------
>
> Key: ISPN-12207
> URL: https://issues.redhat.com/browse/ISPN-12207
> Project: Infinispan
> Issue Type: Bug
> Components: API, Core
> Affects Versions: 11.0.3.Final
> Reporter: Sammy Chu
> Priority: Critical
> Attachments: ISPN-12207.source.zip, ISPN-12207.war
>
>
> Functional Map API not working properly when working with invalidation cache. When in a clustered environment, expecting cache put and cache remove will notify cluster members to do cache eviction when dealing when invalidation cache. But observed no cache eviction occurred in cluster members when using Functional Map APIs to deal with invalidation cache.
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
5 years, 8 months