Re: [infinispan-dev] Strict Expiration

Is it possible/feasible to skip the notification from the backups to the primary (and back) when there is no clustered expiration listener?

Dan On Tue, Jul 21, 2015 at 5:25 PM, William Burns <mudokonman(a)gmail.com&gt; wrote: > So I wanted to sum up what it looks like the plan is for this in regards to > cluster expiration for ISPN 8. > > First off to not make it ambiguous, maxIdle being used with a clustered > cache will provide undefined and unsupported behavior. This can and will > expire entries on a single node without notifying other cluster members > (essentially it will operate as it does today unchanged). > > This leaves me to talk solely about lifespan cluster expiration. > > Lifespan Expiration events are fired by the primary owner of an expired key > > - when accessing an expired entry. > > - by the reaper thread. > > If the expiration is detected by a node other than the primary owner, an > expiration command is sent to it and null is returned immediately not > waiting for a response. > > Expiration event listeners follow the usual rules for sync/async: in the > case of a sync listener, the handler is invoked while holding the lock, > whereas an async listener will not hold locks. > > It is desirable for expiration events to contain both the key and value. > However currently cache stores do not provide the value when they expire > values. Thus we can only guarantee the value is present when an in memory > expiration event occurs. We could plan on adding this later. > > Also as you may have guessed this doesn't touch strict expiration, which I > think we have come to the conclusion should only work with maxIdle and as > such this is not explored with this iteration. > > Let me know if you guys think this approach is okay. > > Cheers, > > - Will > > On Tue, Jul 14, 2015 at 1:51 PM Radim Vansa <rvansa(a)redhat.com&gt; wrote: >> >> Yes, I know about [1]. I've worked that around by storing timestamp in >> the entry as well and when a new record is added, the 'expired' >> invalidations are purged. But I can't purge that if I don't access it - >> Infinispan needs to handle that internally. >> >> Radim >> >> [1] https://hibernate.atlassian.net/browse/HHH-6219 >> >> On 07/14/2015 05:45 PM, Dennis Reed wrote: >> > On 07/14/2015 11:08 AM, Radim Vansa wrote: >> >> On 07/14/2015 04:19 PM, William Burns wrote: >> >>> >> >>> On Tue, Jul 14, 2015 at 9:37 AM William Burns <mudokonman(a)gmail.com >> >>> <mailto:mudokonman@gmail.com>> wrote: >> >>> >> >>> On Tue, Jul 14, 2015 at 4:41 AM Dan Berindei >> >>> <dan.berindei(a)gmail.com <mailto:dan.berindei@gmail.com>> wrote: >> >>> >> >>> Processing expiration only on the reaper thread sounds nice, >> >>> but I >> >>> have one reservation: processing 1 million entries to see >> >>> that >> >>> 1 of >> >>> them is expired is a lot of work, and in the general case we >> >>> will not >> >>> be able to ensure an expiration precision of less than 1 >> >>> minute (maybe >> >>> more, with a huge SingleFileStore attached). >> >>> >> >>> >> >>> This isn't much different then before. The only difference is >> >>> that if a user touched a value after it expired it wouldn't show >> >>> up (which is unlikely with maxIdle especially). >> >>> >> >>> >> >>> What happens to users who need better precision? In >> >>> particular, I know >> >>> some JCache tests were failing because HotRod was only >> >>> supporting >> >>> 1-second resolution instead of the 1-millisecond resolution >> >>> they were >> >>> expecting. >> >>> >> >>> >> >>> JCache is an interesting piece. The thing about JCache is that >> >>> the spec is only defined for local caches. However I wouldn't >> >>> want to muddy up the waters in regards to it behaving >> >>> differently >> >>> for local/remote. In the JCache scenario we could add an >> >>> interceptor to prevent it returning such values (we do something >> >>> similar already for events). JCache behavior vs ISPN behavior >> >>> seems a bit easier to differentiate. But like you are getting >> >>> at, >> >>> either way is not very appealing. >> >>> >> >>> >> >>> >> >>> I'm even less convinced about the need to guarantee that a >> >>> clustered >> >>> expiration listener will only be triggered once, and that >> >>> the >> >>> entry >> >>> must be null everywhere after that listener was invoked. >> >>> What's the >> >>> use case? >> >>> >> >>> >> >>> Maybe Tristan would know more to answer. To be honest this work >> >>> seems fruitless unless we know what our end users want here. >> >>> Spending time on something for it to thrown out is never fun :( >> >>> >> >>> And the more I thought about this the more I question the >> >>> validity >> >>> of maxIdle even. It seems like a very poor way to prevent >> >>> memory >> >>> exhaustion, which eviction does in a much better way and has >> >>> much >> >>> more flexible algorithms. Does anyone know what maxIdle would >> >>> be >> >>> used for that wouldn't be covered by eviction? The only thing I >> >>> can think of is cleaning up the cache store as well. >> >>> >> >>> >> >>> Actually I guess for session/authentication related information this >> >>> would be important. However maxIdle isn't really as usable in that >> >>> case since most likely you would have a sticky session to go back to >> >>> that node which means you would never refresh the last used date on >> >>> the copies (current implementation). Without cluster expiration you >> >>> could lose that session information on a failover very easily. >> >> I would say that maxIdle can be used as for memory management as kind >> >> of >> >> WeakHashMap - e.g. in 2LC the maxIdle is used to store some record for >> >> a >> >> short while (regular transaction lifespan ~ seconds to minutes), and >> >> regularly the record is removed. However, to make sure that we don't >> >> leak records in this cache (if something goes wrong and the remove does >> >> not occur), it is removed. >> > Note that just relying on maxIdle doesn't guarantee you won't leak >> > records in this use case (specifically with the way the current >> > hibernate-infinispan 2LC implementation uses it). >> > >> > Hibernate-infinispan adds entries to its own Map stored in Infinispan, >> > and expects maxIdle to remove the map if it skips a remove. But in a >> > current case, we found that due to frequent accesses to that same map >> > the entries never idle out and it ends up in OOME). >> > >> > -Dennis >> > >> >> I can guess how long the transaction takes place, but not how many >> >> parallel transactions there are. With eviction algorithms (where I am >> >> not sure about the exact guarantees) I can set the cache to not hold >> >> more than N entries, but I can't know for sure that my record does not >> >> suddenly get evicted after shorter period, possibly causing some >> >> inconsistency. >> >> So this is similar to WeakHashMap by removing the key "when it can't be >> >> used anymore" because I know that the transaction will finish before >> >> the >> >> deadline. I don't care about the exact size, I don't want to tune that, >> >> I just don't want to leak. >> >> >> >> From my POV the non-strict maxIdle and strict expiration would be a >> >> nice compromise. >> >> >> >> Radim >> >> >> >>> Note that this would make the reaper thread less efficient: >> >>> with >> >>> numOwners=2 (best case), half of the entries that the reaper >> >>> touches >> >>> cannot be expired, because the node isn't the primary node. >> >>> And to >> >>> make matters worse, the same reaper thread would have to >> >>> perform a >> >>> (synchronous?) RPC for each entry to ensure it expires >> >>> everywhere. >> >>> >> >>> >> >>> I have debated about this, it could something like a sync >> >>> removeAll which has a special marker to tell it is due to >> >>> expiration (which would raise listeners there), while also >> >>> sending >> >>> a cluster expiration event to other non owners. >> >>> >> >>> >> >>> For maxIdle I'd like to know more information about how >> >>> exactly the >> >>> owners would coordinate to expire an entry. I'm pretty sure >> >>> we >> >>> cannot >> >>> avoid ignoring some reads (expiring an entry immediately >> >>> after >> >>> it was >> >>> read), and ensuring that we don't accidentally extend an >> >>> entry's life >> >>> (like the current code does, when we transfer an entry to a >> >>> new owner) >> >>> also sounds problematic. >> >>> >> >>> >> >>> For lifespan it is simple, the primary owner just expires it >> >>> when >> >>> it expires there. There is no coordination needed in this case >> >>> it >> >>> just sends the expired remove to owners etc. >> >>> >> >>> Max idle is more complicated as we all know. The primary owner >> >>> would send a request for the last used time for a given key or >> >>> set >> >>> of keys. Then the owner would take those times and check for a >> >>> new access it isn't aware of. If there isn't then it would send >> >>> a >> >>> remove command for the key(s). If there is a new access the >> >>> owner >> >>> would instead send the last used time to all of the owners. The >> >>> expiration obviously would have a window that if a read occurred >> >>> after sending a response that could be ignored. This could be >> >>> resolved by using some sort of 2PC and blocking reads during >> >>> that >> >>> period but I would say it isn't worth it. >> >>> >> >>> The issue with transferring to a new node refreshing the last >> >>> update/lifespan seems like just a bug we need to fix >> >>> irrespective >> >>> of this issue IMO. >> >>> >> >>> >> >>> I'm not saying expiring entries on each node independently >> >>> is >> >>> perfect, >> >>> far from it. But I wouldn't want us to provide new >> >>> guarantees that >> >>> could hurt performance without a really good use case. >> >>> >> >>> >> >>> I would guess that user perceived performance should be a little >> >>> faster with this. But this also depends on an alternative that >> >>> we >> >>> decided on :) >> >>> >> >>> Also the expiration thread pool is set to min priority atm so it >> >>> may delay removal of said objects but hopefully (if the jvm >> >>> supports) it wouldn't overrun a CPU while processing unless it >> >>> has >> >>> availability. >> >>> >> >>> >> >>> Cheers >> >>> Dan >> >>> >> >>> >> >>> On Mon, Jul 13, 2015 at 9:25 PM, Tristan Tarrant >> >>> <ttarrant(a)redhat.com <mailto:ttarrant@redhat.com>> wrote: >> >>> > After re-reading the whole original thread, I agree with >> >>> the >> >>> proposal >> >>> > with two caveats: >> >>> > >> >>> > - ensure that we don't break JCache compatibility >> >>> > - ensure that we document this properly >> >>> > >> >>> > Tristan >> >>> > >> >>> > On 13/07/2015 18:41, Sanne Grinovero wrote: >> >>> >> +1 >> >>> >> You had me convinced at the first line, although "A lot >> >>> of >> >>> code can now >> >>> >> be removed and made simpler" makes it look extremely >> >>> nice. >> >>> >> >> >>> >> On 13 Jul 2015 18:14, "William Burns" >> >>> <mudokonman(a)gmail.com >> >>> <mailto:mudokonman@gmail.com> >> >>> >> <mailto:mudokonman@gmail.com >> >> >>> <mailto:mudokonman@gmail.com>>> wrote: >> >>> >> >> >>> >> This is a necro of [1]. >> >>> >> >> >>> >> With Infinispan 8.0 we are adding in clustered >> >>> expiration. That >> >>> >> includes an expiration event raised that is clustered >> >>> as well. >> >>> >> Unfortunately expiration events currently occur >> >>> multiple times (if >> >>> >> numOwners > 1) at different times across nodes in a >> >>> cluster. This >> >>> >> makes coordinating a single cluster expiration event >> >>> quite difficult. >> >>> >> >> >>> >> To work around this I am proposing that the >> >>> expiration >> >>> of an event >> >>> >> is done solely by the owner of the given key that is >> >>> now expired. >> >>> >> This would fix the issue of having multiple events >> >>> and >> >>> the event can >> >>> >> be raised while holding the lock for the given key so >> >>> concurrent >> >>> >> modifications would not be an issue. >> >>> >> >> >>> >> The problem arises when you have other nodes that >> >>> have >> >>> expiration >> >>> >> set but expire at different times. Max idle is the >> >>> biggest offender >> >>> >> with this as a read on an owner only refreshes the >> >>> owners timestamp, >> >>> >> meaning other owners would not be updated and expire >> >>> preemptively. >> >>> >> To have expiration work properly in this case you >> >>> would >> >>> need >> >>> >> coordination between the owners to see if anyone has >> >>> a >> >>> higher >> >>> >> value. This requires blocking and would have to be >> >>> done while >> >>> >> accessing a key that is expired to be sure if >> >>> expiration happened or >> >>> >> not. >> >>> >> >> >>> >> The linked dev listing proposed instead to only >> >>> expire >> >>> an entry by >> >>> >> the reaper thread and not on access. In this case a >> >>> read will >> >>> >> return a non null value until it is fully expired, >> >>> increasing hit >> >>> >> ratios possibly. >> >>> >> >> >>> >> Their are quire a bit of real benefits for this: >> >>> >> >> >>> >> 1. Cluster cache reads would be much simpler and >> >>> wouldn't have to >> >>> >> block to verify the object exists or not since this >> >>> would only be >> >>> >> done by the reaper thread (note this would have only >> >>> happened if the >> >>> >> entry was expired locally). An access would just >> >>> return the value >> >>> >> immediately. >> >>> >> 2. Each node only expires entries it owns in the >> >>> reaper >> >>> thread >> >>> >> reducing how many entries they must check or remove. >> >>> This also >> >>> >> provides a single point where events would be raised >> >>> as >> >>> we need. >> >>> >> 3. A lot of code can now be removed and made simpler >> >>> as >> >>> it no longer >> >>> >> has to check for expiration. The expiration check >> >>> would only be >> >>> >> done in 1 place, the expiration reaper thread. >> >>> >> >> >>> >> The main issue with this proposal is as the other >> >>> listing mentions >> >>> >> is if user code expects the value to be gone after >> >>> expiration for >> >>> >> correctness. I would say this use case is not as >> >>> compelling for >> >>> >> maxIdle, especially since we never supported it >> >>> properly. And in >> >>> >> the case of lifespan the user could very easily store >> >>> the expiration >> >>> >> time in the object that they can check after a get as >> >>> pointed out in >> >>> >> the other thread. >> >>> >> >> >>> >> [1] >> >>> >> >> >>> >> >>> http://infinispan-developer-list.980875.n3.nabble.com/infinispan-dev-stri... >> >>> >> >> >>> >> _______________________________________________ >> >>> >> infinispan-dev mailing list >> >>> >> infinispan-dev(a)lists.jboss.org >> >>> <mailto:infinispan-dev@lists.jboss.org> >> >>> <mailto:infinispan-dev@lists.jboss.org >> >>> <mailto:infinispan-dev@lists.jboss.org>> >> >>> >> https://lists.jboss.org/mailman/listinfo/infinispan-dev >> >>> >> >> >>> >> >> >>> >> >> >>> >> _______________________________________________ >> >>> >> infinispan-dev mailing list >> >>> >> infinispan-dev(a)lists.jboss.org >> >>> <mailto:infinispan-dev@lists.jboss.org> >> >>> >> https://lists.jboss.org/mailman/listinfo/infinispan-dev >> >>> >> >> >>> > >> >>> > -- >> >>> > Tristan Tarrant >> >>> > Infinispan Lead >> >>> > JBoss, a division of Red Hat >> >>> > _______________________________________________ >> >>> > infinispan-dev mailing list >> >>> > infinispan-dev(a)lists.jboss.org >> >>> <mailto:infinispan-dev@lists.jboss.org> >> >>> > https://lists.jboss.org/mailman/listinfo/infinispan-dev >> >>> _______________________________________________ >> >>> infinispan-dev mailing list >> >>> infinispan-dev(a)lists.jboss.org >> >>> <mailto:infinispan-dev@lists.jboss.org> >> >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >> >>> >> >>> >> >>> >> >>> _______________________________________________ >> >>> infinispan-dev mailing list >> >>> infinispan-dev(a)lists.jboss.org >> >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >> >> >> > _______________________________________________ >> > infinispan-dev mailing list >> > infinispan-dev(a)lists.jboss.org >> > https://lists.jboss.org/mailman/listinfo/infinispan-dev >> >> >> -- >> Radim Vansa <rvansa(a)redhat.com&gt; >> JBoss Performance Team >> >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev > > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev