12:13 p.m.
This is a necro of [1].
With Infinispan 8.0 we are adding in clustered expiration. That includes
an expiration event raised that is clustered as well. Unfortunately
expiration events currently occur multiple times (if numOwners > 1) at
different times across nodes in a cluster. This makes coordinating a
single cluster expiration event quite difficult.
To work around this I am proposing that the expiration of an event is done
solely by the owner of the given key that is now expired. This would fix
the issue of having multiple events and the event can be raised while
holding the lock for the given key so concurrent modifications would not be
an issue.
The problem arises when you have other nodes that have expiration set but
expire at different times. Max idle is the biggest offender with this as a
read on an owner only refreshes the owners timestamp, meaning other owners
would not be updated and expire preemptively. To have expiration work
properly in this case you would need coordination between the owners to see
if anyone has a higher value. This requires blocking and would have to be
done while accessing a key that is expired to be sure if expiration
happened or not.
The linked dev listing proposed instead to only expire an entry by the
reaper thread and not on access. In this case a read will return a non
null value until it is fully expired, increasing hit ratios possibly.
Their are quire a bit of real benefits for this:
1. Cluster cache reads would be much simpler and wouldn't have to block to
verify the object exists or not since this would only be done by the reaper
thread (note this would have only happened if the entry was expired
locally). An access would just return the value immediately.
2. Each node only expires entries it owns in the reaper thread reducing how
many entries they must check or remove. This also provides a single point
where events would be raised as we need.
3. A lot of code can now be removed and made simpler as it no longer has to
check for expiration. The expiration check would only be done in 1 place,
the expiration reaper thread.
The main issue with this proposal is as the other listing mentions is if
user code expects the value to be gone after expiration for correctness. I
would say this use case is not as compelling for maxIdle, especially since
we never supported it properly. And in the case of lifespan the user could
very easily store the expiration time in the object that they can check
after a get as pointed out in the other thread.
[1]
http://infinispan-developer-list.980875.n3.nabble.com/infinispan-dev-stri...
12:41 p.m.
+1
You had me convinced at the first line, although "A lot of code can now be
removed and made simpler" makes it look extremely nice.
On 13 Jul 2015 18:14, "William Burns" <mudokonman(a)gmail.com> wrote:
This is a necro of [1].
With Infinispan 8.0 we are adding in clustered expiration. That includes
an expiration event raised that is clustered as well. Unfortunately
expiration events currently occur multiple times (if numOwners > 1) at
different times across nodes in a cluster. This makes coordinating a
single cluster expiration event quite difficult.
To work around this I am proposing that the expiration of an event is done
solely by the owner of the given key that is now expired. This would fix
the issue of having multiple events and the event can be raised while
holding the lock for the given key so concurrent modifications would not be
an issue.
The problem arises when you have other nodes that have expiration set but
expire at different times. Max idle is the biggest offender with this as a
read on an owner only refreshes the owners timestamp, meaning other owners
would not be updated and expire preemptively. To have expiration work
properly in this case you would need coordination between the owners to see
if anyone has a higher value. This requires blocking and would have to be
done while accessing a key that is expired to be sure if expiration
happened or not.
The linked dev listing proposed instead to only expire an entry by the
reaper thread and not on access. In this case a read will return a non
null value until it is fully expired, increasing hit ratios possibly.
Their are quire a bit of real benefits for this:
1. Cluster cache reads would be much simpler and wouldn't have to block to
verify the object exists or not since this would only be done by the reaper
thread (note this would have only happened if the entry was expired
locally). An access would just return the value immediately.
2. Each node only expires entries it owns in the reaper thread reducing
how many entries they must check or remove. This also provides a single
point where events would be raised as we need.
3. A lot of code can now be removed and made simpler as it no longer has
to check for expiration. The expiration check would only be done in 1
place, the expiration reaper thread.
The main issue with this proposal is as the other listing mentions is if
user code expects the value to be gone after expiration for correctness. I
would say this use case is not as compelling for maxIdle, especially since
we never supported it properly. And in the case of lifespan the user could
very easily store the expiration time in the object that they can check
after a get as pointed out in the other thread.
[1]
http://infinispan-developer-list.980875.n3.nabble.com/infinispan-dev-stri...
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
1:25 p.m.
After re-reading the whole original thread, I agree with the proposal
with two caveats:
- ensure that we don't break JCache compatibility
- ensure that we document this properly
Tristan
On 13/07/2015 18:41, Sanne Grinovero wrote:
+1
You had me convinced at the first line, although "A lot of code can now
be removed and made simpler" makes it look extremely nice.
On 13 Jul 2015 18:14, "William Burns" <mudokonman(a)gmail.com
<mailto:mudokonman@gmail.com>> wrote:
This is a necro of [1].
With Infinispan 8.0 we are adding in clustered expiration. That
includes an expiration event raised that is clustered as well.
Unfortunately expiration events currently occur multiple times (if
numOwners > 1) at different times across nodes in a cluster. This
makes coordinating a single cluster expiration event quite difficult.
To work around this I am proposing that the expiration of an event
is done solely by the owner of the given key that is now expired.
This would fix the issue of having multiple events and the event can
be raised while holding the lock for the given key so concurrent
modifications would not be an issue.
The problem arises when you have other nodes that have expiration
set but expire at different times. Max idle is the biggest offender
with this as a read on an owner only refreshes the owners timestamp,
meaning other owners would not be updated and expire preemptively.
To have expiration work properly in this case you would need
coordination between the owners to see if anyone has a higher
value. This requires blocking and would have to be done while
accessing a key that is expired to be sure if expiration happened or
not.
The linked dev listing proposed instead to only expire an entry by
the reaper thread and not on access. In this case a read will
return a non null value until it is fully expired, increasing hit
ratios possibly.
Their are quire a bit of real benefits for this:
1. Cluster cache reads would be much simpler and wouldn't have to
block to verify the object exists or not since this would only be
done by the reaper thread (note this would have only happened if the
entry was expired locally). An access would just return the value
immediately.
2. Each node only expires entries it owns in the reaper thread
reducing how many entries they must check or remove. This also
provides a single point where events would be raised as we need.
3. A lot of code can now be removed and made simpler as it no longer
has to check for expiration. The expiration check would only be
done in 1 place, the expiration reaper thread.
The main issue with this proposal is as the other listing mentions
is if user code expects the value to be gone after expiration for
correctness. I would say this use case is not as compelling for
maxIdle, especially since we never supported it properly. And in
the case of lifespan the user could very easily store the expiration
time in the object that they can check after a get as pointed out in
the other thread.
[1]
http://infinispan-developer-list.980875.n3.nabble.com/infinispan-dev-stri...
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org <mailto:infinispan-dev@lists.jboss.org>
https://lists.jboss.org/mailman/listinfo/infinispan-dev
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
--
Tristan Tarrant
Infinispan Lead
JBoss, a division of Red Hat
3:41 a.m.
Processing expiration only on the reaper thread sounds nice, but I
have one reservation: processing 1 million entries to see that 1 of
them is expired is a lot of work, and in the general case we will not
be able to ensure an expiration precision of less than 1 minute (maybe
more, with a huge SingleFileStore attached).
What happens to users who need better precision? In particular, I know
some JCache tests were failing because HotRod was only supporting
1-second resolution instead of the 1-millisecond resolution they were
expecting.
I'm even less convinced about the need to guarantee that a clustered
expiration listener will only be triggered once, and that the entry
must be null everywhere after that listener was invoked. What's the
use case?
Note that this would make the reaper thread less efficient: with
numOwners=2 (best case), half of the entries that the reaper touches
cannot be expired, because the node isn't the primary node. And to
make matters worse, the same reaper thread would have to perform a
(synchronous?) RPC for each entry to ensure it expires everywhere.
For maxIdle I'd like to know more information about how exactly the
owners would coordinate to expire an entry. I'm pretty sure we cannot
avoid ignoring some reads (expiring an entry immediately after it was
read), and ensuring that we don't accidentally extend an entry's life
(like the current code does, when we transfer an entry to a new owner)
also sounds problematic.
I'm not saying expiring entries on each node independently is perfect,
far from it. But I wouldn't want us to provide new guarantees that
could hurt performance without a really good use case.
Cheers
Dan
On Mon, Jul 13, 2015 at 9:25 PM, Tristan Tarrant <ttarrant(a)redhat.com> wrote:
After re-reading the whole original thread, I agree with the
proposal
with two caveats:
- ensure that we don't break JCache compatibility
- ensure that we document this properly
Tristan
On 13/07/2015 18:41, Sanne Grinovero wrote:
> +1
> You had me convinced at the first line, although "A lot of code can now
> be removed and made simpler" makes it look extremely nice.
>
> On 13 Jul 2015 18:14, "William Burns" <mudokonman(a)gmail.com
> <mailto:mudokonman@gmail.com>> wrote:
>
> This is a necro of [1].
>
> With Infinispan 8.0 we are adding in clustered expiration. That
> includes an expiration event raised that is clustered as well.
> Unfortunately expiration events currently occur multiple times (if
> numOwners > 1) at different times across nodes in a cluster. This
> makes coordinating a single cluster expiration event quite difficult.
>
> To work around this I am proposing that the expiration of an event
> is done solely by the owner of the given key that is now expired.
> This would fix the issue of having multiple events and the event can
> be raised while holding the lock for the given key so concurrent
> modifications would not be an issue.
>
> The problem arises when you have other nodes that have expiration
> set but expire at different times. Max idle is the biggest offender
> with this as a read on an owner only refreshes the owners timestamp,
> meaning other owners would not be updated and expire preemptively.
> To have expiration work properly in this case you would need
> coordination between the owners to see if anyone has a higher
> value. This requires blocking and would have to be done while
> accessing a key that is expired to be sure if expiration happened or
> not.
>
> The linked dev listing proposed instead to only expire an entry by
> the reaper thread and not on access. In this case a read will
> return a non null value until it is fully expired, increasing hit
> ratios possibly.
>
> Their are quire a bit of real benefits for this:
>
> 1. Cluster cache reads would be much simpler and wouldn't have to
> block to verify the object exists or not since this would only be
> done by the reaper thread (note this would have only happened if the
> entry was expired locally). An access would just return the value
> immediately.
> 2. Each node only expires entries it owns in the reaper thread
> reducing how many entries they must check or remove. This also
> provides a single point where events would be raised as we need.
> 3. A lot of code can now be removed and made simpler as it no longer
> has to check for expiration. The expiration check would only be
> done in 1 place, the expiration reaper thread.
>
> The main issue with this proposal is as the other listing mentions
> is if user code expects the value to be gone after expiration for
> correctness. I would say this use case is not as compelling for
> maxIdle, especially since we never supported it properly. And in
> the case of lifespan the user could very easily store the expiration
> time in the object that they can check after a get as pointed out in
> the other thread.
>
> [1]
>
http://infinispan-developer-list.980875.n3.nabble.com/infinispan-dev-stri...
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev(a)lists.jboss.org <mailto:infinispan-dev@lists.jboss.org>
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
>
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
--
Tristan Tarrant
Infinispan Lead
JBoss, a division of Red Hat
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
4:48 a.m.
Actually, soon after sending my e-mail yesterday, I've been having other
thoughts on this.
I think we should really distinguish between maxIdle (which IMO
semantically only makes sense in a pure caching environment) and
lifespan. While it is acceptable to have stale data for the former (and
therefore justify a best-effort approach on expiration), I don't think
we can apply that blanket statement to the latter, since a user might
have legitimate reasons to depend on a fixed expiration time. While
offloading the burden of actually checking for staleness on the user
code might seem like the nice thing to do from our point of view, I'm
not sure the user would think the same. Also, it doesn't seem correct
for things like putIfAbsent and computeIfAbsent.
+1 to only have the primary owner generate the expiration event.
Tristan
On 14/07/2015 09:41, Dan Berindei wrote:
Processing expiration only on the reaper thread sounds nice, but I
have one reservation: processing 1 million entries to see that 1 of
them is expired is a lot of work, and in the general case we will not
be able to ensure an expiration precision of less than 1 minute (maybe
more, with a huge SingleFileStore attached).
What happens to users who need better precision? In particular, I know
some JCache tests were failing because HotRod was only supporting
1-second resolution instead of the 1-millisecond resolution they were
expecting.
I'm even less convinced about the need to guarantee that a clustered
expiration listener will only be triggered once, and that the entry
must be null everywhere after that listener was invoked. What's the
use case?
Note that this would make the reaper thread less efficient: with
numOwners=2 (best case), half of the entries that the reaper touches
cannot be expired, because the node isn't the primary node. And to
make matters worse, the same reaper thread would have to perform a
(synchronous?) RPC for each entry to ensure it expires everywhere.
For maxIdle I'd like to know more information about how exactly the
owners would coordinate to expire an entry. I'm pretty sure we cannot
avoid ignoring some reads (expiring an entry immediately after it was
read), and ensuring that we don't accidentally extend an entry's life
(like the current code does, when we transfer an entry to a new owner)
also sounds problematic.
I'm not saying expiring entries on each node independently is perfect,
far from it. But I wouldn't want us to provide new guarantees that
could hurt performance without a really good use case.
Cheers
Dan
On Mon, Jul 13, 2015 at 9:25 PM, Tristan Tarrant <ttarrant(a)redhat.com> wrote:
> After re-reading the whole original thread, I agree with the proposal
> with two caveats:
>
> - ensure that we don't break JCache compatibility
> - ensure that we document this properly
>
> Tristan
>
> On 13/07/2015 18:41, Sanne Grinovero wrote:
>> +1
>> You had me convinced at the first line, although "A lot of code can now
>> be removed and made simpler" makes it look extremely nice.
>>
>> On 13 Jul 2015 18:14, "William Burns" <mudokonman(a)gmail.com
>> <mailto:mudokonman@gmail.com>> wrote:
>>
>> This is a necro of [1].
>>
>> With Infinispan 8.0 we are adding in clustered expiration. That
>> includes an expiration event raised that is clustered as well.
>> Unfortunately expiration events currently occur multiple times (if
>> numOwners > 1) at different times across nodes in a cluster. This
>> makes coordinating a single cluster expiration event quite difficult.
>>
>> To work around this I am proposing that the expiration of an event
>> is done solely by the owner of the given key that is now expired.
>> This would fix the issue of having multiple events and the event can
>> be raised while holding the lock for the given key so concurrent
>> modifications would not be an issue.
>>
>> The problem arises when you have other nodes that have expiration
>> set but expire at different times. Max idle is the biggest offender
>> with this as a read on an owner only refreshes the owners timestamp,
>> meaning other owners would not be updated and expire preemptively.
>> To have expiration work properly in this case you would need
>> coordination between the owners to see if anyone has a higher
>> value. This requires blocking and would have to be done while
>> accessing a key that is expired to be sure if expiration happened or
>> not.
>>
>> The linked dev listing proposed instead to only expire an entry by
>> the reaper thread and not on access. In this case a read will
>> return a non null value until it is fully expired, increasing hit
>> ratios possibly.
>>
>> Their are quire a bit of real benefits for this:
>>
>> 1. Cluster cache reads would be much simpler and wouldn't have to
>> block to verify the object exists or not since this would only be
>> done by the reaper thread (note this would have only happened if the
>> entry was expired locally). An access would just return the value
>> immediately.
>> 2. Each node only expires entries it owns in the reaper thread
>> reducing how many entries they must check or remove. This also
>> provides a single point where events would be raised as we need.
>> 3. A lot of code can now be removed and made simpler as it no longer
>> has to check for expiration. The expiration check would only be
>> done in 1 place, the expiration reaper thread.
>>
>> The main issue with this proposal is as the other listing mentions
>> is if user code expects the value to be gone after expiration for
>> correctness. I would say this use case is not as compelling for
>> maxIdle, especially since we never supported it properly. And in
>> the case of lifespan the user could very easily store the expiration
>> time in the object that they can check after a get as pointed out in
>> the other thread.
>>
>> [1]
>>
http://infinispan-developer-list.980875.n3.nabble.com/infinispan-dev-stri...
>>
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev(a)lists.jboss.org
<mailto:infinispan-dev@lists.jboss.org>
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>
>>
>>
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev(a)lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>
>
> --
> Tristan Tarrant
> Infinispan Lead
> JBoss, a division of Red Hat
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
--
Tristan Tarrant
Infinispan Lead
JBoss, a division of Red Hat
8:37 a.m.
On Tue, Jul 14, 2015 at 4:41 AM Dan Berindei <dan.berindei(a)gmail.com> wrote:
Processing expiration only on the reaper thread sounds nice, but I
have one reservation: processing 1 million entries to see that 1 of
them is expired is a lot of work, and in the general case we will not
be able to ensure an expiration precision of less than 1 minute (maybe
more, with a huge SingleFileStore attached).
This isn't much different then before. The only difference is that if a
user touched a value after it expired it wouldn't show up (which is
unlikely with maxIdle especially).
What happens to users who need better precision? In particular, I know
some JCache tests were failing because HotRod was only supporting
1-second resolution instead of the 1-millisecond resolution they were
expecting.
JCache is an interesting piece. The thing about JCache is that the spec is
only defined for local caches. However I wouldn't want to muddy up the
waters in regards to it behaving differently for local/remote. In the
JCache scenario we could add an interceptor to prevent it returning such
values (we do something similar already for events). JCache behavior vs
ISPN behavior seems a bit easier to differentiate. But like you are
getting at, either way is not very appealing.
I'm even less convinced about the need to guarantee that a clustered
expiration listener will only be triggered once, and that the entry
must be null everywhere after that listener was invoked. What's the
use case?
Maybe Tristan would know more to answer. To be honest this work seems
fruitless unless we know what our end users want here. Spending time on
something for it to thrown out is never fun :(
And the more I thought about this the more I question the validity of
maxIdle even. It seems like a very poor way to prevent memory exhaustion,
which eviction does in a much better way and has much more flexible
algorithms. Does anyone know what maxIdle would be used for that wouldn't
be covered by eviction? The only thing I can think of is cleaning up the
cache store as well.
Note that this would make the reaper thread less efficient: with
numOwners=2 (best case), half of the entries that the reaper touches
cannot be expired, because the node isn't the primary node. And to
make matters worse, the same reaper thread would have to perform a
(synchronous?) RPC for each entry to ensure it expires everywhere.
I have debated about this, it could something like a sync removeAll which
has a special marker to tell it is due to expiration (which would raise
listeners there), while also sending a cluster expiration event to other
non owners.
For maxIdle I'd like to know more information about how exactly the
owners would coordinate to expire an entry. I'm pretty sure we cannot
avoid ignoring some reads (expiring an entry immediately after it was
read), and ensuring that we don't accidentally extend an entry's life
(like the current code does, when we transfer an entry to a new owner)
also sounds problematic.
For lifespan it is simple, the primary owner just expires it when it
expires there. There is no coordination needed in this case it just sends
the expired remove to owners etc.
Max idle is more complicated as we all know. The primary owner would send
a request for the last used time for a given key or set of keys. Then the
owner would take those times and check for a new access it isn't aware of.
If there isn't then it would send a remove command for the key(s). If
there is a new access the owner would instead send the last used time to
all of the owners. The expiration obviously would have a window that if a
read occurred after sending a response that could be ignored. This could
be resolved by using some sort of 2PC and blocking reads during that period
but I would say it isn't worth it.
The issue with transferring to a new node refreshing the last
update/lifespan seems like just a bug we need to fix irrespective of this
issue IMO.
I'm not saying expiring entries on each node independently is perfect,
far from it. But I wouldn't want us to provide new guarantees that
could hurt performance without a really good use case.
I would guess that user perceived performance should be a little faster
with this. But this also depends on an alternative that we decided on :)
Also the expiration thread pool is set to min priority atm so it may delay
removal of said objects but hopefully (if the jvm supports) it wouldn't
overrun a CPU while processing unless it has availability.
Cheers
Dan
On Mon, Jul 13, 2015 at 9:25 PM, Tristan Tarrant <ttarrant(a)redhat.com>
wrote:
> After re-reading the whole original thread, I agree with the proposal
> with two caveats:
>
> - ensure that we don't break JCache compatibility
> - ensure that we document this properly
>
> Tristan
>
> On 13/07/2015 18:41, Sanne Grinovero wrote:
>> +1
>> You had me convinced at the first line, although "A lot of code can now
>> be removed and made simpler" makes it look extremely nice.
>>
>> On 13 Jul 2015 18:14, "William Burns" <mudokonman(a)gmail.com
>> <mailto:mudokonman@gmail.com>> wrote:
>>
>> This is a necro of [1].
>>
>> With Infinispan 8.0 we are adding in clustered expiration. That
>> includes an expiration event raised that is clustered as well.
>> Unfortunately expiration events currently occur multiple times (if
>> numOwners > 1) at different times across nodes in a cluster. This
>> makes coordinating a single cluster expiration event quite
difficult.
>>
>> To work around this I am proposing that the expiration of an event
>> is done solely by the owner of the given key that is now expired.
>> This would fix the issue of having multiple events and the event can
>> be raised while holding the lock for the given key so concurrent
>> modifications would not be an issue.
>>
>> The problem arises when you have other nodes that have expiration
>> set but expire at different times. Max idle is the biggest offender
>> with this as a read on an owner only refreshes the owners timestamp,
>> meaning other owners would not be updated and expire preemptively.
>> To have expiration work properly in this case you would need
>> coordination between the owners to see if anyone has a higher
>> value. This requires blocking and would have to be done while
>> accessing a key that is expired to be sure if expiration happened or
>> not.
>>
>> The linked dev listing proposed instead to only expire an entry by
>> the reaper thread and not on access. In this case a read will
>> return a non null value until it is fully expired, increasing hit
>> ratios possibly.
>>
>> Their are quire a bit of real benefits for this:
>>
>> 1. Cluster cache reads would be much simpler and wouldn't have to
>> block to verify the object exists or not since this would only be
>> done by the reaper thread (note this would have only happened if the
>> entry was expired locally). An access would just return the value
>> immediately.
>> 2. Each node only expires entries it owns in the reaper thread
>> reducing how many entries they must check or remove. This also
>> provides a single point where events would be raised as we need.
>> 3. A lot of code can now be removed and made simpler as it no longer
>> has to check for expiration. The expiration check would only be
>> done in 1 place, the expiration reaper thread.
>>
>> The main issue with this proposal is as the other listing mentions
>> is if user code expects the value to be gone after expiration for
>> correctness. I would say this use case is not as compelling for
>> maxIdle, especially since we never supported it properly. And in
>> the case of lifespan the user could very easily store the expiration
>> time in the object that they can check after a get as pointed out in
>> the other thread.
>>
>> [1]
>>
http://infinispan-developer-list.980875.n3.nabble.com/infinispan-dev-stri...
>>
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev(a)lists.jboss.org <mailto:
infinispan-dev(a)lists.jboss.org>
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>
>>
>>
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev(a)lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>
>
> --
> Tristan Tarrant
> Infinispan Lead
> JBoss, a division of Red Hat
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
9:19 a.m.
On Tue, Jul 14, 2015 at 9:37 AM William Burns <mudokonman(a)gmail.com> wrote:
On Tue, Jul 14, 2015 at 4:41 AM Dan Berindei
<dan.berindei(a)gmail.com>
wrote:
> Processing expiration only on the reaper thread sounds nice, but I
> have one reservation: processing 1 million entries to see that 1 of
> them is expired is a lot of work, and in the general case we will not
> be able to ensure an expiration precision of less than 1 minute (maybe
> more, with a huge SingleFileStore attached).
>
This isn't much different then before. The only difference is that if a
user touched a value after it expired it wouldn't show up (which is
unlikely with maxIdle especially).
>
> What happens to users who need better precision? In particular, I know
> some JCache tests were failing because HotRod was only supporting
> 1-second resolution instead of the 1-millisecond resolution they were
> expecting.
>
JCache is an interesting piece. The thing about JCache is that the spec
is only defined for local caches. However I wouldn't want to muddy up the
waters in regards to it behaving differently for local/remote. In the
JCache scenario we could add an interceptor to prevent it returning such
values (we do something similar already for events). JCache behavior vs
ISPN behavior seems a bit easier to differentiate. But like you are
getting at, either way is not very appealing.
>
>
> I'm even less convinced about the need to guarantee that a clustered
> expiration listener will only be triggered once, and that the entry
> must be null everywhere after that listener was invoked. What's the
> use case?
>
Maybe Tristan would know more to answer. To be honest this work seems
fruitless unless we know what our end users want here. Spending time on
something for it to thrown out is never fun :(
And the more I thought about this the more I question the validity of
maxIdle even. It seems like a very poor way to prevent memory exhaustion,
which eviction does in a much better way and has much more flexible
algorithms. Does anyone know what maxIdle would be used for that wouldn't
be covered by eviction? The only thing I can think of is cleaning up the
cache store as well.
Actually I guess for session/authentication related information this would
be important. However maxIdle isn't really as usable in that case since
most likely you would have a sticky session to go back to that node which
means you would never refresh the last used date on the copies (current
implementation). Without cluster expiration you could lose that session
information on a failover very easily.
>
> Note that this would make the reaper thread less efficient: with
> numOwners=2 (best case), half of the entries that the reaper touches
> cannot be expired, because the node isn't the primary node. And to
> make matters worse, the same reaper thread would have to perform a
> (synchronous?) RPC for each entry to ensure it expires everywhere.
>
I have debated about this, it could something like a sync removeAll which
has a special marker to tell it is due to expiration (which would raise
listeners there), while also sending a cluster expiration event to other
non owners.
>
> For maxIdle I'd like to know more information about how exactly the
> owners would coordinate to expire an entry. I'm pretty sure we cannot
> avoid ignoring some reads (expiring an entry immediately after it was
> read), and ensuring that we don't accidentally extend an entry's life
> (like the current code does, when we transfer an entry to a new owner)
> also sounds problematic.
>
For lifespan it is simple, the primary owner just expires it when it
expires there. There is no coordination needed in this case it just sends
the expired remove to owners etc.
Max idle is more complicated as we all know. The primary owner would send
a request for the last used time for a given key or set of keys. Then the
owner would take those times and check for a new access it isn't aware of.
If there isn't then it would send a remove command for the key(s). If
there is a new access the owner would instead send the last used time to
all of the owners. The expiration obviously would have a window that if a
read occurred after sending a response that could be ignored. This could
be resolved by using some sort of 2PC and blocking reads during that period
but I would say it isn't worth it.
The issue with transferring to a new node refreshing the last
update/lifespan seems like just a bug we need to fix irrespective of this
issue IMO.
>
> I'm not saying expiring entries on each node independently is perfect,
> far from it. But I wouldn't want us to provide new guarantees that
> could hurt performance without a really good use case.
>
I would guess that user perceived performance should be a little faster
with this. But this also depends on an alternative that we decided on :)
Also the expiration thread pool is set to min priority atm so it may delay
removal of said objects but hopefully (if the jvm supports) it wouldn't
overrun a CPU while processing unless it has availability.
>
> Cheers
> Dan
>
>
> On Mon, Jul 13, 2015 at 9:25 PM, Tristan Tarrant <ttarrant(a)redhat.com>
> wrote:
> > After re-reading the whole original thread, I agree with the proposal
> > with two caveats:
> >
> > - ensure that we don't break JCache compatibility
> > - ensure that we document this properly
> >
> > Tristan
> >
> > On 13/07/2015 18:41, Sanne Grinovero wrote:
> >> +1
> >> You had me convinced at the first line, although "A lot of code can
now
> >> be removed and made simpler" makes it look extremely nice.
> >>
> >> On 13 Jul 2015 18:14, "William Burns" <mudokonman(a)gmail.com
> >> <mailto:mudokonman@gmail.com>> wrote:
> >>
> >> This is a necro of [1].
> >>
> >> With Infinispan 8.0 we are adding in clustered expiration. That
> >> includes an expiration event raised that is clustered as well.
> >> Unfortunately expiration events currently occur multiple times (if
> >> numOwners > 1) at different times across nodes in a cluster. This
> >> makes coordinating a single cluster expiration event quite
> difficult.
> >>
> >> To work around this I am proposing that the expiration of an event
> >> is done solely by the owner of the given key that is now expired.
> >> This would fix the issue of having multiple events and the event
> can
> >> be raised while holding the lock for the given key so concurrent
> >> modifications would not be an issue.
> >>
> >> The problem arises when you have other nodes that have expiration
> >> set but expire at different times. Max idle is the biggest
> offender
> >> with this as a read on an owner only refreshes the owners
> timestamp,
> >> meaning other owners would not be updated and expire preemptively.
> >> To have expiration work properly in this case you would need
> >> coordination between the owners to see if anyone has a higher
> >> value. This requires blocking and would have to be done while
> >> accessing a key that is expired to be sure if expiration happened
> or
> >> not.
> >>
> >> The linked dev listing proposed instead to only expire an entry by
> >> the reaper thread and not on access. In this case a read will
> >> return a non null value until it is fully expired, increasing hit
> >> ratios possibly.
> >>
> >> Their are quire a bit of real benefits for this:
> >>
> >> 1. Cluster cache reads would be much simpler and wouldn't have to
> >> block to verify the object exists or not since this would only be
> >> done by the reaper thread (note this would have only happened if
> the
> >> entry was expired locally). An access would just return the value
> >> immediately.
> >> 2. Each node only expires entries it owns in the reaper thread
> >> reducing how many entries they must check or remove. This also
> >> provides a single point where events would be raised as we need.
> >> 3. A lot of code can now be removed and made simpler as it no
> longer
> >> has to check for expiration. The expiration check would only be
> >> done in 1 place, the expiration reaper thread.
> >>
> >> The main issue with this proposal is as the other listing mentions
> >> is if user code expects the value to be gone after expiration for
> >> correctness. I would say this use case is not as compelling for
> >> maxIdle, especially since we never supported it properly. And in
> >> the case of lifespan the user could very easily store the
> expiration
> >> time in the object that they can check after a get as pointed out
> in
> >> the other thread.
> >>
> >> [1]
> >>
>
http://infinispan-developer-list.980875.n3.nabble.com/infinispan-dev-stri...
> >>
> >> _______________________________________________
> >> infinispan-dev mailing list
> >> infinispan-dev(a)lists.jboss.org <mailto:
> infinispan-dev(a)lists.jboss.org>
> >> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >>
> >>
> >>
> >> _______________________________________________
> >> infinispan-dev mailing list
> >> infinispan-dev(a)lists.jboss.org
> >> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >>
> >
> > --
> > Tristan Tarrant
> > Infinispan Lead
> > JBoss, a division of Red Hat
> > _______________________________________________
> > infinispan-dev mailing list
> > infinispan-dev(a)lists.jboss.org
> > https://lists.jboss.org/mailman/listinfo/infinispan-dev
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
10:08 a.m.
On 07/14/2015 04:19 PM, William Burns wrote:
On Tue, Jul 14, 2015 at 9:37 AM William Burns <mudokonman(a)gmail.com
<mailto:mudokonman@gmail.com>> wrote:
On Tue, Jul 14, 2015 at 4:41 AM Dan Berindei
<dan.berindei(a)gmail.com <mailto:dan.berindei@gmail.com>> wrote:
Processing expiration only on the reaper thread sounds nice, but I
have one reservation: processing 1 million entries to see that
1 of
them is expired is a lot of work, and in the general case we
will not
be able to ensure an expiration precision of less than 1
minute (maybe
more, with a huge SingleFileStore attached).
This isn't much different then before. The only difference is
that if a user touched a value after it expired it wouldn't show
up (which is unlikely with maxIdle especially).
What happens to users who need better precision? In
particular, I know
some JCache tests were failing because HotRod was only supporting
1-second resolution instead of the 1-millisecond resolution
they were
expecting.
JCache is an interesting piece. The thing about JCache is that
the spec is only defined for local caches. However I wouldn't
want to muddy up the waters in regards to it behaving differently
for local/remote. In the JCache scenario we could add an
interceptor to prevent it returning such values (we do something
similar already for events). JCache behavior vs ISPN behavior
seems a bit easier to differentiate. But like you are getting at,
either way is not very appealing.
I'm even less convinced about the need to guarantee that a
clustered
expiration listener will only be triggered once, and that the
entry
must be null everywhere after that listener was invoked.
What's the
use case?
Maybe Tristan would know more to answer. To be honest this work
seems fruitless unless we know what our end users want here.
Spending time on something for it to thrown out is never fun :(
And the more I thought about this the more I question the validity
of maxIdle even. It seems like a very poor way to prevent memory
exhaustion, which eviction does in a much better way and has much
more flexible algorithms. Does anyone know what maxIdle would be
used for that wouldn't be covered by eviction? The only thing I
can think of is cleaning up the cache store as well.
Actually I guess for session/authentication related information this
would be important. However maxIdle isn't really as usable in that
case since most likely you would have a sticky session to go back to
that node which means you would never refresh the last used date on
the copies (current implementation). Without cluster expiration you
could lose that session information on a failover very easily.
I would say that maxIdle can be used as for memory management as kind of
WeakHashMap - e.g. in 2LC the maxIdle is used to store some record for a
short while (regular transaction lifespan ~ seconds to minutes), and
regularly the record is removed. However, to make sure that we don't
leak records in this cache (if something goes wrong and the remove does
not occur), it is removed.
I can guess how long the transaction takes place, but not how many
parallel transactions there are. With eviction algorithms (where I am
not sure about the exact guarantees) I can set the cache to not hold
more than N entries, but I can't know for sure that my record does not
suddenly get evicted after shorter period, possibly causing some
inconsistency.
So this is similar to WeakHashMap by removing the key "when it can't be
used anymore" because I know that the transaction will finish before the
deadline. I don't care about the exact size, I don't want to tune that,
I just don't want to leak.
From my POV the non-strict maxIdle and strict expiration would be a
nice compromise.
Radim
Note that this would make the reaper thread less efficient: with
numOwners=2 (best case), half of the entries that the reaper
touches
cannot be expired, because the node isn't the primary node. And to
make matters worse, the same reaper thread would have to perform a
(synchronous?) RPC for each entry to ensure it expires everywhere.
I have debated about this, it could something like a sync
removeAll which has a special marker to tell it is due to
expiration (which would raise listeners there), while also sending
a cluster expiration event to other non owners.
For maxIdle I'd like to know more information about how
exactly the
owners would coordinate to expire an entry. I'm pretty sure we
cannot
avoid ignoring some reads (expiring an entry immediately after
it was
read), and ensuring that we don't accidentally extend an
entry's life
(like the current code does, when we transfer an entry to a
new owner)
also sounds problematic.
For lifespan it is simple, the primary owner just expires it when
it expires there. There is no coordination needed in this case it
just sends the expired remove to owners etc.
Max idle is more complicated as we all know. The primary owner
would send a request for the last used time for a given key or set
of keys. Then the owner would take those times and check for a
new access it isn't aware of. If there isn't then it would send a
remove command for the key(s). If there is a new access the owner
would instead send the last used time to all of the owners. The
expiration obviously would have a window that if a read occurred
after sending a response that could be ignored. This could be
resolved by using some sort of 2PC and blocking reads during that
period but I would say it isn't worth it.
The issue with transferring to a new node refreshing the last
update/lifespan seems like just a bug we need to fix irrespective
of this issue IMO.
I'm not saying expiring entries on each node independently is
perfect,
far from it. But I wouldn't want us to provide new guarantees that
could hurt performance without a really good use case.
I would guess that user perceived performance should be a little
faster with this. But this also depends on an alternative that we
decided on :)
Also the expiration thread pool is set to min priority atm so it
may delay removal of said objects but hopefully (if the jvm
supports) it wouldn't overrun a CPU while processing unless it has
availability.
Cheers
Dan
On Mon, Jul 13, 2015 at 9:25 PM, Tristan Tarrant
<ttarrant(a)redhat.com <mailto:ttarrant@redhat.com>> wrote:
> After re-reading the whole original thread, I agree with the
proposal
> with two caveats:
>
> - ensure that we don't break JCache compatibility
> - ensure that we document this properly
>
> Tristan
>
> On 13/07/2015 18:41, Sanne Grinovero wrote:
>> +1
>> You had me convinced at the first line, although "A lot of
code can now
>> be removed and made simpler" makes it look extremely nice.
>>
>> On 13 Jul 2015 18:14, "William Burns"
<mudokonman(a)gmail.com
<mailto:mudokonman@gmail.com>
>> <mailto:mudokonman@gmail.com
<mailto:mudokonman@gmail.com>>> wrote:
>>
>> This is a necro of [1].
>>
>> With Infinispan 8.0 we are adding in clustered
expiration. That
>> includes an expiration event raised that is clustered
as well.
>> Unfortunately expiration events currently occur
multiple times (if
>> numOwners > 1) at different times across nodes in a
cluster. This
>> makes coordinating a single cluster expiration event
quite difficult.
>>
>> To work around this I am proposing that the expiration
of an event
>> is done solely by the owner of the given key that is
now expired.
>> This would fix the issue of having multiple events and
the event can
>> be raised while holding the lock for the given key so
concurrent
>> modifications would not be an issue.
>>
>> The problem arises when you have other nodes that have
expiration
>> set but expire at different times. Max idle is the
biggest offender
>> with this as a read on an owner only refreshes the
owners timestamp,
>> meaning other owners would not be updated and expire
preemptively.
>> To have expiration work properly in this case you would
need
>> coordination between the owners to see if anyone has a
higher
>> value. This requires blocking and would have to be
done while
>> accessing a key that is expired to be sure if
expiration happened or
>> not.
>>
>> The linked dev listing proposed instead to only expire
an entry by
>> the reaper thread and not on access. In this case a
read will
>> return a non null value until it is fully expired,
increasing hit
>> ratios possibly.
>>
>> Their are quire a bit of real benefits for this:
>>
>> 1. Cluster cache reads would be much simpler and
wouldn't have to
>> block to verify the object exists or not since this
would only be
>> done by the reaper thread (note this would have only
happened if the
>> entry was expired locally). An access would just
return the value
>> immediately.
>> 2. Each node only expires entries it owns in the reaper
thread
>> reducing how many entries they must check or remove.
This also
>> provides a single point where events would be raised as
we need.
>> 3. A lot of code can now be removed and made simpler as
it no longer
>> has to check for expiration. The expiration check
would only be
>> done in 1 place, the expiration reaper thread.
>>
>> The main issue with this proposal is as the other
listing mentions
>> is if user code expects the value to be gone after
expiration for
>> correctness. I would say this use case is not as
compelling for
>> maxIdle, especially since we never supported it
properly. And in
>> the case of lifespan the user could very easily store
the expiration
>> time in the object that they can check after a get as
pointed out in
>> the other thread.
>>
>> [1]
>>
http://infinispan-developer-list.980875.n3.nabble.com/infinispan-dev-stri...
>>
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev(a)lists.jboss.org
<mailto:infinispan-dev@lists.jboss.org>
<mailto:infinispan-dev@lists.jboss.org
<mailto:infinispan-dev@lists.jboss.org>>
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>
>>
>>
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev(a)lists.jboss.org
<mailto:infinispan-dev@lists.jboss.org>
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>
>
> --
> Tristan Tarrant
> Infinispan Lead
> JBoss, a division of Red Hat
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev(a)lists.jboss.org
<mailto:infinispan-dev@lists.jboss.org>
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
<mailto:infinispan-dev@lists.jboss.org>
https://lists.jboss.org/mailman/listinfo/infinispan-dev
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
--
Radim Vansa <rvansa(a)redhat.com>
JBoss Performance Team
10:45 a.m.
On 07/14/2015 11:08 AM, Radim Vansa wrote:
On 07/14/2015 04:19 PM, William Burns wrote:
>
>
> On Tue, Jul 14, 2015 at 9:37 AM William Burns <mudokonman(a)gmail.com
> <mailto:mudokonman@gmail.com>> wrote:
>
> On Tue, Jul 14, 2015 at 4:41 AM Dan Berindei
> <dan.berindei(a)gmail.com <mailto:dan.berindei@gmail.com>> wrote:
>
> Processing expiration only on the reaper thread sounds nice, but I
> have one reservation: processing 1 million entries to see that
> 1 of
> them is expired is a lot of work, and in the general case we
> will not
> be able to ensure an expiration precision of less than 1
> minute (maybe
> more, with a huge SingleFileStore attached).
>
>
> This isn't much different then before. The only difference is
> that if a user touched a value after it expired it wouldn't show
> up (which is unlikely with maxIdle especially).
>
>
> What happens to users who need better precision? In
> particular, I know
> some JCache tests were failing because HotRod was only supporting
> 1-second resolution instead of the 1-millisecond resolution
> they were
> expecting.
>
>
> JCache is an interesting piece. The thing about JCache is that
> the spec is only defined for local caches. However I wouldn't
> want to muddy up the waters in regards to it behaving differently
> for local/remote. In the JCache scenario we could add an
> interceptor to prevent it returning such values (we do something
> similar already for events). JCache behavior vs ISPN behavior
> seems a bit easier to differentiate. But like you are getting at,
> either way is not very appealing.
>
>
>
> I'm even less convinced about the need to guarantee that a
> clustered
> expiration listener will only be triggered once, and that the
> entry
> must be null everywhere after that listener was invoked.
> What's the
> use case?
>
>
> Maybe Tristan would know more to answer. To be honest this work
> seems fruitless unless we know what our end users want here.
> Spending time on something for it to thrown out is never fun :(
>
> And the more I thought about this the more I question the validity
> of maxIdle even. It seems like a very poor way to prevent memory
> exhaustion, which eviction does in a much better way and has much
> more flexible algorithms. Does anyone know what maxIdle would be
> used for that wouldn't be covered by eviction? The only thing I
> can think of is cleaning up the cache store as well.
>
>
> Actually I guess for session/authentication related information this
> would be important. However maxIdle isn't really as usable in that
> case since most likely you would have a sticky session to go back to
> that node which means you would never refresh the last used date on
> the copies (current implementation). Without cluster expiration you
> could lose that session information on a failover very easily.
I would say that maxIdle can be used as for memory management as kind of
WeakHashMap - e.g. in 2LC the maxIdle is used to store some record for a
short while (regular transaction lifespan ~ seconds to minutes), and
regularly the record is removed. However, to make sure that we don't
leak records in this cache (if something goes wrong and the remove does
not occur), it is removed.
Note that just relying on maxIdle doesn't guarantee you won't leak
records in this use case (specifically with the way the current
hibernate-infinispan 2LC implementation uses it).
Hibernate-infinispan adds entries to its own Map stored in Infinispan,
and expects maxIdle to remove the map if it skips a remove. But in a
current case, we found that due to frequent accesses to that same map
the entries never idle out and it ends up in OOME).
-Dennis
I can guess how long the transaction takes place, but not how many
parallel transactions there are. With eviction algorithms (where I am
not sure about the exact guarantees) I can set the cache to not hold
more than N entries, but I can't know for sure that my record does not
suddenly get evicted after shorter period, possibly causing some
inconsistency.
So this is similar to WeakHashMap by removing the key "when it can't be
used anymore" because I know that the transaction will finish before the
deadline. I don't care about the exact size, I don't want to tune that,
I just don't want to leak.
From my POV the non-strict maxIdle and strict expiration would be a
nice compromise.
Radim
>
> Note that this would make the reaper thread less efficient: with
> numOwners=2 (best case), half of the entries that the reaper
> touches
> cannot be expired, because the node isn't the primary node. And to
> make matters worse, the same reaper thread would have to perform a
> (synchronous?) RPC for each entry to ensure it expires everywhere.
>
>
> I have debated about this, it could something like a sync
> removeAll which has a special marker to tell it is due to
> expiration (which would raise listeners there), while also sending
> a cluster expiration event to other non owners.
>
>
> For maxIdle I'd like to know more information about how
> exactly the
> owners would coordinate to expire an entry. I'm pretty sure we
> cannot
> avoid ignoring some reads (expiring an entry immediately after
> it was
> read), and ensuring that we don't accidentally extend an
> entry's life
> (like the current code does, when we transfer an entry to a
> new owner)
> also sounds problematic.
>
>
> For lifespan it is simple, the primary owner just expires it when
> it expires there. There is no coordination needed in this case it
> just sends the expired remove to owners etc.
>
> Max idle is more complicated as we all know. The primary owner
> would send a request for the last used time for a given key or set
> of keys. Then the owner would take those times and check for a
> new access it isn't aware of. If there isn't then it would send a
> remove command for the key(s). If there is a new access the owner
> would instead send the last used time to all of the owners. The
> expiration obviously would have a window that if a read occurred
> after sending a response that could be ignored. This could be
> resolved by using some sort of 2PC and blocking reads during that
> period but I would say it isn't worth it.
>
> The issue with transferring to a new node refreshing the last
> update/lifespan seems like just a bug we need to fix irrespective
> of this issue IMO.
>
>
> I'm not saying expiring entries on each node independently is
> perfect,
> far from it. But I wouldn't want us to provide new guarantees that
> could hurt performance without a really good use case.
>
>
> I would guess that user perceived performance should be a little
> faster with this. But this also depends on an alternative that we
> decided on :)
>
> Also the expiration thread pool is set to min priority atm so it
> may delay removal of said objects but hopefully (if the jvm
> supports) it wouldn't overrun a CPU while processing unless it has
> availability.
>
>
> Cheers
> Dan
>
>
> On Mon, Jul 13, 2015 at 9:25 PM, Tristan Tarrant
> <ttarrant(a)redhat.com <mailto:ttarrant@redhat.com>> wrote:
> > After re-reading the whole original thread, I agree with the
> proposal
> > with two caveats:
> >
> > - ensure that we don't break JCache compatibility
> > - ensure that we document this properly
> >
> > Tristan
> >
> > On 13/07/2015 18:41, Sanne Grinovero wrote:
> >> +1
> >> You had me convinced at the first line, although "A lot of
> code can now
> >> be removed and made simpler" makes it look extremely nice.
> >>
> >> On 13 Jul 2015 18:14, "William Burns"
<mudokonman(a)gmail.com
> <mailto:mudokonman@gmail.com>
> >> <mailto:mudokonman@gmail.com
> <mailto:mudokonman@gmail.com>>> wrote:
> >>
> >> This is a necro of [1].
> >>
> >> With Infinispan 8.0 we are adding in clustered
> expiration. That
> >> includes an expiration event raised that is clustered
> as well.
> >> Unfortunately expiration events currently occur
> multiple times (if
> >> numOwners > 1) at different times across nodes in a
> cluster. This
> >> makes coordinating a single cluster expiration event
> quite difficult.
> >>
> >> To work around this I am proposing that the expiration
> of an event
> >> is done solely by the owner of the given key that is
> now expired.
> >> This would fix the issue of having multiple events and
> the event can
> >> be raised while holding the lock for the given key so
> concurrent
> >> modifications would not be an issue.
> >>
> >> The problem arises when you have other nodes that have
> expiration
> >> set but expire at different times. Max idle is the
> biggest offender
> >> with this as a read on an owner only refreshes the
> owners timestamp,
> >> meaning other owners would not be updated and expire
> preemptively.
> >> To have expiration work properly in this case you would
> need
> >> coordination between the owners to see if anyone has a
> higher
> >> value. This requires blocking and would have to be
> done while
> >> accessing a key that is expired to be sure if
> expiration happened or
> >> not.
> >>
> >> The linked dev listing proposed instead to only expire
> an entry by
> >> the reaper thread and not on access. In this case a
> read will
> >> return a non null value until it is fully expired,
> increasing hit
> >> ratios possibly.
> >>
> >> Their are quire a bit of real benefits for this:
> >>
> >> 1. Cluster cache reads would be much simpler and
> wouldn't have to
> >> block to verify the object exists or not since this
> would only be
> >> done by the reaper thread (note this would have only
> happened if the
> >> entry was expired locally). An access would just
> return the value
> >> immediately.
> >> 2. Each node only expires entries it owns in the reaper
> thread
> >> reducing how many entries they must check or remove.
> This also
> >> provides a single point where events would be raised as
> we need.
> >> 3. A lot of code can now be removed and made simpler as
> it no longer
> >> has to check for expiration. The expiration check
> would only be
> >> done in 1 place, the expiration reaper thread.
> >>
> >> The main issue with this proposal is as the other
> listing mentions
> >> is if user code expects the value to be gone after
> expiration for
> >> correctness. I would say this use case is not as
> compelling for
> >> maxIdle, especially since we never supported it
> properly. And in
> >> the case of lifespan the user could very easily store
> the expiration
> >> time in the object that they can check after a get as
> pointed out in
> >> the other thread.
> >>
> >> [1]
> >>
>
http://infinispan-developer-list.980875.n3.nabble.com/infinispan-dev-stri...
> >>
> >> _______________________________________________
> >> infinispan-dev mailing list
> >> infinispan-dev(a)lists.jboss.org
> <mailto:infinispan-dev@lists.jboss.org>
> <mailto:infinispan-dev@lists.jboss.org
> <mailto:infinispan-dev@lists.jboss.org>>
> >> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >>
> >>
> >>
> >> _______________________________________________
> >> infinispan-dev mailing list
> >> infinispan-dev(a)lists.jboss.org
> <mailto:infinispan-dev@lists.jboss.org>
> >> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >>
> >
> > --
> > Tristan Tarrant
> > Infinispan Lead
> > JBoss, a division of Red Hat
> > _______________________________________________
> > infinispan-dev mailing list
> > infinispan-dev(a)lists.jboss.org
> <mailto:infinispan-dev@lists.jboss.org>
> > https://lists.jboss.org/mailman/listinfo/infinispan-dev
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev(a)lists.jboss.org
> <mailto:infinispan-dev@lists.jboss.org>
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
>
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
12:51 p.m.
Yes, I know about [1]. I've worked that around by storing timestamp in
the entry as well and when a new record is added, the 'expired'
invalidations are purged. But I can't purge that if I don't access it -
Infinispan needs to handle that internally.
Radim
[1] https://hibernate.atlassian.net/browse/HHH-6219
On 07/14/2015 05:45 PM, Dennis Reed wrote:
On 07/14/2015 11:08 AM, Radim Vansa wrote:
> On 07/14/2015 04:19 PM, William Burns wrote:
>>
>> On Tue, Jul 14, 2015 at 9:37 AM William Burns <mudokonman(a)gmail.com
>> <mailto:mudokonman@gmail.com>> wrote:
>>
>> On Tue, Jul 14, 2015 at 4:41 AM Dan Berindei
>> <dan.berindei(a)gmail.com <mailto:dan.berindei@gmail.com>>
wrote:
>>
>> Processing expiration only on the reaper thread sounds nice, but I
>> have one reservation: processing 1 million entries to see that
>> 1 of
>> them is expired is a lot of work, and in the general case we
>> will not
>> be able to ensure an expiration precision of less than 1
>> minute (maybe
>> more, with a huge SingleFileStore attached).
>>
>>
>> This isn't much different then before. The only difference is
>> that if a user touched a value after it expired it wouldn't show
>> up (which is unlikely with maxIdle especially).
>>
>>
>> What happens to users who need better precision? In
>> particular, I know
>> some JCache tests were failing because HotRod was only supporting
>> 1-second resolution instead of the 1-millisecond resolution
>> they were
>> expecting.
>>
>>
>> JCache is an interesting piece. The thing about JCache is that
>> the spec is only defined for local caches. However I wouldn't
>> want to muddy up the waters in regards to it behaving differently
>> for local/remote. In the JCache scenario we could add an
>> interceptor to prevent it returning such values (we do something
>> similar already for events). JCache behavior vs ISPN behavior
>> seems a bit easier to differentiate. But like you are getting at,
>> either way is not very appealing.
>>
>>
>>
>> I'm even less convinced about the need to guarantee that a
>> clustered
>> expiration listener will only be triggered once, and that the
>> entry
>> must be null everywhere after that listener was invoked.
>> What's the
>> use case?
>>
>>
>> Maybe Tristan would know more to answer. To be honest this work
>> seems fruitless unless we know what our end users want here.
>> Spending time on something for it to thrown out is never fun :(
>>
>> And the more I thought about this the more I question the validity
>> of maxIdle even. It seems like a very poor way to prevent memory
>> exhaustion, which eviction does in a much better way and has much
>> more flexible algorithms. Does anyone know what maxIdle would be
>> used for that wouldn't be covered by eviction? The only thing I
>> can think of is cleaning up the cache store as well.
>>
>>
>> Actually I guess for session/authentication related information this
>> would be important. However maxIdle isn't really as usable in that
>> case since most likely you would have a sticky session to go back to
>> that node which means you would never refresh the last used date on
>> the copies (current implementation). Without cluster expiration you
>> could lose that session information on a failover very easily.
> I would say that maxIdle can be used as for memory management as kind of
> WeakHashMap - e.g. in 2LC the maxIdle is used to store some record for a
> short while (regular transaction lifespan ~ seconds to minutes), and
> regularly the record is removed. However, to make sure that we don't
> leak records in this cache (if something goes wrong and the remove does
> not occur), it is removed.
Note that just relying on maxIdle doesn't guarantee you won't leak
records in this use case (specifically with the way the current
hibernate-infinispan 2LC implementation uses it).
Hibernate-infinispan adds entries to its own Map stored in Infinispan,
and expects maxIdle to remove the map if it skips a remove. But in a
current case, we found that due to frequent accesses to that same map
the entries never idle out and it ends up in OOME).
-Dennis
> I can guess how long the transaction takes place, but not how many
> parallel transactions there are. With eviction algorithms (where I am
> not sure about the exact guarantees) I can set the cache to not hold
> more than N entries, but I can't know for sure that my record does not
> suddenly get evicted after shorter period, possibly causing some
> inconsistency.
> So this is similar to WeakHashMap by removing the key "when it can't be
> used anymore" because I know that the transaction will finish before the
> deadline. I don't care about the exact size, I don't want to tune that,
> I just don't want to leak.
>
> From my POV the non-strict maxIdle and strict expiration would be a
> nice compromise.
>
> Radim
>
>> Note that this would make the reaper thread less efficient: with
>> numOwners=2 (best case), half of the entries that the reaper
>> touches
>> cannot be expired, because the node isn't the primary node. And to
>> make matters worse, the same reaper thread would have to perform a
>> (synchronous?) RPC for each entry to ensure it expires everywhere.
>>
>>
>> I have debated about this, it could something like a sync
>> removeAll which has a special marker to tell it is due to
>> expiration (which would raise listeners there), while also sending
>> a cluster expiration event to other non owners.
>>
>>
>> For maxIdle I'd like to know more information about how
>> exactly the
>> owners would coordinate to expire an entry. I'm pretty sure we
>> cannot
>> avoid ignoring some reads (expiring an entry immediately after
>> it was
>> read), and ensuring that we don't accidentally extend an
>> entry's life
>> (like the current code does, when we transfer an entry to a
>> new owner)
>> also sounds problematic.
>>
>>
>> For lifespan it is simple, the primary owner just expires it when
>> it expires there. There is no coordination needed in this case it
>> just sends the expired remove to owners etc.
>>
>> Max idle is more complicated as we all know. The primary owner
>> would send a request for the last used time for a given key or set
>> of keys. Then the owner would take those times and check for a
>> new access it isn't aware of. If there isn't then it would send a
>> remove command for the key(s). If there is a new access the owner
>> would instead send the last used time to all of the owners. The
>> expiration obviously would have a window that if a read occurred
>> after sending a response that could be ignored. This could be
>> resolved by using some sort of 2PC and blocking reads during that
>> period but I would say it isn't worth it.
>>
>> The issue with transferring to a new node refreshing the last
>> update/lifespan seems like just a bug we need to fix irrespective
>> of this issue IMO.
>>
>>
>> I'm not saying expiring entries on each node independently is
>> perfect,
>> far from it. But I wouldn't want us to provide new guarantees that
>> could hurt performance without a really good use case.
>>
>>
>> I would guess that user perceived performance should be a little
>> faster with this. But this also depends on an alternative that we
>> decided on :)
>>
>> Also the expiration thread pool is set to min priority atm so it
>> may delay removal of said objects but hopefully (if the jvm
>> supports) it wouldn't overrun a CPU while processing unless it has
>> availability.
>>
>>
>> Cheers
>> Dan
>>
>>
>> On Mon, Jul 13, 2015 at 9:25 PM, Tristan Tarrant
>> <ttarrant(a)redhat.com <mailto:ttarrant@redhat.com>> wrote:
>> > After re-reading the whole original thread, I agree with the
>> proposal
>> > with two caveats:
>> >
>> > - ensure that we don't break JCache compatibility
>> > - ensure that we document this properly
>> >
>> > Tristan
>> >
>> > On 13/07/2015 18:41, Sanne Grinovero wrote:
>> >> +1
>> >> You had me convinced at the first line, although "A lot
of
>> code can now
>> >> be removed and made simpler" makes it look extremely
nice.
>> >>
>> >> On 13 Jul 2015 18:14, "William Burns"
<mudokonman(a)gmail.com
>> <mailto:mudokonman@gmail.com>
>> >> <mailto:mudokonman@gmail.com
>> <mailto:mudokonman@gmail.com>>> wrote:
>> >>
>> >> This is a necro of [1].
>> >>
>> >> With Infinispan 8.0 we are adding in clustered
>> expiration. That
>> >> includes an expiration event raised that is clustered
>> as well.
>> >> Unfortunately expiration events currently occur
>> multiple times (if
>> >> numOwners > 1) at different times across nodes in a
>> cluster. This
>> >> makes coordinating a single cluster expiration event
>> quite difficult.
>> >>
>> >> To work around this I am proposing that the expiration
>> of an event
>> >> is done solely by the owner of the given key that is
>> now expired.
>> >> This would fix the issue of having multiple events and
>> the event can
>> >> be raised while holding the lock for the given key so
>> concurrent
>> >> modifications would not be an issue.
>> >>
>> >> The problem arises when you have other nodes that have
>> expiration
>> >> set but expire at different times. Max idle is the
>> biggest offender
>> >> with this as a read on an owner only refreshes the
>> owners timestamp,
>> >> meaning other owners would not be updated and expire
>> preemptively.
>> >> To have expiration work properly in this case you would
>> need
>> >> coordination between the owners to see if anyone has a
>> higher
>> >> value. This requires blocking and would have to be
>> done while
>> >> accessing a key that is expired to be sure if
>> expiration happened or
>> >> not.
>> >>
>> >> The linked dev listing proposed instead to only expire
>> an entry by
>> >> the reaper thread and not on access. In this case a
>> read will
>> >> return a non null value until it is fully expired,
>> increasing hit
>> >> ratios possibly.
>> >>
>> >> Their are quire a bit of real benefits for this:
>> >>
>> >> 1. Cluster cache reads would be much simpler and
>> wouldn't have to
>> >> block to verify the object exists or not since this
>> would only be
>> >> done by the reaper thread (note this would have only
>> happened if the
>> >> entry was expired locally). An access would just
>> return the value
>> >> immediately.
>> >> 2. Each node only expires entries it owns in the reaper
>> thread
>> >> reducing how many entries they must check or remove.
>> This also
>> >> provides a single point where events would be raised as
>> we need.
>> >> 3. A lot of code can now be removed and made simpler as
>> it no longer
>> >> has to check for expiration. The expiration check
>> would only be
>> >> done in 1 place, the expiration reaper thread.
>> >>
>> >> The main issue with this proposal is as the other
>> listing mentions
>> >> is if user code expects the value to be gone after
>> expiration for
>> >> correctness. I would say this use case is not as
>> compelling for
>> >> maxIdle, especially since we never supported it
>> properly. And in
>> >> the case of lifespan the user could very easily store
>> the expiration
>> >> time in the object that they can check after a get as
>> pointed out in
>> >> the other thread.
>> >>
>> >> [1]
>> >>
>>
http://infinispan-developer-list.980875.n3.nabble.com/infinispan-dev-stri...
>> >>
>> >> _______________________________________________
>> >> infinispan-dev mailing list
>> >> infinispan-dev(a)lists.jboss.org
>> <mailto:infinispan-dev@lists.jboss.org>
>> <mailto:infinispan-dev@lists.jboss.org
>> <mailto:infinispan-dev@lists.jboss.org>>
>> >> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>> >>
>> >>
>> >>
>> >> _______________________________________________
>> >> infinispan-dev mailing list
>> >> infinispan-dev(a)lists.jboss.org
>> <mailto:infinispan-dev@lists.jboss.org>
>> >> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>> >>
>> >
>> > --
>> > Tristan Tarrant
>> > Infinispan Lead
>> > JBoss, a division of Red Hat
>> > _______________________________________________
>> > infinispan-dev mailing list
>> > infinispan-dev(a)lists.jboss.org
>> <mailto:infinispan-dev@lists.jboss.org>
>> > https://lists.jboss.org/mailman/listinfo/infinispan-dev
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev(a)lists.jboss.org
>> <mailto:infinispan-dev@lists.jboss.org>
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>
>>
>>
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev(a)lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
--
Radim Vansa <rvansa(a)redhat.com>
JBoss Performance Team
9:25 a.m.
So I wanted to sum up what it looks like the plan is for this in regards to
cluster expiration for ISPN 8.
First off to not make it ambiguous, maxIdle being used with a clustered
cache will provide undefined and unsupported behavior. This can and will
expire entries on a single node without notifying other cluster members
(essentially it will operate as it does today unchanged).
This leaves me to talk solely about lifespan cluster expiration.
Lifespan Expiration events are fired by the primary owner of an expired key
- when accessing an expired entry.
- by the reaper thread.
If the expiration is detected by a node other than the primary owner, an
expiration command is sent to it and null is returned immediately not
waiting for a response.
Expiration event listeners follow the usual rules for sync/async: in the
case of a sync listener, the handler is invoked while holding the lock,
whereas an async listener will not hold locks.
It is desirable for expiration events to contain both the key and value.
However currently cache stores do not provide the value when they expire
values. Thus we can only guarantee the value is present when an in memory
expiration event occurs. We could plan on adding this later.
Also as you may have guessed this doesn't touch strict expiration, which I
think we have come to the conclusion should only work with maxIdle and as
such this is not explored with this iteration.
Let me know if you guys think this approach is okay.
Cheers,
- Will
On Tue, Jul 14, 2015 at 1:51 PM Radim Vansa <rvansa(a)redhat.com> wrote:
Yes, I know about [1]. I've worked that around by storing
timestamp in
the entry as well and when a new record is added, the 'expired'
invalidations are purged. But I can't purge that if I don't access it -
Infinispan needs to handle that internally.
Radim
[1] https://hibernate.atlassian.net/browse/HHH-6219
On 07/14/2015 05:45 PM, Dennis Reed wrote:
> On 07/14/2015 11:08 AM, Radim Vansa wrote:
>> On 07/14/2015 04:19 PM, William Burns wrote:
>>>
>>> On Tue, Jul 14, 2015 at 9:37 AM William Burns <mudokonman(a)gmail.com
>>> <mailto:mudokonman@gmail.com>> wrote:
>>>
>>> On Tue, Jul 14, 2015 at 4:41 AM Dan Berindei
>>> <dan.berindei(a)gmail.com <mailto:dan.berindei@gmail.com>>
wrote:
>>>
>>> Processing expiration only on the reaper thread sounds nice,
but I
>>> have one reservation: processing 1 million entries to see
that
>>> 1 of
>>> them is expired is a lot of work, and in the general case we
>>> will not
>>> be able to ensure an expiration precision of less than 1
>>> minute (maybe
>>> more, with a huge SingleFileStore attached).
>>>
>>>
>>> This isn't much different then before. The only difference is
>>> that if a user touched a value after it expired it wouldn't show
>>> up (which is unlikely with maxIdle especially).
>>>
>>>
>>> What happens to users who need better precision? In
>>> particular, I know
>>> some JCache tests were failing because HotRod was only
supporting
>>> 1-second resolution instead of the 1-millisecond resolution
>>> they were
>>> expecting.
>>>
>>>
>>> JCache is an interesting piece. The thing about JCache is that
>>> the spec is only defined for local caches. However I wouldn't
>>> want to muddy up the waters in regards to it behaving differently
>>> for local/remote. In the JCache scenario we could add an
>>> interceptor to prevent it returning such values (we do something
>>> similar already for events). JCache behavior vs ISPN behavior
>>> seems a bit easier to differentiate. But like you are getting
at,
>>> either way is not very appealing.
>>>
>>>
>>>
>>> I'm even less convinced about the need to guarantee that a
>>> clustered
>>> expiration listener will only be triggered once, and that the
>>> entry
>>> must be null everywhere after that listener was invoked.
>>> What's the
>>> use case?
>>>
>>>
>>> Maybe Tristan would know more to answer. To be honest this work
>>> seems fruitless unless we know what our end users want here.
>>> Spending time on something for it to thrown out is never fun :(
>>>
>>> And the more I thought about this the more I question the
validity
>>> of maxIdle even. It seems like a very poor way to prevent memory
>>> exhaustion, which eviction does in a much better way and has much
>>> more flexible algorithms. Does anyone know what maxIdle would be
>>> used for that wouldn't be covered by eviction? The only thing I
>>> can think of is cleaning up the cache store as well.
>>>
>>>
>>> Actually I guess for session/authentication related information this
>>> would be important. However maxIdle isn't really as usable in that
>>> case since most likely you would have a sticky session to go back to
>>> that node which means you would never refresh the last used date on
>>> the copies (current implementation). Without cluster expiration you
>>> could lose that session information on a failover very easily.
>> I would say that maxIdle can be used as for memory management as kind of
>> WeakHashMap - e.g. in 2LC the maxIdle is used to store some record for a
>> short while (regular transaction lifespan ~ seconds to minutes), and
>> regularly the record is removed. However, to make sure that we don't
>> leak records in this cache (if something goes wrong and the remove does
>> not occur), it is removed.
> Note that just relying on maxIdle doesn't guarantee you won't leak
> records in this use case (specifically with the way the current
> hibernate-infinispan 2LC implementation uses it).
>
> Hibernate-infinispan adds entries to its own Map stored in Infinispan,
> and expects maxIdle to remove the map if it skips a remove. But in a
> current case, we found that due to frequent accesses to that same map
> the entries never idle out and it ends up in OOME).
>
> -Dennis
>
>> I can guess how long the transaction takes place, but not how many
>> parallel transactions there are. With eviction algorithms (where I am
>> not sure about the exact guarantees) I can set the cache to not hold
>> more than N entries, but I can't know for sure that my record does not
>> suddenly get evicted after shorter period, possibly causing some
>> inconsistency.
>> So this is similar to WeakHashMap by removing the key "when it can't
be
>> used anymore" because I know that the transaction will finish before the
>> deadline. I don't care about the exact size, I don't want to tune that,
>> I just don't want to leak.
>>
>> From my POV the non-strict maxIdle and strict expiration would be a
>> nice compromise.
>>
>> Radim
>>
>>> Note that this would make the reaper thread less efficient:
with
>>> numOwners=2 (best case), half of the entries that the reaper
>>> touches
>>> cannot be expired, because the node isn't the primary node.
And to
>>> make matters worse, the same reaper thread would have to
perform a
>>> (synchronous?) RPC for each entry to ensure it expires
everywhere.
>>>
>>>
>>> I have debated about this, it could something like a sync
>>> removeAll which has a special marker to tell it is due to
>>> expiration (which would raise listeners there), while also
sending
>>> a cluster expiration event to other non owners.
>>>
>>>
>>> For maxIdle I'd like to know more information about how
>>> exactly the
>>> owners would coordinate to expire an entry. I'm pretty sure
we
>>> cannot
>>> avoid ignoring some reads (expiring an entry immediately
after
>>> it was
>>> read), and ensuring that we don't accidentally extend an
>>> entry's life
>>> (like the current code does, when we transfer an entry to a
>>> new owner)
>>> also sounds problematic.
>>>
>>>
>>> For lifespan it is simple, the primary owner just expires it when
>>> it expires there. There is no coordination needed in this case
it
>>> just sends the expired remove to owners etc.
>>>
>>> Max idle is more complicated as we all know. The primary owner
>>> would send a request for the last used time for a given key or
set
>>> of keys. Then the owner would take those times and check for a
>>> new access it isn't aware of. If there isn't then it would
send
a
>>> remove command for the key(s). If there is a new access the
owner
>>> would instead send the last used time to all of the owners. The
>>> expiration obviously would have a window that if a read occurred
>>> after sending a response that could be ignored. This could be
>>> resolved by using some sort of 2PC and blocking reads during that
>>> period but I would say it isn't worth it.
>>>
>>> The issue with transferring to a new node refreshing the last
>>> update/lifespan seems like just a bug we need to fix irrespective
>>> of this issue IMO.
>>>
>>>
>>> I'm not saying expiring entries on each node independently is
>>> perfect,
>>> far from it. But I wouldn't want us to provide new
guarantees that
>>> could hurt performance without a really good use case.
>>>
>>>
>>> I would guess that user perceived performance should be a little
>>> faster with this. But this also depends on an alternative that
we
>>> decided on :)
>>>
>>> Also the expiration thread pool is set to min priority atm so it
>>> may delay removal of said objects but hopefully (if the jvm
>>> supports) it wouldn't overrun a CPU while processing unless it
has
>>> availability.
>>>
>>>
>>> Cheers
>>> Dan
>>>
>>>
>>> On Mon, Jul 13, 2015 at 9:25 PM, Tristan Tarrant
>>> <ttarrant(a)redhat.com <mailto:ttarrant@redhat.com>>
wrote:
>>> > After re-reading the whole original thread, I agree with
the
>>> proposal
>>> > with two caveats:
>>> >
>>> > - ensure that we don't break JCache compatibility
>>> > - ensure that we document this properly
>>> >
>>> > Tristan
>>> >
>>> > On 13/07/2015 18:41, Sanne Grinovero wrote:
>>> >> +1
>>> >> You had me convinced at the first line, although "A
lot of
>>> code can now
>>> >> be removed and made simpler" makes it look extremely
nice.
>>> >>
>>> >> On 13 Jul 2015 18:14, "William Burns" <
mudokonman(a)gmail.com
>>> <mailto:mudokonman@gmail.com>
>>> >> <mailto:mudokonman@gmail.com
>>> <mailto:mudokonman@gmail.com>>> wrote:
>>> >>
>>> >> This is a necro of [1].
>>> >>
>>> >> With Infinispan 8.0 we are adding in clustered
>>> expiration. That
>>> >> includes an expiration event raised that is
clustered
>>> as well.
>>> >> Unfortunately expiration events currently occur
>>> multiple times (if
>>> >> numOwners > 1) at different times across nodes in
a
>>> cluster. This
>>> >> makes coordinating a single cluster expiration event
>>> quite difficult.
>>> >>
>>> >> To work around this I am proposing that the
expiration
>>> of an event
>>> >> is done solely by the owner of the given key that is
>>> now expired.
>>> >> This would fix the issue of having multiple events
and
>>> the event can
>>> >> be raised while holding the lock for the given key
so
>>> concurrent
>>> >> modifications would not be an issue.
>>> >>
>>> >> The problem arises when you have other nodes that
have
>>> expiration
>>> >> set but expire at different times. Max idle is the
>>> biggest offender
>>> >> with this as a read on an owner only refreshes the
>>> owners timestamp,
>>> >> meaning other owners would not be updated and expire
>>> preemptively.
>>> >> To have expiration work properly in this case you
would
>>> need
>>> >> coordination between the owners to see if anyone has
a
>>> higher
>>> >> value. This requires blocking and would have to be
>>> done while
>>> >> accessing a key that is expired to be sure if
>>> expiration happened or
>>> >> not.
>>> >>
>>> >> The linked dev listing proposed instead to only
expire
>>> an entry by
>>> >> the reaper thread and not on access. In this case a
>>> read will
>>> >> return a non null value until it is fully expired,
>>> increasing hit
>>> >> ratios possibly.
>>> >>
>>> >> Their are quire a bit of real benefits for this:
>>> >>
>>> >> 1. Cluster cache reads would be much simpler and
>>> wouldn't have to
>>> >> block to verify the object exists or not since this
>>> would only be
>>> >> done by the reaper thread (note this would have only
>>> happened if the
>>> >> entry was expired locally). An access would just
>>> return the value
>>> >> immediately.
>>> >> 2. Each node only expires entries it owns in the
reaper
>>> thread
>>> >> reducing how many entries they must check or remove.
>>> This also
>>> >> provides a single point where events would be raised
as
>>> we need.
>>> >> 3. A lot of code can now be removed and made simpler
as
>>> it no longer
>>> >> has to check for expiration. The expiration check
>>> would only be
>>> >> done in 1 place, the expiration reaper thread.
>>> >>
>>> >> The main issue with this proposal is as the other
>>> listing mentions
>>> >> is if user code expects the value to be gone after
>>> expiration for
>>> >> correctness. I would say this use case is not as
>>> compelling for
>>> >> maxIdle, especially since we never supported it
>>> properly. And in
>>> >> the case of lifespan the user could very easily
store
>>> the expiration
>>> >> time in the object that they can check after a get
as
>>> pointed out in
>>> >> the other thread.
>>> >>
>>> >> [1]
>>> >>
>>>
http://infinispan-developer-list.980875.n3.nabble.com/infinispan-dev-stri...
>>> >>
>>> >> _______________________________________________
>>> >> infinispan-dev mailing list
>>> >> infinispan-dev(a)lists.jboss.org
>>> <mailto:infinispan-dev@lists.jboss.org>
>>> <mailto:infinispan-dev@lists.jboss.org
>>> <mailto:infinispan-dev@lists.jboss.org>>
>>> >> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>> >>
>>> >>
>>> >>
>>> >> _______________________________________________
>>> >> infinispan-dev mailing list
>>> >> infinispan-dev(a)lists.jboss.org
>>> <mailto:infinispan-dev@lists.jboss.org>
>>> >> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>> >>
>>> >
>>> > --
>>> > Tristan Tarrant
>>> > Infinispan Lead
>>> > JBoss, a division of Red Hat
>>> > _______________________________________________
>>> > infinispan-dev mailing list
>>> > infinispan-dev(a)lists.jboss.org
>>> <mailto:infinispan-dev@lists.jboss.org>
>>> > https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>> _______________________________________________
>>> infinispan-dev mailing list
>>> infinispan-dev(a)lists.jboss.org
>>> <mailto:infinispan-dev@lists.jboss.org>
>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>>
>>>
>>>
>>> _______________________________________________
>>> infinispan-dev mailing list
>>> infinispan-dev(a)lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
--
Radim Vansa <rvansa(a)redhat.com>
JBoss Performance Team
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
9:53 a.m.
Is it possible/feasible to skip the notification from the backups to
the primary (and back) when there is no clustered expiration listener?
Dan
On Tue, Jul 21, 2015 at 5:25 PM, William Burns <mudokonman(a)gmail.com> wrote:
So I wanted to sum up what it looks like the plan is for this in
regards to
cluster expiration for ISPN 8.
First off to not make it ambiguous, maxIdle being used with a clustered
cache will provide undefined and unsupported behavior. This can and will
expire entries on a single node without notifying other cluster members
(essentially it will operate as it does today unchanged).
This leaves me to talk solely about lifespan cluster expiration.
Lifespan Expiration events are fired by the primary owner of an expired key
- when accessing an expired entry.
- by the reaper thread.
If the expiration is detected by a node other than the primary owner, an
expiration command is sent to it and null is returned immediately not
waiting for a response.
Expiration event listeners follow the usual rules for sync/async: in the
case of a sync listener, the handler is invoked while holding the lock,
whereas an async listener will not hold locks.
It is desirable for expiration events to contain both the key and value.
However currently cache stores do not provide the value when they expire
values. Thus we can only guarantee the value is present when an in memory
expiration event occurs. We could plan on adding this later.
Also as you may have guessed this doesn't touch strict expiration, which I
think we have come to the conclusion should only work with maxIdle and as
such this is not explored with this iteration.
Let me know if you guys think this approach is okay.
Cheers,
- Will
On Tue, Jul 14, 2015 at 1:51 PM Radim Vansa <rvansa(a)redhat.com> wrote:
>
> Yes, I know about [1]. I've worked that around by storing timestamp in
> the entry as well and when a new record is added, the 'expired'
> invalidations are purged. But I can't purge that if I don't access it -
> Infinispan needs to handle that internally.
>
> Radim
>
> [1] https://hibernate.atlassian.net/browse/HHH-6219
>
> On 07/14/2015 05:45 PM, Dennis Reed wrote:
> > On 07/14/2015 11:08 AM, Radim Vansa wrote:
> >> On 07/14/2015 04:19 PM, William Burns wrote:
> >>>
> >>> On Tue, Jul 14, 2015 at 9:37 AM William Burns <mudokonman(a)gmail.com
> >>> <mailto:mudokonman@gmail.com>> wrote:
> >>>
> >>> On Tue, Jul 14, 2015 at 4:41 AM Dan Berindei
> >>> <dan.berindei(a)gmail.com
<mailto:dan.berindei@gmail.com>> wrote:
> >>>
> >>> Processing expiration only on the reaper thread sounds nice,
> >>> but I
> >>> have one reservation: processing 1 million entries to see
> >>> that
> >>> 1 of
> >>> them is expired is a lot of work, and in the general case we
> >>> will not
> >>> be able to ensure an expiration precision of less than 1
> >>> minute (maybe
> >>> more, with a huge SingleFileStore attached).
> >>>
> >>>
> >>> This isn't much different then before. The only difference
is
> >>> that if a user touched a value after it expired it wouldn't
show
> >>> up (which is unlikely with maxIdle especially).
> >>>
> >>>
> >>> What happens to users who need better precision? In
> >>> particular, I know
> >>> some JCache tests were failing because HotRod was only
> >>> supporting
> >>> 1-second resolution instead of the 1-millisecond resolution
> >>> they were
> >>> expecting.
> >>>
> >>>
> >>> JCache is an interesting piece. The thing about JCache is that
> >>> the spec is only defined for local caches. However I
wouldn't
> >>> want to muddy up the waters in regards to it behaving
> >>> differently
> >>> for local/remote. In the JCache scenario we could add an
> >>> interceptor to prevent it returning such values (we do something
> >>> similar already for events). JCache behavior vs ISPN behavior
> >>> seems a bit easier to differentiate. But like you are getting
> >>> at,
> >>> either way is not very appealing.
> >>>
> >>>
> >>>
> >>> I'm even less convinced about the need to guarantee that
a
> >>> clustered
> >>> expiration listener will only be triggered once, and that
> >>> the
> >>> entry
> >>> must be null everywhere after that listener was invoked.
> >>> What's the
> >>> use case?
> >>>
> >>>
> >>> Maybe Tristan would know more to answer. To be honest this work
> >>> seems fruitless unless we know what our end users want here.
> >>> Spending time on something for it to thrown out is never fun :(
> >>>
> >>> And the more I thought about this the more I question the
> >>> validity
> >>> of maxIdle even. It seems like a very poor way to prevent
> >>> memory
> >>> exhaustion, which eviction does in a much better way and has
> >>> much
> >>> more flexible algorithms. Does anyone know what maxIdle would
> >>> be
> >>> used for that wouldn't be covered by eviction? The only thing
I
> >>> can think of is cleaning up the cache store as well.
> >>>
> >>>
> >>> Actually I guess for session/authentication related information this
> >>> would be important. However maxIdle isn't really as usable in that
> >>> case since most likely you would have a sticky session to go back to
> >>> that node which means you would never refresh the last used date on
> >>> the copies (current implementation). Without cluster expiration you
> >>> could lose that session information on a failover very easily.
> >> I would say that maxIdle can be used as for memory management as kind
> >> of
> >> WeakHashMap - e.g. in 2LC the maxIdle is used to store some record for
> >> a
> >> short while (regular transaction lifespan ~ seconds to minutes), and
> >> regularly the record is removed. However, to make sure that we don't
> >> leak records in this cache (if something goes wrong and the remove does
> >> not occur), it is removed.
> > Note that just relying on maxIdle doesn't guarantee you won't leak
> > records in this use case (specifically with the way the current
> > hibernate-infinispan 2LC implementation uses it).
> >
> > Hibernate-infinispan adds entries to its own Map stored in Infinispan,
> > and expects maxIdle to remove the map if it skips a remove. But in a
> > current case, we found that due to frequent accesses to that same map
> > the entries never idle out and it ends up in OOME).
> >
> > -Dennis
> >
> >> I can guess how long the transaction takes place, but not how many
> >> parallel transactions there are. With eviction algorithms (where I am
> >> not sure about the exact guarantees) I can set the cache to not hold
> >> more than N entries, but I can't know for sure that my record does not
> >> suddenly get evicted after shorter period, possibly causing some
> >> inconsistency.
> >> So this is similar to WeakHashMap by removing the key "when it
can't be
> >> used anymore" because I know that the transaction will finish before
> >> the
> >> deadline. I don't care about the exact size, I don't want to tune
that,
> >> I just don't want to leak.
> >>
> >> From my POV the non-strict maxIdle and strict expiration would be a
> >> nice compromise.
> >>
> >> Radim
> >>
> >>> Note that this would make the reaper thread less efficient:
> >>> with
> >>> numOwners=2 (best case), half of the entries that the reaper
> >>> touches
> >>> cannot be expired, because the node isn't the primary
node.
> >>> And to
> >>> make matters worse, the same reaper thread would have to
> >>> perform a
> >>> (synchronous?) RPC for each entry to ensure it expires
> >>> everywhere.
> >>>
> >>>
> >>> I have debated about this, it could something like a sync
> >>> removeAll which has a special marker to tell it is due to
> >>> expiration (which would raise listeners there), while also
> >>> sending
> >>> a cluster expiration event to other non owners.
> >>>
> >>>
> >>> For maxIdle I'd like to know more information about how
> >>> exactly the
> >>> owners would coordinate to expire an entry. I'm pretty
sure
> >>> we
> >>> cannot
> >>> avoid ignoring some reads (expiring an entry immediately
> >>> after
> >>> it was
> >>> read), and ensuring that we don't accidentally extend an
> >>> entry's life
> >>> (like the current code does, when we transfer an entry to a
> >>> new owner)
> >>> also sounds problematic.
> >>>
> >>>
> >>> For lifespan it is simple, the primary owner just expires it
> >>> when
> >>> it expires there. There is no coordination needed in this case
> >>> it
> >>> just sends the expired remove to owners etc.
> >>>
> >>> Max idle is more complicated as we all know. The primary owner
> >>> would send a request for the last used time for a given key or
> >>> set
> >>> of keys. Then the owner would take those times and check for a
> >>> new access it isn't aware of. If there isn't then it
would send
> >>> a
> >>> remove command for the key(s). If there is a new access the
> >>> owner
> >>> would instead send the last used time to all of the owners. The
> >>> expiration obviously would have a window that if a read occurred
> >>> after sending a response that could be ignored. This could be
> >>> resolved by using some sort of 2PC and blocking reads during
> >>> that
> >>> period but I would say it isn't worth it.
> >>>
> >>> The issue with transferring to a new node refreshing the last
> >>> update/lifespan seems like just a bug we need to fix
> >>> irrespective
> >>> of this issue IMO.
> >>>
> >>>
> >>> I'm not saying expiring entries on each node
independently
> >>> is
> >>> perfect,
> >>> far from it. But I wouldn't want us to provide new
> >>> guarantees that
> >>> could hurt performance without a really good use case.
> >>>
> >>>
> >>> I would guess that user perceived performance should be a little
> >>> faster with this. But this also depends on an alternative that
> >>> we
> >>> decided on :)
> >>>
> >>> Also the expiration thread pool is set to min priority atm so it
> >>> may delay removal of said objects but hopefully (if the jvm
> >>> supports) it wouldn't overrun a CPU while processing unless
it
> >>> has
> >>> availability.
> >>>
> >>>
> >>> Cheers
> >>> Dan
> >>>
> >>>
> >>> On Mon, Jul 13, 2015 at 9:25 PM, Tristan Tarrant
> >>> <ttarrant(a)redhat.com <mailto:ttarrant@redhat.com>>
wrote:
> >>> > After re-reading the whole original thread, I agree with
> >>> the
> >>> proposal
> >>> > with two caveats:
> >>> >
> >>> > - ensure that we don't break JCache compatibility
> >>> > - ensure that we document this properly
> >>> >
> >>> > Tristan
> >>> >
> >>> > On 13/07/2015 18:41, Sanne Grinovero wrote:
> >>> >> +1
> >>> >> You had me convinced at the first line, although
"A lot
> >>> of
> >>> code can now
> >>> >> be removed and made simpler" makes it look
extremely
> >>> nice.
> >>> >>
> >>> >> On 13 Jul 2015 18:14, "William Burns"
> >>> <mudokonman(a)gmail.com
> >>> <mailto:mudokonman@gmail.com>
> >>> >> <mailto:mudokonman@gmail.com
>
> >>> <mailto:mudokonman@gmail.com>>> wrote:
> >>> >>
> >>> >> This is a necro of [1].
> >>> >>
> >>> >> With Infinispan 8.0 we are adding in clustered
> >>> expiration. That
> >>> >> includes an expiration event raised that is
clustered
> >>> as well.
> >>> >> Unfortunately expiration events currently occur
> >>> multiple times (if
> >>> >> numOwners > 1) at different times across nodes
in a
> >>> cluster. This
> >>> >> makes coordinating a single cluster expiration
event
> >>> quite difficult.
> >>> >>
> >>> >> To work around this I am proposing that the
> >>> expiration
> >>> of an event
> >>> >> is done solely by the owner of the given key that
is
> >>> now expired.
> >>> >> This would fix the issue of having multiple
events
> >>> and
> >>> the event can
> >>> >> be raised while holding the lock for the given
key so
> >>> concurrent
> >>> >> modifications would not be an issue.
> >>> >>
> >>> >> The problem arises when you have other nodes
that
> >>> have
> >>> expiration
> >>> >> set but expire at different times. Max idle is
the
> >>> biggest offender
> >>> >> with this as a read on an owner only refreshes
the
> >>> owners timestamp,
> >>> >> meaning other owners would not be updated and
expire
> >>> preemptively.
> >>> >> To have expiration work properly in this case
you
> >>> would
> >>> need
> >>> >> coordination between the owners to see if anyone
has
> >>> a
> >>> higher
> >>> >> value. This requires blocking and would have to
be
> >>> done while
> >>> >> accessing a key that is expired to be sure if
> >>> expiration happened or
> >>> >> not.
> >>> >>
> >>> >> The linked dev listing proposed instead to only
> >>> expire
> >>> an entry by
> >>> >> the reaper thread and not on access. In this
case a
> >>> read will
> >>> >> return a non null value until it is fully
expired,
> >>> increasing hit
> >>> >> ratios possibly.
> >>> >>
> >>> >> Their are quire a bit of real benefits for this:
> >>> >>
> >>> >> 1. Cluster cache reads would be much simpler and
> >>> wouldn't have to
> >>> >> block to verify the object exists or not since
this
> >>> would only be
> >>> >> done by the reaper thread (note this would have
only
> >>> happened if the
> >>> >> entry was expired locally). An access would
just
> >>> return the value
> >>> >> immediately.
> >>> >> 2. Each node only expires entries it owns in the
> >>> reaper
> >>> thread
> >>> >> reducing how many entries they must check or
remove.
> >>> This also
> >>> >> provides a single point where events would be
raised
> >>> as
> >>> we need.
> >>> >> 3. A lot of code can now be removed and made
simpler
> >>> as
> >>> it no longer
> >>> >> has to check for expiration. The expiration
check
> >>> would only be
> >>> >> done in 1 place, the expiration reaper thread.
> >>> >>
> >>> >> The main issue with this proposal is as the
other
> >>> listing mentions
> >>> >> is if user code expects the value to be gone
after
> >>> expiration for
> >>> >> correctness. I would say this use case is not
as
> >>> compelling for
> >>> >> maxIdle, especially since we never supported it
> >>> properly. And in
> >>> >> the case of lifespan the user could very easily
store
> >>> the expiration
> >>> >> time in the object that they can check after a
get as
> >>> pointed out in
> >>> >> the other thread.
> >>> >>
> >>> >> [1]
> >>> >>
> >>>
> >>>
http://infinispan-developer-list.980875.n3.nabble.com/infinispan-dev-stri...
> >>> >>
> >>> >> _______________________________________________
> >>> >> infinispan-dev mailing list
> >>> >> infinispan-dev(a)lists.jboss.org
> >>> <mailto:infinispan-dev@lists.jboss.org>
> >>> <mailto:infinispan-dev@lists.jboss.org
> >>> <mailto:infinispan-dev@lists.jboss.org>>
> >>> >>
https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >>> >>
> >>> >>
> >>> >>
> >>> >> _______________________________________________
> >>> >> infinispan-dev mailing list
> >>> >> infinispan-dev(a)lists.jboss.org
> >>> <mailto:infinispan-dev@lists.jboss.org>
> >>> >>
https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >>> >>
> >>> >
> >>> > --
> >>> > Tristan Tarrant
> >>> > Infinispan Lead
> >>> > JBoss, a division of Red Hat
> >>> > _______________________________________________
> >>> > infinispan-dev mailing list
> >>> > infinispan-dev(a)lists.jboss.org
> >>> <mailto:infinispan-dev@lists.jboss.org>
> >>> > https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >>> _______________________________________________
> >>> infinispan-dev mailing list
> >>> infinispan-dev(a)lists.jboss.org
> >>> <mailto:infinispan-dev@lists.jboss.org>
> >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >>>
> >>>
> >>>
> >>> _______________________________________________
> >>> infinispan-dev mailing list
> >>> infinispan-dev(a)lists.jboss.org
> >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >>
> > _______________________________________________
> > infinispan-dev mailing list
> > infinispan-dev(a)lists.jboss.org
> > https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
>
> --
> Radim Vansa <rvansa(a)redhat.com>
> JBoss Performance Team
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
10:06 a.m.
On Wed, Jul 22, 2015 at 10:53 AM Dan Berindei <dan.berindei(a)gmail.com>
wrote:
Is it possible/feasible to skip the notification from the backups to
the primary (and back) when there is no clustered expiration listener?
Unfortunately there is no way to distinguish whether or a listener is
create, modify, remove or expiration. So this would only work if there are
no clustered listeners.
This however should be feasible. This shouldn't be hard to add.
The only thing I would have to figure out is what happens in the case of a
rehash and the node that removed the value is now the primary owner and
some nodes have the old value and someone registers an expiration
listener. I am thinking I should only raise the event if the primary owner
still has the value.
Dan
On Tue, Jul 21, 2015 at 5:25 PM, William Burns <mudokonman(a)gmail.com>
wrote:
> So I wanted to sum up what it looks like the plan is for this in regards
to
> cluster expiration for ISPN 8.
>
> First off to not make it ambiguous, maxIdle being used with a clustered
> cache will provide undefined and unsupported behavior. This can and will
> expire entries on a single node without notifying other cluster members
> (essentially it will operate as it does today unchanged).
>
> This leaves me to talk solely about lifespan cluster expiration.
>
> Lifespan Expiration events are fired by the primary owner of an expired
key
>
> - when accessing an expired entry.
>
> - by the reaper thread.
>
> If the expiration is detected by a node other than the primary owner, an
> expiration command is sent to it and null is returned immediately not
> waiting for a response.
>
> Expiration event listeners follow the usual rules for sync/async: in the
> case of a sync listener, the handler is invoked while holding the lock,
> whereas an async listener will not hold locks.
>
> It is desirable for expiration events to contain both the key and value.
> However currently cache stores do not provide the value when they expire
> values. Thus we can only guarantee the value is present when an in
memory
> expiration event occurs. We could plan on adding this later.
>
> Also as you may have guessed this doesn't touch strict expiration, which
I
> think we have come to the conclusion should only work with maxIdle and as
> such this is not explored with this iteration.
>
> Let me know if you guys think this approach is okay.
>
> Cheers,
>
> - Will
>
> On Tue, Jul 14, 2015 at 1:51 PM Radim Vansa <rvansa(a)redhat.com> wrote:
>>
>> Yes, I know about [1]. I've worked that around by storing timestamp in
>> the entry as well and when a new record is added, the 'expired'
>> invalidations are purged. But I can't purge that if I don't access it -
>> Infinispan needs to handle that internally.
>>
>> Radim
>>
>> [1] https://hibernate.atlassian.net/browse/HHH-6219
>>
>> On 07/14/2015 05:45 PM, Dennis Reed wrote:
>> > On 07/14/2015 11:08 AM, Radim Vansa wrote:
>> >> On 07/14/2015 04:19 PM, William Burns wrote:
>> >>>
>> >>> On Tue, Jul 14, 2015 at 9:37 AM William Burns
<mudokonman(a)gmail.com
>> >>> <mailto:mudokonman@gmail.com>> wrote:
>> >>>
>> >>> On Tue, Jul 14, 2015 at 4:41 AM Dan Berindei
>> >>> <dan.berindei(a)gmail.com
<mailto:dan.berindei@gmail.com>>
wrote:
>> >>>
>> >>> Processing expiration only on the reaper thread sounds
nice,
>> >>> but I
>> >>> have one reservation: processing 1 million entries to
see
>> >>> that
>> >>> 1 of
>> >>> them is expired is a lot of work, and in the general
case
we
>> >>> will not
>> >>> be able to ensure an expiration precision of less than 1
>> >>> minute (maybe
>> >>> more, with a huge SingleFileStore attached).
>> >>>
>> >>>
>> >>> This isn't much different then before. The only
difference is
>> >>> that if a user touched a value after it expired it
wouldn't
show
>> >>> up (which is unlikely with maxIdle especially).
>> >>>
>> >>>
>> >>> What happens to users who need better precision? In
>> >>> particular, I know
>> >>> some JCache tests were failing because HotRod was only
>> >>> supporting
>> >>> 1-second resolution instead of the 1-millisecond
resolution
>> >>> they were
>> >>> expecting.
>> >>>
>> >>>
>> >>> JCache is an interesting piece. The thing about JCache is
that
>> >>> the spec is only defined for local caches. However I
wouldn't
>> >>> want to muddy up the waters in regards to it behaving
>> >>> differently
>> >>> for local/remote. In the JCache scenario we could add an
>> >>> interceptor to prevent it returning such values (we do
something
>> >>> similar already for events). JCache behavior vs ISPN
behavior
>> >>> seems a bit easier to differentiate. But like you are
getting
>> >>> at,
>> >>> either way is not very appealing.
>> >>>
>> >>>
>> >>>
>> >>> I'm even less convinced about the need to guarantee
that a
>> >>> clustered
>> >>> expiration listener will only be triggered once, and
that
>> >>> the
>> >>> entry
>> >>> must be null everywhere after that listener was invoked.
>> >>> What's the
>> >>> use case?
>> >>>
>> >>>
>> >>> Maybe Tristan would know more to answer. To be honest this
work
>> >>> seems fruitless unless we know what our end users want here.
>> >>> Spending time on something for it to thrown out is never fun
:(
>> >>>
>> >>> And the more I thought about this the more I question the
>> >>> validity
>> >>> of maxIdle even. It seems like a very poor way to prevent
>> >>> memory
>> >>> exhaustion, which eviction does in a much better way and has
>> >>> much
>> >>> more flexible algorithms. Does anyone know what maxIdle
would
>> >>> be
>> >>> used for that wouldn't be covered by eviction? The only
thing I
>> >>> can think of is cleaning up the cache store as well.
>> >>>
>> >>>
>> >>> Actually I guess for session/authentication related information
this
>> >>> would be important. However maxIdle isn't really as usable in
that
>> >>> case since most likely you would have a sticky session to go back
to
>> >>> that node which means you would never refresh the last used date
on
>> >>> the copies (current implementation). Without cluster expiration
you
>> >>> could lose that session information on a failover very easily.
>> >> I would say that maxIdle can be used as for memory management as kind
>> >> of
>> >> WeakHashMap - e.g. in 2LC the maxIdle is used to store some record
for
>> >> a
>> >> short while (regular transaction lifespan ~ seconds to minutes), and
>> >> regularly the record is removed. However, to make sure that we
don't
>> >> leak records in this cache (if something goes wrong and the remove
does
>> >> not occur), it is removed.
>> > Note that just relying on maxIdle doesn't guarantee you won't leak
>> > records in this use case (specifically with the way the current
>> > hibernate-infinispan 2LC implementation uses it).
>> >
>> > Hibernate-infinispan adds entries to its own Map stored in Infinispan,
>> > and expects maxIdle to remove the map if it skips a remove. But in a
>> > current case, we found that due to frequent accesses to that same map
>> > the entries never idle out and it ends up in OOME).
>> >
>> > -Dennis
>> >
>> >> I can guess how long the transaction takes place, but not how many
>> >> parallel transactions there are. With eviction algorithms (where I am
>> >> not sure about the exact guarantees) I can set the cache to not hold
>> >> more than N entries, but I can't know for sure that my record does
not
>> >> suddenly get evicted after shorter period, possibly causing some
>> >> inconsistency.
>> >> So this is similar to WeakHashMap by removing the key "when it
can't
be
>> >> used anymore" because I know that the transaction will finish
before
>> >> the
>> >> deadline. I don't care about the exact size, I don't want to
tune
that,
>> >> I just don't want to leak.
>> >>
>> >> From my POV the non-strict maxIdle and strict expiration would be
a
>> >> nice compromise.
>> >>
>> >> Radim
>> >>
>> >>> Note that this would make the reaper thread less
efficient:
>> >>> with
>> >>> numOwners=2 (best case), half of the entries that the
reaper
>> >>> touches
>> >>> cannot be expired, because the node isn't the
primary
node.
>> >>> And to
>> >>> make matters worse, the same reaper thread would have to
>> >>> perform a
>> >>> (synchronous?) RPC for each entry to ensure it expires
>> >>> everywhere.
>> >>>
>> >>>
>> >>> I have debated about this, it could something like a sync
>> >>> removeAll which has a special marker to tell it is due to
>> >>> expiration (which would raise listeners there), while also
>> >>> sending
>> >>> a cluster expiration event to other non owners.
>> >>>
>> >>>
>> >>> For maxIdle I'd like to know more information about
how
>> >>> exactly the
>> >>> owners would coordinate to expire an entry. I'm
pretty
sure
>> >>> we
>> >>> cannot
>> >>> avoid ignoring some reads (expiring an entry immediately
>> >>> after
>> >>> it was
>> >>> read), and ensuring that we don't accidentally extend
an
>> >>> entry's life
>> >>> (like the current code does, when we transfer an entry
to
a
>> >>> new owner)
>> >>> also sounds problematic.
>> >>>
>> >>>
>> >>> For lifespan it is simple, the primary owner just expires it
>> >>> when
>> >>> it expires there. There is no coordination needed in this
case
>> >>> it
>> >>> just sends the expired remove to owners etc.
>> >>>
>> >>> Max idle is more complicated as we all know. The primary
owner
>> >>> would send a request for the last used time for a given key
or
>> >>> set
>> >>> of keys. Then the owner would take those times and check
for
a
>> >>> new access it isn't aware of. If there isn't then it
would
send
>> >>> a
>> >>> remove command for the key(s). If there is a new access the
>> >>> owner
>> >>> would instead send the last used time to all of the owners.
The
>> >>> expiration obviously would have a window that if a read
occurred
>> >>> after sending a response that could be ignored. This could
be
>> >>> resolved by using some sort of 2PC and blocking reads during
>> >>> that
>> >>> period but I would say it isn't worth it.
>> >>>
>> >>> The issue with transferring to a new node refreshing the
last
>> >>> update/lifespan seems like just a bug we need to fix
>> >>> irrespective
>> >>> of this issue IMO.
>> >>>
>> >>>
>> >>> I'm not saying expiring entries on each node
independently
>> >>> is
>> >>> perfect,
>> >>> far from it. But I wouldn't want us to provide new
>> >>> guarantees that
>> >>> could hurt performance without a really good use case.
>> >>>
>> >>>
>> >>> I would guess that user perceived performance should be a
little
>> >>> faster with this. But this also depends on an alternative
that
>> >>> we
>> >>> decided on :)
>> >>>
>> >>> Also the expiration thread pool is set to min priority atm
so
it
>> >>> may delay removal of said objects but hopefully (if the jvm
>> >>> supports) it wouldn't overrun a CPU while processing
unless it
>> >>> has
>> >>> availability.
>> >>>
>> >>>
>> >>> Cheers
>> >>> Dan
>> >>>
>> >>>
>> >>> On Mon, Jul 13, 2015 at 9:25 PM, Tristan Tarrant
>> >>> <ttarrant(a)redhat.com
<mailto:ttarrant@redhat.com>> wrote:
>> >>> > After re-reading the whole original thread, I agree
with
>> >>> the
>> >>> proposal
>> >>> > with two caveats:
>> >>> >
>> >>> > - ensure that we don't break JCache
compatibility
>> >>> > - ensure that we document this properly
>> >>> >
>> >>> > Tristan
>> >>> >
>> >>> > On 13/07/2015 18:41, Sanne Grinovero wrote:
>> >>> >> +1
>> >>> >> You had me convinced at the first line, although
"A lot
>> >>> of
>> >>> code can now
>> >>> >> be removed and made simpler" makes it look
extremely
>> >>> nice.
>> >>> >>
>> >>> >> On 13 Jul 2015 18:14, "William Burns"
>> >>> <mudokonman(a)gmail.com
>> >>> <mailto:mudokonman@gmail.com>
>> >>> >> <mailto:mudokonman@gmail.com
>>
>> >>> <mailto:mudokonman@gmail.com>>> wrote:
>> >>> >>
>> >>> >> This is a necro of [1].
>> >>> >>
>> >>> >> With Infinispan 8.0 we are adding in
clustered
>> >>> expiration. That
>> >>> >> includes an expiration event raised that is
clustered
>> >>> as well.
>> >>> >> Unfortunately expiration events currently
occur
>> >>> multiple times (if
>> >>> >> numOwners > 1) at different times across
nodes in a
>> >>> cluster. This
>> >>> >> makes coordinating a single cluster
expiration
event
>> >>> quite difficult.
>> >>> >>
>> >>> >> To work around this I am proposing that the
>> >>> expiration
>> >>> of an event
>> >>> >> is done solely by the owner of the given key
that
is
>> >>> now expired.
>> >>> >> This would fix the issue of having multiple
events
>> >>> and
>> >>> the event can
>> >>> >> be raised while holding the lock for the
given key
so
>> >>> concurrent
>> >>> >> modifications would not be an issue.
>> >>> >>
>> >>> >> The problem arises when you have other nodes
that
>> >>> have
>> >>> expiration
>> >>> >> set but expire at different times. Max idle
is the
>> >>> biggest offender
>> >>> >> with this as a read on an owner only
refreshes the
>> >>> owners timestamp,
>> >>> >> meaning other owners would not be updated
and
expire
>> >>> preemptively.
>> >>> >> To have expiration work properly in this
case you
>> >>> would
>> >>> need
>> >>> >> coordination between the owners to see if
anyone
has
>> >>> a
>> >>> higher
>> >>> >> value. This requires blocking and would
have to be
>> >>> done while
>> >>> >> accessing a key that is expired to be sure
if
>> >>> expiration happened or
>> >>> >> not.
>> >>> >>
>> >>> >> The linked dev listing proposed instead to
only
>> >>> expire
>> >>> an entry by
>> >>> >> the reaper thread and not on access. In
this case
a
>> >>> read will
>> >>> >> return a non null value until it is fully
expired,
>> >>> increasing hit
>> >>> >> ratios possibly.
>> >>> >>
>> >>> >> Their are quire a bit of real benefits for
this:
>> >>> >>
>> >>> >> 1. Cluster cache reads would be much simpler
and
>> >>> wouldn't have to
>> >>> >> block to verify the object exists or not
since this
>> >>> would only be
>> >>> >> done by the reaper thread (note this would
have
only
>> >>> happened if the
>> >>> >> entry was expired locally). An access would
just
>> >>> return the value
>> >>> >> immediately.
>> >>> >> 2. Each node only expires entries it owns in
the
>> >>> reaper
>> >>> thread
>> >>> >> reducing how many entries they must check
or
remove.
>> >>> This also
>> >>> >> provides a single point where events would
be
raised
>> >>> as
>> >>> we need.
>> >>> >> 3. A lot of code can now be removed and
made
simpler
>> >>> as
>> >>> it no longer
>> >>> >> has to check for expiration. The expiration
check
>> >>> would only be
>> >>> >> done in 1 place, the expiration reaper
thread.
>> >>> >>
>> >>> >> The main issue with this proposal is as the
other
>> >>> listing mentions
>> >>> >> is if user code expects the value to be gone
after
>> >>> expiration for
>> >>> >> correctness. I would say this use case is
not as
>> >>> compelling for
>> >>> >> maxIdle, especially since we never supported
it
>> >>> properly. And in
>> >>> >> the case of lifespan the user could very
easily
store
>> >>> the expiration
>> >>> >> time in the object that they can check after
a get
as
>> >>> pointed out in
>> >>> >> the other thread.
>> >>> >>
>> >>> >> [1]
>> >>> >>
>> >>>
>> >>>
http://infinispan-developer-list.980875.n3.nabble.com/infinispan-dev-stri...
>> >>> >>
>> >>> >>
_______________________________________________
>> >>> >> infinispan-dev mailing list
>> >>> >> infinispan-dev(a)lists.jboss.org
>> >>> <mailto:infinispan-dev@lists.jboss.org>
>> >>> <mailto:infinispan-dev@lists.jboss.org
>> >>> <mailto:infinispan-dev@lists.jboss.org>>
>> >>> >>
https://lists.jboss.org/mailman/listinfo/infinispan-dev
>> >>> >>
>> >>> >>
>> >>> >>
>> >>> >> _______________________________________________
>> >>> >> infinispan-dev mailing list
>> >>> >> infinispan-dev(a)lists.jboss.org
>> >>> <mailto:infinispan-dev@lists.jboss.org>
>> >>> >>
https://lists.jboss.org/mailman/listinfo/infinispan-dev
>> >>> >>
>> >>> >
>> >>> > --
>> >>> > Tristan Tarrant
>> >>> > Infinispan Lead
>> >>> > JBoss, a division of Red Hat
>> >>> > _______________________________________________
>> >>> > infinispan-dev mailing list
>> >>> > infinispan-dev(a)lists.jboss.org
>> >>> <mailto:infinispan-dev@lists.jboss.org>
>> >>> >
https://lists.jboss.org/mailman/listinfo/infinispan-dev
>> >>> _______________________________________________
>> >>> infinispan-dev mailing list
>> >>> infinispan-dev(a)lists.jboss.org
>> >>> <mailto:infinispan-dev@lists.jboss.org>
>> >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>> >>>
>> >>>
>> >>>
>> >>> _______________________________________________
>> >>> infinispan-dev mailing list
>> >>> infinispan-dev(a)lists.jboss.org
>> >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>> >>
>> > _______________________________________________
>> > infinispan-dev mailing list
>> > infinispan-dev(a)lists.jboss.org
>> > https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>
>>
>> --
>> Radim Vansa <rvansa(a)redhat.com>
>> JBoss Performance Team
>>
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev(a)lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
7:37 a.m.
New subject: Fwd: Strict Expiration
I actually found another hiccup with cache stores. It seems currently we
only allow for a callback when an entry is expired from a cache store when
using the reaper thread [1]. However we don't allow for such a callback on
a read which finds an expired entry and wants to remove it [2].
Interestingly our cache stores in general don't even expire entries on load
with the few exceptions below:
1. SingleCacheStore returns true for an expired entry on contains
2. SingleCacheStore removes expired entries on load
3. RemoteStore does not need to worry about expiration since it is handled
by another remote server.
Of all of the other stores I have looked at they return false properly for
expired entries and only purge elements from within reaper thread.
I propose we change SingleCacheStore to behave as the other cache stores.
This doesn't require any API changes. We would then rely on store expiring
elements only during reaper thread or if the element expires in memory. We
should also guarantee that when a cache store is used that the reaper
thread is enabled (throw exception if not enabled and store is present at
init). Should I worry about when only a RemoteStore is used (this seems a
bit fragile)?
To be honest we would need to revamp the CacheLoader/Writer API at a later
point to allow for values to be optionally provided for expiration anyways,
so I would say to do that in addition to allowing loader/stores to expire
on access.
[1]
https://github.com/infinispan/infinispan/blob/master/core/src/main/java/o...
[2]
https://github.com/infinispan/infinispan/blob/master/core/src/main/java/o...
---------- Forwarded message ---------
From: William Burns <mudokonman(a)gmail.com>
Date: Wed, Jul 22, 2015 at 11:06 AM
Subject: Re: [infinispan-dev] Strict Expiration
To: infinispan -Dev List <infinispan-dev(a)lists.jboss.org>
On Wed, Jul 22, 2015 at 10:53 AM Dan Berindei <dan.berindei(a)gmail.com>
wrote:
Is it possible/feasible to skip the notification from the backups to
the primary (and back) when there is no clustered expiration listener?
Unfortunately there is no way to distinguish whether or a listener is
create, modify, remove or expiration. So this would only work if there are
no clustered listeners.
This however should be feasible. This shouldn't be hard to add.
The only thing I would have to figure out is what happens in the case of a
rehash and the node that removed the value is now the primary owner and
some nodes have the old value and someone registers an expiration
listener. I am thinking I should only raise the event if the primary owner
still has the value.
Dan
On Tue, Jul 21, 2015 at 5:25 PM, William Burns <mudokonman(a)gmail.com>
wrote:
> So I wanted to sum up what it looks like the plan is for this in regards
to
> cluster expiration for ISPN 8.
>
> First off to not make it ambiguous, maxIdle being used with a clustered
> cache will provide undefined and unsupported behavior. This can and will
> expire entries on a single node without notifying other cluster members
> (essentially it will operate as it does today unchanged).
>
> This leaves me to talk solely about lifespan cluster expiration.
>
> Lifespan Expiration events are fired by the primary owner of an expired
key
>
> - when accessing an expired entry.
>
> - by the reaper thread.
>
> If the expiration is detected by a node other than the primary owner, an
> expiration command is sent to it and null is returned immediately not
> waiting for a response.
>
> Expiration event listeners follow the usual rules for sync/async: in the
> case of a sync listener, the handler is invoked while holding the lock,
> whereas an async listener will not hold locks.
>
> It is desirable for expiration events to contain both the key and value.
> However currently cache stores do not provide the value when they expire
> values. Thus we can only guarantee the value is present when an in
memory
> expiration event occurs. We could plan on adding this later.
>
> Also as you may have guessed this doesn't touch strict expiration, which
I
> think we have come to the conclusion should only work with maxIdle and as
> such this is not explored with this iteration.
>
> Let me know if you guys think this approach is okay.
>
> Cheers,
>
> - Will
>
> On Tue, Jul 14, 2015 at 1:51 PM Radim Vansa <rvansa(a)redhat.com> wrote:
>>
>> Yes, I know about [1]. I've worked that around by storing timestamp in
>> the entry as well and when a new record is added, the 'expired'
>> invalidations are purged. But I can't purge that if I don't access it -
>> Infinispan needs to handle that internally.
>>
>> Radim
>>
>> [1] https://hibernate.atlassian.net/browse/HHH-6219
>>
>> On 07/14/2015 05:45 PM, Dennis Reed wrote:
>> > On 07/14/2015 11:08 AM, Radim Vansa wrote:
>> >> On 07/14/2015 04:19 PM, William Burns wrote:
>> >>>
>> >>> On Tue, Jul 14, 2015 at 9:37 AM William Burns
<mudokonman(a)gmail.com
>> >>> <mailto:mudokonman@gmail.com>> wrote:
>> >>>
>> >>> On Tue, Jul 14, 2015 at 4:41 AM Dan Berindei
>> >>> <dan.berindei(a)gmail.com
<mailto:dan.berindei@gmail.com>>
wrote:
>> >>>
>> >>> Processing expiration only on the reaper thread sounds
nice,
>> >>> but I
>> >>> have one reservation: processing 1 million entries to
see
>> >>> that
>> >>> 1 of
>> >>> them is expired is a lot of work, and in the general
case
we
>> >>> will not
>> >>> be able to ensure an expiration precision of less than 1
>> >>> minute (maybe
>> >>> more, with a huge SingleFileStore attached).
>> >>>
>> >>>
>> >>> This isn't much different then before. The only
difference is
>> >>> that if a user touched a value after it expired it
wouldn't
show
>> >>> up (which is unlikely with maxIdle especially).
>> >>>
>> >>>
>> >>> What happens to users who need better precision? In
>> >>> particular, I know
>> >>> some JCache tests were failing because HotRod was only
>> >>> supporting
>> >>> 1-second resolution instead of the 1-millisecond
resolution
>> >>> they were
>> >>> expecting.
>> >>>
>> >>>
>> >>> JCache is an interesting piece. The thing about JCache is
that
>> >>> the spec is only defined for local caches. However I
wouldn't
>> >>> want to muddy up the waters in regards to it behaving
>> >>> differently
>> >>> for local/remote. In the JCache scenario we could add an
>> >>> interceptor to prevent it returning such values (we do
something
>> >>> similar already for events). JCache behavior vs ISPN
behavior
>> >>> seems a bit easier to differentiate. But like you are
getting
>> >>> at,
>> >>> either way is not very appealing.
>> >>>
>> >>>
>> >>>
>> >>> I'm even less convinced about the need to guarantee
that a
>> >>> clustered
>> >>> expiration listener will only be triggered once, and
that
>> >>> the
>> >>> entry
>> >>> must be null everywhere after that listener was invoked.
>> >>> What's the
>> >>> use case?
>> >>>
>> >>>
>> >>> Maybe Tristan would know more to answer. To be honest this
work
>> >>> seems fruitless unless we know what our end users want here.
>> >>> Spending time on something for it to thrown out is never fun
:(
>> >>>
>> >>> And the more I thought about this the more I question the
>> >>> validity
>> >>> of maxIdle even. It seems like a very poor way to prevent
>> >>> memory
>> >>> exhaustion, which eviction does in a much better way and has
>> >>> much
>> >>> more flexible algorithms. Does anyone know what maxIdle
would
>> >>> be
>> >>> used for that wouldn't be covered by eviction? The only
thing I
>> >>> can think of is cleaning up the cache store as well.
>> >>>
>> >>>
>> >>> Actually I guess for session/authentication related information
this
>> >>> would be important. However maxIdle isn't really as usable in
that
>> >>> case since most likely you would have a sticky session to go back
to
>> >>> that node which means you would never refresh the last used date
on
>> >>> the copies (current implementation). Without cluster expiration
you
>> >>> could lose that session information on a failover very easily.
>> >> I would say that maxIdle can be used as for memory management as kind
>> >> of
>> >> WeakHashMap - e.g. in 2LC the maxIdle is used to store some record
for
>> >> a
>> >> short while (regular transaction lifespan ~ seconds to minutes), and
>> >> regularly the record is removed. However, to make sure that we
don't
>> >> leak records in this cache (if something goes wrong and the remove
does
>> >> not occur), it is removed.
>> > Note that just relying on maxIdle doesn't guarantee you won't leak
>> > records in this use case (specifically with the way the current
>> > hibernate-infinispan 2LC implementation uses it).
>> >
>> > Hibernate-infinispan adds entries to its own Map stored in Infinispan,
>> > and expects maxIdle to remove the map if it skips a remove. But in a
>> > current case, we found that due to frequent accesses to that same map
>> > the entries never idle out and it ends up in OOME).
>> >
>> > -Dennis
>> >
>> >> I can guess how long the transaction takes place, but not how many
>> >> parallel transactions there are. With eviction algorithms (where I am
>> >> not sure about the exact guarantees) I can set the cache to not hold
>> >> more than N entries, but I can't know for sure that my record does
not
>> >> suddenly get evicted after shorter period, possibly causing some
>> >> inconsistency.
>> >> So this is similar to WeakHashMap by removing the key "when it
can't
be
>> >> used anymore" because I know that the transaction will finish
before
>> >> the
>> >> deadline. I don't care about the exact size, I don't want to
tune
that,
>> >> I just don't want to leak.
>> >>
>> >> From my POV the non-strict maxIdle and strict expiration would be
a
>> >> nice compromise.
>> >>
>> >> Radim
>> >>
>> >>> Note that this would make the reaper thread less
efficient:
>> >>> with
>> >>> numOwners=2 (best case), half of the entries that the
reaper
>> >>> touches
>> >>> cannot be expired, because the node isn't the
primary
node.
>> >>> And to
>> >>> make matters worse, the same reaper thread would have to
>> >>> perform a
>> >>> (synchronous?) RPC for each entry to ensure it expires
>> >>> everywhere.
>> >>>
>> >>>
>> >>> I have debated about this, it could something like a sync
>> >>> removeAll which has a special marker to tell it is due to
>> >>> expiration (which would raise listeners there), while also
>> >>> sending
>> >>> a cluster expiration event to other non owners.
>> >>>
>> >>>
>> >>> For maxIdle I'd like to know more information about
how
>> >>> exactly the
>> >>> owners would coordinate to expire an entry. I'm
pretty
sure
>> >>> we
>> >>> cannot
>> >>> avoid ignoring some reads (expiring an entry immediately
>> >>> after
>> >>> it was
>> >>> read), and ensuring that we don't accidentally extend
an
>> >>> entry's life
>> >>> (like the current code does, when we transfer an entry
to
a
>> >>> new owner)
>> >>> also sounds problematic.
>> >>>
>> >>>
>> >>> For lifespan it is simple, the primary owner just expires it
>> >>> when
>> >>> it expires there. There is no coordination needed in this
case
>> >>> it
>> >>> just sends the expired remove to owners etc.
>> >>>
>> >>> Max idle is more complicated as we all know. The primary
owner
>> >>> would send a request for the last used time for a given key
or
>> >>> set
>> >>> of keys. Then the owner would take those times and check
for
a
>> >>> new access it isn't aware of. If there isn't then it
would
send
>> >>> a
>> >>> remove command for the key(s). If there is a new access the
>> >>> owner
>> >>> would instead send the last used time to all of the owners.
The
>> >>> expiration obviously would have a window that if a read
occurred
>> >>> after sending a response that could be ignored. This could
be
>> >>> resolved by using some sort of 2PC and blocking reads during
>> >>> that
>> >>> period but I would say it isn't worth it.
>> >>>
>> >>> The issue with transferring to a new node refreshing the
last
>> >>> update/lifespan seems like just a bug we need to fix
>> >>> irrespective
>> >>> of this issue IMO.
>> >>>
>> >>>
>> >>> I'm not saying expiring entries on each node
independently
>> >>> is
>> >>> perfect,
>> >>> far from it. But I wouldn't want us to provide new
>> >>> guarantees that
>> >>> could hurt performance without a really good use case.
>> >>>
>> >>>
>> >>> I would guess that user perceived performance should be a
little
>> >>> faster with this. But this also depends on an alternative
that
>> >>> we
>> >>> decided on :)
>> >>>
>> >>> Also the expiration thread pool is set to min priority atm
so
it
>> >>> may delay removal of said objects but hopefully (if the jvm
>> >>> supports) it wouldn't overrun a CPU while processing
unless it
>> >>> has
>> >>> availability.
>> >>>
>> >>>
>> >>> Cheers
>> >>> Dan
>> >>>
>> >>>
>> >>> On Mon, Jul 13, 2015 at 9:25 PM, Tristan Tarrant
>> >>> <ttarrant(a)redhat.com
<mailto:ttarrant@redhat.com>> wrote:
>> >>> > After re-reading the whole original thread, I agree
with
>> >>> the
>> >>> proposal
>> >>> > with two caveats:
>> >>> >
>> >>> > - ensure that we don't break JCache
compatibility
>> >>> > - ensure that we document this properly
>> >>> >
>> >>> > Tristan
>> >>> >
>> >>> > On 13/07/2015 18:41, Sanne Grinovero wrote:
>> >>> >> +1
>> >>> >> You had me convinced at the first line, although
"A lot
>> >>> of
>> >>> code can now
>> >>> >> be removed and made simpler" makes it look
extremely
>> >>> nice.
>> >>> >>
>> >>> >> On 13 Jul 2015 18:14, "William Burns"
>> >>> <mudokonman(a)gmail.com
>> >>> <mailto:mudokonman@gmail.com>
>> >>> >> <mailto:mudokonman@gmail.com
>>
>> >>> <mailto:mudokonman@gmail.com>>> wrote:
>> >>> >>
>> >>> >> This is a necro of [1].
>> >>> >>
>> >>> >> With Infinispan 8.0 we are adding in
clustered
>> >>> expiration. That
>> >>> >> includes an expiration event raised that is
clustered
>> >>> as well.
>> >>> >> Unfortunately expiration events currently
occur
>> >>> multiple times (if
>> >>> >> numOwners > 1) at different times across
nodes in a
>> >>> cluster. This
>> >>> >> makes coordinating a single cluster
expiration
event
>> >>> quite difficult.
>> >>> >>
>> >>> >> To work around this I am proposing that the
>> >>> expiration
>> >>> of an event
>> >>> >> is done solely by the owner of the given key
that
is
>> >>> now expired.
>> >>> >> This would fix the issue of having multiple
events
>> >>> and
>> >>> the event can
>> >>> >> be raised while holding the lock for the
given key
so
>> >>> concurrent
>> >>> >> modifications would not be an issue.
>> >>> >>
>> >>> >> The problem arises when you have other nodes
that
>> >>> have
>> >>> expiration
>> >>> >> set but expire at different times. Max idle
is the
>> >>> biggest offender
>> >>> >> with this as a read on an owner only
refreshes the
>> >>> owners timestamp,
>> >>> >> meaning other owners would not be updated
and
expire
>> >>> preemptively.
>> >>> >> To have expiration work properly in this
case you
>> >>> would
>> >>> need
>> >>> >> coordination between the owners to see if
anyone
has
>> >>> a
>> >>> higher
>> >>> >> value. This requires blocking and would
have to be
>> >>> done while
>> >>> >> accessing a key that is expired to be sure
if
>> >>> expiration happened or
>> >>> >> not.
>> >>> >>
>> >>> >> The linked dev listing proposed instead to
only
>> >>> expire
>> >>> an entry by
>> >>> >> the reaper thread and not on access. In
this case
a
>> >>> read will
>> >>> >> return a non null value until it is fully
expired,
>> >>> increasing hit
>> >>> >> ratios possibly.
>> >>> >>
>> >>> >> Their are quire a bit of real benefits for
this:
>> >>> >>
>> >>> >> 1. Cluster cache reads would be much simpler
and
>> >>> wouldn't have to
>> >>> >> block to verify the object exists or not
since this
>> >>> would only be
>> >>> >> done by the reaper thread (note this would
have
only
>> >>> happened if the
>> >>> >> entry was expired locally). An access would
just
>> >>> return the value
>> >>> >> immediately.
>> >>> >> 2. Each node only expires entries it owns in
the
>> >>> reaper
>> >>> thread
>> >>> >> reducing how many entries they must check
or
remove.
>> >>> This also
>> >>> >> provides a single point where events would
be
raised
>> >>> as
>> >>> we need.
>> >>> >> 3. A lot of code can now be removed and
made
simpler
>> >>> as
>> >>> it no longer
>> >>> >> has to check for expiration. The expiration
check
>> >>> would only be
>> >>> >> done in 1 place, the expiration reaper
thread.
>> >>> >>
>> >>> >> The main issue with this proposal is as the
other
>> >>> listing mentions
>> >>> >> is if user code expects the value to be gone
after
>> >>> expiration for
>> >>> >> correctness. I would say this use case is
not as
>> >>> compelling for
>> >>> >> maxIdle, especially since we never supported
it
>> >>> properly. And in
>> >>> >> the case of lifespan the user could very
easily
store
>> >>> the expiration
>> >>> >> time in the object that they can check after
a get
as
>> >>> pointed out in
>> >>> >> the other thread.
>> >>> >>
>> >>> >> [1]
>> >>> >>
>> >>>
>> >>>
http://infinispan-developer-list.980875.n3.nabble.com/infinispan-dev-stri...
>> >>> >>
>> >>> >>
_______________________________________________
>> >>> >> infinispan-dev mailing list
>> >>> >> infinispan-dev(a)lists.jboss.org
>> >>> <mailto:infinispan-dev@lists.jboss.org>
>> >>> <mailto:infinispan-dev@lists.jboss.org
>> >>> <mailto:infinispan-dev@lists.jboss.org>>
>> >>> >>
https://lists.jboss.org/mailman/listinfo/infinispan-dev
>> >>> >>
>> >>> >>
>> >>> >>
>> >>> >> _______________________________________________
>> >>> >> infinispan-dev mailing list
>> >>> >> infinispan-dev(a)lists.jboss.org
>> >>> <mailto:infinispan-dev@lists.jboss.org>
>> >>> >>
https://lists.jboss.org/mailman/listinfo/infinispan-dev
>> >>> >>
>> >>> >
>> >>> > --
>> >>> > Tristan Tarrant
>> >>> > Infinispan Lead
>> >>> > JBoss, a division of Red Hat
>> >>> > _______________________________________________
>> >>> > infinispan-dev mailing list
>> >>> > infinispan-dev(a)lists.jboss.org
>> >>> <mailto:infinispan-dev@lists.jboss.org>
>> >>> >
https://lists.jboss.org/mailman/listinfo/infinispan-dev
>> >>> _______________________________________________
>> >>> infinispan-dev mailing list
>> >>> infinispan-dev(a)lists.jboss.org
>> >>> <mailto:infinispan-dev@lists.jboss.org>
>> >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>> >>>
>> >>>
>> >>>
>> >>> _______________________________________________
>> >>> infinispan-dev mailing list
>> >>> infinispan-dev(a)lists.jboss.org
>> >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>> >>
>> > _______________________________________________
>> > infinispan-dev mailing list
>> > infinispan-dev(a)lists.jboss.org
>> > https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>
>>
>> --
>> Radim Vansa <rvansa(a)redhat.com>
>> JBoss Performance Team
>>
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev(a)lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
8:59 a.m.
New subject: Fwd: Strict Expiration
On Thu, Jul 23, 2015 at 3:37 PM, William Burns <mudokonman(a)gmail.com> wrote:
I actually found another hiccup with cache stores. It seems
currently we
only allow for a callback when an entry is expired from a cache store when
using the reaper thread [1]. However we don't allow for such a callback on
a read which finds an expired entry and wants to remove it [2].
Interestingly our cache stores in general don't even expire entries on load
with the few exceptions below:
1. SingleCacheStore returns true for an expired entry on contains
2. SingleCacheStore removes expired entries on load
3. RemoteStore does not need to worry about expiration since it is handled
by another remote server.
Of all of the other stores I have looked at they return false properly for
expired entries and only purge elements from within reaper thread.
I propose we change SingleCacheStore to behave as the other cache stores.
This doesn't require any API changes. We would then rely on store expiring
elements only during reaper thread or if the element expires in memory. We
should also guarantee that when a cache store is used that the reaper thread
is enabled (throw exception if not enabled and store is present at init).
Should I worry about when only a RemoteStore is used (this seems a bit
fragile)?
+1, I wouldn't add a special case for RemoteStore.
To be honest we would need to revamp the CacheLoader/Writer API at a later
point to allow for values to be optionally provided for expiration anyways,
so I would say to do that in addition to allowing loader/stores to expire on
access.
Sounds good.
[1]
https://github.com/infinispan/infinispan/blob/master/core/src/main/java/o...
[2]
https://github.com/infinispan/infinispan/blob/master/core/src/main/java/o...
---------- Forwarded message ---------
From: William Burns <mudokonman(a)gmail.com>
Date: Wed, Jul 22, 2015 at 11:06 AM
Subject: Re: [infinispan-dev] Strict Expiration
To: infinispan -Dev List <infinispan-dev(a)lists.jboss.org>
On Wed, Jul 22, 2015 at 10:53 AM Dan Berindei <dan.berindei(a)gmail.com>
wrote:
>
> Is it possible/feasible to skip the notification from the backups to
> the primary (and back) when there is no clustered expiration listener?
Unfortunately there is no way to distinguish whether or a listener is
create, modify, remove or expiration. So this would only work if there are
no clustered listeners.
This however should be feasible. This shouldn't be hard to add.
It should be good enough.
The only thing I would have to figure out is what happens in the case of a
rehash and the node that removed the value is now the primary owner and some
nodes have the old value and someone registers an expiration listener. I am
thinking I should only raise the event if the primary owner still has the
value.
+1
>
>
> Dan
>
>
> On Tue, Jul 21, 2015 at 5:25 PM, William Burns <mudokonman(a)gmail.com>
> wrote:
> > So I wanted to sum up what it looks like the plan is for this in regards
> > to
> > cluster expiration for ISPN 8.
> >
> > First off to not make it ambiguous, maxIdle being used with a clustered
> > cache will provide undefined and unsupported behavior. This can and
> > will
> > expire entries on a single node without notifying other cluster members
> > (essentially it will operate as it does today unchanged).
> >
> > This leaves me to talk solely about lifespan cluster expiration.
> >
> > Lifespan Expiration events are fired by the primary owner of an expired
> > key
> >
> > - when accessing an expired entry.
> >
> > - by the reaper thread.
> >
> > If the expiration is detected by a node other than the primary owner, an
> > expiration command is sent to it and null is returned immediately not
> > waiting for a response.
> >
> > Expiration event listeners follow the usual rules for sync/async: in the
> > case of a sync listener, the handler is invoked while holding the lock,
> > whereas an async listener will not hold locks.
> >
> > It is desirable for expiration events to contain both the key and value.
> > However currently cache stores do not provide the value when they expire
> > values. Thus we can only guarantee the value is present when an in
> > memory
> > expiration event occurs. We could plan on adding this later.
> >
> > Also as you may have guessed this doesn't touch strict expiration, which
> > I
> > think we have come to the conclusion should only work with maxIdle and
> > as
> > such this is not explored with this iteration.
> >
> > Let me know if you guys think this approach is okay.
> >
> > Cheers,
> >
> > - Will
> >
> > On Tue, Jul 14, 2015 at 1:51 PM Radim Vansa <rvansa(a)redhat.com> wrote:
> >>
> >> Yes, I know about [1]. I've worked that around by storing timestamp in
> >> the entry as well and when a new record is added, the 'expired'
> >> invalidations are purged. But I can't purge that if I don't access
it -
> >> Infinispan needs to handle that internally.
> >>
> >> Radim
> >>
> >> [1] https://hibernate.atlassian.net/browse/HHH-6219
> >>
> >> On 07/14/2015 05:45 PM, Dennis Reed wrote:
> >> > On 07/14/2015 11:08 AM, Radim Vansa wrote:
> >> >> On 07/14/2015 04:19 PM, William Burns wrote:
> >> >>>
> >> >>> On Tue, Jul 14, 2015 at 9:37 AM William Burns
<mudokonman(a)gmail.com
> >> >>> <mailto:mudokonman@gmail.com>> wrote:
> >> >>>
> >> >>> On Tue, Jul 14, 2015 at 4:41 AM Dan Berindei
> >> >>> <dan.berindei(a)gmail.com
<mailto:dan.berindei@gmail.com>>
> >> >>> wrote:
> >> >>>
> >> >>> Processing expiration only on the reaper thread
sounds
> >> >>> nice,
> >> >>> but I
> >> >>> have one reservation: processing 1 million entries to
see
> >> >>> that
> >> >>> 1 of
> >> >>> them is expired is a lot of work, and in the general
case
> >> >>> we
> >> >>> will not
> >> >>> be able to ensure an expiration precision of less
than 1
> >> >>> minute (maybe
> >> >>> more, with a huge SingleFileStore attached).
> >> >>>
> >> >>>
> >> >>> This isn't much different then before. The only
difference
> >> >>> is
> >> >>> that if a user touched a value after it expired it
wouldn't
> >> >>> show
> >> >>> up (which is unlikely with maxIdle especially).
> >> >>>
> >> >>>
> >> >>> What happens to users who need better precision? In
> >> >>> particular, I know
> >> >>> some JCache tests were failing because HotRod was
only
> >> >>> supporting
> >> >>> 1-second resolution instead of the 1-millisecond
> >> >>> resolution
> >> >>> they were
> >> >>> expecting.
> >> >>>
> >> >>>
> >> >>> JCache is an interesting piece. The thing about JCache
is
> >> >>> that
> >> >>> the spec is only defined for local caches. However I
> >> >>> wouldn't
> >> >>> want to muddy up the waters in regards to it behaving
> >> >>> differently
> >> >>> for local/remote. In the JCache scenario we could add
an
> >> >>> interceptor to prevent it returning such values (we do
> >> >>> something
> >> >>> similar already for events). JCache behavior vs ISPN
> >> >>> behavior
> >> >>> seems a bit easier to differentiate. But like you are
> >> >>> getting
> >> >>> at,
> >> >>> either way is not very appealing.
> >> >>>
> >> >>>
> >> >>>
> >> >>> I'm even less convinced about the need to
guarantee that
> >> >>> a
> >> >>> clustered
> >> >>> expiration listener will only be triggered once, and
that
> >> >>> the
> >> >>> entry
> >> >>> must be null everywhere after that listener was
invoked.
> >> >>> What's the
> >> >>> use case?
> >> >>>
> >> >>>
> >> >>> Maybe Tristan would know more to answer. To be honest
this
> >> >>> work
> >> >>> seems fruitless unless we know what our end users want
here.
> >> >>> Spending time on something for it to thrown out is never
fun
> >> >>> :(
> >> >>>
> >> >>> And the more I thought about this the more I question
the
> >> >>> validity
> >> >>> of maxIdle even. It seems like a very poor way to
prevent
> >> >>> memory
> >> >>> exhaustion, which eviction does in a much better way and
has
> >> >>> much
> >> >>> more flexible algorithms. Does anyone know what maxIdle
> >> >>> would
> >> >>> be
> >> >>> used for that wouldn't be covered by eviction? The
only
> >> >>> thing I
> >> >>> can think of is cleaning up the cache store as well.
> >> >>>
> >> >>>
> >> >>> Actually I guess for session/authentication related
information
> >> >>> this
> >> >>> would be important. However maxIdle isn't really as usable
in that
> >> >>> case since most likely you would have a sticky session to go
back
> >> >>> to
> >> >>> that node which means you would never refresh the last used
date on
> >> >>> the copies (current implementation). Without cluster
expiration
> >> >>> you
> >> >>> could lose that session information on a failover very easily.
> >> >> I would say that maxIdle can be used as for memory management as
> >> >> kind
> >> >> of
> >> >> WeakHashMap - e.g. in 2LC the maxIdle is used to store some record
> >> >> for
> >> >> a
> >> >> short while (regular transaction lifespan ~ seconds to minutes),
and
> >> >> regularly the record is removed. However, to make sure that we
don't
> >> >> leak records in this cache (if something goes wrong and the remove
> >> >> does
> >> >> not occur), it is removed.
> >> > Note that just relying on maxIdle doesn't guarantee you won't
leak
> >> > records in this use case (specifically with the way the current
> >> > hibernate-infinispan 2LC implementation uses it).
> >> >
> >> > Hibernate-infinispan adds entries to its own Map stored in
> >> > Infinispan,
> >> > and expects maxIdle to remove the map if it skips a remove. But in a
> >> > current case, we found that due to frequent accesses to that same map
> >> > the entries never idle out and it ends up in OOME).
> >> >
> >> > -Dennis
> >> >
> >> >> I can guess how long the transaction takes place, but not how many
> >> >> parallel transactions there are. With eviction algorithms (where I
> >> >> am
> >> >> not sure about the exact guarantees) I can set the cache to not
hold
> >> >> more than N entries, but I can't know for sure that my record
does
> >> >> not
> >> >> suddenly get evicted after shorter period, possibly causing some
> >> >> inconsistency.
> >> >> So this is similar to WeakHashMap by removing the key "when it
can't
> >> >> be
> >> >> used anymore" because I know that the transaction will finish
before
> >> >> the
> >> >> deadline. I don't care about the exact size, I don't want
to tune
> >> >> that,
> >> >> I just don't want to leak.
> >> >>
> >> >> From my POV the non-strict maxIdle and strict expiration would
be
> >> >> a
> >> >> nice compromise.
> >> >>
> >> >> Radim
> >> >>
> >> >>> Note that this would make the reaper thread less
> >> >>> efficient:
> >> >>> with
> >> >>> numOwners=2 (best case), half of the entries that
the
> >> >>> reaper
> >> >>> touches
> >> >>> cannot be expired, because the node isn't the
primary
> >> >>> node.
> >> >>> And to
> >> >>> make matters worse, the same reaper thread would have
to
> >> >>> perform a
> >> >>> (synchronous?) RPC for each entry to ensure it
expires
> >> >>> everywhere.
> >> >>>
> >> >>>
> >> >>> I have debated about this, it could something like a
sync
> >> >>> removeAll which has a special marker to tell it is due
to
> >> >>> expiration (which would raise listeners there), while
also
> >> >>> sending
> >> >>> a cluster expiration event to other non owners.
> >> >>>
> >> >>>
> >> >>> For maxIdle I'd like to know more information
about how
> >> >>> exactly the
> >> >>> owners would coordinate to expire an entry. I'm
pretty
> >> >>> sure
> >> >>> we
> >> >>> cannot
> >> >>> avoid ignoring some reads (expiring an entry
immediately
> >> >>> after
> >> >>> it was
> >> >>> read), and ensuring that we don't accidentally
extend an
> >> >>> entry's life
> >> >>> (like the current code does, when we transfer an
entry to
> >> >>> a
> >> >>> new owner)
> >> >>> also sounds problematic.
> >> >>>
> >> >>>
> >> >>> For lifespan it is simple, the primary owner just expires
it
> >> >>> when
> >> >>> it expires there. There is no coordination needed in
this
> >> >>> case
> >> >>> it
> >> >>> just sends the expired remove to owners etc.
> >> >>>
> >> >>> Max idle is more complicated as we all know. The
primary
> >> >>> owner
> >> >>> would send a request for the last used time for a given
key
> >> >>> or
> >> >>> set
> >> >>> of keys. Then the owner would take those times and check
for
> >> >>> a
> >> >>> new access it isn't aware of. If there isn't
then it would
> >> >>> send
> >> >>> a
> >> >>> remove command for the key(s). If there is a new access
the
> >> >>> owner
> >> >>> would instead send the last used time to all of the
owners.
> >> >>> The
> >> >>> expiration obviously would have a window that if a read
> >> >>> occurred
> >> >>> after sending a response that could be ignored. This
could
> >> >>> be
> >> >>> resolved by using some sort of 2PC and blocking reads
during
> >> >>> that
> >> >>> period but I would say it isn't worth it.
> >> >>>
> >> >>> The issue with transferring to a new node refreshing the
last
> >> >>> update/lifespan seems like just a bug we need to fix
> >> >>> irrespective
> >> >>> of this issue IMO.
> >> >>>
> >> >>>
> >> >>> I'm not saying expiring entries on each node
> >> >>> independently
> >> >>> is
> >> >>> perfect,
> >> >>> far from it. But I wouldn't want us to provide
new
> >> >>> guarantees that
> >> >>> could hurt performance without a really good use
case.
> >> >>>
> >> >>>
> >> >>> I would guess that user perceived performance should be
a
> >> >>> little
> >> >>> faster with this. But this also depends on an
alternative
> >> >>> that
> >> >>> we
> >> >>> decided on :)
> >> >>>
> >> >>> Also the expiration thread pool is set to min priority
atm so
> >> >>> it
> >> >>> may delay removal of said objects but hopefully (if the
jvm
> >> >>> supports) it wouldn't overrun a CPU while processing
unless
> >> >>> it
> >> >>> has
> >> >>> availability.
> >> >>>
> >> >>>
> >> >>> Cheers
> >> >>> Dan
> >> >>>
> >> >>>
> >> >>> On Mon, Jul 13, 2015 at 9:25 PM, Tristan Tarrant
> >> >>> <ttarrant(a)redhat.com
<mailto:ttarrant@redhat.com>> wrote:
> >> >>> > After re-reading the whole original thread, I
agree
> >> >>> with
> >> >>> the
> >> >>> proposal
> >> >>> > with two caveats:
> >> >>> >
> >> >>> > - ensure that we don't break JCache
compatibility
> >> >>> > - ensure that we document this properly
> >> >>> >
> >> >>> > Tristan
> >> >>> >
> >> >>> > On 13/07/2015 18:41, Sanne Grinovero wrote:
> >> >>> >> +1
> >> >>> >> You had me convinced at the first line,
although "A
> >> >>> lot
> >> >>> of
> >> >>> code can now
> >> >>> >> be removed and made simpler" makes it
look extremely
> >> >>> nice.
> >> >>> >>
> >> >>> >> On 13 Jul 2015 18:14, "William
Burns"
> >> >>> <mudokonman(a)gmail.com
> >> >>> <mailto:mudokonman@gmail.com>
> >> >>> >> <mailto:mudokonman@gmail.com
> >>
> >> >>> <mailto:mudokonman@gmail.com>>> wrote:
> >> >>> >>
> >> >>> >> This is a necro of [1].
> >> >>> >>
> >> >>> >> With Infinispan 8.0 we are adding in
clustered
> >> >>> expiration. That
> >> >>> >> includes an expiration event raised that
is
> >> >>> clustered
> >> >>> as well.
> >> >>> >> Unfortunately expiration events
currently occur
> >> >>> multiple times (if
> >> >>> >> numOwners > 1) at different times
across nodes in
> >> >>> a
> >> >>> cluster. This
> >> >>> >> makes coordinating a single cluster
expiration
> >> >>> event
> >> >>> quite difficult.
> >> >>> >>
> >> >>> >> To work around this I am proposing that
the
> >> >>> expiration
> >> >>> of an event
> >> >>> >> is done solely by the owner of the given
key that
> >> >>> is
> >> >>> now expired.
> >> >>> >> This would fix the issue of having
multiple events
> >> >>> and
> >> >>> the event can
> >> >>> >> be raised while holding the lock for the
given key
> >> >>> so
> >> >>> concurrent
> >> >>> >> modifications would not be an issue.
> >> >>> >>
> >> >>> >> The problem arises when you have other
nodes that
> >> >>> have
> >> >>> expiration
> >> >>> >> set but expire at different times. Max
idle is
> >> >>> the
> >> >>> biggest offender
> >> >>> >> with this as a read on an owner only
refreshes the
> >> >>> owners timestamp,
> >> >>> >> meaning other owners would not be
updated and
> >> >>> expire
> >> >>> preemptively.
> >> >>> >> To have expiration work properly in this
case you
> >> >>> would
> >> >>> need
> >> >>> >> coordination between the owners to see
if anyone
> >> >>> has
> >> >>> a
> >> >>> higher
> >> >>> >> value. This requires blocking and would
have to
> >> >>> be
> >> >>> done while
> >> >>> >> accessing a key that is expired to be
sure if
> >> >>> expiration happened or
> >> >>> >> not.
> >> >>> >>
> >> >>> >> The linked dev listing proposed instead
to only
> >> >>> expire
> >> >>> an entry by
> >> >>> >> the reaper thread and not on access. In
this case
> >> >>> a
> >> >>> read will
> >> >>> >> return a non null value until it is
fully expired,
> >> >>> increasing hit
> >> >>> >> ratios possibly.
> >> >>> >>
> >> >>> >> Their are quire a bit of real benefits
for this:
> >> >>> >>
> >> >>> >> 1. Cluster cache reads would be much
simpler and
> >> >>> wouldn't have to
> >> >>> >> block to verify the object exists or not
since
> >> >>> this
> >> >>> would only be
> >> >>> >> done by the reaper thread (note this
would have
> >> >>> only
> >> >>> happened if the
> >> >>> >> entry was expired locally). An access
would just
> >> >>> return the value
> >> >>> >> immediately.
> >> >>> >> 2. Each node only expires entries it
owns in the
> >> >>> reaper
> >> >>> thread
> >> >>> >> reducing how many entries they must
check or
> >> >>> remove.
> >> >>> This also
> >> >>> >> provides a single point where events
would be
> >> >>> raised
> >> >>> as
> >> >>> we need.
> >> >>> >> 3. A lot of code can now be removed and
made
> >> >>> simpler
> >> >>> as
> >> >>> it no longer
> >> >>> >> has to check for expiration. The
expiration check
> >> >>> would only be
> >> >>> >> done in 1 place, the expiration reaper
thread.
> >> >>> >>
> >> >>> >> The main issue with this proposal is as
the other
> >> >>> listing mentions
> >> >>> >> is if user code expects the value to be
gone after
> >> >>> expiration for
> >> >>> >> correctness. I would say this use case
is not as
> >> >>> compelling for
> >> >>> >> maxIdle, especially since we never
supported it
> >> >>> properly. And in
> >> >>> >> the case of lifespan the user could very
easily
> >> >>> store
> >> >>> the expiration
> >> >>> >> time in the object that they can check
after a get
> >> >>> as
> >> >>> pointed out in
> >> >>> >> the other thread.
> >> >>> >>
> >> >>> >> [1]
> >> >>> >>
> >> >>>
> >> >>>
> >> >>>
http://infinispan-developer-list.980875.n3.nabble.com/infinispan-dev-stri...
> >> >>> >>
> >> >>> >>
_______________________________________________
> >> >>> >> infinispan-dev mailing list
> >> >>> >> infinispan-dev(a)lists.jboss.org
> >> >>> <mailto:infinispan-dev@lists.jboss.org>
> >> >>> <mailto:infinispan-dev@lists.jboss.org
> >> >>> <mailto:infinispan-dev@lists.jboss.org>>
> >> >>> >>
> >> >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >> >>> >>
> >> >>> >>
> >> >>> >>
> >> >>> >>
_______________________________________________
> >> >>> >> infinispan-dev mailing list
> >> >>> >> infinispan-dev(a)lists.jboss.org
> >> >>> <mailto:infinispan-dev@lists.jboss.org>
> >> >>> >>
> >> >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >> >>> >>
> >> >>> >
> >> >>> > --
> >> >>> > Tristan Tarrant
> >> >>> > Infinispan Lead
> >> >>> > JBoss, a division of Red Hat
> >> >>> > _______________________________________________
> >> >>> > infinispan-dev mailing list
> >> >>> > infinispan-dev(a)lists.jboss.org
> >> >>> <mailto:infinispan-dev@lists.jboss.org>
> >> >>> >
https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >> >>> _______________________________________________
> >> >>> infinispan-dev mailing list
> >> >>> infinispan-dev(a)lists.jboss.org
> >> >>> <mailto:infinispan-dev@lists.jboss.org>
> >> >>>
https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >> >>>
> >> >>>
> >> >>>
> >> >>> _______________________________________________
> >> >>> infinispan-dev mailing list
> >> >>> infinispan-dev(a)lists.jboss.org
> >> >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >> >>
> >> > _______________________________________________
> >> > infinispan-dev mailing list
> >> > infinispan-dev(a)lists.jboss.org
> >> > https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >>
> >>
> >> --
> >> Radim Vansa <rvansa(a)redhat.com>
> >> JBoss Performance Team
> >>
> >> _______________________________________________
> >> infinispan-dev mailing list
> >> infinispan-dev(a)lists.jboss.org
> >> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >
> >
> > _______________________________________________
> > infinispan-dev mailing list
> > infinispan-dev(a)lists.jboss.org
> > https://lists.jboss.org/mailman/listinfo/infinispan-dev
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
11:54 a.m.
New subject: Fwd: Strict Expiration
When you're into the stores & expiration: any plans for handling [1]?
Radim
[1] https://issues.jboss.org/browse/ISPN-3202
On 07/23/2015 02:37 PM, William Burns wrote:
I actually found another hiccup with cache stores. It seems currently
we only allow for a callback when an entry is expired from a cache
store when using the reaper thread [1]. However we don't allow for
such a callback on a read which finds an expired entry and wants to
remove it [2].
Interestingly our cache stores in general don't even expire entries on
load with the few exceptions below:
1. SingleCacheStore returns true for an expired entry on contains
2. SingleCacheStore removes expired entries on load
3. RemoteStore does not need to worry about expiration since it is
handled by another remote server.
Of all of the other stores I have looked at they return false properly
for expired entries and only purge elements from within reaper thread.
I propose we change SingleCacheStore to behave as the other cache
stores. This doesn't require any API changes. We would then rely on
store expiring elements only during reaper thread or if the element
expires in memory. We should also guarantee that when a cache store is
used that the reaper thread is enabled (throw exception if not enabled
and store is present at init). Should I worry about when only a
RemoteStore is used (this seems a bit fragile)?
To be honest we would need to revamp the CacheLoader/Writer API at a
later point to allow for values to be optionally provided for
expiration anyways, so I would say to do that in addition to allowing
loader/stores to expire on access.
[1]
https://github.com/infinispan/infinispan/blob/master/core/src/main/java/o...
[2]
https://github.com/infinispan/infinispan/blob/master/core/src/main/java/o...
---------- Forwarded message ---------
From: William Burns <mudokonman(a)gmail.com <mailto:mudokonman@gmail.com>>
Date: Wed, Jul 22, 2015 at 11:06 AM
Subject: Re: [infinispan-dev] Strict Expiration
To: infinispan -Dev List <infinispan-dev(a)lists.jboss.org
<mailto:infinispan-dev@lists.jboss.org>>
On Wed, Jul 22, 2015 at 10:53 AM Dan Berindei <dan.berindei(a)gmail.com
<mailto:dan.berindei@gmail.com>> wrote:
Is it possible/feasible to skip the notification from the backups to
the primary (and back) when there is no clustered expiration listener?
Unfortunately there is no way to distinguish whether or a listener is
create, modify, remove or expiration. So this would only work if
there are no clustered listeners.
This however should be feasible. This shouldn't be hard to add.
The only thing I would have to figure out is what happens in the case
of a rehash and the node that removed the value is now the primary
owner and some nodes have the old value and someone registers an
expiration listener. I am thinking I should only raise the event if
the primary owner still has the value.
Dan
On Tue, Jul 21, 2015 at 5:25 PM, William Burns
<mudokonman(a)gmail.com <mailto:mudokonman@gmail.com>> wrote:
> So I wanted to sum up what it looks like the plan is for this in
regards to
> cluster expiration for ISPN 8.
>
> First off to not make it ambiguous, maxIdle being used with a
clustered
> cache will provide undefined and unsupported behavior. This can
and will
> expire entries on a single node without notifying other cluster
members
> (essentially it will operate as it does today unchanged).
>
> This leaves me to talk solely about lifespan cluster expiration.
>
> Lifespan Expiration events are fired by the primary owner of an
expired key
>
> - when accessing an expired entry.
>
> - by the reaper thread.
>
> If the expiration is detected by a node other than the primary
owner, an
> expiration command is sent to it and null is returned
immediately not
> waiting for a response.
>
> Expiration event listeners follow the usual rules for
sync/async: in the
> case of a sync listener, the handler is invoked while holding
the lock,
> whereas an async listener will not hold locks.
>
> It is desirable for expiration events to contain both the key
and value.
> However currently cache stores do not provide the value when
they expire
> values. Thus we can only guarantee the value is present when an
in memory
> expiration event occurs. We could plan on adding this later.
>
> Also as you may have guessed this doesn't touch strict
expiration, which I
> think we have come to the conclusion should only work with
maxIdle and as
> such this is not explored with this iteration.
>
> Let me know if you guys think this approach is okay.
>
> Cheers,
>
> - Will
>
> On Tue, Jul 14, 2015 at 1:51 PM Radim Vansa <rvansa(a)redhat.com
<mailto:rvansa@redhat.com>> wrote:
>>
>> Yes, I know about [1]. I've worked that around by storing
timestamp in
>> the entry as well and when a new record is added, the 'expired'
>> invalidations are purged. But I can't purge that if I don't
access it -
>> Infinispan needs to handle that internally.
>>
>> Radim
>>
>> [1] https://hibernate.atlassian.net/browse/HHH-6219
>>
>> On 07/14/2015 05:45 PM, Dennis Reed wrote:
>> > On 07/14/2015 11:08 AM, Radim Vansa wrote:
>> >> On 07/14/2015 04:19 PM, William Burns wrote:
>> >>>
>> >>> On Tue, Jul 14, 2015 at 9:37 AM William Burns
<mudokonman(a)gmail.com <mailto:mudokonman@gmail.com>
>> >>> <mailto:mudokonman@gmail.com
<mailto:mudokonman@gmail.com>>> wrote:
>> >>>
>> >>> On Tue, Jul 14, 2015 at 4:41 AM Dan Berindei
>> >>> <dan.berindei(a)gmail.com
<mailto:dan.berindei@gmail.com> <mailto:dan.berindei@gmail.com
<mailto:dan.berindei@gmail.com>>> wrote:
>> >>>
>> >>> Processing expiration only on the reaper thread
sounds nice,
>> >>> but I
>> >>> have one reservation: processing 1 million
entries to see
>> >>> that
>> >>> 1 of
>> >>> them is expired is a lot of work, and in the
general case we
>> >>> will not
>> >>> be able to ensure an expiration precision of less
than 1
>> >>> minute (maybe
>> >>> more, with a huge SingleFileStore attached).
>> >>>
>> >>>
>> >>> This isn't much different then before. The only
difference is
>> >>> that if a user touched a value after it expired it
wouldn't show
>> >>> up (which is unlikely with maxIdle especially).
>> >>>
>> >>>
>> >>> What happens to users who need better precision? In
>> >>> particular, I know
>> >>> some JCache tests were failing because HotRod was
only
>> >>> supporting
>> >>> 1-second resolution instead of the 1-millisecond
resolution
>> >>> they were
>> >>> expecting.
>> >>>
>> >>>
>> >>> JCache is an interesting piece. The thing about
JCache is that
>> >>> the spec is only defined for local caches. However I
wouldn't
>> >>> want to muddy up the waters in regards to it behaving
>> >>> differently
>> >>> for local/remote. In the JCache scenario we could add an
>> >>> interceptor to prevent it returning such values (we
do something
>> >>> similar already for events). JCache behavior vs ISPN
behavior
>> >>> seems a bit easier to differentiate. But like you
are getting
>> >>> at,
>> >>> either way is not very appealing.
>> >>>
>> >>>
>> >>>
>> >>> I'm even less convinced about the need to
guarantee that a
>> >>> clustered
>> >>> expiration listener will only be triggered once,
and that
>> >>> the
>> >>> entry
>> >>> must be null everywhere after that listener was
invoked.
>> >>> What's the
>> >>> use case?
>> >>>
>> >>>
>> >>> Maybe Tristan would know more to answer. To be
honest this work
>> >>> seems fruitless unless we know what our end users
want here.
>> >>> Spending time on something for it to thrown out is
never fun :(
>> >>>
>> >>> And the more I thought about this the more I question
the
>> >>> validity
>> >>> of maxIdle even. It seems like a very poor way to
prevent
>> >>> memory
>> >>> exhaustion, which eviction does in a much better way
and has
>> >>> much
>> >>> more flexible algorithms. Does anyone know what
maxIdle would
>> >>> be
>> >>> used for that wouldn't be covered by eviction? The
only thing I
>> >>> can think of is cleaning up the cache store as well.
>> >>>
>> >>>
>> >>> Actually I guess for session/authentication related
information this
>> >>> would be important. However maxIdle isn't really as usable
in that
>> >>> case since most likely you would have a sticky session to
go back to
>> >>> that node which means you would never refresh the last used
date on
>> >>> the copies (current implementation). Without cluster
expiration you
>> >>> could lose that session information on a failover very easily.
>> >> I would say that maxIdle can be used as for memory
management as kind
>> >> of
>> >> WeakHashMap - e.g. in 2LC the maxIdle is used to store some
record for
>> >> a
>> >> short while (regular transaction lifespan ~ seconds to
minutes), and
>> >> regularly the record is removed. However, to make sure that
we don't
>> >> leak records in this cache (if something goes wrong and the
remove does
>> >> not occur), it is removed.
>> > Note that just relying on maxIdle doesn't guarantee you won't
leak
>> > records in this use case (specifically with the way the current
>> > hibernate-infinispan 2LC implementation uses it).
>> >
>> > Hibernate-infinispan adds entries to its own Map stored in
Infinispan,
>> > and expects maxIdle to remove the map if it skips a remove.
But in a
>> > current case, we found that due to frequent accesses to that
same map
>> > the entries never idle out and it ends up in OOME).
>> >
>> > -Dennis
>> >
>> >> I can guess how long the transaction takes place, but not
how many
>> >> parallel transactions there are. With eviction algorithms
(where I am
>> >> not sure about the exact guarantees) I can set the cache to
not hold
>> >> more than N entries, but I can't know for sure that my
record does not
>> >> suddenly get evicted after shorter period, possibly causing some
>> >> inconsistency.
>> >> So this is similar to WeakHashMap by removing the key "when
it can't be
>> >> used anymore" because I know that the transaction will
finish before
>> >> the
>> >> deadline. I don't care about the exact size, I don't want
to
tune that,
>> >> I just don't want to leak.
>> >>
>> >> From my POV the non-strict maxIdle and strict expiration
would be a
>> >> nice compromise.
>> >>
>> >> Radim
>> >>
>> >>> Note that this would make the reaper thread less
efficient:
>> >>> with
>> >>> numOwners=2 (best case), half of the entries that
the reaper
>> >>> touches
>> >>> cannot be expired, because the node isn't the
primary node.
>> >>> And to
>> >>> make matters worse, the same reaper thread would
have to
>> >>> perform a
>> >>> (synchronous?) RPC for each entry to ensure it
expires
>> >>> everywhere.
>> >>>
>> >>>
>> >>> I have debated about this, it could something like a
sync
>> >>> removeAll which has a special marker to tell it is due
to
>> >>> expiration (which would raise listeners there), while
also
>> >>> sending
>> >>> a cluster expiration event to other non owners.
>> >>>
>> >>>
>> >>> For maxIdle I'd like to know more information
about how
>> >>> exactly the
>> >>> owners would coordinate to expire an entry. I'm
pretty sure
>> >>> we
>> >>> cannot
>> >>> avoid ignoring some reads (expiring an entry
immediately
>> >>> after
>> >>> it was
>> >>> read), and ensuring that we don't accidentally
extend an
>> >>> entry's life
>> >>> (like the current code does, when we transfer an
entry to a
>> >>> new owner)
>> >>> also sounds problematic.
>> >>>
>> >>>
>> >>> For lifespan it is simple, the primary owner just
expires it
>> >>> when
>> >>> it expires there. There is no coordination needed in
this case
>> >>> it
>> >>> just sends the expired remove to owners etc.
>> >>>
>> >>> Max idle is more complicated as we all know. The
primary owner
>> >>> would send a request for the last used time for a
given key or
>> >>> set
>> >>> of keys. Then the owner would take those times and
check for a
>> >>> new access it isn't aware of. If there isn't
then it
would send
>> >>> a
>> >>> remove command for the key(s). If there is a new
access the
>> >>> owner
>> >>> would instead send the last used time to all of the
owners. The
>> >>> expiration obviously would have a window that if a
read occurred
>> >>> after sending a response that could be ignored. This
could be
>> >>> resolved by using some sort of 2PC and blocking reads
during
>> >>> that
>> >>> period but I would say it isn't worth it.
>> >>>
>> >>> The issue with transferring to a new node refreshing
the last
>> >>> update/lifespan seems like just a bug we need to fix
>> >>> irrespective
>> >>> of this issue IMO.
>> >>>
>> >>>
>> >>> I'm not saying expiring entries on each node
independently
>> >>> is
>> >>> perfect,
>> >>> far from it. But I wouldn't want us to provide
new
>> >>> guarantees that
>> >>> could hurt performance without a really good use
case.
>> >>>
>> >>>
>> >>> I would guess that user perceived performance should
be a little
>> >>> faster with this. But this also depends on an
alternative that
>> >>> we
>> >>> decided on :)
>> >>>
>> >>> Also the expiration thread pool is set to min
priority atm so it
>> >>> may delay removal of said objects but hopefully (if
the jvm
>> >>> supports) it wouldn't overrun a CPU while processing
unless it
>> >>> has
>> >>> availability.
>> >>>
>> >>>
>> >>> Cheers
>> >>> Dan
>> >>>
>> >>>
>> >>> On Mon, Jul 13, 2015 at 9:25 PM, Tristan Tarrant
>> >>> <ttarrant(a)redhat.com
<mailto:ttarrant@redhat.com>
<mailto:ttarrant@redhat.com <mailto:ttarrant@redhat.com>>> wrote:
>> >>> > After re-reading the whole original thread, I
agree with
>> >>> the
>> >>> proposal
>> >>> > with two caveats:
>> >>> >
>> >>> > - ensure that we don't break JCache
compatibility
>> >>> > - ensure that we document this properly
>> >>> >
>> >>> > Tristan
>> >>> >
>> >>> > On 13/07/2015 18:41, Sanne Grinovero wrote:
>> >>> >> +1
>> >>> >> You had me convinced at the first line,
although "A lot
>> >>> of
>> >>> code can now
>> >>> >> be removed and made simpler" makes it
look
extremely
>> >>> nice.
>> >>> >>
>> >>> >> On 13 Jul 2015 18:14, "William
Burns"
>> >>> <mudokonman(a)gmail.com <mailto:mudokonman@gmail.com>
>> >>> <mailto:mudokonman@gmail.com
<mailto:mudokonman@gmail.com>>
>> >>> >> <mailto:mudokonman@gmail.com
<mailto:mudokonman@gmail.com>
>>
>> >>> <mailto:mudokonman@gmail.com
<mailto:mudokonman@gmail.com>>>> wrote:
>> >>> >>
>> >>> >> This is a necro of [1].
>> >>> >>
>> >>> >> With Infinispan 8.0 we are adding in
clustered
>> >>> expiration. That
>> >>> >> includes an expiration event raised that
is
clustered
>> >>> as well.
>> >>> >> Unfortunately expiration events currently
occur
>> >>> multiple times (if
>> >>> >> numOwners > 1) at different times
across
nodes in a
>> >>> cluster. This
>> >>> >> makes coordinating a single cluster
expiration event
>> >>> quite difficult.
>> >>> >>
>> >>> >> To work around this I am proposing that
the
>> >>> expiration
>> >>> of an event
>> >>> >> is done solely by the owner of the given
key
that is
>> >>> now expired.
>> >>> >> This would fix the issue of having
multiple
events
>> >>> and
>> >>> the event can
>> >>> >> be raised while holding the lock for the
given key so
>> >>> concurrent
>> >>> >> modifications would not be an issue.
>> >>> >>
>> >>> >> The problem arises when you have other
nodes that
>> >>> have
>> >>> expiration
>> >>> >> set but expire at different times. Max
idle
is the
>> >>> biggest offender
>> >>> >> with this as a read on an owner only
refreshes the
>> >>> owners timestamp,
>> >>> >> meaning other owners would not be updated
and
expire
>> >>> preemptively.
>> >>> >> To have expiration work properly in this
case you
>> >>> would
>> >>> need
>> >>> >> coordination between the owners to see if
anyone has
>> >>> a
>> >>> higher
>> >>> >> value. This requires blocking and would
have
to be
>> >>> done while
>> >>> >> accessing a key that is expired to be sure
if
>> >>> expiration happened or
>> >>> >> not.
>> >>> >>
>> >>> >> The linked dev listing proposed instead to
only
>> >>> expire
>> >>> an entry by
>> >>> >> the reaper thread and not on access. In
this
case a
>> >>> read will
>> >>> >> return a non null value until it is fully
expired,
>> >>> increasing hit
>> >>> >> ratios possibly.
>> >>> >>
>> >>> >> Their are quire a bit of real benefits for
this:
>> >>> >>
>> >>> >> 1. Cluster cache reads would be much
simpler and
>> >>> wouldn't have to
>> >>> >> block to verify the object exists or not
since this
>> >>> would only be
>> >>> >> done by the reaper thread (note this would
have only
>> >>> happened if the
>> >>> >> entry was expired locally). An access
would just
>> >>> return the value
>> >>> >> immediately.
>> >>> >> 2. Each node only expires entries it owns
in the
>> >>> reaper
>> >>> thread
>> >>> >> reducing how many entries they must check
or
remove.
>> >>> This also
>> >>> >> provides a single point where events would
be
raised
>> >>> as
>> >>> we need.
>> >>> >> 3. A lot of code can now be removed and
made
simpler
>> >>> as
>> >>> it no longer
>> >>> >> has to check for expiration. The
expiration
check
>> >>> would only be
>> >>> >> done in 1 place, the expiration reaper
thread.
>> >>> >>
>> >>> >> The main issue with this proposal is as the
other
>> >>> listing mentions
>> >>> >> is if user code expects the value to be
gone
after
>> >>> expiration for
>> >>> >> correctness. I would say this use case is
not as
>> >>> compelling for
>> >>> >> maxIdle, especially since we never
supported it
>> >>> properly. And in
>> >>> >> the case of lifespan the user could very
easily store
>> >>> the expiration
>> >>> >> time in the object that they can check
after
a get as
>> >>> pointed out in
>> >>> >> the other thread.
>> >>> >>
>> >>> >> [1]
>> >>> >>
>> >>>
>> >>>
http://infinispan-developer-list.980875.n3.nabble.com/infinispan-dev-stri...
>> >>> >>
>> >>> >>
_______________________________________________
>> >>> >> infinispan-dev mailing list
>> >>> >> infinispan-dev(a)lists.jboss.org
<mailto:infinispan-dev@lists.jboss.org>
>> >>> <mailto:infinispan-dev@lists.jboss.org
<mailto:infinispan-dev@lists.jboss.org>>
>> >>> <mailto:infinispan-dev@lists.jboss.org
<mailto:infinispan-dev@lists.jboss.org>
>> >>> <mailto:infinispan-dev@lists.jboss.org
<mailto:infinispan-dev@lists.jboss.org>>>
>> >>> >>
https://lists.jboss.org/mailman/listinfo/infinispan-dev
>> >>> >>
>> >>> >>
>> >>> >>
>> >>> >>
_______________________________________________
>> >>> >> infinispan-dev mailing list
>> >>> >> infinispan-dev(a)lists.jboss.org
<mailto:infinispan-dev@lists.jboss.org>
>> >>> <mailto:infinispan-dev@lists.jboss.org
<mailto:infinispan-dev@lists.jboss.org>>
>> >>> >>
https://lists.jboss.org/mailman/listinfo/infinispan-dev
>> >>> >>
>> >>> >
>> >>> > --
>> >>> > Tristan Tarrant
>> >>> > Infinispan Lead
>> >>> > JBoss, a division of Red Hat
>> >>> > _______________________________________________
>> >>> > infinispan-dev mailing list
>> >>> > infinispan-dev(a)lists.jboss.org
<mailto:infinispan-dev@lists.jboss.org>
>> >>> <mailto:infinispan-dev@lists.jboss.org
<mailto:infinispan-dev@lists.jboss.org>>
>> >>> >
https://lists.jboss.org/mailman/listinfo/infinispan-dev
>> >>> _______________________________________________
>> >>> infinispan-dev mailing list
>> >>> infinispan-dev(a)lists.jboss.org
<mailto:infinispan-dev@lists.jboss.org>
>> >>> <mailto:infinispan-dev@lists.jboss.org
<mailto:infinispan-dev@lists.jboss.org>>
>> >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>> >>>
>> >>>
>> >>>
>> >>> _______________________________________________
>> >>> infinispan-dev mailing list
>> >>> infinispan-dev(a)lists.jboss.org
<mailto:infinispan-dev@lists.jboss.org>
>> >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>> >>
>> > _______________________________________________
>> > infinispan-dev mailing list
>> > infinispan-dev(a)lists.jboss.org
<mailto:infinispan-dev@lists.jboss.org>
>> > https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>
>>
>> --
>> Radim Vansa <rvansa(a)redhat.com <mailto:rvansa@redhat.com>>
>> JBoss Performance Team
>>
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev(a)lists.jboss.org
<mailto:infinispan-dev@lists.jboss.org>
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev(a)lists.jboss.org
<mailto:infinispan-dev@lists.jboss.org>
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org <mailto:infinispan-dev@lists.jboss.org>
https://lists.jboss.org/mailman/listinfo/infinispan-dev
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
--
Radim Vansa <rvansa(a)redhat.com>
JBoss Performance Team
12:03 p.m.
New subject: Fwd: Strict Expiration
On Thu, Jul 23, 2015 at 12:54 PM Radim Vansa <rvansa(a)redhat.com> wrote:
When you're into the stores & expiration: any plans for
handling [1]?
Radim
[1] https://issues.jboss.org/browse/ISPN-3202
I am not planning on it. This is yet another thing I wasn't aware of that
makes me dislike maxIdle.
On 07/23/2015 02:37 PM, William Burns wrote:
> I actually found another hiccup with cache stores. It seems currently
> we only allow for a callback when an entry is expired from a cache
> store when using the reaper thread [1]. However we don't allow for
> such a callback on a read which finds an expired entry and wants to
> remove it [2].
>
> Interestingly our cache stores in general don't even expire entries on
> load with the few exceptions below:
>
> 1. SingleCacheStore returns true for an expired entry on contains
> 2. SingleCacheStore removes expired entries on load
> 3. RemoteStore does not need to worry about expiration since it is
> handled by another remote server.
>
> Of all of the other stores I have looked at they return false properly
> for expired entries and only purge elements from within reaper thread.
>
> I propose we change SingleCacheStore to behave as the other cache
> stores. This doesn't require any API changes. We would then rely on
> store expiring elements only during reaper thread or if the element
> expires in memory. We should also guarantee that when a cache store is
> used that the reaper thread is enabled (throw exception if not enabled
> and store is present at init). Should I worry about when only a
> RemoteStore is used (this seems a bit fragile)?
>
> To be honest we would need to revamp the CacheLoader/Writer API at a
> later point to allow for values to be optionally provided for
> expiration anyways, so I would say to do that in addition to allowing
> loader/stores to expire on access.
>
> [1]
>
https://github.com/infinispan/infinispan/blob/master/core/src/main/java/o...
>
> [2]
>
https://github.com/infinispan/infinispan/blob/master/core/src/main/java/o...
>
> ---------- Forwarded message ---------
> From: William Burns <mudokonman(a)gmail.com
<mailto:mudokonman@gmail.com>>
> Date: Wed, Jul 22, 2015 at 11:06 AM
> Subject: Re: [infinispan-dev] Strict Expiration
> To: infinispan -Dev List <infinispan-dev(a)lists.jboss.org
> <mailto:infinispan-dev@lists.jboss.org>>
>
>
> On Wed, Jul 22, 2015 at 10:53 AM Dan Berindei <dan.berindei(a)gmail.com
> <mailto:dan.berindei@gmail.com>> wrote:
>
> Is it possible/feasible to skip the notification from the backups to
> the primary (and back) when there is no clustered expiration
listener?
>
>
> Unfortunately there is no way to distinguish whether or a listener is
> create, modify, remove or expiration. So this would only work if
> there are no clustered listeners.
>
> This however should be feasible. This shouldn't be hard to add.
>
> The only thing I would have to figure out is what happens in the case
> of a rehash and the node that removed the value is now the primary
> owner and some nodes have the old value and someone registers an
> expiration listener. I am thinking I should only raise the event if
> the primary owner still has the value.
>
>
> Dan
>
>
> On Tue, Jul 21, 2015 at 5:25 PM, William Burns
> <mudokonman(a)gmail.com <mailto:mudokonman@gmail.com>> wrote:
> > So I wanted to sum up what it looks like the plan is for this in
> regards to
> > cluster expiration for ISPN 8.
> >
> > First off to not make it ambiguous, maxIdle being used with a
> clustered
> > cache will provide undefined and unsupported behavior. This can
> and will
> > expire entries on a single node without notifying other cluster
> members
> > (essentially it will operate as it does today unchanged).
> >
> > This leaves me to talk solely about lifespan cluster expiration.
> >
> > Lifespan Expiration events are fired by the primary owner of an
> expired key
> >
> > - when accessing an expired entry.
> >
> > - by the reaper thread.
> >
> > If the expiration is detected by a node other than the primary
> owner, an
> > expiration command is sent to it and null is returned
> immediately not
> > waiting for a response.
> >
> > Expiration event listeners follow the usual rules for
> sync/async: in the
> > case of a sync listener, the handler is invoked while holding
> the lock,
> > whereas an async listener will not hold locks.
> >
> > It is desirable for expiration events to contain both the key
> and value.
> > However currently cache stores do not provide the value when
> they expire
> > values. Thus we can only guarantee the value is present when an
> in memory
> > expiration event occurs. We could plan on adding this later.
> >
> > Also as you may have guessed this doesn't touch strict
> expiration, which I
> > think we have come to the conclusion should only work with
> maxIdle and as
> > such this is not explored with this iteration.
> >
> > Let me know if you guys think this approach is okay.
> >
> > Cheers,
> >
> > - Will
> >
> > On Tue, Jul 14, 2015 at 1:51 PM Radim Vansa <rvansa(a)redhat.com
> <mailto:rvansa@redhat.com>> wrote:
> >>
> >> Yes, I know about [1]. I've worked that around by storing
> timestamp in
> >> the entry as well and when a new record is added, the
'expired'
> >> invalidations are purged. But I can't purge that if I don't
> access it -
> >> Infinispan needs to handle that internally.
> >>
> >> Radim
> >>
> >> [1] https://hibernate.atlassian.net/browse/HHH-6219
> >>
> >> On 07/14/2015 05:45 PM, Dennis Reed wrote:
> >> > On 07/14/2015 11:08 AM, Radim Vansa wrote:
> >> >> On 07/14/2015 04:19 PM, William Burns wrote:
> >> >>>
> >> >>> On Tue, Jul 14, 2015 at 9:37 AM William Burns
> <mudokonman(a)gmail.com <mailto:mudokonman@gmail.com>
> >> >>> <mailto:mudokonman@gmail.com
> <mailto:mudokonman@gmail.com>>> wrote:
> >> >>>
> >> >>> On Tue, Jul 14, 2015 at 4:41 AM Dan Berindei
> >> >>> <dan.berindei(a)gmail.com
> <mailto:dan.berindei@gmail.com> <mailto:dan.berindei@gmail.com
> <mailto:dan.berindei@gmail.com>>> wrote:
> >> >>>
> >> >>> Processing expiration only on the reaper thread
> sounds nice,
> >> >>> but I
> >> >>> have one reservation: processing 1 million
> entries to see
> >> >>> that
> >> >>> 1 of
> >> >>> them is expired is a lot of work, and in the
> general case we
> >> >>> will not
> >> >>> be able to ensure an expiration precision of
less
> than 1
> >> >>> minute (maybe
> >> >>> more, with a huge SingleFileStore attached).
> >> >>>
> >> >>>
> >> >>> This isn't much different then before. The
only
> difference is
> >> >>> that if a user touched a value after it expired it
> wouldn't show
> >> >>> up (which is unlikely with maxIdle especially).
> >> >>>
> >> >>>
> >> >>> What happens to users who need better precision?
In
> >> >>> particular, I know
> >> >>> some JCache tests were failing because HotRod
was
> only
> >> >>> supporting
> >> >>> 1-second resolution instead of the
1-millisecond
> resolution
> >> >>> they were
> >> >>> expecting.
> >> >>>
> >> >>>
> >> >>> JCache is an interesting piece. The thing about
> JCache is that
> >> >>> the spec is only defined for local caches. However
I
> wouldn't
> >> >>> want to muddy up the waters in regards to it
behaving
> >> >>> differently
> >> >>> for local/remote. In the JCache scenario we could
add an
> >> >>> interceptor to prevent it returning such values (we
> do something
> >> >>> similar already for events). JCache behavior vs
ISPN
> behavior
> >> >>> seems a bit easier to differentiate. But like you
> are getting
> >> >>> at,
> >> >>> either way is not very appealing.
> >> >>>
> >> >>>
> >> >>>
> >> >>> I'm even less convinced about the need to
> guarantee that a
> >> >>> clustered
> >> >>> expiration listener will only be triggered
once,
> and that
> >> >>> the
> >> >>> entry
> >> >>> must be null everywhere after that listener was
> invoked.
> >> >>> What's the
> >> >>> use case?
> >> >>>
> >> >>>
> >> >>> Maybe Tristan would know more to answer. To be
> honest this work
> >> >>> seems fruitless unless we know what our end users
> want here.
> >> >>> Spending time on something for it to thrown out is
> never fun :(
> >> >>>
> >> >>> And the more I thought about this the more I
question
the
> >> >>> validity
> >> >>> of maxIdle even. It seems like a very poor way to
prevent
> >> >>> memory
> >> >>> exhaustion, which eviction does in a much better
way
> and has
> >> >>> much
> >> >>> more flexible algorithms. Does anyone know what
> maxIdle would
> >> >>> be
> >> >>> used for that wouldn't be covered by eviction?
The
> only thing I
> >> >>> can think of is cleaning up the cache store as
well.
> >> >>>
> >> >>>
> >> >>> Actually I guess for session/authentication related
> information this
> >> >>> would be important. However maxIdle isn't really as
usable
> in that
> >> >>> case since most likely you would have a sticky session to
> go back to
> >> >>> that node which means you would never refresh the last
used
> date on
> >> >>> the copies (current implementation). Without cluster
> expiration you
> >> >>> could lose that session information on a failover very
easily.
> >> >> I would say that maxIdle can be used as for memory
> management as kind
> >> >> of
> >> >> WeakHashMap - e.g. in 2LC the maxIdle is used to store some
> record for
> >> >> a
> >> >> short while (regular transaction lifespan ~ seconds to
> minutes), and
> >> >> regularly the record is removed. However, to make sure that
> we don't
> >> >> leak records in this cache (if something goes wrong and the
> remove does
> >> >> not occur), it is removed.
> >> > Note that just relying on maxIdle doesn't guarantee you
won't
> leak
> >> > records in this use case (specifically with the way the current
> >> > hibernate-infinispan 2LC implementation uses it).
> >> >
> >> > Hibernate-infinispan adds entries to its own Map stored in
> Infinispan,
> >> > and expects maxIdle to remove the map if it skips a remove.
> But in a
> >> > current case, we found that due to frequent accesses to that
> same map
> >> > the entries never idle out and it ends up in OOME).
> >> >
> >> > -Dennis
> >> >
> >> >> I can guess how long the transaction takes place, but not
> how many
> >> >> parallel transactions there are. With eviction algorithms
> (where I am
> >> >> not sure about the exact guarantees) I can set the cache to
> not hold
> >> >> more than N entries, but I can't know for sure that my
> record does not
> >> >> suddenly get evicted after shorter period, possibly causing
some
> >> >> inconsistency.
> >> >> So this is similar to WeakHashMap by removing the key
"when
> it can't be
> >> >> used anymore" because I know that the transaction will
> finish before
> >> >> the
> >> >> deadline. I don't care about the exact size, I don't
want to
> tune that,
> >> >> I just don't want to leak.
> >> >>
> >> >> From my POV the non-strict maxIdle and strict expiration
> would be a
> >> >> nice compromise.
> >> >>
> >> >> Radim
> >> >>
> >> >>> Note that this would make the reaper thread
less
> efficient:
> >> >>> with
> >> >>> numOwners=2 (best case), half of the entries
that
> the reaper
> >> >>> touches
> >> >>> cannot be expired, because the node isn't
the
> primary node.
> >> >>> And to
> >> >>> make matters worse, the same reaper thread
would
> have to
> >> >>> perform a
> >> >>> (synchronous?) RPC for each entry to ensure it
> expires
> >> >>> everywhere.
> >> >>>
> >> >>>
> >> >>> I have debated about this, it could something like
a
sync
> >> >>> removeAll which has a special marker to tell it is
due
to
> >> >>> expiration (which would raise listeners there),
while
> also
> >> >>> sending
> >> >>> a cluster expiration event to other non owners.
> >> >>>
> >> >>>
> >> >>> For maxIdle I'd like to know more
information
> about how
> >> >>> exactly the
> >> >>> owners would coordinate to expire an entry.
I'm
> pretty sure
> >> >>> we
> >> >>> cannot
> >> >>> avoid ignoring some reads (expiring an entry
> immediately
> >> >>> after
> >> >>> it was
> >> >>> read), and ensuring that we don't
accidentally
> extend an
> >> >>> entry's life
> >> >>> (like the current code does, when we transfer
an
> entry to a
> >> >>> new owner)
> >> >>> also sounds problematic.
> >> >>>
> >> >>>
> >> >>> For lifespan it is simple, the primary owner just
> expires it
> >> >>> when
> >> >>> it expires there. There is no coordination needed
in
> this case
> >> >>> it
> >> >>> just sends the expired remove to owners etc.
> >> >>>
> >> >>> Max idle is more complicated as we all know. The
> primary owner
> >> >>> would send a request for the last used time for a
> given key or
> >> >>> set
> >> >>> of keys. Then the owner would take those times and
> check for a
> >> >>> new access it isn't aware of. If there
isn't then it
> would send
> >> >>> a
> >> >>> remove command for the key(s). If there is a new
> access the
> >> >>> owner
> >> >>> would instead send the last used time to all of the
> owners. The
> >> >>> expiration obviously would have a window that if a
> read occurred
> >> >>> after sending a response that could be ignored.
This
> could be
> >> >>> resolved by using some sort of 2PC and blocking
reads
> during
> >> >>> that
> >> >>> period but I would say it isn't worth it.
> >> >>>
> >> >>> The issue with transferring to a new node
refreshing
> the last
> >> >>> update/lifespan seems like just a bug we need to
fix
> >> >>> irrespective
> >> >>> of this issue IMO.
> >> >>>
> >> >>>
> >> >>> I'm not saying expiring entries on each
node
> independently
> >> >>> is
> >> >>> perfect,
> >> >>> far from it. But I wouldn't want us to
provide new
> >> >>> guarantees that
> >> >>> could hurt performance without a really good
use
> case.
> >> >>>
> >> >>>
> >> >>> I would guess that user perceived performance
should
> be a little
> >> >>> faster with this. But this also depends on an
> alternative that
> >> >>> we
> >> >>> decided on :)
> >> >>>
> >> >>> Also the expiration thread pool is set to min
> priority atm so it
> >> >>> may delay removal of said objects but hopefully (if
> the jvm
> >> >>> supports) it wouldn't overrun a CPU while
processing
> unless it
> >> >>> has
> >> >>> availability.
> >> >>>
> >> >>>
> >> >>> Cheers
> >> >>> Dan
> >> >>>
> >> >>>
> >> >>> On Mon, Jul 13, 2015 at 9:25 PM, Tristan
Tarrant
> >> >>> <ttarrant(a)redhat.com
<mailto:ttarrant@redhat.com>
> <mailto:ttarrant@redhat.com <mailto:ttarrant@redhat.com>>>
wrote:
> >> >>> > After re-reading the whole original thread,
I
> agree with
> >> >>> the
> >> >>> proposal
> >> >>> > with two caveats:
> >> >>> >
> >> >>> > - ensure that we don't break JCache
compatibility
> >> >>> > - ensure that we document this properly
> >> >>> >
> >> >>> > Tristan
> >> >>> >
> >> >>> > On 13/07/2015 18:41, Sanne Grinovero
wrote:
> >> >>> >> +1
> >> >>> >> You had me convinced at the first
line,
> although "A lot
> >> >>> of
> >> >>> code can now
> >> >>> >> be removed and made simpler" makes
it look
> extremely
> >> >>> nice.
> >> >>> >>
> >> >>> >> On 13 Jul 2015 18:14, "William
Burns"
> >> >>> <mudokonman(a)gmail.com
<mailto:mudokonman@gmail.com>
> >> >>> <mailto:mudokonman@gmail.com
> <mailto:mudokonman@gmail.com>>
> >> >>> >> <mailto:mudokonman@gmail.com
> <mailto:mudokonman@gmail.com>
> >>
> >> >>> <mailto:mudokonman@gmail.com
> <mailto:mudokonman@gmail.com>>>> wrote:
> >> >>> >>
> >> >>> >> This is a necro of [1].
> >> >>> >>
> >> >>> >> With Infinispan 8.0 we are adding in
clustered
> >> >>> expiration. That
> >> >>> >> includes an expiration event raised
that is
> clustered
> >> >>> as well.
> >> >>> >> Unfortunately expiration events
currently occur
> >> >>> multiple times (if
> >> >>> >> numOwners > 1) at different times
across
> nodes in a
> >> >>> cluster. This
> >> >>> >> makes coordinating a single cluster
> expiration event
> >> >>> quite difficult.
> >> >>> >>
> >> >>> >> To work around this I am proposing
that the
> >> >>> expiration
> >> >>> of an event
> >> >>> >> is done solely by the owner of the
given key
> that is
> >> >>> now expired.
> >> >>> >> This would fix the issue of having
multiple
> events
> >> >>> and
> >> >>> the event can
> >> >>> >> be raised while holding the lock for
the
> given key so
> >> >>> concurrent
> >> >>> >> modifications would not be an issue.
> >> >>> >>
> >> >>> >> The problem arises when you have other
nodes
that
> >> >>> have
> >> >>> expiration
> >> >>> >> set but expire at different times.
Max idle
> is the
> >> >>> biggest offender
> >> >>> >> with this as a read on an owner only
> refreshes the
> >> >>> owners timestamp,
> >> >>> >> meaning other owners would not be
updated and
> expire
> >> >>> preemptively.
> >> >>> >> To have expiration work properly in
this case
you
> >> >>> would
> >> >>> need
> >> >>> >> coordination between the owners to see
if
> anyone has
> >> >>> a
> >> >>> higher
> >> >>> >> value. This requires blocking and
would have
> to be
> >> >>> done while
> >> >>> >> accessing a key that is expired to be
sure if
> >> >>> expiration happened or
> >> >>> >> not.
> >> >>> >>
> >> >>> >> The linked dev listing proposed
instead to only
> >> >>> expire
> >> >>> an entry by
> >> >>> >> the reaper thread and not on access.
In this
> case a
> >> >>> read will
> >> >>> >> return a non null value until it is
fully
> expired,
> >> >>> increasing hit
> >> >>> >> ratios possibly.
> >> >>> >>
> >> >>> >> Their are quire a bit of real benefits
for this:
> >> >>> >>
> >> >>> >> 1. Cluster cache reads would be much
simpler and
> >> >>> wouldn't have to
> >> >>> >> block to verify the object exists or
not
> since this
> >> >>> would only be
> >> >>> >> done by the reaper thread (note this
would
> have only
> >> >>> happened if the
> >> >>> >> entry was expired locally). An access
would
just
> >> >>> return the value
> >> >>> >> immediately.
> >> >>> >> 2. Each node only expires entries it
owns in the
> >> >>> reaper
> >> >>> thread
> >> >>> >> reducing how many entries they must
check or
> remove.
> >> >>> This also
> >> >>> >> provides a single point where events
would be
> raised
> >> >>> as
> >> >>> we need.
> >> >>> >> 3. A lot of code can now be removed
and made
> simpler
> >> >>> as
> >> >>> it no longer
> >> >>> >> has to check for expiration. The
expiration
> check
> >> >>> would only be
> >> >>> >> done in 1 place, the expiration reaper
thread.
> >> >>> >>
> >> >>> >> The main issue with this proposal is
as the
other
> >> >>> listing mentions
> >> >>> >> is if user code expects the value to
be gone
> after
> >> >>> expiration for
> >> >>> >> correctness. I would say this use
case is not
as
> >> >>> compelling for
> >> >>> >> maxIdle, especially since we never
supported it
> >> >>> properly. And in
> >> >>> >> the case of lifespan the user could
very
> easily store
> >> >>> the expiration
> >> >>> >> time in the object that they can check
after
> a get as
> >> >>> pointed out in
> >> >>> >> the other thread.
> >> >>> >>
> >> >>> >> [1]
> >> >>> >>
> >> >>>
> >> >>>
>
http://infinispan-developer-list.980875.n3.nabble.com/infinispan-dev-stri...
> >> >>> >>
> >> >>> >>
_______________________________________________
> >> >>> >> infinispan-dev mailing list
> >> >>> >> infinispan-dev(a)lists.jboss.org
> <mailto:infinispan-dev@lists.jboss.org>
> >> >>> <mailto:infinispan-dev@lists.jboss.org
> <mailto:infinispan-dev@lists.jboss.org>>
> >> >>> <mailto:infinispan-dev@lists.jboss.org
> <mailto:infinispan-dev@lists.jboss.org>
> >> >>> <mailto:infinispan-dev@lists.jboss.org
> <mailto:infinispan-dev@lists.jboss.org>>>
> >> >>> >>
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >> >>> >>
> >> >>> >>
> >> >>> >>
> >> >>> >>
_______________________________________________
> >> >>> >> infinispan-dev mailing list
> >> >>> >> infinispan-dev(a)lists.jboss.org
> <mailto:infinispan-dev@lists.jboss.org>
> >> >>> <mailto:infinispan-dev@lists.jboss.org
> <mailto:infinispan-dev@lists.jboss.org>>
> >> >>> >>
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >> >>> >>
> >> >>> >
> >> >>> > --
> >> >>> > Tristan Tarrant
> >> >>> > Infinispan Lead
> >> >>> > JBoss, a division of Red Hat
> >> >>> >
_______________________________________________
> >> >>> > infinispan-dev mailing list
> >> >>> > infinispan-dev(a)lists.jboss.org
> <mailto:infinispan-dev@lists.jboss.org>
> >> >>> <mailto:infinispan-dev@lists.jboss.org
> <mailto:infinispan-dev@lists.jboss.org>>
> >> >>> >
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >> >>> _______________________________________________
> >> >>> infinispan-dev mailing list
> >> >>> infinispan-dev(a)lists.jboss.org
> <mailto:infinispan-dev@lists.jboss.org>
> >> >>> <mailto:infinispan-dev@lists.jboss.org
> <mailto:infinispan-dev@lists.jboss.org>>
> >> >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >> >>>
> >> >>>
> >> >>>
> >> >>> _______________________________________________
> >> >>> infinispan-dev mailing list
> >> >>> infinispan-dev(a)lists.jboss.org
> <mailto:infinispan-dev@lists.jboss.org>
> >> >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >> >>
> >> > _______________________________________________
> >> > infinispan-dev mailing list
> >> > infinispan-dev(a)lists.jboss.org
> <mailto:infinispan-dev@lists.jboss.org>
> >> > https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >>
> >>
> >> --
> >> Radim Vansa <rvansa(a)redhat.com <mailto:rvansa@redhat.com>>
> >> JBoss Performance Team
> >>
> >> _______________________________________________
> >> infinispan-dev mailing list
> >> infinispan-dev(a)lists.jboss.org
> <mailto:infinispan-dev@lists.jboss.org>
> >> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >
> >
> > _______________________________________________
> > infinispan-dev mailing list
> > infinispan-dev(a)lists.jboss.org
> <mailto:infinispan-dev@lists.jboss.org>
> > https://lists.jboss.org/mailman/listinfo/infinispan-dev
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev(a)lists.jboss.org <mailto:
infinispan-dev(a)lists.jboss.org>
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
>
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
--
Radim Vansa <rvansa(a)redhat.com>
JBoss Performance Team
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
10:58 a.m.
On 14/07/2015 14:37, William Burns wrote:
I'm even less convinced about the need to guarantee that a
clustered
expiration listener will only be triggered once, and that the entry
must be null everywhere after that listener was invoked. What's the
use case?
Maybe Tristan would know more to answer. To be honest this work seems
fruitless unless we know what our end users want here. Spending time on
something for it to thrown out is never fun :(
I see two use-cases related to taking action on expiration:
1. do something with the data before it definitely is removed. This
could be something like archiving to secondary storage. This particular
case would also require still having the value to pass to the event
handler. I don't think we need special guarantees about "only once" but
it would be a nice-to-have. Also null-after-the-fact wouldn't be a
requirement.
2. pre-emptive memoization, i.e. a way to implement refreshOnExpiration
akin to computeIfAbsent. "Only once" would definitely be preferable here
and. "Null-after-the-fact" is also irrelevant since the code would
probably end up replacing the existing value.
Tristan
--
Tristan Tarrant
Infinispan Lead
JBoss, a division of Red Hat
3519
days inactive
3529
days old
infinispan-dev@lists.jboss.org
17 comments
6 participants
participants (6)
-
Dan Berindei -
Dennis Reed -
Radim Vansa -
Sanne Grinovero -
Tristan Tarrant -
William Burns