[JBoss JIRA] (ISPN-10666) Eviction get stuck with <off-heap> configuration
by Will Burns (Jira)
[ https://issues.jboss.org/browse/ISPN-10666?page=com.atlassian.jira.plugin... ]
Will Burns commented on ISPN-10666:
-----------------------------------
I can reproduce this. It looks like somehow the head pointer is pointing to an address that existing prior, but is no longer there. Unfortunately by the time it is in this state it is hard to figure out how it got there. Trace will hopefully show though :)
> Eviction get stuck with <off-heap> configuration
> ------------------------------------------------
>
> Key: ISPN-10666
> URL: https://issues.jboss.org/browse/ISPN-10666
> Project: Infinispan
> Issue Type: Bug
> Reporter: Wolf-Dieter Fink
> Assignee: Will Burns
> Priority: Major
>
> If the cache is configured as followed adding entries every second will end up in an unresponsive server.
> The client will stop after 120sec (eviction count reached) with
> Exception in thread "main" org.infinispan.client.hotrod.exceptions.TransportException:: java.net.SocketTimeoutException: PutOperation{ExpirationCache, key=[B0x033E17323031392D..[26], value=[B0x033F280058585858..[10244], flags=4} timed out after 60000 ms
> at org.infinispan.client.hotrod.impl.Util.rewrap(Util.java:54)
> at org.infinispan.client.hotrod.impl.Util.await(Util.java:27)
> at org.infinispan.client.hotrod.impl.RemoteCacheImpl.put(RemoteCacheImpl.java:335)
> at org.infinispan.client.hotrod.impl.RemoteCacheSupport.put(RemoteCacheSupport.java:129)
> at org.infinispan.wfink.hotrod.client.HotRodConstantLoadExpirationClient.putWithExpiration(HotRodConstantLoadExpirationClient.java:48)
> at org.infinispan.wfink.hotrod.client.HotRodConstantLoadExpirationClient.testLoop(HotRodConstantLoadExpirationClient.java:84)
> at org.infinispan.wfink.hotrod.client.HotRodConstantLoadExpirationClient.main(HotRodConstantLoadExpirationClient.java:160)
> Caused by: java.net.SocketTimeoutException: PutOperation{ExpirationCache, key=[B0x033E17323031392D..[26], value=[B0x033F280058585858..[10244], flags=4} timed out after 60000 ms
> at org.infinispan.client.hotrod.impl.operations.HotRodOperation.run(HotRodOperation.java:172)
> at io.netty.util.concurrent.PromiseTask$RunnableAdapter.call(PromiseTask.java:38)
> at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:127)
> at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
> at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404)
> at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:322)
> at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
5 years, 5 months
[JBoss JIRA] (ISPN-10645) Configured eviction bases on COUNT will unexpected start eviction to early and might evict until the cache is empty
by Will Burns (Jira)
[ https://issues.jboss.org/browse/ISPN-10645?page=com.atlassian.jira.plugin... ]
Will Burns edited comment on ISPN-10645 at 9/25/19 12:12 PM:
-------------------------------------------------------------
This is expected behavior.
The 59 is due to entries expiring after 60 seconds, which means with an insertion rate of 1 per second will only have 59-60 entries in the cache at a time.
The cache dropping to a size of 2 is because of the access pattern and the fact that the expiration reaper is disabled. Caffeine uses the TinyLFU algorithm for its eviction. Effectively what happened is that it is Caffeine uses an eden space of 2 that store the most recently added entries. Once this fills up it will add that entry into the main space. Due to the access pattern it has determined the new entry promoted to the main space should be immediately evicted. This causes the cache to have 2 "new" entries that aren't expired and 118 that are expired and aren't being evicted. If you enable the reaper it will clean these out sooner and never get to that point.
was (Author: william.burns):
This is expected behavior.
The 59 is due to entries expiring after 60 seconds, which means with an insertion rate of 1 per second will only have 59-60 entries in the cache at a time.
The cache dropping to a size of 2 is because of the access pattern and the fact that the expiration reaper is disabled. Caffeine uses the TinyLFU algorithm for its eviction. Effectively what happened is that it is Caffeine uses an eden space of 2 that store the most recently added entries. Once this fills up it will add that entry into the main space. Due to the access pattern it has determined the new entry should be immediately evicted. This causes the cache to have 2 "new" entries that aren't expired an 118 that are expired and aren't being evicted. If you enable the reaper it will clean these out sooner and never get to that point.
> Configured eviction bases on COUNT will unexpected start eviction to early and might evict until the cache is empty
> -------------------------------------------------------------------------------------------------------------------
>
> Key: ISPN-10645
> URL: https://issues.jboss.org/browse/ISPN-10645
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Affects Versions: 10.0.0.CR2
> Reporter: Wolf-Dieter Fink
> Assignee: Will Burns
> Priority: Major
>
> If a cache is populated and configured with eviction and maybe expiration the eviction will start earlier than expected and the size count will go down to more or less 0.
> Assume the cache is configured like this
> <distributed-cache name="ExpirationCache">
> <memory>
> <binary eviction="COUNT" size="120"/>
> </memory>
> <expiration lifespan="60000" interval="-1"/>
> </distributed-cache>
> The client is simple and add a new entry every second to keep the monitoring simple
> minute 1 - entries are added up to ~59 without eviction
> minute 2 - entries are added but the cache.size() is still 59
> CLI check attribute number_of_entries and evictions for the cache
> shows the same size and no eviction
> minute 3 - continue adding, cache.size()==59 but
> CLI shows the same size and increasing evictions
> minute 4 - still adding but size() decrease as well as CLI n-o-e and evictions grow
> The fact is that having less than 120 entries in the cache is unexpected as well as remove more entries after a time.
> The issue remain for
> <off-heap> and <object> with object count.
> as well as without exiration element or configured with interval - here the entries are expiring but eviction will have effects as well and drop the cache content unexpected.
> So it seems not releated to expiration at all
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
5 years, 5 months
[JBoss JIRA] (ISPN-10666) Eviction get stuck with <off-heap> configuration
by Wolf-Dieter Fink (Jira)
[ https://issues.jboss.org/browse/ISPN-10666?page=com.atlassian.jira.plugin... ]
Wolf-Dieter Fink reassigned ISPN-10666:
---------------------------------------
Assignee: Will Burns
> Eviction get stuck with <off-heap> configuration
> ------------------------------------------------
>
> Key: ISPN-10666
> URL: https://issues.jboss.org/browse/ISPN-10666
> Project: Infinispan
> Issue Type: Bug
> Reporter: Wolf-Dieter Fink
> Assignee: Will Burns
> Priority: Major
>
> If the cache is configured as followed adding entries every second will end up in an unresponsive server.
> The client will stop after 120sec (eviction count reached) with
> Exception in thread "main" org.infinispan.client.hotrod.exceptions.TransportException:: java.net.SocketTimeoutException: PutOperation{ExpirationCache, key=[B0x033E17323031392D..[26], value=[B0x033F280058585858..[10244], flags=4} timed out after 60000 ms
> at org.infinispan.client.hotrod.impl.Util.rewrap(Util.java:54)
> at org.infinispan.client.hotrod.impl.Util.await(Util.java:27)
> at org.infinispan.client.hotrod.impl.RemoteCacheImpl.put(RemoteCacheImpl.java:335)
> at org.infinispan.client.hotrod.impl.RemoteCacheSupport.put(RemoteCacheSupport.java:129)
> at org.infinispan.wfink.hotrod.client.HotRodConstantLoadExpirationClient.putWithExpiration(HotRodConstantLoadExpirationClient.java:48)
> at org.infinispan.wfink.hotrod.client.HotRodConstantLoadExpirationClient.testLoop(HotRodConstantLoadExpirationClient.java:84)
> at org.infinispan.wfink.hotrod.client.HotRodConstantLoadExpirationClient.main(HotRodConstantLoadExpirationClient.java:160)
> Caused by: java.net.SocketTimeoutException: PutOperation{ExpirationCache, key=[B0x033E17323031392D..[26], value=[B0x033F280058585858..[10244], flags=4} timed out after 60000 ms
> at org.infinispan.client.hotrod.impl.operations.HotRodOperation.run(HotRodOperation.java:172)
> at io.netty.util.concurrent.PromiseTask$RunnableAdapter.call(PromiseTask.java:38)
> at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:127)
> at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
> at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404)
> at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:322)
> at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
5 years, 5 months
[JBoss JIRA] (ISPN-10666) Eviction get stuck with <off-heap> configuration
by Wolf-Dieter Fink (Jira)
Wolf-Dieter Fink created ISPN-10666:
---------------------------------------
Summary: Eviction get stuck with <off-heap> configuration
Key: ISPN-10666
URL: https://issues.jboss.org/browse/ISPN-10666
Project: Infinispan
Issue Type: Bug
Reporter: Wolf-Dieter Fink
If the cache is configured as followed adding entries every second will end up in an unresponsive server.
The client will stop after 120sec (eviction count reached) with
Exception in thread "main" org.infinispan.client.hotrod.exceptions.TransportException:: java.net.SocketTimeoutException: PutOperation{ExpirationCache, key=[B0x033E17323031392D..[26], value=[B0x033F280058585858..[10244], flags=4} timed out after 60000 ms
at org.infinispan.client.hotrod.impl.Util.rewrap(Util.java:54)
at org.infinispan.client.hotrod.impl.Util.await(Util.java:27)
at org.infinispan.client.hotrod.impl.RemoteCacheImpl.put(RemoteCacheImpl.java:335)
at org.infinispan.client.hotrod.impl.RemoteCacheSupport.put(RemoteCacheSupport.java:129)
at org.infinispan.wfink.hotrod.client.HotRodConstantLoadExpirationClient.putWithExpiration(HotRodConstantLoadExpirationClient.java:48)
at org.infinispan.wfink.hotrod.client.HotRodConstantLoadExpirationClient.testLoop(HotRodConstantLoadExpirationClient.java:84)
at org.infinispan.wfink.hotrod.client.HotRodConstantLoadExpirationClient.main(HotRodConstantLoadExpirationClient.java:160)
Caused by: java.net.SocketTimeoutException: PutOperation{ExpirationCache, key=[B0x033E17323031392D..[26], value=[B0x033F280058585858..[10244], flags=4} timed out after 60000 ms
at org.infinispan.client.hotrod.impl.operations.HotRodOperation.run(HotRodOperation.java:172)
at io.netty.util.concurrent.PromiseTask$RunnableAdapter.call(PromiseTask.java:38)
at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:127)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:322)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
5 years, 5 months
[JBoss JIRA] (ISPN-10645) Configured eviction bases on COUNT will unexpected start eviction to early and might evict until the cache is empty
by Will Burns (Jira)
[ https://issues.jboss.org/browse/ISPN-10645?page=com.atlassian.jira.plugin... ]
Will Burns resolved ISPN-10645.
-------------------------------
Resolution: Rejected
> Configured eviction bases on COUNT will unexpected start eviction to early and might evict until the cache is empty
> -------------------------------------------------------------------------------------------------------------------
>
> Key: ISPN-10645
> URL: https://issues.jboss.org/browse/ISPN-10645
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Affects Versions: 10.0.0.CR2
> Reporter: Wolf-Dieter Fink
> Assignee: Will Burns
> Priority: Major
>
> If a cache is populated and configured with eviction and maybe expiration the eviction will start earlier than expected and the size count will go down to more or less 0.
> Assume the cache is configured like this
> <distributed-cache name="ExpirationCache">
> <memory>
> <binary eviction="COUNT" size="120"/>
> </memory>
> <expiration lifespan="60000" interval="-1"/>
> </distributed-cache>
> The client is simple and add a new entry every second to keep the monitoring simple
> minute 1 - entries are added up to ~59 without eviction
> minute 2 - entries are added but the cache.size() is still 59
> CLI check attribute number_of_entries and evictions for the cache
> shows the same size and no eviction
> minute 3 - continue adding, cache.size()==59 but
> CLI shows the same size and increasing evictions
> minute 4 - still adding but size() decrease as well as CLI n-o-e and evictions grow
> The fact is that having less than 120 entries in the cache is unexpected as well as remove more entries after a time.
> The issue remain for
> <off-heap> and <object> with object count.
> as well as without exiration element or configured with interval - here the entries are expiring but eviction will have effects as well and drop the cache content unexpected.
> So it seems not releated to expiration at all
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
5 years, 5 months
[JBoss JIRA] (ISPN-10645) Configured eviction bases on COUNT will unexpected start eviction to early and might evict until the cache is empty
by Will Burns (Jira)
[ https://issues.jboss.org/browse/ISPN-10645?page=com.atlassian.jira.plugin... ]
Will Burns commented on ISPN-10645:
-----------------------------------
This is expected behavior.
The 59 is due to entries expiring after 60 seconds, which means with an insertion rate of 1 per second will only have 59-60 entries in the cache at a time.
The cache dropping to a size of 2 is because of the access pattern and the fact that the expiration reaper is disabled. Caffeine uses the TinyLFU algorithm for its eviction. Effectively what happened is that it is Caffeine uses an eden space of 2 that store the most recently added entries. Once this fills up it will add that entry into the main space. Due to the access pattern it has determined the new entry should be immediately evicted. This causes the cache to have 2 "new" entries that aren't expired an 118 that are expired and aren't being evicted. If you enable the reaper it will clean these out sooner and never get to that point.
> Configured eviction bases on COUNT will unexpected start eviction to early and might evict until the cache is empty
> -------------------------------------------------------------------------------------------------------------------
>
> Key: ISPN-10645
> URL: https://issues.jboss.org/browse/ISPN-10645
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Affects Versions: 10.0.0.CR2
> Reporter: Wolf-Dieter Fink
> Assignee: Will Burns
> Priority: Major
>
> If a cache is populated and configured with eviction and maybe expiration the eviction will start earlier than expected and the size count will go down to more or less 0.
> Assume the cache is configured like this
> <distributed-cache name="ExpirationCache">
> <memory>
> <binary eviction="COUNT" size="120"/>
> </memory>
> <expiration lifespan="60000" interval="-1"/>
> </distributed-cache>
> The client is simple and add a new entry every second to keep the monitoring simple
> minute 1 - entries are added up to ~59 without eviction
> minute 2 - entries are added but the cache.size() is still 59
> CLI check attribute number_of_entries and evictions for the cache
> shows the same size and no eviction
> minute 3 - continue adding, cache.size()==59 but
> CLI shows the same size and increasing evictions
> minute 4 - still adding but size() decrease as well as CLI n-o-e and evictions grow
> The fact is that having less than 120 entries in the cache is unexpected as well as remove more entries after a time.
> The issue remain for
> <off-heap> and <object> with object count.
> as well as without exiration element or configured with interval - here the entries are expiring but eviction will have effects as well and drop the cache content unexpected.
> So it seems not releated to expiration at all
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
5 years, 5 months
[JBoss JIRA] (ISPN-10650) Expiration should not be replicated cross site
by Pedro Ruivo (Jira)
[ https://issues.jboss.org/browse/ISPN-10650?page=com.atlassian.jira.plugin... ]
Pedro Ruivo commented on ISPN-10650:
------------------------------------
10.x (master) PR integrated!
> Expiration should not be replicated cross site
> ----------------------------------------------
>
> Key: ISPN-10650
> URL: https://issues.jboss.org/browse/ISPN-10650
> Project: Infinispan
> Issue Type: Bug
> Components: Cross-Site Replication, Expiration
> Affects Versions: 9.4.16.Final, 10.0.0.CR2
> Reporter: Will Burns
> Assignee: Will Burns
> Priority: Major
> Fix For: 10.0.0.CR3, 9.4.17.Final
>
>
> There is no reason currently to replicate expiration commands across site.
> Lifespan will work just fine without without replicating as the entry is expired on either side properly, irrespective of system clocks.
> However max idle doesn't support the given semantics with xsite. We currently only confirm from other nodes in the same cluster for max idle before removing. It would be extremely costly to query all the cross sites to see if they have a read on the same entry before expiring. Also this requires having system clocks somewhat close in time, which can be even more problematic with clusters in disparate locations. We need to document that this doesn't work as well.
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
5 years, 5 months