Very interesting ! I will take a look at it later.<div><br></div><div>Right now, I check the size of the cache after each write and launch a thread to evict the entries...</div><div><br></div><div><div> </div><div> // calculate cache size</div>
<div> cacheSize = 0;</div><div> int maxCacheSizeInMB = 20;</div><div> final int percentToEvict = 2;</div><div> final int durationToConsiderOldInSeconds = 20 * 60; // (twenty</div>
<div> // minutes)</div><div> for (InternalCacheEntry ice : cache.getAdvancedCache().getDataContainer()) {</div><div> final int size = ((byte[]) ice.getValue()).length;</div><div>
<a href="http://logger.info">logger.info</a>("Cache entry size " + size);</div><div> cacheSize += size;</div><div> }</div><div><br></div><div> <a href="http://logger.info">logger.info</a>("Cache size " + cacheSize);</div>
<div><br></div><div> int ONEMB = 1024 * 1024;</div><div><br></div><div> final int maxCacheSizeInBytes = maxCacheSizeInMB * ONEMB;</div><div> if (cacheSize > maxCacheSizeInBytes) {</div>
<div> <a href="http://logger.info">logger.info</a>("Cache too big (in MB)" + cacheSize / ONEMB);</div><div><br></div><div><br></div><div> new Thread("Qi4j - Eviction ") {</div>
<div><br></div><div> @Override</div><div> public void run() {</div><div><br></div><div> <a href="http://logger.info">logger.info</a>("Started eviction thread");</div>
<div> // ok let's evict a %age</div><div> Iterator<InternalCacheEntry> it = cache.getAdvancedCache().getDataContainer()</div><div> .iterator();</div>
<div> // inflate cache size to the percentage to remove</div><div> cacheSize += (cacheSize * percentToEvict) / 100;</div><div> while (cacheSize > maxCacheSizeInBytes && it.hasNext()) {</div>
<div> InternalCacheEntry ice = it.next();</div><div> final long diffTime = currentTimeMillis - ice.getLastUsed();</div><div> final long oldTreshold = durationToConsiderOldInSeconds * 1000;</div>
<div> if (diffTime > oldTreshold) {</div><div> cache.evict(ice.getKey() + "");</div><div> cacheSize -= ((byte[]) ice.getValue()).length;</div>
<div> }</div><div> }</div><div> <a href="http://logger.info">logger.info</a>("new cache size (in MB)" + cacheSize / 1024);</div><div> }</div>
<div> }.start();</div><div><br></div><div> }</div><div> </div><div><br></div><div><br></div><div><br><div class="gmail_quote">On Mon, May 17, 2010 at 4:11 PM, Vladimir Blagojevic <span dir="ltr"><<a href="mailto:vblagoje@redhat.com">vblagoje@redhat.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;"><div style="word-wrap:break-word">How did you do this? Did you write your own EvictionManager. Notice that a contract between EvictionManager and DataContainer changed slightly in Alpha2. DataContainer fulfils eviction contract with EvictionManager by letting EvictionManager know the evicted entries (DataContainer#getEvictionCandidates). You can also select eviction thread policy (DEFAULT or PIGGYBACK)[1]. Have a look at [2] for more details.<div>
<br></div><div><br></div><div>[1] <a href="http://docs.jboss.org/infinispan/4.1/apidocs/config.html#ce_default_eviction" target="_blank">http://docs.jboss.org/infinispan/4.1/apidocs/config.html#ce_default_eviction</a><div>
[2] <a href="http://infinispan.blogspot.com/2010/03/infinispan-eviction-batching-updates.html" target="_blank">http://infinispan.blogspot.com/2010/03/infinispan-eviction-batching-updates.html</a><div><div></div><div class="h5">
<br><div><br></div><div><br><div><div>On 2010-05-17, at 10:03 AM, Philippe Van Dyck wrote:</div><br><blockquote type="cite">Well... to be honest, I am not ;-)<div>I run my own eviction trigger (based on the total cache size) and I evict any entry older than 20 minutes until the size of the cache is reduced enough (usually 2%). Since I need the 'lastUsed' value, I need the LRU strategy.</div>
<div><br></div><div>phil</div><div><br><div class="gmail_quote">On Mon, May 17, 2010 at 3:55 PM, Vladimir Blagojevic <span dir="ltr"><<a href="mailto:vblagoje@redhat.com" target="_blank">vblagoje@redhat.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="word-wrap:break-word">We have to handle -1 case. I'll look into this. I am glad that you are exercising our new container with eviction. Keep pounding at it :)<div><div></div><div><div><br><div>
<div>On 2010-05-17, at 9:49 AM, Philippe Van Dyck wrote:</div><br><blockquote type="cite">Ok, working now. Thanks again Vladimir. The memory problem was surely coming from there (I will investigate it later) - I am back on BETA1.<div>
<br><div>BTW, "maxEntries=-1" is not working anymore (update xml config doc ?)</div>
<div><br></div><div><div>Caused by: java.lang.IllegalArgumentException</div><div><span style="white-space:pre"> </span>at org.infinispan.util.concurrent.BoundedConcurrentHashMap.<init>(BoundedConcurrentHashMap.java:1139)</div>
<div><span style="white-space:pre"> </span>at org.infinispan.container.DefaultDataContainer.<init>(DefaultDataContainer.java:92)</div><div><br></div><div>cheers,</div><div><br></div><div>phil</div>
<br><div class="gmail_quote">On Mon, May 17, 2010 at 3:30 PM, Philippe Van Dyck <span dir="ltr"><<a href="mailto:pvdyck@gmail.com" target="_blank">pvdyck@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Thanks Vladimir... <div><br></div><div>May I suppose that this limitation was not verified in alpha1 ?</div><div>I will test this right away !</div><div><br></div><div>cheers,</div><div><br></div><div>phil</div><div><div>
</div><div><div><br></div>
<div><br><div class="gmail_quote">On Mon, May 17, 2010 at 3:26 PM, Vladimir Blagojevic <span dir="ltr"><<a href="mailto:vblagoje@redhat.com" target="_blank">vblagoje@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="word-wrap:break-word">I think the problem is related to the fact that you have maxEntries = 1 specified in configuration for your container. <div><div></div><div><br><div><div>On 2010-05-17, at 9:24 AM, Philippe Van Dyck wrote:</div>
<br><blockquote type="cite">Confirmed - when I go back to alpha1 the problem disappears.<div><br></div><div>Could anyone explain with alpha3 (the problem is already there) there is only one entry in getDataContainer ?</div>
<div>
<br></div><div><div> for (InternalCacheEntry ice : cache.getAdvancedCache().getDataContainer()) {</div><div> final int size = ((byte[]) ice.getValue()).length;</div><div> <a href="http://logger.info/" target="_blank">logger.info</a>("Cache entry size " + size);</div>
<div> cacheSize += size;</div><div> }</div><div><br></div><div> <a href="http://logger.info/" target="_blank">logger.info</a>("Cache size " + cacheSize);</div></div>
<div><br></div><div>
<br></div><div>cheers </div><div><br></div><div>phil<br><br><div class="gmail_quote">On Mon, May 17, 2010 at 2:57 PM, Manik Surtani <span dir="ltr"><<a href="mailto:manik@jboss.org" target="_blank">manik@jboss.org</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word">Wow, no idea. Any thread dumps, stack traces? Logging?<div><div></div><div>
<div><br><div><div>On 17 May 2010, at 13:48, Philippe Van Dyck wrote:</div><br><blockquote type="cite">Update - trashed & crashed as planned.<div>Done some debugging : something strange... my cache seems to contain only one entry (???)</div>
<div>Any clue ?</div><div><br></div><div>phil<br><br><div class="gmail_quote">On Mon, May 17, 2010 at 2:21 PM, Philippe Van Dyck <span dir="ltr"><<a href="mailto:pvdyck@gmail.com" target="_blank">pvdyck@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">I don't have any resource available to setup profiling in prepod right now.<div>Looking at the changes from alpha1 to beta1, I only see jclouds and some guava libs updated.</div>
<div>Load on the server went berserk these 10 last minutes, it will probably trash & crash in the next hour.</div>
<div>Will probably go back to ALPHA1. <br><br></div><div>phil</div><div><div></div><div><div><br><div class="gmail_quote">On Mon, May 17, 2010 at 2:13 PM, Manik Surtani <span dir="ltr"><<a href="mailto:manik@jboss.org" target="_blank">manik@jboss.org</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word">Have you tried profiling stuff? Nothing really should have changed in Beta1 to affect such a config, except perhaps the version of JClouds and some JClouds-related code.<div>
<div></div><div><div><br><div><div>On 17 May 2010, at 13:07, Philippe Van Dyck wrote:</div><br><blockquote type="cite"><div><?xml version="1.0" encoding="UTF-8"?></div><div><br></div><div>
<infinispan xmlns:xsi="<a href="http://www.w3.org/2001/XMLSchema-instance" target="_blank">http://www.w3.org/2001/XMLSchema-instance</a>"</div>
<div> xmlns="urn:infinispan:config:4.0"></div><div> <global></div><div> <transport</div><div> transportClass="org.infinispan.remoting.transport.jgroups.JGroupsTransport"></div>
<div> <properties></div><div> <property name="configurationFile" value="jgroupsprod.xml"/></div><div> </properties></div><div> </transport></div>
<div> <globalJmxStatistics enabled="true" allowDuplicateDomains="true"/></div><div> </global></div><div><br></div><div><br></div><div> <namedCache name="qi4j"></div>
<div> <jmxStatistics enabled="true"/></div><div> <transaction</div><div> transactionManagerLookupClass="org.qi4j.entitystore.s3jclouds.AtomikosTransactionManagerLookup"/></div>
<div> <clustering mode="distribution"></div><div> <l1 enabled="true" lifespan="100000"/></div><div> <hash numOwners="2" rehashRpcTimeout="120000"/></div>
<div> </clustering></div><div><br></div><div> <loaders passivation="false" shared="true" preload="false"></div><div> </div><div> <loader</div><div> class="org.infinispan...CloudCacheStore"</div>
<div> fetchPersistentState="false" ignoreModifications="false"</div><div> purgeOnStartup="false" purgeSynchronously="true"></div><div><br></div><div>
<properties></div>
<div> <property name="identity" value="***"/></div><div> <property name="password" value="***"/></div><div> <property name="bucketPrefix" value="store2"/></div>
<div> <property name="cloudService" value="s3"/></div><div> </properties></div><div> </loader></div><div> </loaders></div><div><br></div><div>
<eviction strategy="LRU" wakeUpInterval="-1" maxEntries="1"/></div><div><br></div><div> <locking lockAcquisitionTimeout="60000" useLockStriping="true"/></div>
<div><br></div><div><br></div><div> <unsafe unreliableReturnValues="true"/></div><div><br></div><div> </namedCache></div><div> </div><div></infinispan></div><div><br></div><br><div class="gmail_quote">
On Mon, May 17, 2010 at 1:55 PM, Manik Surtani <span dir="ltr"><<a href="mailto:manik@jboss.org" target="_blank">manik@jboss.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
What configuration do you use?<br>
<div><div></div><div><br>
On 17 May 2010, at 12:46, Philippe Van Dyck wrote:<br>
<br>
> FYI, I upgraded from ALPHA1 to BETA1 on a preproduction system this morning.<br>
><br>
> Take a look at the graphic attached, the server is restarted everyday around 1 am (blue and green lines crossing).<br>
><br>
> Users began to use the system around 9 am.... look at today's pattern and the previous day pattern !<br>
><br>
> Anything I should know or I missed ?<br>
><br>
> cheers,<br>
><br>
> phil<br>
</div></div>> <memleak.tiff>_______________________________________________<br>
> infinispan-dev mailing list<br>
> <a href="mailto:infinispan-dev@lists.jboss.org" target="_blank">infinispan-dev@lists.jboss.org</a><br>
> <a href="https://lists.jboss.org/mailman/listinfo/infinispan-dev" target="_blank">https://lists.jboss.org/mailman/listinfo/infinispan-dev</a><br>
<br>
--<br>
Manik Surtani<br>
<a href="mailto:manik@jboss.org" target="_blank">manik@jboss.org</a><br>
Lead, Infinispan<br>
Lead, JBoss Cache<br>
<a href="http://www.infinispan.org/" target="_blank">http://www.infinispan.org</a><br>
<a href="http://www.jbosscache.org/" target="_blank">http://www.jbosscache.org</a><br>
<br>
<br>
<br>
<br>
<br>
_______________________________________________<br>
infinispan-dev mailing list<br>
<a href="mailto:infinispan-dev@lists.jboss.org" target="_blank">infinispan-dev@lists.jboss.org</a><br>
<a href="https://lists.jboss.org/mailman/listinfo/infinispan-dev" target="_blank">https://lists.jboss.org/mailman/listinfo/infinispan-dev</a><br>
</blockquote></div><br>
_______________________________________________<br>infinispan-dev mailing list<br><a href="mailto:infinispan-dev@lists.jboss.org" target="_blank">infinispan-dev@lists.jboss.org</a><br><a href="https://lists.jboss.org/mailman/listinfo/infinispan-dev" target="_blank">https://lists.jboss.org/mailman/listinfo/infinispan-dev</a></blockquote>
</div><br><div>
<span style="border-collapse:separate;color:rgb(0, 0, 0);font-family:Helvetica;font-size:12px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:auto;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><span style="border-collapse:separate;color:rgb(0, 0, 0);font-family:Helvetica;font-size:12px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><div style="word-wrap:break-word">
<div>--</div><div>Manik Surtani</div><div><a href="mailto:manik@jboss.org" target="_blank">manik@jboss.org</a></div><div>Lead, Infinispan</div><div>Lead, JBoss Cache</div><div><a href="http://www.infinispan.org/" target="_blank">http://www.infinispan.org</a></div>
<div><a href="http://www.jbosscache.org/" target="_blank">http://www.jbosscache.org</a></div><div><br></div></div></span><br></span><br>
</div>
<br></div></div></div></div><br>_______________________________________________<br>
infinispan-dev mailing list<br>
<a href="mailto:infinispan-dev@lists.jboss.org" target="_blank">infinispan-dev@lists.jboss.org</a><br>
<a href="https://lists.jboss.org/mailman/listinfo/infinispan-dev" target="_blank">https://lists.jboss.org/mailman/listinfo/infinispan-dev</a><br></blockquote></div><br></div>
</div></div></blockquote></div><br></div>
_______________________________________________<br>infinispan-dev mailing list<br><a href="mailto:infinispan-dev@lists.jboss.org" target="_blank">infinispan-dev@lists.jboss.org</a><br><a href="https://lists.jboss.org/mailman/listinfo/infinispan-dev" target="_blank">https://lists.jboss.org/mailman/listinfo/infinispan-dev</a></blockquote>
</div><br><div>
<span style="border-collapse:separate;color:rgb(0, 0, 0);font-family:Helvetica;font-size:12px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:auto;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><span style="border-collapse:separate;color:rgb(0, 0, 0);font-family:Helvetica;font-size:12px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><div style="word-wrap:break-word">
<div>--</div><div>Manik Surtani</div><div><a href="mailto:manik@jboss.org" target="_blank">manik@jboss.org</a></div><div>Lead, Infinispan</div><div>Lead, JBoss Cache</div><div><a href="http://www.infinispan.org/" target="_blank">http://www.infinispan.org</a></div>
<div><a href="http://www.jbosscache.org/" target="_blank">http://www.jbosscache.org</a></div><div><br></div></div></span><br></span><br>
</div>
<br></div></div></div></div><br>_______________________________________________<br>
infinispan-dev mailing list<br>
<a href="mailto:infinispan-dev@lists.jboss.org" target="_blank">infinispan-dev@lists.jboss.org</a><br>
<a href="https://lists.jboss.org/mailman/listinfo/infinispan-dev" target="_blank">https://lists.jboss.org/mailman/listinfo/infinispan-dev</a><br></blockquote></div><br></div>
_______________________________________________<br>infinispan-dev mailing list<br><a href="mailto:infinispan-dev@lists.jboss.org" target="_blank">infinispan-dev@lists.jboss.org</a><br><a href="https://lists.jboss.org/mailman/listinfo/infinispan-dev" target="_blank">https://lists.jboss.org/mailman/listinfo/infinispan-dev</a></blockquote>
</div><br></div></div><font color="#888888"><div>
<span style="border-collapse:separate;color:rgb(0, 0, 0);font-family:Helvetica;font-size:medium;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:auto;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><span style="border-collapse:separate;color:rgb(0, 0, 0);font-family:Helvetica;font-size:medium;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><div style="word-wrap:break-word">
<div>Vladimir Blagojevic</div><div><div>JBoss Clustering Team</div><div>JBoss by Red Hat</div></div></div></span><br></span><br>
</div>
<br></font></div><br>_______________________________________________<br>
infinispan-dev mailing list<br>
<a href="mailto:infinispan-dev@lists.jboss.org" target="_blank">infinispan-dev@lists.jboss.org</a><br>
<a href="https://lists.jboss.org/mailman/listinfo/infinispan-dev" target="_blank">https://lists.jboss.org/mailman/listinfo/infinispan-dev</a><br></blockquote></div><br></div>
</div></div></blockquote></div><br></div></div>
_______________________________________________<br>infinispan-dev mailing list<br><a href="mailto:infinispan-dev@lists.jboss.org" target="_blank">infinispan-dev@lists.jboss.org</a><br><a href="https://lists.jboss.org/mailman/listinfo/infinispan-dev" target="_blank">https://lists.jboss.org/mailman/listinfo/infinispan-dev</a></blockquote>
</div><br>
<br></div></div></div></div><br>_______________________________________________<br>
infinispan-dev mailing list<br>
<a href="mailto:infinispan-dev@lists.jboss.org" target="_blank">infinispan-dev@lists.jboss.org</a><br>
<a href="https://lists.jboss.org/mailman/listinfo/infinispan-dev" target="_blank">https://lists.jboss.org/mailman/listinfo/infinispan-dev</a><br></blockquote></div><br></div>
_______________________________________________<br>infinispan-dev mailing list<br><a href="mailto:infinispan-dev@lists.jboss.org" target="_blank">infinispan-dev@lists.jboss.org</a><br><a href="https://lists.jboss.org/mailman/listinfo/infinispan-dev" target="_blank">https://lists.jboss.org/mailman/listinfo/infinispan-dev</a></blockquote>
</div><br><div>
<span style="border-collapse:separate;color:rgb(0, 0, 0);font-family:Helvetica;font-size:medium;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:auto;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><span style="border-collapse:separate;color:rgb(0, 0, 0);font-family:Helvetica;font-size:medium;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><div style="word-wrap:break-word">
<div>Vladimir Blagojevic</div><div><div>JBoss Clustering Team</div><div>JBoss by Red Hat</div></div></div></span><br></span><br>
</div>
<br></div></div></div></div></div></div><br>_______________________________________________<br>
infinispan-dev mailing list<br>
<a href="mailto:infinispan-dev@lists.jboss.org">infinispan-dev@lists.jboss.org</a><br>
<a href="https://lists.jboss.org/mailman/listinfo/infinispan-dev" target="_blank">https://lists.jboss.org/mailman/listinfo/infinispan-dev</a><br></blockquote></div><br></div></div>