FYI, I upgraded from ALPHA1 to BETA1 on a preproduction system this morning. Take a look at the graphic attached, the server is restarted everyday around 1 am (blue and green lines crossing). Users began to use the system around 9 am.... look at today's pattern and the previous day pattern ! Anything I should know or I missed ? cheers, phil <memleak.tiff>_______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

<?xml version="1.0" encoding="UTF-8"?> <infinispan xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="urn:infinispan:config:4.0"> <global> <transport transportClass="org.infinispan.remoting.transport.jgroups.JGroupsTransport"> <properties> <property name="configurationFile" value="jgroupsprod.xml"/> </properties> </transport> <globalJmxStatistics enabled="true" allowDuplicateDomains="true"/> </global> <namedCache name="qi4j"> <jmxStatistics enabled="true"/> <transaction transactionManagerLookupClass="org.qi4j.entitystore.s3jclouds.AtomikosTransactionManagerLookup"/> <clustering mode="distribution"> <l1 enabled="true" lifespan="100000"/> <hash numOwners="2" rehashRpcTimeout="120000"/> </clustering> <loaders passivation="false" shared="true" preload="false"> <loader class="org.infinispan...CloudCacheStore" fetchPersistentState="false" ignoreModifications="false" purgeOnStartup="false" purgeSynchronously="true"> <properties> <property name="identity" value="***"/> <property name="password" value="***"/> <property name="bucketPrefix" value="store2"/> <property name="cloudService" value="s3"/> </properties> </loader> </loaders> <eviction strategy="LRU" wakeUpInterval="-1" maxEntries="1"/> <locking lockAcquisitionTimeout="60000" useLockStriping="true"/> <unsafe unreliableReturnValues="true"/> </namedCache> </infinispan> On Mon, May 17, 2010 at 1:55 PM, Manik Surtani <manik(a)jboss.org&gt; wrote: What configuration do you use? On 17 May 2010, at 12:46, Philippe Van Dyck wrote: > FYI, I upgraded from ALPHA1 to BETA1 on a preproduction system this morning. > > Take a look at the graphic attached, the server is restarted everyday around 1 am (blue and green lines crossing). > > Users began to use the system around 9 am.... look at today's pattern and the previous day pattern ! > > Anything I should know or I missed ? > > cheers, > > phil > <memleak.tiff>_______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev -- Manik Surtani manik(a)jboss.org Lead, Infinispan Lead, JBoss Cache http://www.infinispan.org http://www.jbosscache.org _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

I don't have any resource available to setup profiling in prepod right now. Looking at the changes from alpha1 to beta1, I only see jclouds and some guava libs updated. Load on the server went berserk these 10 last minutes, it will probably trash & crash in the next hour. Will probably go back to ALPHA1. phil On Mon, May 17, 2010 at 2:13 PM, Manik Surtani <manik(a)jboss.org&gt; wrote: > Have you tried profiling stuff? Nothing really should have changed in > Beta1 to affect such a config, except perhaps the version of JClouds and > some JClouds-related code. > > On 17 May 2010, at 13:07, Philippe Van Dyck wrote: > > <?xml version="1.0" encoding="UTF-8"?> > > <infinispan xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" > xmlns="urn:infinispan:config:4.0"> > <global> > <transport > > transportClass="org.infinispan.remoting.transport.jgroups.JGroupsTransport"> > <properties> > <property name="configurationFile" value="jgroupsprod.xml"/> > </properties> > </transport> > <globalJmxStatistics enabled="true" allowDuplicateDomains="true"/> > </global> > > > <namedCache name="qi4j"> > <jmxStatistics enabled="true"/> > <transaction > > transactionManagerLookupClass="org.qi4j.entitystore.s3jclouds.AtomikosTransactionManagerLookup"/> > <clustering mode="distribution"> > <l1 enabled="true" lifespan="100000"/> > <hash numOwners="2" rehashRpcTimeout="120000"/> > </clustering> > > <loaders passivation="false" shared="true" preload="false"> > > <loader > class="org.infinispan...CloudCacheStore" > fetchPersistentState="false" ignoreModifications="false" > purgeOnStartup="false" purgeSynchronously="true"> > > <properties> > <property name="identity" value="***"/> > <property name="password" value="***"/> > <property name="bucketPrefix" value="store2"/> > <property name="cloudService" value="s3"/> > </properties> > </loader> > </loaders> > > <eviction strategy="LRU" wakeUpInterval="-1" maxEntries="1"/> > > <locking lockAcquisitionTimeout="60000" useLockStriping="true"/> > > > <unsafe unreliableReturnValues="true"/> > > </namedCache> > > </infinispan> > > > On Mon, May 17, 2010 at 1:55 PM, Manik Surtani <manik(a)jboss.org&gt; wrote: > >> What configuration do you use? >> >> On 17 May 2010, at 12:46, Philippe Van Dyck wrote: >> >> > FYI, I upgraded from ALPHA1 to BETA1 on a preproduction system this >> morning. >> > >> > Take a look at the graphic attached, the server is restarted everyday >> around 1 am (blue and green lines crossing). >> > >> > Users began to use the system around 9 am.... look at today's pattern >> and the previous day pattern ! >> > >> > Anything I should know or I missed ? >> > >> > cheers, >> > >> > phil >> > <memleak.tiff>_______________________________________________ >> > infinispan-dev mailing list >> > infinispan-dev(a)lists.jboss.org >> > https://lists.jboss.org/mailman/listinfo/infinispan-dev >> >> -- >> Manik Surtani >> manik(a)jboss.org >> Lead, Infinispan >> Lead, JBoss Cache >> http://www.infinispan.org >> http://www.jbosscache.org >> >> >> >> >> >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev >> > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev > > > -- > Manik Surtani > manik(a)jboss.org > Lead, Infinispan > Lead, JBoss Cache > http://www.infinispan.org > http://www.jbosscache.org > > > > > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev >

Wow, no idea. Any thread dumps, stack traces? Logging? On 17 May 2010, at 13:48, Philippe Van Dyck wrote: Update - trashed & crashed as planned. Done some debugging : something strange... my cache seems to contain only one entry (???) Any clue ? phil On Mon, May 17, 2010 at 2:21 PM, Philippe Van Dyck <pvdyck(a)gmail.com&gt;wrote: > I don't have any resource available to setup profiling in prepod right > now. > Looking at the changes from alpha1 to beta1, I only see jclouds and some > guava libs updated. > Load on the server went berserk these 10 last minutes, it will probably > trash & crash in the next hour. > Will probably go back to ALPHA1. > > phil > > On Mon, May 17, 2010 at 2:13 PM, Manik Surtani <manik(a)jboss.org&gt; wrote: > >> Have you tried profiling stuff? Nothing really should have changed in >> Beta1 to affect such a config, except perhaps the version of JClouds and >> some JClouds-related code. >> >> On 17 May 2010, at 13:07, Philippe Van Dyck wrote: >> >> <?xml version="1.0" encoding="UTF-8"?> >> >> <infinispan xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" >> xmlns="urn:infinispan:config:4.0"> >> <global> >> <transport >> >> transportClass="org.infinispan.remoting.transport.jgroups.JGroupsTransport"> >> <properties> >> <property name="configurationFile" value="jgroupsprod.xml"/> >> </properties> >> </transport> >> <globalJmxStatistics enabled="true" allowDuplicateDomains="true"/> >> </global> >> >> >> <namedCache name="qi4j"> >> <jmxStatistics enabled="true"/> >> <transaction >> >> transactionManagerLookupClass="org.qi4j.entitystore.s3jclouds.AtomikosTransactionManagerLookup"/> >> <clustering mode="distribution"> >> <l1 enabled="true" lifespan="100000"/> >> <hash numOwners="2" rehashRpcTimeout="120000"/> >> </clustering> >> >> <loaders passivation="false" shared="true" preload="false"> >> >> <loader >> class="org.infinispan...CloudCacheStore" >> fetchPersistentState="false" ignoreModifications="false" >> purgeOnStartup="false" purgeSynchronously="true"> >> >> <properties> >> <property name="identity" value="***"/> >> <property name="password" value="***"/> >> <property name="bucketPrefix" value="store2"/> >> <property name="cloudService" value="s3"/> >> </properties> >> </loader> >> </loaders> >> >> <eviction strategy="LRU" wakeUpInterval="-1" maxEntries="1"/> >> >> <locking lockAcquisitionTimeout="60000" useLockStriping="true"/> >> >> >> <unsafe unreliableReturnValues="true"/> >> >> </namedCache> >> >> </infinispan> >> >> >> On Mon, May 17, 2010 at 1:55 PM, Manik Surtani <manik(a)jboss.org&gt; wrote: >> >>> What configuration do you use? >>> >>> On 17 May 2010, at 12:46, Philippe Van Dyck wrote: >>> >>> > FYI, I upgraded from ALPHA1 to BETA1 on a preproduction system this >>> morning. >>> > >>> > Take a look at the graphic attached, the server is restarted everyday >>> around 1 am (blue and green lines crossing). >>> > >>> > Users began to use the system around 9 am.... look at today's pattern >>> and the previous day pattern ! >>> > >>> > Anything I should know or I missed ? >>> > >>> > cheers, >>> > >>> > phil >>> > <memleak.tiff>_______________________________________________ >>> > infinispan-dev mailing list >>> > infinispan-dev(a)lists.jboss.org >>> > https://lists.jboss.org/mailman/listinfo/infinispan-dev >>> >>> -- >>> Manik Surtani >>> manik(a)jboss.org >>> Lead, Infinispan >>> Lead, JBoss Cache >>> http://www.infinispan.org >>> http://www.jbosscache.org >>> >>> >>> >>> >>> >>> _______________________________________________ >>> infinispan-dev mailing list >>> infinispan-dev(a)lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>> >> >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev >> >> >> -- >> Manik Surtani >> manik(a)jboss.org >> Lead, Infinispan >> Lead, JBoss Cache >> http://www.infinispan.org >> http://www.jbosscache.org >> >> >> >> >> >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev >> > > _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev -- Manik Surtani manik(a)jboss.org Lead, Infinispan Lead, JBoss Cache http://www.infinispan.org http://www.jbosscache.org _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

I think the problem is related to the fact that you have maxEntries = 1 specified in configuration for your container. On 2010-05-17, at 9:24 AM, Philippe Van Dyck wrote: Confirmed - when I go back to alpha1 the problem disappears. Could anyone explain with alpha3 (the problem is already there) there is only one entry in getDataContainer ? for (InternalCacheEntry ice : cache.getAdvancedCache().getDataContainer()) { final int size = ((byte[]) ice.getValue()).length; logger.info("Cache entry size " + size); cacheSize += size; } logger.info("Cache size " + cacheSize); cheers phil On Mon, May 17, 2010 at 2:57 PM, Manik Surtani <manik(a)jboss.org&gt; wrote: > Wow, no idea. Any thread dumps, stack traces? Logging? > > On 17 May 2010, at 13:48, Philippe Van Dyck wrote: > > Update - trashed & crashed as planned. > Done some debugging : something strange... my cache seems to contain only > one entry (???) > Any clue ? > > phil > > On Mon, May 17, 2010 at 2:21 PM, Philippe Van Dyck <pvdyck(a)gmail.com&gt;wrote: > >> I don't have any resource available to setup profiling in prepod right >> now. >> Looking at the changes from alpha1 to beta1, I only see jclouds and some >> guava libs updated. >> Load on the server went berserk these 10 last minutes, it will probably >> trash & crash in the next hour. >> Will probably go back to ALPHA1. >> >> phil >> >> On Mon, May 17, 2010 at 2:13 PM, Manik Surtani <manik(a)jboss.org&gt; wrote: >> >>> Have you tried profiling stuff? Nothing really should have changed in >>> Beta1 to affect such a config, except perhaps the version of JClouds and >>> some JClouds-related code. >>> >>> On 17 May 2010, at 13:07, Philippe Van Dyck wrote: >>> >>> <?xml version="1.0" encoding="UTF-8"?> >>> >>> <infinispan xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" >>> xmlns="urn:infinispan:config:4.0"> >>> <global> >>> <transport >>> >>> transportClass="org.infinispan.remoting.transport.jgroups.JGroupsTransport"> >>> <properties> >>> <property name="configurationFile" value="jgroupsprod.xml"/> >>> </properties> >>> </transport> >>> <globalJmxStatistics enabled="true" allowDuplicateDomains="true"/> >>> </global> >>> >>> >>> <namedCache name="qi4j"> >>> <jmxStatistics enabled="true"/> >>> <transaction >>> >>> transactionManagerLookupClass="org.qi4j.entitystore.s3jclouds.AtomikosTransactionManagerLookup"/> >>> <clustering mode="distribution"> >>> <l1 enabled="true" lifespan="100000"/> >>> <hash numOwners="2" rehashRpcTimeout="120000"/> >>> </clustering> >>> >>> <loaders passivation="false" shared="true" preload="false"> >>> >>> <loader >>> class="org.infinispan...CloudCacheStore" >>> fetchPersistentState="false" ignoreModifications="false" >>> purgeOnStartup="false" purgeSynchronously="true"> >>> >>> <properties> >>> <property name="identity" value="***"/> >>> <property name="password" value="***"/> >>> <property name="bucketPrefix" value="store2"/> >>> <property name="cloudService" value="s3"/> >>> </properties> >>> </loader> >>> </loaders> >>> >>> <eviction strategy="LRU" wakeUpInterval="-1" maxEntries="1"/> >>> >>> <locking lockAcquisitionTimeout="60000" useLockStriping="true"/> >>> >>> >>> <unsafe unreliableReturnValues="true"/> >>> >>> </namedCache> >>> >>> </infinispan> >>> >>> >>> On Mon, May 17, 2010 at 1:55 PM, Manik Surtani <manik(a)jboss.org&gt; wrote: >>> >>>> What configuration do you use? >>>> >>>> On 17 May 2010, at 12:46, Philippe Van Dyck wrote: >>>> >>>> > FYI, I upgraded from ALPHA1 to BETA1 on a preproduction system this >>>> morning. >>>> > >>>> > Take a look at the graphic attached, the server is restarted everyday >>>> around 1 am (blue and green lines crossing). >>>> > >>>> > Users began to use the system around 9 am.... look at today's pattern >>>> and the previous day pattern ! >>>> > >>>> > Anything I should know or I missed ? >>>> > >>>> > cheers, >>>> > >>>> > phil >>>> > <memleak.tiff>_______________________________________________ >>>> > infinispan-dev mailing list >>>> > infinispan-dev(a)lists.jboss.org >>>> > https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>> >>>> -- >>>> Manik Surtani >>>> manik(a)jboss.org >>>> Lead, Infinispan >>>> Lead, JBoss Cache >>>> http://www.infinispan.org >>>> http://www.jbosscache.org >>>> >>>> >>>> >>>> >>>> >>>> _______________________________________________ >>>> infinispan-dev mailing list >>>> infinispan-dev(a)lists.jboss.org >>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>> >>> >>> _______________________________________________ >>> infinispan-dev mailing list >>> infinispan-dev(a)lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>> >>> >>> -- >>> Manik Surtani >>> manik(a)jboss.org >>> Lead, Infinispan >>> Lead, JBoss Cache >>> http://www.infinispan.org >>> http://www.jbosscache.org >>> >>> >>> >>> >>> >>> _______________________________________________ >>> infinispan-dev mailing list >>> infinispan-dev(a)lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>> >> >> > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev > > > -- > Manik Surtani > manik(a)jboss.org > Lead, Infinispan > Lead, JBoss Cache > http://www.infinispan.org > http://www.jbosscache.org > > > > > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev > _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev Vladimir Blagojevic JBoss Clustering Team JBoss by Red Hat _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

Ok, working now. Thanks again Vladimir. The memory problem was surely coming from there (I will investigate it later) - I am back on BETA1. BTW, "maxEntries=-1" is not working anymore (update xml config doc ?)

Caused by: java.lang.IllegalArgumentException at org.infinispan.util.concurrent.BoundedConcurrentHashMap.<init>(BoundedConcurrentHashMap.java:1139) at org.infinispan.container.DefaultDataContainer.<init>(DefaultDataContainer.java:92) cheers, phil On Mon, May 17, 2010 at 3:30 PM, Philippe Van Dyck <pvdyck(a)gmail.com&gt; wrote: Thanks Vladimir... May I suppose that this limitation was not verified in alpha1 ? I will test this right away ! cheers, phil On Mon, May 17, 2010 at 3:26 PM, Vladimir Blagojevic <vblagoje(a)redhat.com&gt; wrote: I think the problem is related to the fact that you have maxEntries = 1 specified in configuration for your container. On 2010-05-17, at 9:24 AM, Philippe Van Dyck wrote: > Confirmed - when I go back to alpha1 the problem disappears. > > Could anyone explain with alpha3 (the problem is already there) there is only one entry in getDataContainer ? > > for (InternalCacheEntry ice : cache.getAdvancedCache().getDataContainer()) { > final int size = ((byte[]) ice.getValue()).length; > logger.info("Cache entry size " + size); > cacheSize += size; > } > > logger.info("Cache size " + cacheSize); > > > cheers > > phil > > On Mon, May 17, 2010 at 2:57 PM, Manik Surtani <manik(a)jboss.org&gt; wrote: > Wow, no idea. Any thread dumps, stack traces? Logging? > > On 17 May 2010, at 13:48, Philippe Van Dyck wrote: > >> Update - trashed & crashed as planned. >> Done some debugging : something strange... my cache seems to contain only one entry (???) >> Any clue ? >> >> phil >> >> On Mon, May 17, 2010 at 2:21 PM, Philippe Van Dyck <pvdyck(a)gmail.com&gt; wrote: >> I don't have any resource available to setup profiling in prepod right now. >> Looking at the changes from alpha1 to beta1, I only see jclouds and some guava libs updated. >> Load on the server went berserk these 10 last minutes, it will probably trash & crash in the next hour. >> Will probably go back to ALPHA1. >> >> phil >> >> On Mon, May 17, 2010 at 2:13 PM, Manik Surtani <manik(a)jboss.org&gt; wrote: >> Have you tried profiling stuff? Nothing really should have changed in Beta1 to affect such a config, except perhaps the version of JClouds and some JClouds-related code. >> >> On 17 May 2010, at 13:07, Philippe Van Dyck wrote: >> >>> <?xml version="1.0" encoding="UTF-8"?> >>> >>> <infinispan xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" >>> xmlns="urn:infinispan:config:4.0"> >>> <global> >>> <transport >>> transportClass="org.infinispan.remoting.transport.jgroups.JGroupsTransport"> >>> <properties> >>> <property name="configurationFile" value="jgroupsprod.xml"/> >>> </properties> >>> </transport> >>> <globalJmxStatistics enabled="true" allowDuplicateDomains="true"/> >>> </global> >>> >>> >>> <namedCache name="qi4j"> >>> <jmxStatistics enabled="true"/> >>> <transaction >>> transactionManagerLookupClass="org.qi4j.entitystore.s3jclouds.AtomikosTransactionManagerLookup"/> >>> <clustering mode="distribution"> >>> <l1 enabled="true" lifespan="100000"/> >>> <hash numOwners="2" rehashRpcTimeout="120000"/> >>> </clustering> >>> >>> <loaders passivation="false" shared="true" preload="false"> >>> >>> <loader >>> class="org.infinispan...CloudCacheStore" >>> fetchPersistentState="false" ignoreModifications="false" >>> purgeOnStartup="false" purgeSynchronously="true"> >>> >>> <properties> >>> <property name="identity" value="***"/> >>> <property name="password" value="***"/> >>> <property name="bucketPrefix" value="store2"/> >>> <property name="cloudService" value="s3"/> >>> </properties> >>> </loader> >>> </loaders> >>> >>> <eviction strategy="LRU" wakeUpInterval="-1" maxEntries="1"/> >>> >>> <locking lockAcquisitionTimeout="60000" useLockStriping="true"/> >>> >>> >>> <unsafe unreliableReturnValues="true"/> >>> >>> </namedCache> >>> >>> </infinispan> >>> >>> >>> On Mon, May 17, 2010 at 1:55 PM, Manik Surtani <manik(a)jboss.org&gt; wrote: >>> What configuration do you use? >>> >>> On 17 May 2010, at 12:46, Philippe Van Dyck wrote: >>> >>> > FYI, I upgraded from ALPHA1 to BETA1 on a preproduction system this morning. >>> > >>> > Take a look at the graphic attached, the server is restarted everyday around 1 am (blue and green lines crossing). >>> > >>> > Users began to use the system around 9 am.... look at today's pattern and the previous day pattern ! >>> > >>> > Anything I should know or I missed ? >>> > >>> > cheers, >>> > >>> > phil >>> > <memleak.tiff>_______________________________________________ >>> > infinispan-dev mailing list >>> > infinispan-dev(a)lists.jboss.org >>> > https://lists.jboss.org/mailman/listinfo/infinispan-dev >>> >>> -- >>> Manik Surtani >>> manik(a)jboss.org >>> Lead, Infinispan >>> Lead, JBoss Cache >>> http://www.infinispan.org >>> http://www.jbosscache.org >>> >>> >>> >>> >>> >>> _______________________________________________ >>> infinispan-dev mailing list >>> infinispan-dev(a)lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>> >>> _______________________________________________ >>> infinispan-dev mailing list >>> infinispan-dev(a)lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >> >> -- >> Manik Surtani >> manik(a)jboss.org >> Lead, Infinispan >> Lead, JBoss Cache >> http://www.infinispan.org >> http://www.jbosscache.org >> >> >> >> >> >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev >> >> >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev > > -- > Manik Surtani > manik(a)jboss.org > Lead, Infinispan > Lead, JBoss Cache > http://www.infinispan.org > http://www.jbosscache.org > > > > > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev Vladimir Blagojevic JBoss Clustering Team JBoss by Red Hat _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

On 2010-05-17, at 9:54 AM, Manik Surtani wrote: > > On 17 May 2010, at 14:49, Philippe Van Dyck wrote: > >> Ok, working now. Thanks again Vladimir. The memory problem was surely coming from there (I will investigate it later) - I am back on BETA1. >> >> BTW, "maxEntries=-1" is not working anymore (update xml config doc ?) > > This is a bug. This should work - Vladimir, care to take this on? How should we handle this use case? Ignore eviction algorithm setting and use unbounded container?

We have to handle -1 case. I'll look into this. I am glad that you are exercising our new container with eviction. Keep pounding at it :) On 2010-05-17, at 9:49 AM, Philippe Van Dyck wrote: Ok, working now. Thanks again Vladimir. The memory problem was surely coming from there (I will investigate it later) - I am back on BETA1. BTW, "maxEntries=-1" is not working anymore (update xml config doc ?) Caused by: java.lang.IllegalArgumentException at org.infinispan.util.concurrent.BoundedConcurrentHashMap.<init>(BoundedConcurrentHashMap.java:1139) at org.infinispan.container.DefaultDataContainer.<init>(DefaultDataContainer.java:92) cheers, phil On Mon, May 17, 2010 at 3:30 PM, Philippe Van Dyck <pvdyck(a)gmail.com&gt;wrote: > Thanks Vladimir... > > May I suppose that this limitation was not verified in alpha1 ? > I will test this right away ! > > cheers, > > phil > > > On Mon, May 17, 2010 at 3:26 PM, Vladimir Blagojevic <vblagoje(a)redhat.com > > wrote: > >> I think the problem is related to the fact that you have maxEntries = 1 >> specified in configuration for your container. >> >> On 2010-05-17, at 9:24 AM, Philippe Van Dyck wrote: >> >> Confirmed - when I go back to alpha1 the problem disappears. >> >> Could anyone explain with alpha3 (the problem is already there) there is >> only one entry in getDataContainer ? >> >> for (InternalCacheEntry ice : >> cache.getAdvancedCache().getDataContainer()) { >> final int size = ((byte[]) ice.getValue()).length; >> logger.info("Cache entry size " + size); >> cacheSize += size; >> } >> >> logger.info("Cache size " + cacheSize); >> >> >> cheers >> >> phil >> >> On Mon, May 17, 2010 at 2:57 PM, Manik Surtani <manik(a)jboss.org&gt; wrote: >> >>> Wow, no idea. Any thread dumps, stack traces? Logging? >>> >>> On 17 May 2010, at 13:48, Philippe Van Dyck wrote: >>> >>> Update - trashed & crashed as planned. >>> Done some debugging : something strange... my cache seems to contain >>> only one entry (???) >>> Any clue ? >>> >>> phil >>> >>> On Mon, May 17, 2010 at 2:21 PM, Philippe Van Dyck <pvdyck(a)gmail.com&gt;wrote: >>> >>>> I don't have any resource available to setup profiling in prepod right >>>> now. >>>> Looking at the changes from alpha1 to beta1, I only see jclouds and >>>> some guava libs updated. >>>> Load on the server went berserk these 10 last minutes, it will probably >>>> trash & crash in the next hour. >>>> Will probably go back to ALPHA1. >>>> >>>> phil >>>> >>>> On Mon, May 17, 2010 at 2:13 PM, Manik Surtani <manik(a)jboss.org&gt;wrote: >>>> >>>>> Have you tried profiling stuff? Nothing really should have changed in >>>>> Beta1 to affect such a config, except perhaps the version of JClouds and >>>>> some JClouds-related code. >>>>> >>>>> On 17 May 2010, at 13:07, Philippe Van Dyck wrote: >>>>> >>>>> <?xml version="1.0" encoding="UTF-8"?> >>>>> >>>>> <infinispan xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" >>>>> xmlns="urn:infinispan:config:4.0"> >>>>> <global> >>>>> <transport >>>>> >>>>> transportClass="org.infinispan.remoting.transport.jgroups.JGroupsTransport"> >>>>> <properties> >>>>> <property name="configurationFile" >>>>> value="jgroupsprod.xml"/> >>>>> </properties> >>>>> </transport> >>>>> <globalJmxStatistics enabled="true" >>>>> allowDuplicateDomains="true"/> >>>>> </global> >>>>> >>>>> >>>>> <namedCache name="qi4j"> >>>>> <jmxStatistics enabled="true"/> >>>>> <transaction >>>>> >>>>> transactionManagerLookupClass="org.qi4j.entitystore.s3jclouds.AtomikosTransactionManagerLookup"/> >>>>> <clustering mode="distribution"> >>>>> <l1 enabled="true" lifespan="100000"/> >>>>> <hash numOwners="2" rehashRpcTimeout="120000"/> >>>>> </clustering> >>>>> >>>>> <loaders passivation="false" shared="true" preload="false"> >>>>> >>>>> <loader >>>>> class="org.infinispan...CloudCacheStore" >>>>> fetchPersistentState="false" >>>>> ignoreModifications="false" >>>>> purgeOnStartup="false" purgeSynchronously="true"> >>>>> >>>>> <properties> >>>>> <property name="identity" value="***"/> >>>>> <property name="password" value="***"/> >>>>> <property name="bucketPrefix" value="store2"/> >>>>> <property name="cloudService" value="s3"/> >>>>> </properties> >>>>> </loader> >>>>> </loaders> >>>>> >>>>> <eviction strategy="LRU" wakeUpInterval="-1" maxEntries="1"/> >>>>> >>>>> <locking lockAcquisitionTimeout="60000" useLockStriping="true"/> >>>>> >>>>> >>>>> <unsafe unreliableReturnValues="true"/> >>>>> >>>>> </namedCache> >>>>> >>>>> </infinispan> >>>>> >>>>> >>>>> On Mon, May 17, 2010 at 1:55 PM, Manik Surtani <manik(a)jboss.org&gt;wrote: >>>>> >>>>>> What configuration do you use? >>>>>> >>>>>> On 17 May 2010, at 12:46, Philippe Van Dyck wrote: >>>>>> >>>>>> > FYI, I upgraded from ALPHA1 to BETA1 on a preproduction system this >>>>>> morning. >>>>>> > >>>>>> > Take a look at the graphic attached, the server is restarted >>>>>> everyday around 1 am (blue and green lines crossing). >>>>>> > >>>>>> > Users began to use the system around 9 am.... look at today's >>>>>> pattern and the previous day pattern ! >>>>>> > >>>>>> > Anything I should know or I missed ? >>>>>> > >>>>>> > cheers, >>>>>> > >>>>>> > phil >>>>>> > <memleak.tiff>_______________________________________________ >>>>>> > infinispan-dev mailing list >>>>>> > infinispan-dev(a)lists.jboss.org >>>>>> > https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>>>> >>>>>> -- >>>>>> Manik Surtani >>>>>> manik(a)jboss.org >>>>>> Lead, Infinispan >>>>>> Lead, JBoss Cache >>>>>> http://www.infinispan.org >>>>>> http://www.jbosscache.org >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> infinispan-dev mailing list >>>>>> infinispan-dev(a)lists.jboss.org >>>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>>>> >>>>> >>>>> _______________________________________________ >>>>> infinispan-dev mailing list >>>>> infinispan-dev(a)lists.jboss.org >>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>>> >>>>> >>>>> -- >>>>> Manik Surtani >>>>> manik(a)jboss.org >>>>> Lead, Infinispan >>>>> Lead, JBoss Cache >>>>> http://www.infinispan.org >>>>> http://www.jbosscache.org >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> infinispan-dev mailing list >>>>> infinispan-dev(a)lists.jboss.org >>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>>> >>>> >>>> >>> _______________________________________________ >>> infinispan-dev mailing list >>> infinispan-dev(a)lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>> >>> >>> -- >>> Manik Surtani >>> manik(a)jboss.org >>> Lead, Infinispan >>> Lead, JBoss Cache >>> http://www.infinispan.org >>> http://www.jbosscache.org >>> >>> >>> >>> >>> >>> _______________________________________________ >>> infinispan-dev mailing list >>> infinispan-dev(a)lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>> >> >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev >> >> >> Vladimir Blagojevic >> JBoss Clustering Team >> JBoss by Red Hat >> >> >> >> >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev >> > > _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

How did you do this? Did you write your own EvictionManager. Notice that a contract between EvictionManager and DataContainer changed slightly in Alpha2. DataContainer fulfils eviction contract with EvictionManager by letting EvictionManager know the evicted entries (DataContainer#getEvictionCandidates). You can also select eviction thread policy (DEFAULT or PIGGYBACK)[1]. Have a look at [2] for more details. [1] http://docs.jboss.org/infinispan/4.1/apidocs/config.html#ce_default_eviction [2] http://infinispan.blogspot.com/2010/03/infinispan-eviction-batching-updat... On 2010-05-17, at 10:03 AM, Philippe Van Dyck wrote: Well... to be honest, I am not ;-) I run my own eviction trigger (based on the total cache size) and I evict any entry older than 20 minutes until the size of the cache is reduced enough (usually 2%). Since I need the 'lastUsed' value, I need the LRU strategy. phil On Mon, May 17, 2010 at 3:55 PM, Vladimir Blagojevic <vblagoje(a)redhat.com&gt;wrote: > We have to handle -1 case. I'll look into this. I am glad that you are > exercising our new container with eviction. Keep pounding at it :) > > On 2010-05-17, at 9:49 AM, Philippe Van Dyck wrote: > > Ok, working now. Thanks again Vladimir. The memory problem was surely > coming from there (I will investigate it later) - I am back on BETA1. > > BTW, "maxEntries=-1" is not working anymore (update xml config doc ?) > > Caused by: java.lang.IllegalArgumentException > at > org.infinispan.util.concurrent.BoundedConcurrentHashMap.<init>(BoundedConcurrentHashMap.java:1139) > at > org.infinispan.container.DefaultDataContainer.<init>(DefaultDataContainer.java:92) > > cheers, > > phil > > On Mon, May 17, 2010 at 3:30 PM, Philippe Van Dyck <pvdyck(a)gmail.com&gt;wrote: > >> Thanks Vladimir... >> >> May I suppose that this limitation was not verified in alpha1 ? >> I will test this right away ! >> >> cheers, >> >> phil >> >> >> On Mon, May 17, 2010 at 3:26 PM, Vladimir Blagojevic < >> vblagoje(a)redhat.com&gt; wrote: >> >>> I think the problem is related to the fact that you have maxEntries = 1 >>> specified in configuration for your container. >>> >>> On 2010-05-17, at 9:24 AM, Philippe Van Dyck wrote: >>> >>> Confirmed - when I go back to alpha1 the problem disappears. >>> >>> Could anyone explain with alpha3 (the problem is already there) there is >>> only one entry in getDataContainer ? >>> >>> for (InternalCacheEntry ice : >>> cache.getAdvancedCache().getDataContainer()) { >>> final int size = ((byte[]) ice.getValue()).length; >>> logger.info("Cache entry size " + size); >>> cacheSize += size; >>> } >>> >>> logger.info("Cache size " + cacheSize); >>> >>> >>> cheers >>> >>> phil >>> >>> On Mon, May 17, 2010 at 2:57 PM, Manik Surtani <manik(a)jboss.org&gt; wrote: >>> >>>> Wow, no idea. Any thread dumps, stack traces? Logging? >>>> >>>> On 17 May 2010, at 13:48, Philippe Van Dyck wrote: >>>> >>>> Update - trashed & crashed as planned. >>>> Done some debugging : something strange... my cache seems to contain >>>> only one entry (???) >>>> Any clue ? >>>> >>>> phil >>>> >>>> On Mon, May 17, 2010 at 2:21 PM, Philippe Van Dyck <pvdyck(a)gmail.com&gt;wrote: >>>> >>>>> I don't have any resource available to setup profiling in prepod right >>>>> now. >>>>> Looking at the changes from alpha1 to beta1, I only see jclouds and >>>>> some guava libs updated. >>>>> Load on the server went berserk these 10 last minutes, it will >>>>> probably trash & crash in the next hour. >>>>> Will probably go back to ALPHA1. >>>>> >>>>> phil >>>>> >>>>> On Mon, May 17, 2010 at 2:13 PM, Manik Surtani <manik(a)jboss.org&gt;wrote: >>>>> >>>>>> Have you tried profiling stuff? Nothing really should have changed >>>>>> in Beta1 to affect such a config, except perhaps the version of JClouds and >>>>>> some JClouds-related code. >>>>>> >>>>>> On 17 May 2010, at 13:07, Philippe Van Dyck wrote: >>>>>> >>>>>> <?xml version="1.0" encoding="UTF-8"?> >>>>>> >>>>>> <infinispan xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" >>>>>> xmlns="urn:infinispan:config:4.0"> >>>>>> <global> >>>>>> <transport >>>>>> >>>>>> transportClass="org.infinispan.remoting.transport.jgroups.JGroupsTransport"> >>>>>> <properties> >>>>>> <property name="configurationFile" >>>>>> value="jgroupsprod.xml"/> >>>>>> </properties> >>>>>> </transport> >>>>>> <globalJmxStatistics enabled="true" >>>>>> allowDuplicateDomains="true"/> >>>>>> </global> >>>>>> >>>>>> >>>>>> <namedCache name="qi4j"> >>>>>> <jmxStatistics enabled="true"/> >>>>>> <transaction >>>>>> >>>>>> transactionManagerLookupClass="org.qi4j.entitystore.s3jclouds.AtomikosTransactionManagerLookup"/> >>>>>> <clustering mode="distribution"> >>>>>> <l1 enabled="true" lifespan="100000"/> >>>>>> <hash numOwners="2" rehashRpcTimeout="120000"/> >>>>>> </clustering> >>>>>> >>>>>> <loaders passivation="false" shared="true" preload="false"> >>>>>> >>>>>> <loader >>>>>> class="org.infinispan...CloudCacheStore" >>>>>> fetchPersistentState="false" >>>>>> ignoreModifications="false" >>>>>> purgeOnStartup="false" purgeSynchronously="true"> >>>>>> >>>>>> <properties> >>>>>> <property name="identity" value="***"/> >>>>>> <property name="password" value="***"/> >>>>>> <property name="bucketPrefix" value="store2"/> >>>>>> <property name="cloudService" value="s3"/> >>>>>> </properties> >>>>>> </loader> >>>>>> </loaders> >>>>>> >>>>>> <eviction strategy="LRU" wakeUpInterval="-1" maxEntries="1"/> >>>>>> >>>>>> <locking lockAcquisitionTimeout="60000" >>>>>> useLockStriping="true"/> >>>>>> >>>>>> >>>>>> <unsafe unreliableReturnValues="true"/> >>>>>> >>>>>> </namedCache> >>>>>> >>>>>> </infinispan> >>>>>> >>>>>> >>>>>> On Mon, May 17, 2010 at 1:55 PM, Manik Surtani <manik(a)jboss.org&gt;wrote: >>>>>> >>>>>>> What configuration do you use? >>>>>>> >>>>>>> On 17 May 2010, at 12:46, Philippe Van Dyck wrote: >>>>>>> >>>>>>> > FYI, I upgraded from ALPHA1 to BETA1 on a preproduction system >>>>>>> this morning. >>>>>>> > >>>>>>> > Take a look at the graphic attached, the server is restarted >>>>>>> everyday around 1 am (blue and green lines crossing). >>>>>>> > >>>>>>> > Users began to use the system around 9 am.... look at today's >>>>>>> pattern and the previous day pattern ! >>>>>>> > >>>>>>> > Anything I should know or I missed ? >>>>>>> > >>>>>>> > cheers, >>>>>>> > >>>>>>> > phil >>>>>>> > <memleak.tiff>_______________________________________________ >>>>>>> > infinispan-dev mailing list >>>>>>> > infinispan-dev(a)lists.jboss.org >>>>>>> > https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>>>>> >>>>>>> -- >>>>>>> Manik Surtani >>>>>>> manik(a)jboss.org >>>>>>> Lead, Infinispan >>>>>>> Lead, JBoss Cache >>>>>>> http://www.infinispan.org >>>>>>> http://www.jbosscache.org >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> infinispan-dev mailing list >>>>>>> infinispan-dev(a)lists.jboss.org >>>>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> infinispan-dev mailing list >>>>>> infinispan-dev(a)lists.jboss.org >>>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>>>> >>>>>> >>>>>> -- >>>>>> Manik Surtani >>>>>> manik(a)jboss.org >>>>>> Lead, Infinispan >>>>>> Lead, JBoss Cache >>>>>> http://www.infinispan.org >>>>>> http://www.jbosscache.org >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> infinispan-dev mailing list >>>>>> infinispan-dev(a)lists.jboss.org >>>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>>>> >>>>> >>>>> >>>> _______________________________________________ >>>> infinispan-dev mailing list >>>> infinispan-dev(a)lists.jboss.org >>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>> >>>> >>>> -- >>>> Manik Surtani >>>> manik(a)jboss.org >>>> Lead, Infinispan >>>> Lead, JBoss Cache >>>> http://www.infinispan.org >>>> http://www.jbosscache.org >>>> >>>> >>>> >>>> >>>> >>>> _______________________________________________ >>>> infinispan-dev mailing list >>>> infinispan-dev(a)lists.jboss.org >>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>> >>> >>> _______________________________________________ >>> infinispan-dev mailing list >>> infinispan-dev(a)lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>> >>> >>> Vladimir Blagojevic >>> JBoss Clustering Team >>> JBoss by Red Hat >>> >>> >>> >>> >>> _______________________________________________ >>> infinispan-dev mailing list >>> infinispan-dev(a)lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>> >> >> > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev > > > > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev > _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev Vladimir Blagojevic JBoss Clustering Team JBoss by Red Hat _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev