Re: [infinispan-dev] ISPN-863 - Thoughts / Questions

It was my problem. If -1 is set for maxEntries, then I am creating a BoundedConcurrentHashMap with a size of Integer.MAX_VALUE. Then when I ran the tests all of them ran out of memory. I should have looked at the stack trace a little more closely. I am going to have to put logic in so that the maximum capacity is determined by the amount of free memory when the cache is created. Previously, if -1 was set for maxEntries, a ConcurrentHashMap was created with maximum capacity of Integer.MAX_VALUE. I have not looked at the code, but my guess is that it works because it grows over time and does not pre-allocate all structures ahead of time.

Any thoughts on logic for determining maximum capacity at creation time?

> From: galder(a)redhat.com > Date: Wed, 19 Jan 2011 10:32:42 +0100 > To: infinispan-dev(a)lists.jboss.org > Subject: Re: [infinispan-dev] ISPN-863 - Thoughts / Questions > > Out of curiosity, what is a code problem on your side? Or a set up issue? > > On Jan 19, 2011, at 4:29 AM, david marion wrote: > > > Disregard, I found the issue. Sorry for the spam. > > > > > From: dlmarion(a)hotmail.com > > > To: infinispan-dev(a)lists.jboss.org > > > Date: Tue, 18 Jan 2011 20:39:21 -0500 > > > Subject: Re: [infinispan-dev] ISPN-863 - Thoughts / Questions > > > > > > Hah, might be because the POM is configured to run tests in parallel with a > > > max heap size of 1GB. Can someone update > > > http://community.jboss.org/wiki/ParallelTestSuite or > > > http://community.jboss.org/wiki/InfinispanQuickStartGuideforDevelopers#Testi > > > ng with instructions on how to run test serially (I didn’t find anything wrt > > > Maven surefire plugin)? Thanks > > > > > > -----Original Message----- > > > From: infinispan-dev-bounces(a)lists.jboss.org > > > [mailto:infinispan-dev-bounces@lists.jboss.org] On Behalf Of Galder > > > Zamarreño > > > Sent: Tuesday, January 18, 2011 6:12 AM > > > To: infinispan -Dev List > > > Subject: Re: [infinispan-dev] ISPN-863 - Thoughts / Questions > > > > > > Hmmm, try increasing -Xmx? Start with -XX:+HeapDumpOnOutOfMemoryError > > > -XX:HeapDumpPath=/tmp/java_heap and inspect the dump with Eclipse MAT?... > > > > > > On Jan 17, 2011, at 4:51 PM, david marion wrote: > > > > > > > same result. > > > > > > > > Date: Mon, 17 Jan 2011 11:54:15 -0300 > > > > From: vblagoje(a)redhat.com > > > > To: infinispan-dev(a)lists.jboss.org > > > > Subject: Re: [infinispan-dev] ISPN-863 - Thoughts / Questions > > > > > > > > Its easier to run everything from command line for two reason I can think > > > of: > > > > - everyone has same env to compare results with you > > > > - it is all ready to go > > > > > > > > See http://community.jboss.org/wiki/InfinispanandMaven > > > > > > > > On 11-01-17 11:38 AM, Dave wrote: > > > > All, > > > > > > > > I am trying to test some changes that I made using Eclipse and the TestNG > > > plugin. I am not familiar with TestNG. Are there any settings that I should > > > make? Any ideas? > > > > > > > > java.lang.OutOfMemoryError: Java heap space > > > > at > > > org.infinispan.util.concurrent.BoundedConcurrentHashMap$HashEntry.newArray(B > > > oundedConcurrentHashMap.java:295) > > > > at > > > org.infinispan.util.concurrent.BoundedConcurrentHashMap$Segment.<init>(Bound > > > edConcurrentHashMap.java:898) > > > > at > > > org.infinispan.util.concurrent.BoundedConcurrentHashMap.<init>(BoundedConcur > > > rentHashMap.java:1367) > > > > at > > > org.infinispan.container.DefaultDataContainer.<init>(DefaultDataContainer.ja > > > va:78) > > > > at > > > org.infinispan.container.DefaultDataContainer.<init>(DefaultDataContainer.ja > > > va:47) > > > > at > > > org.infinispan.container.DefaultDataContainer.unBoundedDataContainer(Default > > > DataContainer.java:93) > > > > at > > > org.infinispan.factories.DataContainerFactory.construct(DataContainerFactory > > > .java:53) > > > > at > > > org.infinispan.factories.AbstractComponentRegistry.getOrCreateComponent(Abst > > > ractComponentRegistry.java:315) > > > > at > > > org.infinispan.factories.AbstractComponentRegistry.invokeInjectionMethod(Abs > > > tractComponentRegistry.java:251) > > > > at > > > org.infinispan.factories.AbstractComponentRegistry$Component.injectDependenc > > > ies(AbstractComponentRegistry.java:840) > > > > at > > > org.infinispan.factories.AbstractComponentRegistry.registerComponent(Abstrac > > > tComponentRegistry.java:225) > > > > at > > > org.infinispan.factories.ComponentRegistry.registerComponent(ComponentRegist > > > ry.java:120) > > > > at > > > org.infinispan.factories.AbstractComponentRegistry.registerComponent(Abstrac > > > tComponentRegistry.java:192) > > > > at > > > org.infinispan.factories.AbstractComponentRegistry.getOrCreateComponent(Abst > > > ractComponentRegistry.java:323) > > > > at > > > org.infinispan.factories.AbstractComponentRegistry.invokeInjectionMethod(Abs > > > tractComponentRegistry.java:251) > > > > at > > > org.infinispan.factories.AbstractComponentRegistry$Component.injectDependenc > > > ies(AbstractComponentRegistry.java:840) > > > > at > > > org.infinispan.factories.AbstractComponentRegistry.registerComponent(Abstrac > > > tComponentRegistry.java:225) > > > > at > > > org.infinispan.factories.ComponentRegistry.registerComponent(ComponentRegist > > > ry.java:120) > > > > at > > > org.infinispan.factories.AbstractComponentRegistry.registerComponent(Abstrac > > > tComponentRegistry.java:192) > > > > at > > > org.infinispan.factories.AbstractComponentRegistry.getOrCreateComponent(Abst > > > ractComponentRegistry.java:323) > > > > at > > > org.infinispan.factories.AbstractComponentRegistry.invokeInjectionMethod(Abs > > > tractComponentRegistry.java:251) > > > > at > > > org.infinispan.factories.AbstractComponentRegistry$Component.injectDependenc > > > ies(AbstractComponentRegistry.java:840) > > > > at > > > org.infinispan.factories.AbstractComponentRegistry.registerComponent(Abstrac > > > tComponentRegistry.java:225) > > > > at > > > org.infinispan.factories.ComponentRegistry.registerComponent(ComponentRegist > > > ry.java:120) > > > > at > > > org.infinispan.factories.AbstractComponentRegistry.registerComponent(Abstrac > > > tComponentRegistry.java:192) > > > > at > > > org.infinispan.factories.InternalCacheFactory.bootstrap(InternalCacheFactory > > > .java:92) > > > > at > > > org.infinispan.factories.InternalCacheFactory.createAndWire(InternalCacheFac > > > tory.java:78) > > > > at > > > org.infinispan.factories.InternalCacheFactory.createCache(InternalCacheFacto > > > ry.java:62) > > > > at > > > org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.j > > > ava:510) > > > > at > > > org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java > > > :440) > > > > at > > > org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java > > > :409) > > > > at > > > org.infinispan.config.ConfigurationValidationTest.testDefaultMemoryGuardConf > > > iguration(ConfigurationValidationTest.java:100) > > > > > > > > > > > > > > > > -----Original Message----- > > > > From: infinispan-dev-bounces(a)lists.jboss.org > > > [mailto:infinispan-dev-bounces@lists.jboss.org] On Behalf Of Vladimir > > > Blagojevic > > > > Sent: Friday, January 14, 2011 11:05 AM > > > > To: infinispan -Dev List > > > > Subject: Re: [infinispan-dev] ISPN-863 - Thoughts / Questions > > > > > > > > Hey David, > > > > > > > > First of all kudos for taking on such a non-trivial task! > > > > > > > > On 11-01-14 12:22 AM, david marion wrote: > > > > 1. Modify configuration in some way so that the following can be > > > specified: > > > > a. The percentage value of used JVM memory (i.e. 95) at which > > > entries should be evicted to try and avoid an OOM error. > > > > b. The number of items that should be evicted when memory reaches > > > this threshold > > > > 2. Modify LRU and LIRS Eviction class so that the accessQueue member > > > can be accessed by the new Eviction class so that two access queues don’t > > > have to be maintained. > > > > 3. Create a new Eviction class, a subclass of LIRS, where the > > > accessQueue is used from the Eviction strategy the user specifies and the > > > for loop in the execute method is exited when the evicted set equals value > > > from 1.b above. > > > > 4. Modify DataContainerFactory.construct() to call > > > DefaultDataContainer.boundedDataContainer() regardless of eviction policy. > > > This will always create a BoundedConcurrentHashMap > > > > > > > > I don't think you have to extend LRU and LIRS, as long as you implement > > > EvictionPolicy you are fullfilling the contract. What extending LRU and LIRS > > > can do is give you some order in selecting proper elements for eviction. I > > > think the precise technical term in research literature is "eviction > > > precision". > > > > > > > > > > > > 1. > > > > 2. Create an instance of the new Eviction class in each segment. > > > > 3. Modify BoundedConcurrentHashMap.Segment put and replace methods > > > such that when new values are going to be put into the Segment, the memory > > > usage is checked and the execute method is called on the new Eviction class. > > > > > > > > Sounds right! > > > > > > > > > > > > 1. > > > > > > > > Questions: > > > > > > > > 1. What are the implications of using a BoundedConcurrentHashMap > > > instead of a ConcurrentHashMap when maxEntries is set to -1? > > > > > > > > I think none except you turn on eviction by using > > > BoundedConcurrentHashMap. > > > > > > > > > > > > 1. > > > > > > > > Thoughts > > > > > > > > 1. This will not guarantee that an OOM error does not occur. It will > > > attempt to guard against an OOM caused by putting new values into the cache. > > > This will probably be more effective when the cache is being used in > > > client/server mode, and less effective when used in embedded mode as to > > > other code running in the JVM. > > > > > > > > > > > > -- Dave Marion > > > > > > > > > > > > Cheers, > > > > Vladimir > > > > > > > > _______________________________________________ > > > > infinispan-dev mailing list > > > > > > > > infinispan-dev(a)lists.jboss.org > > > > https://lists.jboss.org/mailman/listinfo/infinispan-dev > > > > > > > > > > > > _______________________________________________ infinispan-dev mailing > > > list infinispan-dev(a)lists.jboss.org > > > https://lists.jboss.org/mailman/listinfo/infinispan-dev_____________________ > > > __________________________ > > > > infinispan-dev mailing list > > > > infinispan-dev(a)lists.jboss.org > > > > https://lists.jboss.org/mailman/listinfo/infinispan-dev > > > > > > -- > > > Galder Zamarreño > > > Sr. Software Engineer > > > Infinispan, JBoss Cache > > > > > > > > > _______________________________________________ > > > infinispan-dev mailing list > > > infinispan-dev(a)lists.jboss.org > > > https://lists.jboss.org/mailman/listinfo/infinispan-dev > > > > > > > > > _______________________________________________ > > > infinispan-dev mailing list > > > infinispan-dev(a)lists.jboss.org > > > https://lists.jboss.org/mailman/listinfo/infinispan-dev > > _______________________________________________ > > infinispan-dev mailing list > > infinispan-dev(a)lists.jboss.org > > https://lists.jboss.org/mailman/listinfo/infinispan-dev > > -- > Galder Zamarreño > Sr. Software Engineer > Infinispan, JBoss Cache > > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev