1:34 a.m.
Manik,
I am on the list. I will start getting my dev environment setup according to instructions
on site.
Dave Marion
12:08 p.m.
Excellent, Dave.
You should probably come up with a design first - or maybe a prototype on your fork of the
project on GitHub - and post it here for comment.
Cheers
Manik
On 8 Jan 2011, at 00:34, david marion wrote:
Manik,
I am on the list. I will start getting my dev environment setup according to instructions
on site.
Dave Marion
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
--
Manik Surtani
manik(a)jboss.org
twitter.com/maniksurtani
Lead, Infinispan
http://www.infinispan.org
12:46 p.m.
New subject: Poor man's size-based eviction Re: ISPN-863
Hi Dave
Just saw that you signed the contributor agreement.
So yeah, what you may want to look into is the EvictionManager [1] which will periodically
have its processEviction() [2] method called. This doesn't actually do any evictions
here (named so for legacy reasons) since the current data container is
self-sizing/evicting. However it is still periodically invoked to purge expired entries.
You can piggyback on the same invocation to test upper and lower bounds of memory
utilisation (if enabled), but the bit that I still think you need to figure out is
deciding which entries to evict, and how many of them to evict.
Cheers
Manik
[1]
https://github.com/infinispan/infinispan/blob/master/core/src/main/java/o...
[2]
https://github.com/infinispan/infinispan/blob/master/core/src/main/java/o...
On 10 Jan 2011, at 11:08, Manik Surtani wrote:
Excellent, Dave.
You should probably come up with a design first - or maybe a prototype on your fork of
the project on GitHub - and post it here for comment.
Cheers
Manik
On 8 Jan 2011, at 00:34, david marion wrote:
> Manik,
>
> I am on the list. I will start getting my dev environment setup according to
instructions on site.
>
> Dave Marion
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
--
Manik Surtani
manik(a)jboss.org
twitter.com/maniksurtani
Lead, Infinispan
http://www.infinispan.org
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
--
Manik Surtani
manik(a)jboss.org
twitter.com/maniksurtani
Lead, Infinispan
http://www.infinispan.org
4:22 a.m.
New subject: ISPN-863 - Thoughts / Questions
I have been looking at the code and think I have a starting point on the design. Remember,
the goal here is to be able to create a cache that uses expiration but has no defined
cache size limit. Reading through the code, if ‘-1’ is specified for the eviction
maxEntries, then a ConcurrentHashMap is created and used as the data container and no
eviction is performed. Otherwise a BoundedConcurrentHashMap is created and used as the
data container and eviction is performed according to the specified strategy.
Eviction currently appears to happen in two different ways:
1. If an entry is expired, then it can be evicted.
2. If the number of entries in a segment of the cache contains more entries than
“allowed” by the configuration.
Here are the modifications that I think need to be made. I have not made any code mods
yet, so I don’t know for sure if this will work. Definitely looking for feedback.
Modify configuration in some way so that the following can be specified:
The percentage value of used JVM memory (i.e. 95) at which entries should be evicted to
try and avoid an OOM error.
The number of items that should be evicted when memory reaches this threshold
Modify LRU and LIRS Eviction class so that the accessQueue member can be accessed by the
new Eviction class so that two access queues don’t have to be maintained.
Create a new Eviction class, a subclass of LIRS, where the accessQueue is used from the
Eviction strategy the user specifies and the for loop in the execute method is exited when
the evicted set equals value from 1.b above.
Modify DataContainerFactory.construct() to call
DefaultDataContainer.boundedDataContainer() regardless of eviction policy. This will
always create a BoundedConcurrentHashMap
Create an instance of the new Eviction class in each segment.
Modify BoundedConcurrentHashMap.Segment put and replace methods such that when new values
are going to be put into the Segment, the memory usage is checked and the execute method
is called on the new Eviction class.
Questions:
What are the implications of using a BoundedConcurrentHashMap instead of a
ConcurrentHashMap when maxEntries is set to -1?
Thoughts
This will not guarantee that an OOM error does not occur. It will attempt to guard against
an OOM caused by putting new values into the cache. This will probably be more effective
when the cache is being used in client/server mode, and less effective when used in
embedded mode as to other code running in the JVM.
-- Dave Marion
From: manik(a)jboss.org
Date: Tue, 11 Jan 2011 11:46:40 +0000
To: infinispan-dev(a)lists.jboss.org
Subject: [infinispan-dev] Poor man's size-based eviction Re: ISPN-863
Hi Dave
Just saw that you signed the contributor agreement.
So yeah, what you may want to look into is the EvictionManager [1] which will periodically
have its processEviction() [2] method called. This doesn't actually do any evictions
here (named so for legacy reasons) since the current data container is
self-sizing/evicting. However it is still periodically invoked to purge expired entries.
You can piggyback on the same invocation to test upper and lower bounds of memory
utilisation (if enabled), but the bit that I still think you need to figure out is
deciding which entries to evict, and how many of them to evict.
Cheers
Manik
[1]
https://github.com/infinispan/infinispan/blob/master/core/src/main/java/o...
[2]
https://github.com/infinispan/infinispan/blob/master/core/src/main/java/o...
On 10 Jan 2011, at 11:08, Manik Surtani wrote:
Excellent, Dave.
You should probably come up with a design first - or maybe a prototype on your fork of the
project on GitHub - and post it here for comment.
Cheers
Manik
On 8 Jan 2011, at 00:34, david marion wrote:
Manik,
I am on the list. I will start getting my dev environment setup according to instructions
on site.
Dave Marion _______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
--
Manik Surtani
manik(a)jboss.org
twitter.com/maniksurtani
Lead, Infinispan
http://www.infinispan.org
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
--
Manik Surtani
manik(a)jboss.org
twitter.com/maniksurtani
Lead, Infinispan
http://www.infinispan.org
_______________________________________________ infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev
9:40 a.m.
New subject: ISPN-863 - Thoughts / Questions
Excellent, memory based eviction is very important ! This has been on
the todo list for years !
One thing you might want to think about is - *in addition* to what you
say below - whether you want to add an OOME handler, which evits on OOME
caused by putXXX() methods.
For example, you could wrap all methods which add to the map to catch
OOMEs. When this is the case, kick off the eviction handler to evict
elements immediately. Kind of like a second line of defense against OOMEs...
WDYT ?
On 1/14/11 4:22 AM, david marion wrote:
I have been looking at the code and think I have a starting point on the design.
Remember, the goal here is to be able to create a cache that uses expiration but has no
defined cache size limit. Reading through the code, if ‘-1’ is specified for the eviction
maxEntries, then a ConcurrentHashMap is created and used as the data container and no
eviction is performed. Otherwise a BoundedConcurrentHashMap is created and used as the
data container and eviction is performed according to the specified strategy.
Eviction currently appears to happen in two different ways:
1. If an entry is expired, then it can be evicted.
2. If the number of entries in a segment of the cache contains more entries than
“allowed” by the configuration.
Here are the modifications that I think need to be made. I have not made any code mods
yet, so I don’t know for sure if this will work. Definitely looking for feedback.
Modify configuration in some way so that the following can be specified:
The percentage value of used JVM memory (i.e. 95) at which entries should be evicted to
try and avoid an OOM error.
The number of items that should be evicted when memory reaches this threshold
Modify LRU and LIRS Eviction class so that the accessQueue member can be accessed by the
new Eviction class so that two access queues don’t have to be maintained.
Create a new Eviction class, a subclass of LIRS, where the accessQueue is used from the
Eviction strategy the user specifies and the for loop in the execute method is exited when
the evicted set equals value from 1.b above.
Modify DataContainerFactory.construct() to call
DefaultDataContainer.boundedDataContainer() regardless of eviction policy. This will
always create a BoundedConcurrentHashMap
Create an instance of the new Eviction class in each segment.
Modify BoundedConcurrentHashMap.Segment put and replace methods such that when new values
are going to be put into the Segment, the memory usage is checked and the execute method
is called on the new Eviction class.
Questions:
What are the implications of using a BoundedConcurrentHashMap instead of a
ConcurrentHashMap when maxEntries is set to -1?
Thoughts
This will not guarantee that an OOM error does not occur. It will attempt to guard
against an OOM caused by putting new values into the cache. This will probably be more
effective when the cache is being used in client/server mode, and less effective when used
in embedded mode as to other code running in the JVM.
--
Bela Ban
Lead JGroups / Clustering Team
JBoss
12:40 p.m.
New subject: ISPN-863 - Thoughts / Questions
Nice, I like the "2nd level of defence" here.
On 14 Jan 2011, at 08:40, Bela Ban wrote:
Excellent, memory based eviction is very important ! This has been on
the todo list for years !
One thing you might want to think about is - *in addition* to what you
say below - whether you want to add an OOME handler, which evits on OOME
caused by putXXX() methods.
For example, you could wrap all methods which add to the map to catch
OOMEs. When this is the case, kick off the eviction handler to evict
elements immediately. Kind of like a second line of defense against OOMEs...
WDYT ?
On 1/14/11 4:22 AM, david marion wrote:
>
> I have been looking at the code and think I have a starting point on the design.
Remember, the goal here is to be able to create a cache that uses expiration but has no
defined cache size limit. Reading through the code, if ‘-1’ is specified for the eviction
maxEntries, then a ConcurrentHashMap is created and used as the data container and no
eviction is performed. Otherwise a BoundedConcurrentHashMap is created and used as the
data container and eviction is performed according to the specified strategy.
>
> Eviction currently appears to happen in two different ways:
>
> 1. If an entry is expired, then it can be evicted.
> 2. If the number of entries in a segment of the cache contains more entries
than “allowed” by the configuration.
>
> Here are the modifications that I think need to be made. I have not made any code
mods yet, so I don’t know for sure if this will work. Definitely looking for feedback.
>
>
> Modify configuration in some way so that the following can be specified:
>
> The percentage value of used JVM memory (i.e. 95) at which entries should be evicted
to try and avoid an OOM error.
> The number of items that should be evicted when memory reaches this threshold
> Modify LRU and LIRS Eviction class so that the accessQueue member can be accessed by
the new Eviction class so that two access queues don’t have to be maintained.
> Create a new Eviction class, a subclass of LIRS, where the accessQueue is used from
the Eviction strategy the user specifies and the for loop in the execute method is exited
when the evicted set equals value from 1.b above.
> Modify DataContainerFactory.construct() to call
DefaultDataContainer.boundedDataContainer() regardless of eviction policy. This will
always create a BoundedConcurrentHashMap
> Create an instance of the new Eviction class in each segment.
> Modify BoundedConcurrentHashMap.Segment put and replace methods such that when new
values are going to be put into the Segment, the memory usage is checked and the execute
method is called on the new Eviction class.
>
> Questions:
>
>
> What are the implications of using a BoundedConcurrentHashMap instead of a
ConcurrentHashMap when maxEntries is set to -1?
>
> Thoughts
>
>
> This will not guarantee that an OOM error does not occur. It will attempt to guard
against an OOM caused by putting new values into the cache. This will probably be more
effective when the cache is being used in client/server mode, and less effective when used
in embedded mode as to other code running in the JVM.
--
Bela Ban
Lead JGroups / Clustering Team
JBoss
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
--
Manik Surtani
manik(a)jboss.org
twitter.com/maniksurtani
Lead, Infinispan
http://www.infinispan.org
3:08 p.m.
New subject: ISPN-863 - Thoughts / Questions
-1
If you get an OOME, chances are the system is a pretty bad shape and you can't really
do much else at that point.
On Jan 14, 2011, at 9:40 AM, Bela Ban wrote:
Excellent, memory based eviction is very important ! This has been on
the todo list for years !
One thing you might want to think about is - *in addition* to what you
say below - whether you want to add an OOME handler, which evits on OOME
caused by putXXX() methods.
For example, you could wrap all methods which add to the map to catch
OOMEs. When this is the case, kick off the eviction handler to evict
elements immediately. Kind of like a second line of defense against OOMEs...
WDYT ?
On 1/14/11 4:22 AM, david marion wrote:
>
> I have been looking at the code and think I have a starting point on the design.
Remember, the goal here is to be able to create a cache that uses expiration but has no
defined cache size limit. Reading through the code, if ‘-1’ is specified for the eviction
maxEntries, then a ConcurrentHashMap is created and used as the data container and no
eviction is performed. Otherwise a BoundedConcurrentHashMap is created and used as the
data container and eviction is performed according to the specified strategy.
>
> Eviction currently appears to happen in two different ways:
>
> 1. If an entry is expired, then it can be evicted.
> 2. If the number of entries in a segment of the cache contains more entries
than “allowed” by the configuration.
>
> Here are the modifications that I think need to be made. I have not made any code
mods yet, so I don’t know for sure if this will work. Definitely looking for feedback.
>
>
> Modify configuration in some way so that the following can be specified:
>
> The percentage value of used JVM memory (i.e. 95) at which entries should be evicted
to try and avoid an OOM error.
> The number of items that should be evicted when memory reaches this threshold
> Modify LRU and LIRS Eviction class so that the accessQueue member can be accessed by
the new Eviction class so that two access queues don’t have to be maintained.
> Create a new Eviction class, a subclass of LIRS, where the accessQueue is used from
the Eviction strategy the user specifies and the for loop in the execute method is exited
when the evicted set equals value from 1.b above.
> Modify DataContainerFactory.construct() to call
DefaultDataContainer.boundedDataContainer() regardless of eviction policy. This will
always create a BoundedConcurrentHashMap
> Create an instance of the new Eviction class in each segment.
> Modify BoundedConcurrentHashMap.Segment put and replace methods such that when new
values are going to be put into the Segment, the memory usage is checked and the execute
method is called on the new Eviction class.
>
> Questions:
>
>
> What are the implications of using a BoundedConcurrentHashMap instead of a
ConcurrentHashMap when maxEntries is set to -1?
>
> Thoughts
>
>
> This will not guarantee that an OOM error does not occur. It will attempt to guard
against an OOM caused by putting new values into the cache. This will probably be more
effective when the cache is being used in client/server mode, and less effective when used
in embedded mode as to other code running in the JVM.
--
Bela Ban
Lead JGroups / Clustering Team
JBoss
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
--
Galder Zamarreño
Sr. Software Engineer
Infinispan, JBoss Cache
3:29 p.m.
New subject: ISPN-863 - Thoughts / Questions
On 1/14/11 3:08 PM, Galder Zamarreño wrote:
-1
If you get an OOME, chances are the system is a pretty bad shape and you can't really
do much else at that point.
If a put() is the culprit leading to the OOME, we can definitely do
something !
--
Bela Ban
Lead JGroups / Clustering Team
JBoss
3:35 p.m.
New subject: ISPN-863 - Thoughts / Questions
On 14 Jan 2011, at 14:29, Bela Ban wrote:
On 1/14/11 3:08 PM, Galder Zamarreño wrote:
> -1
>
> If you get an OOME, chances are the system is a pretty bad shape and you can't
really do much else at that point.
If a put() is the culprit leading to the OOME, we can definitely do
something !
Galder is right though, even if the put() is the culprit, concurrent threads elsewhere
(out of our control) would also be seeing OOMs and even if we free up memory by evicting,
other threads may be in an unstable state.
--
Manik Surtani
manik(a)jboss.org
twitter.com/maniksurtani
Lead, Infinispan
http://www.infinispan.org
3:54 p.m.
New subject: ISPN-863 - Thoughts / Questions
On 1/14/11 3:35 PM, Manik Surtani wrote:
> If a put() is the culprit leading to the OOME, we can definitely
do
> something !
Galder is right though, even if the put() is the culprit, concurrent threads elsewhere
(out of our control) would also be seeing OOMs and even if we free up memory by evicting,
other threads may be in an unstable state.
JGroups tries to handle temporary OOMEs, and if Infinispan does that,
too, then it's only the app left which can cause OOMEs.
But at least the middleware (JGroups, Infinispan) should go to great
lengths to make sure it handles OOMEs.
I see a big issue coming when we don't do this: as long as we don't have
a more or less even key/value distribution in Infinispan, there is a
chance we end up with some JVMs lightly loaded (in terms of memory) and
others almost full.
Even if we come up with a better distribution function, we cannot
prevent an application from using objects with uneven sizes that they
stick into the cache, e.g. thousands of smallish values and a few very
large values. This will also cause an uneven memory distribution, even
if the key distribution is even.
One thing we could do on the receiving side is to check memory only if
we receive a value that bigger than a certain size. Problem is what
happens when someone sets a value...
--
Bela Ban
Lead JGroups / Clustering Team
JBoss
3:57 p.m.
New subject: ISPN-863 - Thoughts / Questions
Yeah, I do agree that we should do this (best effort); just that we shouldn't fool
ourselves into thinking that this is going to solve all problems caused by an OOM in
Infinispan.
Also, re: middleware, this is often more than just JGroups + Infinispan (think JBoss AS,
maybe SEAM, some web fwk, or Spring, and a whole bunch of user code that could be left
unstable).
On 14 Jan 2011, at 14:54, Bela Ban wrote:
On 1/14/11 3:35 PM, Manik Surtani wrote:
>> If a put() is the culprit leading to the OOME, we can definitely do
>> something !
>
> Galder is right though, even if the put() is the culprit, concurrent threads
elsewhere (out of our control) would also be seeing OOMs and even if we free up memory by
evicting, other threads may be in an unstable state.
JGroups tries to handle temporary OOMEs, and if Infinispan does that,
too, then it's only the app left which can cause OOMEs.
But at least the middleware (JGroups, Infinispan) should go to great
lengths to make sure it handles OOMEs.
I see a big issue coming when we don't do this: as long as we don't have
a more or less even key/value distribution in Infinispan, there is a
chance we end up with some JVMs lightly loaded (in terms of memory) and
others almost full.
Even if we come up with a better distribution function, we cannot
prevent an application from using objects with uneven sizes that they
stick into the cache, e.g. thousands of smallish values and a few very
large values. This will also cause an uneven memory distribution, even
if the key distribution is even.
One thing we could do on the receiving side is to check memory only if
we receive a value that bigger than a certain size. Problem is what
happens when someone sets a value...
--
Bela Ban
Lead JGroups / Clustering Team
JBoss
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
--
Manik Surtani
manik(a)jboss.org
twitter.com/maniksurtani
Lead, Infinispan
http://www.infinispan.org
10:11 a.m.
New subject: ISPN-863 - Thoughts / Questions
On Jan 14, 2011, at 3:35 PM, Manik Surtani wrote:
On 14 Jan 2011, at 14:29, Bela Ban wrote:
>
>
> On 1/14/11 3:08 PM, Galder Zamarreño wrote:
>> -1
>>
>> If you get an OOME, chances are the system is a pretty bad shape and you
can't really do much else at that point.
>
>
> If a put() is the culprit leading to the OOME, we can definitely do
> something !
Galder is right though, even if the put() is the culprit, concurrent threads elsewhere
(out of our control) would also be seeing OOMs and even if we free up memory by evicting,
other threads may be in an unstable state.
As a side thing, IMO, rather than trying to salvage an OOME, it's better to analise
what when wrong adding these two flags:
-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/java_heap
--
Manik Surtani
manik(a)jboss.org
twitter.com/maniksurtani
Lead, Infinispan
http://www.infinispan.org
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
--
Galder Zamarreño
Sr. Software Engineer
Infinispan, JBoss Cache
3:26 p.m.
New subject: ISPN-863 - Thoughts / Questions
We could definitely try that. I'm not sure what happens internally when an OOM is
raised by the VM.
Date: Fri, 14 Jan 2011 09:40:37 +0100
From: bban(a)redhat.com
To: infinispan-dev(a)lists.jboss.org
Subject: Re: [infinispan-dev] ISPN-863 - Thoughts / Questions
Excellent, memory based eviction is very important ! This has been on
the todo list for years !
One thing you might want to think about is - *in addition* to what you
say below - whether you want to add an OOME handler, which evits on OOME
caused by putXXX() methods.
For example, you could wrap all methods which add to the map to catch
OOMEs. When this is the case, kick off the eviction handler to evict
elements immediately. Kind of like a second line of defense against OOMEs...
WDYT ?
On 1/14/11 4:22 AM, david marion wrote:
>
> I have been looking at the code and think I have a starting point on the design.
Remember, the goal here is to be able to create a cache that uses expiration but has no
defined cache size limit. Reading through the code, if ‘-1’ is specified for the eviction
maxEntries, then a ConcurrentHashMap is created and used as the data container and no
eviction is performed. Otherwise a BoundedConcurrentHashMap is created and used as the
data container and eviction is performed according to the specified strategy.
>
> Eviction currently appears to happen in two different ways:
>
> 1. If an entry is expired, then it can be evicted.
> 2. If the number of entries in a segment of the cache contains more entries than
“allowed” by the configuration.
>
> Here are the modifications that I think need to be made. I have not made any code
mods yet, so I don’t know for sure if this will work. Definitely looking for feedback.
>
>
> Modify configuration in some way so that the following can be specified:
>
> The percentage value of used JVM memory (i.e. 95) at which entries should be evicted
to try and avoid an OOM error.
> The number of items that should be evicted when memory reaches this threshold
> Modify LRU and LIRS Eviction class so that the accessQueue member can be accessed by
the new Eviction class so that two access queues don’t have to be maintained.
> Create a new Eviction class, a subclass of LIRS, where the accessQueue is used from
the Eviction strategy the user specifies and the for loop in the execute method is exited
when the evicted set equals value from 1.b above.
> Modify DataContainerFactory.construct() to call
DefaultDataContainer.boundedDataContainer() regardless of eviction policy. This will
always create a BoundedConcurrentHashMap
> Create an instance of the new Eviction class in each segment.
> Modify BoundedConcurrentHashMap.Segment put and replace methods such that when new
values are going to be put into the Segment, the memory usage is checked and the execute
method is called on the new Eviction class.
>
> Questions:
>
>
> What are the implications of using a BoundedConcurrentHashMap instead of a
ConcurrentHashMap when maxEntries is set to -1?
>
> Thoughts
>
>
> This will not guarantee that an OOM error does not occur. It will attempt to guard
against an OOM caused by putting new values into the cache. This will probably be more
effective when the cache is being used in client/server mode, and less effective when used
in embedded mode as to other code running in the JVM.
--
Bela Ban
Lead JGroups / Clustering Team
JBoss
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
3:31 p.m.
New subject: ISPN-863 - Thoughts / Questions
Very simple: when a malloc() fails, an OOME (runtime exception) is
thrown. We can catch it and see if we can help reducing memory by
evicting elements from the cache. Note that OOMEs are also thrown when
the stack is exhausted (IIRC) and when a new thread cannot be created.
Definitely something worth trying out...
On 1/14/11 3:26 PM, david marion wrote:
We could definitely try that. I'm not sure what happens internally when an OOM is
raised by the VM.
> Date: Fri, 14 Jan 2011 09:40:37 +0100
> From: bban(a)redhat.com
> To: infinispan-dev(a)lists.jboss.org
> Subject: Re: [infinispan-dev] ISPN-863 - Thoughts / Questions
>
> Excellent, memory based eviction is very important ! This has been on
> the todo list for years !
>
> One thing you might want to think about is - *in addition* to what you
> say below - whether you want to add an OOME handler, which evits on OOME
> caused by putXXX() methods.
>
> For example, you could wrap all methods which add to the map to catch
> OOMEs. When this is the case, kick off the eviction handler to evict
> elements immediately. Kind of like a second line of defense against OOMEs...
>
>
> WDYT ?
>
>
>
> On 1/14/11 4:22 AM, david marion wrote:
>>
>> I have been looking at the code and think I have a starting point on the design.
Remember, the goal here is to be able to create a cache that uses expiration but has no
defined cache size limit. Reading through the code, if ‘-1’ is specified for the eviction
maxEntries, then a ConcurrentHashMap is created and used as the data container and no
eviction is performed. Otherwise a BoundedConcurrentHashMap is created and used as the
data container and eviction is performed according to the specified strategy.
>>
>> Eviction currently appears to happen in two different ways:
>>
>> 1. If an entry is expired, then it can be evicted.
>> 2. If the number of entries in a segment of the cache contains more entries than
“allowed” by the configuration.
>>
>> Here are the modifications that I think need to be made. I have not made any code
mods yet, so I don’t know for sure if this will work. Definitely looking for feedback.
>>
>>
>> Modify configuration in some way so that the following can be specified:
>>
>> The percentage value of used JVM memory (i.e. 95) at which entries should be
evicted to try and avoid an OOM error.
>> The number of items that should be evicted when memory reaches this threshold
>> Modify LRU and LIRS Eviction class so that the accessQueue member can be accessed
by the new Eviction class so that two access queues don’t have to be maintained.
>> Create a new Eviction class, a subclass of LIRS, where the accessQueue is used
from the Eviction strategy the user specifies and the for loop in the execute method is
exited when the evicted set equals value from 1.b above.
>> Modify DataContainerFactory.construct() to call
DefaultDataContainer.boundedDataContainer() regardless of eviction policy. This will
always create a BoundedConcurrentHashMap
>> Create an instance of the new Eviction class in each segment.
>> Modify BoundedConcurrentHashMap.Segment put and replace methods such that when
new values are going to be put into the Segment, the memory usage is checked and the
execute method is called on the new Eviction class.
>>
>> Questions:
>>
>>
>> What are the implications of using a BoundedConcurrentHashMap instead of a
ConcurrentHashMap when maxEntries is set to -1?
>>
>> Thoughts
>>
>>
>> This will not guarantee that an OOM error does not occur. It will attempt to
guard against an OOM caused by putting new values into the cache. This will probably be
more effective when the cache is being used in client/server mode, and less effective when
used in embedded mode as to other code running in the JVM.
>
> --
> Bela Ban
> Lead JGroups / Clustering Team
> JBoss
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
--
Bela Ban
Lead JGroups / Clustering Team
JBoss
12:45 p.m.
New subject: ISPN-863 - Thoughts / Questions
I'll let Vladimir comment more as he wrote most of this code originally, but it sounds
OK in principle.
Would you need to check the memory usage on every put and replace? Wouldn't this be
expensive? Couldn't this happen periodically?
On 14 Jan 2011, at 03:22, david marion wrote:
I have been looking at the code and think I have a starting point on
the design. Remember, the goal here is to be able to create a cache that uses expiration
but has no defined cache size limit. Reading through the code, if ‘-1’ is specified for
the eviction maxEntries, then a ConcurrentHashMap is created and used as the data
container and no eviction is performed. Otherwise a BoundedConcurrentHashMap is created
and used as the data container and eviction is performed according to the specified
strategy.
Eviction currently appears to happen in two different ways:
1. If an entry is expired, then it can be evicted.
2. If the number of entries in a segment of the cache contains more entries than
“allowed” by the configuration.
Here are the modifications that I think need to be made. I have not made any code mods
yet, so I don’t know for sure if this will work. Definitely looking for feedback.
Modify configuration in some way so that the following can be specified:
The percentage value of used JVM memory (i.e. 95) at which entries should be evicted to
try and avoid an OOM error.
The number of items that should be evicted when memory reaches this threshold
Modify LRU and LIRS Eviction class so that the accessQueue member can be accessed by the
new Eviction class so that two access queues don’t have to be maintained.
Create a new Eviction class, a subclass of LIRS, where the accessQueue is used from the
Eviction strategy the user specifies and the for loop in the execute method is exited when
the evicted set equals value from 1.b above.
Modify DataContainerFactory.construct() to call
DefaultDataContainer.boundedDataContainer() regardless of eviction policy. This will
always create a BoundedConcurrentHashMap
Create an instance of the new Eviction class in each segment.
Modify BoundedConcurrentHashMap.Segment put and replace methods such that when new values
are going to be put into the Segment, the memory usage is checked and the execute method
is called on the new Eviction class.
Questions:
What are the implications of using a BoundedConcurrentHashMap instead of a
ConcurrentHashMap when maxEntries is set to -1?
Thoughts
This will not guarantee that an OOM error does not occur. It will attempt to guard
against an OOM caused by putting new values into the cache. This will probably be more
effective when the cache is being used in client/server mode, and less effective when used
in embedded mode as to other code running in the JVM.
-- Dave Marion
From: manik(a)jboss.org
Date: Tue, 11 Jan 2011 11:46:40 +0000
To: infinispan-dev(a)lists.jboss.org
Subject: [infinispan-dev] Poor man's size-based eviction Re: ISPN-863
Hi Dave
Just saw that you signed the contributor agreement.
So yeah, what you may want to look into is the EvictionManager [1] which will
periodically have its processEviction() [2] method called. This doesn't actually do
any evictions here (named so for legacy reasons) since the current data container is
self-sizing/evicting. However it is still periodically invoked to purge expired entries.
You can piggyback on the same invocation to test upper and lower bounds of memory
utilisation (if enabled), but the bit that I still think you need to figure out is
deciding which entries to evict, and how many of them to evict.
Cheers
Manik
[1]
https://github.com/infinispan/infinispan/blob/master/core/src/main/java/o...
[2]
https://github.com/infinispan/infinispan/blob/master/core/src/main/java/o...
On 10 Jan 2011, at 11:08, Manik Surtani wrote:
Excellent, Dave.
You should probably come up with a design first - or maybe a prototype on your fork of
the project on GitHub - and post it here for comment.
Cheers
Manik
On 8 Jan 2011, at 00:34, david marion wrote:
Manik,
I am on the list. I will start getting my dev environment setup according to instructions
on site.
Dave Marion
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
--
Manik Surtani
manik(a)jboss.org
twitter.com/maniksurtani
Lead, Infinispan
http://www.infinispan.org
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
--
Manik Surtani
manik(a)jboss.org
twitter.com/maniksurtani
Lead, Infinispan
http://www.infinispan.org
_______________________________________________ infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev__________________...
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
--
Manik Surtani
manik(a)jboss.org
twitter.com/maniksurtani
Lead, Infinispan
http://www.infinispan.org
3:32 p.m.
New subject: ISPN-863 - Thoughts / Questions
I agree. There are two ways to determine what the memory usage is, polling or
notifications. The notification mechanism is not guaranteed, so we could have a background
thread that polls to determine whether or not the threshold has been passed. Then in the
put and replace methods, it would be the cost of checking a boolean value.
From: manik(a)jboss.org
Date: Fri, 14 Jan 2011 11:45:29 +0000
To: infinispan-dev(a)lists.jboss.org
Subject: Re: [infinispan-dev] ISPN-863 - Thoughts / Questions
I'll let Vladimir comment more as he wrote most of this code originally, but it sounds
OK in principle.
Would you need to check the memory usage on every put and replace? Wouldn't this be
expensive? Couldn't this happen periodically?
On 14 Jan 2011, at 03:22, david marion wrote:
I have been looking at the code and think I have a starting point on the design. Remember,
the goal here is to be able to create a cache that uses expiration but has no defined
cache size limit. Reading through the code, if ‘-1’ is specified for the eviction
maxEntries, then a ConcurrentHashMap is created and used as the data container and no
eviction is performed. Otherwise a BoundedConcurrentHashMap is created and used as the
data container and eviction is performed according to the specified strategy.
Eviction currently appears to happen in two different ways:
1. If an entry is expired, then it can be evicted.
2. If the number of entries in a segment of the cache contains more entries than
“allowed” by the configuration.
Here are the modifications that I think need to be made. I have not made any code mods
yet, so I don’t know for sure if this will work. Definitely looking for feedback.
Modify configuration in some way so that the following can be specified:
The percentage value of used JVM memory (i.e. 95) at which entries should be evicted to
try and avoid an OOM error.
The number of items that should be evicted when memory reaches this threshold
Modify LRU and LIRS Eviction class so that the accessQueue member can be accessed by the
new Eviction class so that two access queues don’t have to be maintained.
Create a new Eviction class, a subclass of LIRS, where the accessQueue is used from the
Eviction strategy the user specifies and the for loop in the execute method is exited when
the evicted set equals value from 1.b above.
Modify DataContainerFactory.construct() to call
DefaultDataContainer.boundedDataContainer() regardless of eviction policy. This will
always create a BoundedConcurrentHashMap
Create an instance of the new Eviction class in each segment.
Modify BoundedConcurrentHashMap.Segment put and replace methods such that when new values
are going to be put into the Segment, the memory usage is checked and the execute method
is called on the new Eviction class.
Questions:
What are the implications of using a BoundedConcurrentHashMap instead of a
ConcurrentHashMap when maxEntries is set to -1?
Thoughts
This will not guarantee that an OOM error does not occur. It will attempt to guard against
an OOM caused by putting new values into the cache. This will probably be more effective
when the cache is being used in client/server mode, and less effective when used in
embedded mode as to other code running in the JVM.
-- Dave Marion
From: manik(a)jboss.org
Date: Tue, 11 Jan 2011 11:46:40 +0000
To: infinispan-dev(a)lists.jboss.org
Subject: [infinispan-dev] Poor man's size-based eviction Re: ISPN-863
Hi Dave
Just saw that you signed the contributor agreement.
So yeah, what you may want to look into is the EvictionManager [1] which will periodically
have its processEviction() [2] method called. This doesn't actually do any evictions
here (named so for legacy reasons) since the current data container is
self-sizing/evicting. However it is still periodically invoked to purge expired entries.
You can piggyback on the same invocation to test upper and lower bounds of memory
utilisation (if enabled), but the bit that I still think you need to figure out is
deciding which entries to evict, and how many of them to evict.
Cheers
Manik
[1]
https://github.com/infinispan/infinispan/blob/master/core/src/main/java/o...
[2]
https://github.com/infinispan/infinispan/blob/master/core/src/main/java/o...
On 10 Jan 2011, at 11:08, Manik Surtani wrote:
Excellent, Dave.
You should probably come up with a design first - or maybe a prototype on your fork of the
project on GitHub - and post it here for comment.
Cheers
Manik
On 8 Jan 2011, at 00:34, david marion wrote:
Manik,
I am on the list. I will start getting my dev environment setup according to instructions
on site.
Dave Marion _______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
--
Manik Surtani
manik(a)jboss.org
twitter.com/maniksurtani
Lead, Infinispan
http://www.infinispan.org
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
--
Manik Surtani
manik(a)jboss.org
twitter.com/maniksurtani
Lead, Infinispan
http://www.infinispan.org
_______________________________________________ infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev__________________...
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
--
Manik Surtani
manik(a)jboss.org
twitter.com/maniksurtani
Lead, Infinispan
http://www.infinispan.org
_______________________________________________ infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev
3:46 p.m.
New subject: ISPN-863 - Thoughts / Questions
On 1/14/11 3:32 PM, david marion wrote:
I agree. There are two ways to determine what the memory usage is, polling or
notifications. The notification mechanism is not guaranteed
Are you referring to MemoryMXBean / MemoryPoolMXBean ? What do you mean
by not guaranteed ? Are you suggesting the notfications are not
implemented by all JVM vendors ?
so we could have a background thread that polls to determine whether
or not the threshold has been passed. Then in the put and replace methods, it would be the
cost of checking a boolean value.
When notifications are available, we should use them, otherwise polling
would be fine, I guess.
--
Bela Ban
Lead JGroups / Clustering Team
JBoss
3:48 p.m.
New subject: ISPN-863 - Thoughts / Questions
I should have been more specific, notification delivery is not guaranteed. From
http://download.oracle.com/javase/6/docs/api/java/lang/management/MemoryP...:
"There is no guarantee about when the MemoryMXBean will emit a threshold notification
and when the notification will be delivered. "
Date: Fri, 14 Jan 2011 15:46:02 +0100
From: bban(a)redhat.com
To: infinispan-dev(a)lists.jboss.org
Subject: Re: [infinispan-dev] ISPN-863 - Thoughts / Questions
On 1/14/11 3:32 PM, david marion wrote:
>
>
> I agree. There are two ways to determine what the memory usage is, polling or
notifications. The notification mechanism is not guaranteed
Are you referring to MemoryMXBean / MemoryPoolMXBean ? What do you mean
by not guaranteed ? Are you suggesting the notfications are not
implemented by all JVM vendors ?
> so we could have a background thread that polls to determine whether or not the
threshold has been passed. Then in the put and replace methods, it would be the cost of
checking a boolean value.
When notifications are available, we should use them, otherwise polling
would be fine, I guess.
--
Bela Ban
Lead JGroups / Clustering Team
JBoss
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
3:57 p.m.
New subject: ISPN-863 - Thoughts / Questions
On 1/14/11 3:48 PM, david marion wrote:
I should have been more specific, notification delivery is not guaranteed. From
http://download.oracle.com/javase/6/docs/api/java/lang/management/MemoryP...:
"There is no guarantee about when the MemoryMXBean will emit a threshold
notification and when the notification will be delivered."
Interesting. I suggest we try it out to see what the behavior is. If
this is unreliable, I guess we have to switch to polling
--
Bela Ban
Lead JGroups / Clustering Team
JBoss
5:05 p.m.
New subject: ISPN-863 - Thoughts / Questions
Hey David,
First of all kudos for taking on such a non-trivial task!
On 11-01-14 12:22 AM, david marion wrote:
1. Modify configuration in some way so that the following can be
specified:
1. The percentage value of used JVM memory (i.e. 95) at which
entries should be evicted to try and avoid an OOM error.
2. The number of items that should be evicted when memory
reaches this threshold
2. Modify LRU and LIRS Eviction class so that the accessQueue
member can be accessed by the new Eviction class so that two
access queues don’t have to be maintained.
3. Create a new Eviction class, a subclass of LIRS, where the
accessQueue is used from the Eviction strategy the user
specifies and the for loop in the execute method is exited when
the evicted set equals value from 1.b above.
4. Modify DataContainerFactory.construct() to call
DefaultDataContainer.boundedDataContainer() regardless of
eviction policy. This will always create a BoundedConcurrentHashMap
I don't think you have to extend LRU and LIRS, as long as you implement
EvictionPolicy you are fullfilling the contract. What extending LRU and
LIRS can do is give you some order in selecting proper elements for
eviction. I think the precise technical term in research literature is
"eviction precision".
1.
2. Create an instance of the new Eviction class in each segment.
3. Modify BoundedConcurrentHashMap.Segment put and replace methods
such that when new values are going to be put into the Segment,
the memory usage is checked and the execute method is called on
the new Eviction class.
Sounds right!
1.
Questions:
1. What are the implications of using a BoundedConcurrentHashMap
instead of a ConcurrentHashMap when maxEntries is set to -1?
I think none except you turn on eviction by using BoundedConcurrentHashMap.
1.
Thoughts
1. This will not guarantee that an OOM error does not occur. It
will attempt to guard against an OOM caused by putting new
values into the cache. This will probably be more effective when
the cache is being used in client/server mode, and less
effective when used in embedded mode as to other code running in
the JVM.
-- Dave Marion
Cheers,
Vladimir
3:38 p.m.
New subject: ISPN-863 - Thoughts / Questions
All,
I am trying to test some changes that I made using Eclipse and the TestNG
plugin. I am not familiar with TestNG. Are there any settings that I should
make? Any ideas?
java.lang.OutOfMemoryError: Java heap space
at
org.infinispan.util.concurrent.BoundedConcurrentHashMap$HashEntry.newArray(B
oundedConcurrentHashMap.java:295)
at
org.infinispan.util.concurrent.BoundedConcurrentHashMap$Segment.<init>(Bound
edConcurrentHashMap.java:898)
at
org.infinispan.util.concurrent.BoundedConcurrentHashMap.<init>(BoundedConcur
rentHashMap.java:1367)
at
org.infinispan.container.DefaultDataContainer.<init>(DefaultDataContainer.ja
va:78)
at
org.infinispan.container.DefaultDataContainer.<init>(DefaultDataContainer.ja
va:47)
at
org.infinispan.container.DefaultDataContainer.unBoundedDataContainer(Default
DataContainer.java:93)
at
org.infinispan.factories.DataContainerFactory.construct(DataContainerFactory
.java:53)
at
org.infinispan.factories.AbstractComponentRegistry.getOrCreateComponent(Abst
ractComponentRegistry.java:315)
at
org.infinispan.factories.AbstractComponentRegistry.invokeInjectionMethod(Abs
tractComponentRegistry.java:251)
at
org.infinispan.factories.AbstractComponentRegistry$Component.injectDependenc
ies(AbstractComponentRegistry.java:840)
at
org.infinispan.factories.AbstractComponentRegistry.registerComponent(Abstrac
tComponentRegistry.java:225)
at
org.infinispan.factories.ComponentRegistry.registerComponent(ComponentRegist
ry.java:120)
at
org.infinispan.factories.AbstractComponentRegistry.registerComponent(Abstrac
tComponentRegistry.java:192)
at
org.infinispan.factories.AbstractComponentRegistry.getOrCreateComponent(Abst
ractComponentRegistry.java:323)
at
org.infinispan.factories.AbstractComponentRegistry.invokeInjectionMethod(Abs
tractComponentRegistry.java:251)
at
org.infinispan.factories.AbstractComponentRegistry$Component.injectDependenc
ies(AbstractComponentRegistry.java:840)
at
org.infinispan.factories.AbstractComponentRegistry.registerComponent(Abstrac
tComponentRegistry.java:225)
at
org.infinispan.factories.ComponentRegistry.registerComponent(ComponentRegist
ry.java:120)
at
org.infinispan.factories.AbstractComponentRegistry.registerComponent(Abstrac
tComponentRegistry.java:192)
at
org.infinispan.factories.AbstractComponentRegistry.getOrCreateComponent(Abst
ractComponentRegistry.java:323)
at
org.infinispan.factories.AbstractComponentRegistry.invokeInjectionMethod(Abs
tractComponentRegistry.java:251)
at
org.infinispan.factories.AbstractComponentRegistry$Component.injectDependenc
ies(AbstractComponentRegistry.java:840)
at
org.infinispan.factories.AbstractComponentRegistry.registerComponent(Abstrac
tComponentRegistry.java:225)
at
org.infinispan.factories.ComponentRegistry.registerComponent(ComponentRegist
ry.java:120)
at
org.infinispan.factories.AbstractComponentRegistry.registerComponent(Abstrac
tComponentRegistry.java:192)
at
org.infinispan.factories.InternalCacheFactory.bootstrap(InternalCacheFactory
.java:92)
at
org.infinispan.factories.InternalCacheFactory.createAndWire(InternalCacheFac
tory.java:78)
at
org.infinispan.factories.InternalCacheFactory.createCache(InternalCacheFacto
ry.java:62)
at
org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.j
ava:510)
at
org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java
:440)
at
org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java
:409)
at
org.infinispan.config.ConfigurationValidationTest.testDefaultMemoryGuardConf
iguration(ConfigurationValidationTest.java:100)
-----Original Message-----
From: infinispan-dev-bounces(a)lists.jboss.org
[mailto:infinispan-dev-bounces@lists.jboss.org] On Behalf Of Vladimir
Blagojevic
Sent: Friday, January 14, 2011 11:05 AM
To: infinispan -Dev List
Subject: Re: [infinispan-dev] ISPN-863 - Thoughts / Questions
Hey David,
First of all kudos for taking on such a non-trivial task!
On 11-01-14 12:22 AM, david marion wrote:
1. Modify configuration in some way so that the following can be
specified:
a. The percentage value of used JVM memory (i.e. 95) at which entries
should be evicted to try and avoid an OOM error.
b. The number of items that should be evicted when memory reaches this
threshold
2. Modify LRU and LIRS Eviction class so that the accessQueue member
can be accessed by the new Eviction class so that two access queues don't
have to be maintained.
3. Create a new Eviction class, a subclass of LIRS, where the
accessQueue is used from the Eviction strategy the user specifies and the
for loop in the execute method is exited when the evicted set equals value
from 1.b above.
4. Modify DataContainerFactory.construct() to call
DefaultDataContainer.boundedDataContainer() regardless of eviction policy.
This will always create a BoundedConcurrentHashMap
I don't think you have to extend LRU and LIRS, as long as you implement
EvictionPolicy you are fullfilling the contract. What extending LRU and LIRS
can do is give you some order in selecting proper elements for eviction. I
think the precise technical term in research literature is "eviction
precision".
1.
2. Create an instance of the new Eviction class in each segment.
3. Modify BoundedConcurrentHashMap.Segment put and replace methods such
that when new values are going to be put into the Segment, the memory usage
is checked and the execute method is called on the new Eviction class.
Sounds right!
1.
Questions:
1. What are the implications of using a BoundedConcurrentHashMap
instead of a ConcurrentHashMap when maxEntries is set to -1?
I think none except you turn on eviction by using BoundedConcurrentHashMap.
1.
Thoughts
1. This will not guarantee that an OOM error does not occur. It will
attempt to guard against an OOM caused by putting new values into the cache.
This will probably be more effective when the cache is being used in
client/server mode, and less effective when used in embedded mode as to
other code running in the JVM.
-- Dave Marion
Cheers,
Vladimir
3:54 p.m.
New subject: ISPN-863 - Thoughts / Questions
Its easier to run everything from command line for two reason I can
think of:
- everyone has same env to compare results with you
- it is all ready to go
See http://community.jboss.org/wiki/InfinispanandMaven
On 11-01-17 11:38 AM, Dave wrote:
All,
I am trying to test some changes that I made using Eclipse and the
TestNG plugin. I am not familiar with TestNG. Are there any settings
that I should make? Any ideas?
java.lang.OutOfMemoryError: Java heap space
at
org.infinispan.util.concurrent.BoundedConcurrentHashMap$HashEntry.newArray(_BoundedConcurrentHashMap.java:295_)
at
org.infinispan.util.concurrent.BoundedConcurrentHashMap$Segment.<init>(_BoundedConcurrentHashMap.java:898_)
at
org.infinispan.util.concurrent.BoundedConcurrentHashMap.<init>(_BoundedConcurrentHashMap.java:1367_)
at
org.infinispan.container.DefaultDataContainer.<init>(_DefaultDataContainer.java:78_)
at
org.infinispan.container.DefaultDataContainer.<init>(_DefaultDataContainer.java:47_)
at
org.infinispan.container.DefaultDataContainer.unBoundedDataContainer(_DefaultDataContainer.java:93_)
at
org.infinispan.factories.DataContainerFactory.construct(_DataContainerFactory.java:53_)
at
org.infinispan.factories.AbstractComponentRegistry.getOrCreateComponent(_AbstractComponentRegistry.java:315_)
at
org.infinispan.factories.AbstractComponentRegistry.invokeInjectionMethod(_AbstractComponentRegistry.java:251_)
at
org.infinispan.factories.AbstractComponentRegistry$Component.injectDependencies(_AbstractComponentRegistry.java:840_)
at
org.infinispan.factories.AbstractComponentRegistry.registerComponent(_AbstractComponentRegistry.java:225_)
at
org.infinispan.factories.ComponentRegistry.registerComponent(_ComponentRegistry.java:120_)
at
org.infinispan.factories.AbstractComponentRegistry.registerComponent(_AbstractComponentRegistry.java:192_)
at
org.infinispan.factories.AbstractComponentRegistry.getOrCreateComponent(_AbstractComponentRegistry.java:323_)
at
org.infinispan.factories.AbstractComponentRegistry.invokeInjectionMethod(_AbstractComponentRegistry.java:251_)
at
org.infinispan.factories.AbstractComponentRegistry$Component.injectDependencies(_AbstractComponentRegistry.java:840_)
at
org.infinispan.factories.AbstractComponentRegistry.registerComponent(_AbstractComponentRegistry.java:225_)
at
org.infinispan.factories.ComponentRegistry.registerComponent(_ComponentRegistry.java:120_)
at
org.infinispan.factories.AbstractComponentRegistry.registerComponent(_AbstractComponentRegistry.java:192_)
at
org.infinispan.factories.AbstractComponentRegistry.getOrCreateComponent(_AbstractComponentRegistry.java:323_)
at
org.infinispan.factories.AbstractComponentRegistry.invokeInjectionMethod(_AbstractComponentRegistry.java:251_)
at
org.infinispan.factories.AbstractComponentRegistry$Component.injectDependencies(_AbstractComponentRegistry.java:840_)
at
org.infinispan.factories.AbstractComponentRegistry.registerComponent(_AbstractComponentRegistry.java:225_)
at
org.infinispan.factories.ComponentRegistry.registerComponent(_ComponentRegistry.java:120_)
at
org.infinispan.factories.AbstractComponentRegistry.registerComponent(_AbstractComponentRegistry.java:192_)
at
org.infinispan.factories.InternalCacheFactory.bootstrap(_InternalCacheFactory.java:92_)
at
org.infinispan.factories.InternalCacheFactory.createAndWire(_InternalCacheFactory.java:78_)
at
org.infinispan.factories.InternalCacheFactory.createCache(_InternalCacheFactory.java:62_)
at
org.infinispan.manager.DefaultCacheManager.createCache(_DefaultCacheManager.java:510_)
at
org.infinispan.manager.DefaultCacheManager.getCache(_DefaultCacheManager.java:440_)
at
org.infinispan.manager.DefaultCacheManager.getCache(_DefaultCacheManager.java:409_)
at
org.infinispan.config.ConfigurationValidationTest.testDefaultMemoryGuardConfiguration(_ConfigurationValidationTest.java:100_)
-----Original Message-----
*From:* infinispan-dev-bounces(a)lists.jboss.org
[mailto:infinispan-dev-bounces@lists.jboss.org] *On Behalf Of
*Vladimir Blagojevic
*Sent:* Friday, January 14, 2011 11:05 AM
*To:* infinispan -Dev List
*Subject:* Re: [infinispan-dev] ISPN-863 - Thoughts / Questions
Hey David,
First of all kudos for taking on such a non-trivial task!
On 11-01-14 12:22 AM, david marion wrote:
1.Modify configuration in some way so that the following can be specified:
a.The percentage value of used JVM memory (i.e. 95) at which entries
should be evicted to try and avoid an OOM error.
b.The number of items that should be evicted when memory reaches this
threshold
2.Modify LRU and LIRS Eviction class so that the accessQueue member
can be accessed by the new Eviction class so that two access queues
don't have to be maintained.
3.Create a new Eviction class, a subclass of LIRS, where the
accessQueue is used from the Eviction strategy the user specifies and
the for loop in the execute method is exited when the evicted set
equals value from 1.b above.
4.Modify DataContainerFactory.construct() to call
DefaultDataContainer.boundedDataContainer() regardless of eviction
policy. This will always create a BoundedConcurrentHashMap
I don't think you have to extend LRU and LIRS, as long as you
implement EvictionPolicy you are fullfilling the contract. What
extending LRU and LIRS can do is give you some order in selecting
proper elements for eviction. I think the precise technical term in
research literature is "eviction precision".
1.
2.Create an instance of the new Eviction class in each segment.
3.Modify BoundedConcurrentHashMap.Segment put and replace methods such
that when new values are going to be put into the Segment, the memory
usage is checked and the execute method is called on the new Eviction
class.
Sounds right!
1.
Questions:
1.What are the implications of using a BoundedConcurrentHashMap
instead of a ConcurrentHashMap when maxEntries is set to -1?
I think none except you turn on eviction by using
BoundedConcurrentHashMap.
1.
Thoughts
1.This will not guarantee that an OOM error does not occur. It will
attempt to guard against an OOM caused by putting new values into the
cache. This will probably be more effective when the cache is being
used in client/server mode, and less effective when used in embedded
mode as to other code running in the JVM.
-- Dave Marion
Cheers,
Vladimir
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
4:51 p.m.
New subject: ISPN-863 - Thoughts / Questions
same result.
Date: Mon, 17 Jan 2011 11:54:15 -0300
From: vblagoje(a)redhat.com
To: infinispan-dev(a)lists.jboss.org
Subject: Re: [infinispan-dev] ISPN-863 - Thoughts / Questions
Its easier to run everything from command line for two reason I can think of:
- everyone has same env to compare results with you
- it is all ready to go
See http://community.jboss.org/wiki/InfinispanandMaven
On 11-01-17 11:38 AM, Dave wrote:
All,
I am trying to test some changes that I made using Eclipse and the TestNG plugin. I am not
familiar with TestNG. Are there any settings that I should make? Any ideas?
java.lang.OutOfMemoryError: Java heap space
at
org.infinispan.util.concurrent.BoundedConcurrentHashMap$HashEntry.newArray(BoundedConcurrentHashMap.java:295)
at
org.infinispan.util.concurrent.BoundedConcurrentHashMap$Segment.<init>(BoundedConcurrentHashMap.java:898)
at
org.infinispan.util.concurrent.BoundedConcurrentHashMap.<init>(BoundedConcurrentHashMap.java:1367)
at
org.infinispan.container.DefaultDataContainer.<init>(DefaultDataContainer.java:78)
at
org.infinispan.container.DefaultDataContainer.<init>(DefaultDataContainer.java:47)
at
org.infinispan.container.DefaultDataContainer.unBoundedDataContainer(DefaultDataContainer.java:93)
at
org.infinispan.factories.DataContainerFactory.construct(DataContainerFactory.java:53)
at
org.infinispan.factories.AbstractComponentRegistry.getOrCreateComponent(AbstractComponentRegistry.java:315)
at
org.infinispan.factories.AbstractComponentRegistry.invokeInjectionMethod(AbstractComponentRegistry.java:251)
at
org.infinispan.factories.AbstractComponentRegistry$Component.injectDependencies(AbstractComponentRegistry.java:840)
at
org.infinispan.factories.AbstractComponentRegistry.registerComponent(AbstractComponentRegistry.java:225)
at
org.infinispan.factories.ComponentRegistry.registerComponent(ComponentRegistry.java:120)
at
org.infinispan.factories.AbstractComponentRegistry.registerComponent(AbstractComponentRegistry.java:192)
at
org.infinispan.factories.AbstractComponentRegistry.getOrCreateComponent(AbstractComponentRegistry.java:323)
at
org.infinispan.factories.AbstractComponentRegistry.invokeInjectionMethod(AbstractComponentRegistry.java:251)
at
org.infinispan.factories.AbstractComponentRegistry$Component.injectDependencies(AbstractComponentRegistry.java:840)
at
org.infinispan.factories.AbstractComponentRegistry.registerComponent(AbstractComponentRegistry.java:225)
at
org.infinispan.factories.ComponentRegistry.registerComponent(ComponentRegistry.java:120)
at
org.infinispan.factories.AbstractComponentRegistry.registerComponent(AbstractComponentRegistry.java:192)
at
org.infinispan.factories.AbstractComponentRegistry.getOrCreateComponent(AbstractComponentRegistry.java:323)
at
org.infinispan.factories.AbstractComponentRegistry.invokeInjectionMethod(AbstractComponentRegistry.java:251)
at
org.infinispan.factories.AbstractComponentRegistry$Component.injectDependencies(AbstractComponentRegistry.java:840)
at
org.infinispan.factories.AbstractComponentRegistry.registerComponent(AbstractComponentRegistry.java:225)
at
org.infinispan.factories.ComponentRegistry.registerComponent(ComponentRegistry.java:120)
at
org.infinispan.factories.AbstractComponentRegistry.registerComponent(AbstractComponentRegistry.java:192)
at
org.infinispan.factories.InternalCacheFactory.bootstrap(InternalCacheFactory.java:92)
at
org.infinispan.factories.InternalCacheFactory.createAndWire(InternalCacheFactory.java:78)
at
org.infinispan.factories.InternalCacheFactory.createCache(InternalCacheFactory.java:62)
at
org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:510)
at
org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:440)
at
org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:409)
at
org.infinispan.config.ConfigurationValidationTest.testDefaultMemoryGuardConfiguration(ConfigurationValidationTest.java:100)
-----Original Message-----
From: infinispan-dev-bounces(a)lists.jboss.org
[mailto:infinispan-dev-bounces@lists.jboss.org] On Behalf Of Vladimir Blagojevic
Sent: Friday, January 14, 2011 11:05 AM
To: infinispan -Dev List
Subject: Re: [infinispan-dev] ISPN-863 - Thoughts / Questions
Hey David,
First of all kudos for taking on such a non-trivial task!
On 11-01-14 12:22 AM, david marion wrote:
1. Modify configuration in some way so that the following can be specified:
a. The percentage value of used JVM memory (i.e. 95) at which entries should be
evicted to try and avoid an OOM error.
b. The number of items that should be evicted when memory reaches this threshold
2. Modify LRU and LIRS Eviction class so that the accessQueue member can be accessed
by the new Eviction class so that two access queues don’t have to be maintained.
3. Create a new Eviction class, a subclass of LIRS, where the accessQueue is used
from the Eviction strategy the user specifies and the for loop in the execute method is
exited when the evicted set equals value from 1.b above.
4. Modify DataContainerFactory.construct() to call
DefaultDataContainer.boundedDataContainer() regardless of eviction policy. This will
always create a BoundedConcurrentHashMap
I don't think you have to extend LRU and LIRS, as long as you implement EvictionPolicy
you are fullfilling the contract. What extending LRU and LIRS can do is give you some
order in selecting proper elements for eviction. I think the precise technical term in
research literature is "eviction precision".
1.
2. Create an instance of the new Eviction class in each segment.
3. Modify BoundedConcurrentHashMap.Segment put and replace methods such that when new
values are going to be put into the Segment, the memory usage is checked and the execute
method is called on the new Eviction class.
Sounds right!
1.
Questions:
1. What are the implications of using a BoundedConcurrentHashMap instead of a
ConcurrentHashMap when maxEntries is set to -1?
I think none except you turn on eviction by using BoundedConcurrentHashMap.
1.
Thoughts
1. This will not guarantee that an OOM error does not occur. It will attempt to guard
against an OOM caused by putting new values into the cache. This will probably be more
effective when the cache is being used in client/server mode, and less effective when used
in embedded mode as to other code running in the JVM.
-- Dave Marion
Cheers,
Vladimir
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
_______________________________________________ infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev
12:11 p.m.
New subject: ISPN-863 - Thoughts / Questions
Hmmm, try increasing -Xmx? Start with -XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=/tmp/java_heap and inspect the dump with Eclipse MAT?...
On Jan 17, 2011, at 4:51 PM, david marion wrote:
same result.
Date: Mon, 17 Jan 2011 11:54:15 -0300
From: vblagoje(a)redhat.com
To: infinispan-dev(a)lists.jboss.org
Subject: Re: [infinispan-dev] ISPN-863 - Thoughts / Questions
Its easier to run everything from command line for two reason I can think of:
- everyone has same env to compare results with you
- it is all ready to go
See http://community.jboss.org/wiki/InfinispanandMaven
On 11-01-17 11:38 AM, Dave wrote:
All,
I am trying to test some changes that I made using Eclipse and the TestNG plugin. I am
not familiar with TestNG. Are there any settings that I should make? Any ideas?
java.lang.OutOfMemoryError: Java heap space
at
org.infinispan.util.concurrent.BoundedConcurrentHashMap$HashEntry.newArray(BoundedConcurrentHashMap.java:295)
at
org.infinispan.util.concurrent.BoundedConcurrentHashMap$Segment.<init>(BoundedConcurrentHashMap.java:898)
at
org.infinispan.util.concurrent.BoundedConcurrentHashMap.<init>(BoundedConcurrentHashMap.java:1367)
at
org.infinispan.container.DefaultDataContainer.<init>(DefaultDataContainer.java:78)
at
org.infinispan.container.DefaultDataContainer.<init>(DefaultDataContainer.java:47)
at
org.infinispan.container.DefaultDataContainer.unBoundedDataContainer(DefaultDataContainer.java:93)
at
org.infinispan.factories.DataContainerFactory.construct(DataContainerFactory.java:53)
at
org.infinispan.factories.AbstractComponentRegistry.getOrCreateComponent(AbstractComponentRegistry.java:315)
at
org.infinispan.factories.AbstractComponentRegistry.invokeInjectionMethod(AbstractComponentRegistry.java:251)
at
org.infinispan.factories.AbstractComponentRegistry$Component.injectDependencies(AbstractComponentRegistry.java:840)
at
org.infinispan.factories.AbstractComponentRegistry.registerComponent(AbstractComponentRegistry.java:225)
at
org.infinispan.factories.ComponentRegistry.registerComponent(ComponentRegistry.java:120)
at
org.infinispan.factories.AbstractComponentRegistry.registerComponent(AbstractComponentRegistry.java:192)
at
org.infinispan.factories.AbstractComponentRegistry.getOrCreateComponent(AbstractComponentRegistry.java:323)
at
org.infinispan.factories.AbstractComponentRegistry.invokeInjectionMethod(AbstractComponentRegistry.java:251)
at
org.infinispan.factories.AbstractComponentRegistry$Component.injectDependencies(AbstractComponentRegistry.java:840)
at
org.infinispan.factories.AbstractComponentRegistry.registerComponent(AbstractComponentRegistry.java:225)
at
org.infinispan.factories.ComponentRegistry.registerComponent(ComponentRegistry.java:120)
at
org.infinispan.factories.AbstractComponentRegistry.registerComponent(AbstractComponentRegistry.java:192)
at
org.infinispan.factories.AbstractComponentRegistry.getOrCreateComponent(AbstractComponentRegistry.java:323)
at
org.infinispan.factories.AbstractComponentRegistry.invokeInjectionMethod(AbstractComponentRegistry.java:251)
at
org.infinispan.factories.AbstractComponentRegistry$Component.injectDependencies(AbstractComponentRegistry.java:840)
at
org.infinispan.factories.AbstractComponentRegistry.registerComponent(AbstractComponentRegistry.java:225)
at
org.infinispan.factories.ComponentRegistry.registerComponent(ComponentRegistry.java:120)
at
org.infinispan.factories.AbstractComponentRegistry.registerComponent(AbstractComponentRegistry.java:192)
at
org.infinispan.factories.InternalCacheFactory.bootstrap(InternalCacheFactory.java:92)
at
org.infinispan.factories.InternalCacheFactory.createAndWire(InternalCacheFactory.java:78)
at
org.infinispan.factories.InternalCacheFactory.createCache(InternalCacheFactory.java:62)
at
org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:510)
at
org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:440)
at
org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:409)
at
org.infinispan.config.ConfigurationValidationTest.testDefaultMemoryGuardConfiguration(ConfigurationValidationTest.java:100)
-----Original Message-----
From: infinispan-dev-bounces(a)lists.jboss.org
[mailto:infinispan-dev-bounces@lists.jboss.org] On Behalf Of Vladimir Blagojevic
Sent: Friday, January 14, 2011 11:05 AM
To: infinispan -Dev List
Subject: Re: [infinispan-dev] ISPN-863 - Thoughts / Questions
Hey David,
First of all kudos for taking on such a non-trivial task!
On 11-01-14 12:22 AM, david marion wrote:
1. Modify configuration in some way so that the following can be specified:
a. The percentage value of used JVM memory (i.e. 95) at which entries should be
evicted to try and avoid an OOM error.
b. The number of items that should be evicted when memory reaches this threshold
2. Modify LRU and LIRS Eviction class so that the accessQueue member can be accessed
by the new Eviction class so that two access queues don’t have to be maintained.
3. Create a new Eviction class, a subclass of LIRS, where the accessQueue is used
from the Eviction strategy the user specifies and the for loop in the execute method is
exited when the evicted set equals value from 1.b above.
4. Modify DataContainerFactory.construct() to call
DefaultDataContainer.boundedDataContainer() regardless of eviction policy. This will
always create a BoundedConcurrentHashMap
I don't think you have to extend LRU and LIRS, as long as you implement
EvictionPolicy you are fullfilling the contract. What extending LRU and LIRS can do is
give you some order in selecting proper elements for eviction. I think the precise
technical term in research literature is "eviction precision".
1.
2. Create an instance of the new Eviction class in each segment.
3. Modify BoundedConcurrentHashMap.Segment put and replace methods such that when
new values are going to be put into the Segment, the memory usage is checked and the
execute method is called on the new Eviction class.
Sounds right!
1.
Questions:
1. What are the implications of using a BoundedConcurrentHashMap instead of a
ConcurrentHashMap when maxEntries is set to -1?
I think none except you turn on eviction by using BoundedConcurrentHashMap.
1.
Thoughts
1. This will not guarantee that an OOM error does not occur. It will attempt to
guard against an OOM caused by putting new values into the cache. This will probably be
more effective when the cache is being used in client/server mode, and less effective when
used in embedded mode as to other code running in the JVM.
-- Dave Marion
Cheers,
Vladimir
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
_______________________________________________ infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev__________________...
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
--
Galder Zamarreño
Sr. Software Engineer
Infinispan, JBoss Cache
2:39 a.m.
New subject: ISPN-863 - Thoughts / Questions
Hah, might be because the POM is configured to run tests in parallel with a
max heap size of 1GB. Can someone update
http://community.jboss.org/wiki/ParallelTestSuite or
http://community.jboss.org/wiki/InfinispanQuickStartGuideforDevelopers#Testi
ng with instructions on how to run test serially (I didnt find anything wrt
Maven surefire plugin)? Thanks
-----Original Message-----
From: infinispan-dev-bounces(a)lists.jboss.org
[mailto:infinispan-dev-bounces@lists.jboss.org] On Behalf Of Galder
Zamarreño
Sent: Tuesday, January 18, 2011 6:12 AM
To: infinispan -Dev List
Subject: Re: [infinispan-dev] ISPN-863 - Thoughts / Questions
Hmmm, try increasing -Xmx? Start with -XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=/tmp/java_heap and inspect the dump with Eclipse MAT?...
On Jan 17, 2011, at 4:51 PM, david marion wrote:
same result.
Date: Mon, 17 Jan 2011 11:54:15 -0300
From: vblagoje(a)redhat.com
To: infinispan-dev(a)lists.jboss.org
Subject: Re: [infinispan-dev] ISPN-863 - Thoughts / Questions
Its easier to run everything from command line for two reason I can think
of:
- everyone has same env to compare results with you
- it is all ready to go
See http://community.jboss.org/wiki/InfinispanandMaven
On 11-01-17 11:38 AM, Dave wrote:
All,
I am trying to test some changes that I made using Eclipse and the TestNG
plugin. I
am not familiar with TestNG. Are there any settings that I should
make? Any ideas?
java.lang.OutOfMemoryError: Java heap space
at
org.infinispan.util.concurrent.BoundedConcurrentHashMap$HashEntry.newArray(B
oundedConcurrentHashMap.java:295)
at
org.infinispan.util.concurrent.BoundedConcurrentHashMap$Segment.<init>(Bound
edConcurrentHashMap.java:898)
at
org.infinispan.util.concurrent.BoundedConcurrentHashMap.<init>(BoundedConcur
rentHashMap.java:1367)
at
org.infinispan.container.DefaultDataContainer.<init>(DefaultDataContainer.ja
va:78)
at
org.infinispan.container.DefaultDataContainer.<init>(DefaultDataContainer.ja
va:47)
at
org.infinispan.container.DefaultDataContainer.unBoundedDataContainer(Default
DataContainer.java:93)
at
org.infinispan.factories.DataContainerFactory.construct(DataContainerFactory
.java:53)
at
org.infinispan.factories.AbstractComponentRegistry.getOrCreateComponent(Abst
ractComponentRegistry.java:315)
at
org.infinispan.factories.AbstractComponentRegistry.invokeInjectionMethod(Abs
tractComponentRegistry.java:251)
at
org.infinispan.factories.AbstractComponentRegistry$Component.injectDependenc
ies(AbstractComponentRegistry.java:840)
at
org.infinispan.factories.AbstractComponentRegistry.registerComponent(Abstrac
tComponentRegistry.java:225)
at
org.infinispan.factories.ComponentRegistry.registerComponent(ComponentRegist
ry.java:120)
at
org.infinispan.factories.AbstractComponentRegistry.registerComponent(Abstrac
tComponentRegistry.java:192)
at
org.infinispan.factories.AbstractComponentRegistry.getOrCreateComponent(Abst
ractComponentRegistry.java:323)
at
org.infinispan.factories.AbstractComponentRegistry.invokeInjectionMethod(Abs
tractComponentRegistry.java:251)
at
org.infinispan.factories.AbstractComponentRegistry$Component.injectDependenc
ies(AbstractComponentRegistry.java:840)
at
org.infinispan.factories.AbstractComponentRegistry.registerComponent(Abstrac
tComponentRegistry.java:225)
at
org.infinispan.factories.ComponentRegistry.registerComponent(ComponentRegist
ry.java:120)
at
org.infinispan.factories.AbstractComponentRegistry.registerComponent(Abstrac
tComponentRegistry.java:192)
at
org.infinispan.factories.AbstractComponentRegistry.getOrCreateComponent(Abst
ractComponentRegistry.java:323)
at
org.infinispan.factories.AbstractComponentRegistry.invokeInjectionMethod(Abs
tractComponentRegistry.java:251)
at
org.infinispan.factories.AbstractComponentRegistry$Component.injectDependenc
ies(AbstractComponentRegistry.java:840)
at
org.infinispan.factories.AbstractComponentRegistry.registerComponent(Abstrac
tComponentRegistry.java:225)
at
org.infinispan.factories.ComponentRegistry.registerComponent(ComponentRegist
ry.java:120)
at
org.infinispan.factories.AbstractComponentRegistry.registerComponent(Abstrac
tComponentRegistry.java:192)
at
org.infinispan.factories.InternalCacheFactory.bootstrap(InternalCacheFactory
.java:92)
at
org.infinispan.factories.InternalCacheFactory.createAndWire(InternalCacheFac
tory.java:78)
at
org.infinispan.factories.InternalCacheFactory.createCache(InternalCacheFacto
ry.java:62)
at
org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.j
ava:510)
at
org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java
:440)
at
org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java
:409)
at
org.infinispan.config.ConfigurationValidationTest.testDefaultMemoryGuardConf
iguration(ConfigurationValidationTest.java:100)
-----Original Message-----
From: infinispan-dev-bounces(a)lists.jboss.org
[mailto:infinispan-dev-bounces@lists.jboss.org] On Behalf Of Vladimir
Blagojevic
Sent: Friday, January 14, 2011 11:05 AM
To: infinispan -Dev List
Subject: Re: [infinispan-dev] ISPN-863 - Thoughts / Questions
Hey David,
First of all kudos for taking on such a non-trivial task!
On 11-01-14 12:22 AM, david marion wrote:
1. Modify configuration in some way so that the following can be
specified:
a. The percentage value of used JVM memory (i.e. 95) at which
entries should be evicted to try and avoid an OOM error.
b. The number of items that should be evicted when memory
reaches
this threshold
2. Modify LRU and LIRS Eviction class so that the accessQueue
member
can be accessed by the new Eviction class so that two access queues dont
have to be maintained.
3. Create a new Eviction class, a subclass of LIRS, where the
accessQueue is used from the Eviction strategy the user specifies and the
for loop in the execute method is exited when the evicted set equals value
from 1.b above.
4. Modify DataContainerFactory.construct() to call
DefaultDataContainer.boundedDataContainer() regardless of eviction policy.
This will always create a BoundedConcurrentHashMap
I don't think you have to extend LRU and LIRS, as long as you implement
EvictionPolicy you are fullfilling the contract. What extending LRU and LIRS
can do is give you some order in selecting proper elements for eviction. I
think the precise technical term in research literature is "eviction
precision".
1.
2. Create an instance of the new Eviction class in each segment.
3. Modify BoundedConcurrentHashMap.Segment put and replace methods
such that
when new values are going to be put into the Segment, the memory
usage is checked and the execute method is called on the new Eviction class.
Sounds right!
1.
Questions:
1. What are the implications of using a BoundedConcurrentHashMap
instead of a
ConcurrentHashMap when maxEntries is set to -1?
I think none except you turn on eviction by using
BoundedConcurrentHashMap.
1.
Thoughts
1. This will not guarantee that an OOM error does not occur. It will
attempt
to guard against an OOM caused by putting new values into the cache.
This will probably be more effective when the cache is being used in
client/server mode, and less effective when used in embedded mode as to
other code running in the JVM.
-- Dave Marion
Cheers,
Vladimir
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
_______________________________________________ infinispan-dev mailing
list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev_____________________
__________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
--
Galder Zamarreño
Sr. Software Engineer
Infinispan, JBoss Cache
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
4:29 a.m.
New subject: ISPN-863 - Thoughts / Questions
Disregard, I found the issue. Sorry for the spam.
From: dlmarion(a)hotmail.com
To: infinispan-dev(a)lists.jboss.org
Date: Tue, 18 Jan 2011 20:39:21 -0500
Subject: Re: [infinispan-dev] ISPN-863 - Thoughts / Questions
Hah, might be because the POM is configured to run tests in parallel with a
max heap size of 1GB. Can someone update
http://community.jboss.org/wiki/ParallelTestSuite or
http://community.jboss.org/wiki/InfinispanQuickStartGuideforDevelopers#Testi
ng with instructions on how to run test serially (I didn’t find anything wrt
Maven surefire plugin)? Thanks
-----Original Message-----
From: infinispan-dev-bounces(a)lists.jboss.org
[mailto:infinispan-dev-bounces@lists.jboss.org] On Behalf Of Galder
Zamarreño
Sent: Tuesday, January 18, 2011 6:12 AM
To: infinispan -Dev List
Subject: Re: [infinispan-dev] ISPN-863 - Thoughts / Questions
Hmmm, try increasing -Xmx? Start with -XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=/tmp/java_heap and inspect the dump with Eclipse MAT?...
On Jan 17, 2011, at 4:51 PM, david marion wrote:
> same result.
>
> Date: Mon, 17 Jan 2011 11:54:15 -0300
> From: vblagoje(a)redhat.com
> To: infinispan-dev(a)lists.jboss.org
> Subject: Re: [infinispan-dev] ISPN-863 - Thoughts / Questions
>
> Its easier to run everything from command line for two reason I can think
of:
> - everyone has same env to compare results with you
> - it is all ready to go
>
> See http://community.jboss.org/wiki/InfinispanandMaven
>
> On 11-01-17 11:38 AM, Dave wrote:
> All,
>
> I am trying to test some changes that I made using Eclipse and the TestNG
plugin. I am not familiar with TestNG. Are there any settings that I should
make? Any ideas?
>
> java.lang.OutOfMemoryError: Java heap space
> at
org.infinispan.util.concurrent.BoundedConcurrentHashMap$HashEntry.newArray(B
oundedConcurrentHashMap.java:295)
> at
org.infinispan.util.concurrent.BoundedConcurrentHashMap$Segment.<init>(Bound
edConcurrentHashMap.java:898)
> at
org.infinispan.util.concurrent.BoundedConcurrentHashMap.<init>(BoundedConcur
rentHashMap.java:1367)
> at
org.infinispan.container.DefaultDataContainer.<init>(DefaultDataContainer.ja
va:78)
> at
org.infinispan.container.DefaultDataContainer.<init>(DefaultDataContainer.ja
va:47)
> at
org.infinispan.container.DefaultDataContainer.unBoundedDataContainer(Default
DataContainer.java:93)
> at
org.infinispan.factories.DataContainerFactory.construct(DataContainerFactory
.java:53)
> at
org.infinispan.factories.AbstractComponentRegistry.getOrCreateComponent(Abst
ractComponentRegistry.java:315)
> at
org.infinispan.factories.AbstractComponentRegistry.invokeInjectionMethod(Abs
tractComponentRegistry.java:251)
> at
org.infinispan.factories.AbstractComponentRegistry$Component.injectDependenc
ies(AbstractComponentRegistry.java:840)
> at
org.infinispan.factories.AbstractComponentRegistry.registerComponent(Abstrac
tComponentRegistry.java:225)
> at
org.infinispan.factories.ComponentRegistry.registerComponent(ComponentRegist
ry.java:120)
> at
org.infinispan.factories.AbstractComponentRegistry.registerComponent(Abstrac
tComponentRegistry.java:192)
> at
org.infinispan.factories.AbstractComponentRegistry.getOrCreateComponent(Abst
ractComponentRegistry.java:323)
> at
org.infinispan.factories.AbstractComponentRegistry.invokeInjectionMethod(Abs
tractComponentRegistry.java:251)
> at
org.infinispan.factories.AbstractComponentRegistry$Component.injectDependenc
ies(AbstractComponentRegistry.java:840)
> at
org.infinispan.factories.AbstractComponentRegistry.registerComponent(Abstrac
tComponentRegistry.java:225)
> at
org.infinispan.factories.ComponentRegistry.registerComponent(ComponentRegist
ry.java:120)
> at
org.infinispan.factories.AbstractComponentRegistry.registerComponent(Abstrac
tComponentRegistry.java:192)
> at
org.infinispan.factories.AbstractComponentRegistry.getOrCreateComponent(Abst
ractComponentRegistry.java:323)
> at
org.infinispan.factories.AbstractComponentRegistry.invokeInjectionMethod(Abs
tractComponentRegistry.java:251)
> at
org.infinispan.factories.AbstractComponentRegistry$Component.injectDependenc
ies(AbstractComponentRegistry.java:840)
> at
org.infinispan.factories.AbstractComponentRegistry.registerComponent(Abstrac
tComponentRegistry.java:225)
> at
org.infinispan.factories.ComponentRegistry.registerComponent(ComponentRegist
ry.java:120)
> at
org.infinispan.factories.AbstractComponentRegistry.registerComponent(Abstrac
tComponentRegistry.java:192)
> at
org.infinispan.factories.InternalCacheFactory.bootstrap(InternalCacheFactory
.java:92)
> at
org.infinispan.factories.InternalCacheFactory.createAndWire(InternalCacheFac
tory.java:78)
> at
org.infinispan.factories.InternalCacheFactory.createCache(InternalCacheFacto
ry.java:62)
> at
org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.j
ava:510)
> at
org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java
:440)
> at
org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java
:409)
> at
org.infinispan.config.ConfigurationValidationTest.testDefaultMemoryGuardConf
iguration(ConfigurationValidationTest.java:100)
>
>
>
> -----Original Message-----
> From: infinispan-dev-bounces(a)lists.jboss.org
[mailto:infinispan-dev-bounces@lists.jboss.org] On Behalf Of Vladimir
Blagojevic
> Sent: Friday, January 14, 2011 11:05 AM
> To: infinispan -Dev List
> Subject: Re: [infinispan-dev] ISPN-863 - Thoughts / Questions
>
> Hey David,
>
> First of all kudos for taking on such a non-trivial task!
>
> On 11-01-14 12:22 AM, david marion wrote:
> 1. Modify configuration in some way so that the following can be
specified:
> a. The percentage value of used JVM memory (i.e. 95) at which
entries should be evicted to try and avoid an OOM error.
> b. The number of items that should be evicted when memory reaches
this threshold
> 2. Modify LRU and LIRS Eviction class so that the accessQueue member
can be accessed by the new Eviction class so that two access queues don’t
have to be maintained.
> 3. Create a new Eviction class, a subclass of LIRS, where the
accessQueue is used from the Eviction strategy the user specifies and the
for loop in the execute method is exited when the evicted set equals value
from 1.b above.
> 4. Modify DataContainerFactory.construct() to call
DefaultDataContainer.boundedDataContainer() regardless of eviction policy.
This will always create a BoundedConcurrentHashMap
>
> I don't think you have to extend LRU and LIRS, as long as you implement
EvictionPolicy you are fullfilling the contract. What extending LRU and LIRS
can do is give you some order in selecting proper elements for eviction. I
think the precise technical term in research literature is "eviction
precision".
>
>
> 1.
> 2. Create an instance of the new Eviction class in each segment.
> 3. Modify BoundedConcurrentHashMap.Segment put and replace methods
such that when new values are going to be put into the Segment, the memory
usage is checked and the execute method is called on the new Eviction class.
>
> Sounds right!
>
>
> 1.
>
> Questions:
>
> 1. What are the implications of using a BoundedConcurrentHashMap
instead of a ConcurrentHashMap when maxEntries is set to -1?
>
> I think none except you turn on eviction by using
BoundedConcurrentHashMap.
>
>
> 1.
>
> Thoughts
>
> 1. This will not guarantee that an OOM error does not occur. It will
attempt to guard against an OOM caused by putting new values into the cache.
This will probably be more effective when the cache is being used in
client/server mode, and less effective when used in embedded mode as to
other code running in the JVM.
>
>
> -- Dave Marion
>
>
> Cheers,
> Vladimir
>
> _______________________________________________
> infinispan-dev mailing list
>
> infinispan-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
>
> _______________________________________________ infinispan-dev mailing
list infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev_____________________
__________________________
> infinispan-dev mailing list
> infinispan-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
--
Galder Zamarreño
Sr. Software Engineer
Infinispan, JBoss Cache
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
10:32 a.m.
New subject: ISPN-863 - Thoughts / Questions
Out of curiosity, what is a code problem on your side? Or a set up issue?
On Jan 19, 2011, at 4:29 AM, david marion wrote:
Disregard, I found the issue. Sorry for the spam.
> From: dlmarion(a)hotmail.com
> To: infinispan-dev(a)lists.jboss.org
> Date: Tue, 18 Jan 2011 20:39:21 -0500
> Subject: Re: [infinispan-dev] ISPN-863 - Thoughts / Questions
>
> Hah, might be because the POM is configured to run tests in parallel with a
> max heap size of 1GB. Can someone update
> http://community.jboss.org/wiki/ParallelTestSuite or
> http://community.jboss.org/wiki/InfinispanQuickStartGuideforDevelopers#Testi
> ng with instructions on how to run test serially (I didn’t find anything wrt
> Maven surefire plugin)? Thanks
>
> -----Original Message-----
> From: infinispan-dev-bounces(a)lists.jboss.org
> [mailto:infinispan-dev-bounces@lists.jboss.org] On Behalf Of Galder
> Zamarreño
> Sent: Tuesday, January 18, 2011 6:12 AM
> To: infinispan -Dev List
> Subject: Re: [infinispan-dev] ISPN-863 - Thoughts / Questions
>
> Hmmm, try increasing -Xmx? Start with -XX:+HeapDumpOnOutOfMemoryError
> -XX:HeapDumpPath=/tmp/java_heap and inspect the dump with Eclipse MAT?...
>
> On Jan 17, 2011, at 4:51 PM, david marion wrote:
>
> > same result.
> >
> > Date: Mon, 17 Jan 2011 11:54:15 -0300
> > From: vblagoje(a)redhat.com
> > To: infinispan-dev(a)lists.jboss.org
> > Subject: Re: [infinispan-dev] ISPN-863 - Thoughts / Questions
> >
> > Its easier to run everything from command line for two reason I can think
> of:
> > - everyone has same env to compare results with you
> > - it is all ready to go
> >
> > See http://community.jboss.org/wiki/InfinispanandMaven
> >
> > On 11-01-17 11:38 AM, Dave wrote:
> > All,
> >
> > I am trying to test some changes that I made using Eclipse and the TestNG
> plugin. I am not familiar with TestNG. Are there any settings that I should
> make? Any ideas?
> >
> > java.lang.OutOfMemoryError: Java heap space
> > at
> org.infinispan.util.concurrent.BoundedConcurrentHashMap$HashEntry.newArray(B
> oundedConcurrentHashMap.java:295)
> > at
> org.infinispan.util.concurrent.BoundedConcurrentHashMap$Segment.<init>(Bound
> edConcurrentHashMap.java:898)
> > at
> org.infinispan.util.concurrent.BoundedConcurrentHashMap.<init>(BoundedConcur
> rentHashMap.java:1367)
> > at
> org.infinispan.container.DefaultDataContainer.<init>(DefaultDataContainer.ja
> va:78)
> > at
> org.infinispan.container.DefaultDataContainer.<init>(DefaultDataContainer.ja
> va:47)
> > at
> org.infinispan.container.DefaultDataContainer.unBoundedDataContainer(Default
> DataContainer.java:93)
> > at
> org.infinispan.factories.DataContainerFactory.construct(DataContainerFactory
> .java:53)
> > at
> org.infinispan.factories.AbstractComponentRegistry.getOrCreateComponent(Abst
> ractComponentRegistry.java:315)
> > at
> org.infinispan.factories.AbstractComponentRegistry.invokeInjectionMethod(Abs
> tractComponentRegistry.java:251)
> > at
> org.infinispan.factories.AbstractComponentRegistry$Component.injectDependenc
> ies(AbstractComponentRegistry.java:840)
> > at
> org.infinispan.factories.AbstractComponentRegistry.registerComponent(Abstrac
> tComponentRegistry.java:225)
> > at
> org.infinispan.factories.ComponentRegistry.registerComponent(ComponentRegist
> ry.java:120)
> > at
> org.infinispan.factories.AbstractComponentRegistry.registerComponent(Abstrac
> tComponentRegistry.java:192)
> > at
> org.infinispan.factories.AbstractComponentRegistry.getOrCreateComponent(Abst
> ractComponentRegistry.java:323)
> > at
> org.infinispan.factories.AbstractComponentRegistry.invokeInjectionMethod(Abs
> tractComponentRegistry.java:251)
> > at
> org.infinispan.factories.AbstractComponentRegistry$Component.injectDependenc
> ies(AbstractComponentRegistry.java:840)
> > at
> org.infinispan.factories.AbstractComponentRegistry.registerComponent(Abstrac
> tComponentRegistry.java:225)
> > at
> org.infinispan.factories.ComponentRegistry.registerComponent(ComponentRegist
> ry.java:120)
> > at
> org.infinispan.factories.AbstractComponentRegistry.registerComponent(Abstrac
> tComponentRegistry.java:192)
> > at
> org.infinispan.factories.AbstractComponentRegistry.getOrCreateComponent(Abst
> ractComponentRegistry.java:323)
> > at
> org.infinispan.factories.AbstractComponentRegistry.invokeInjectionMethod(Abs
> tractComponentRegistry.java:251)
> > at
> org.infinispan.factories.AbstractComponentRegistry$Component.injectDependenc
> ies(AbstractComponentRegistry.java:840)
> > at
> org.infinispan.factories.AbstractComponentRegistry.registerComponent(Abstrac
> tComponentRegistry.java:225)
> > at
> org.infinispan.factories.ComponentRegistry.registerComponent(ComponentRegist
> ry.java:120)
> > at
> org.infinispan.factories.AbstractComponentRegistry.registerComponent(Abstrac
> tComponentRegistry.java:192)
> > at
> org.infinispan.factories.InternalCacheFactory.bootstrap(InternalCacheFactory
> .java:92)
> > at
> org.infinispan.factories.InternalCacheFactory.createAndWire(InternalCacheFac
> tory.java:78)
> > at
> org.infinispan.factories.InternalCacheFactory.createCache(InternalCacheFacto
> ry.java:62)
> > at
> org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.j
> ava:510)
> > at
> org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java
> :440)
> > at
> org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java
> :409)
> > at
> org.infinispan.config.ConfigurationValidationTest.testDefaultMemoryGuardConf
> iguration(ConfigurationValidationTest.java:100)
> >
> >
> >
> > -----Original Message-----
> > From: infinispan-dev-bounces(a)lists.jboss.org
> [mailto:infinispan-dev-bounces@lists.jboss.org] On Behalf Of Vladimir
> Blagojevic
> > Sent: Friday, January 14, 2011 11:05 AM
> > To: infinispan -Dev List
> > Subject: Re: [infinispan-dev] ISPN-863 - Thoughts / Questions
> >
> > Hey David,
> >
> > First of all kudos for taking on such a non-trivial task!
> >
> > On 11-01-14 12:22 AM, david marion wrote:
> > 1. Modify configuration in some way so that the following can be
> specified:
> > a. The percentage value of used JVM memory (i.e. 95) at which
> entries should be evicted to try and avoid an OOM error.
> > b. The number of items that should be evicted when memory reaches
> this threshold
> > 2. Modify LRU and LIRS Eviction class so that the accessQueue member
> can be accessed by the new Eviction class so that two access queues don’t
> have to be maintained.
> > 3. Create a new Eviction class, a subclass of LIRS, where the
> accessQueue is used from the Eviction strategy the user specifies and the
> for loop in the execute method is exited when the evicted set equals value
> from 1.b above.
> > 4. Modify DataContainerFactory.construct() to call
> DefaultDataContainer.boundedDataContainer() regardless of eviction policy.
> This will always create a BoundedConcurrentHashMap
> >
> > I don't think you have to extend LRU and LIRS, as long as you implement
> EvictionPolicy you are fullfilling the contract. What extending LRU and LIRS
> can do is give you some order in selecting proper elements for eviction. I
> think the precise technical term in research literature is "eviction
> precision".
> >
> >
> > 1.
> > 2. Create an instance of the new Eviction class in each segment.
> > 3. Modify BoundedConcurrentHashMap.Segment put and replace methods
> such that when new values are going to be put into the Segment, the memory
> usage is checked and the execute method is called on the new Eviction class.
> >
> > Sounds right!
> >
> >
> > 1.
> >
> > Questions:
> >
> > 1. What are the implications of using a BoundedConcurrentHashMap
> instead of a ConcurrentHashMap when maxEntries is set to -1?
> >
> > I think none except you turn on eviction by using
> BoundedConcurrentHashMap.
> >
> >
> > 1.
> >
> > Thoughts
> >
> > 1. This will not guarantee that an OOM error does not occur. It will
> attempt to guard against an OOM caused by putting new values into the cache.
> This will probably be more effective when the cache is being used in
> client/server mode, and less effective when used in embedded mode as to
> other code running in the JVM.
> >
> >
> > -- Dave Marion
> >
> >
> > Cheers,
> > Vladimir
> >
> > _______________________________________________
> > infinispan-dev mailing list
> >
> > infinispan-dev(a)lists.jboss.org
> > https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >
> >
> > _______________________________________________ infinispan-dev mailing
> list infinispan-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev_____________________
> __________________________
> > infinispan-dev mailing list
> > infinispan-dev(a)lists.jboss.org
> > https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
> --
> Galder Zamarreño
> Sr. Software Engineer
> Infinispan, JBoss Cache
>
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
--
Galder Zamarreño
Sr. Software Engineer
Infinispan, JBoss Cache
1:47 p.m.
New subject: ISPN-863 - Thoughts / Questions
It was my problem. If -1 is set for maxEntries, then I am creating a
BoundedConcurrentHashMap with a size of Integer.MAX_VALUE. Then when I ran the tests all
of them ran out of memory. I should have looked at the stack trace a little more closely.
I am going to have to put logic in so that the maximum capacity is determined by the
amount of free memory when the cache is created. Previously, if -1 was set for maxEntries,
a ConcurrentHashMap was created with maximum capacity of Integer.MAX_VALUE. I have not
looked at the code, but my guess is that it works because it grows over time and does not
pre-allocate all structures ahead of time. Any thoughts on logic for determining maximum
capacity at creation time?
From: galder(a)redhat.com
Date: Wed, 19 Jan 2011 10:32:42 +0100
To: infinispan-dev(a)lists.jboss.org
Subject: Re: [infinispan-dev] ISPN-863 - Thoughts / Questions
Out of curiosity, what is a code problem on your side? Or a set up issue?
On Jan 19, 2011, at 4:29 AM, david marion wrote:
> Disregard, I found the issue. Sorry for the spam.
>
> > From: dlmarion(a)hotmail.com
> > To: infinispan-dev(a)lists.jboss.org
> > Date: Tue, 18 Jan 2011 20:39:21 -0500
> > Subject: Re: [infinispan-dev] ISPN-863 - Thoughts / Questions
> >
> > Hah, might be because the POM is configured to run tests in parallel with a
> > max heap size of 1GB. Can someone update
> > http://community.jboss.org/wiki/ParallelTestSuite or
> > http://community.jboss.org/wiki/InfinispanQuickStartGuideforDevelopers#Testi
> > ng with instructions on how to run test serially (I didn’t find anything wrt
> > Maven surefire plugin)? Thanks
> >
> > -----Original Message-----
> > From: infinispan-dev-bounces(a)lists.jboss.org
> > [mailto:infinispan-dev-bounces@lists.jboss.org] On Behalf Of Galder
> > Zamarreño
> > Sent: Tuesday, January 18, 2011 6:12 AM
> > To: infinispan -Dev List
> > Subject: Re: [infinispan-dev] ISPN-863 - Thoughts / Questions
> >
> > Hmmm, try increasing -Xmx? Start with -XX:+HeapDumpOnOutOfMemoryError
> > -XX:HeapDumpPath=/tmp/java_heap and inspect the dump with Eclipse MAT?...
> >
> > On Jan 17, 2011, at 4:51 PM, david marion wrote:
> >
> > > same result.
> > >
> > > Date: Mon, 17 Jan 2011 11:54:15 -0300
> > > From: vblagoje(a)redhat.com
> > > To: infinispan-dev(a)lists.jboss.org
> > > Subject: Re: [infinispan-dev] ISPN-863 - Thoughts / Questions
> > >
> > > Its easier to run everything from command line for two reason I can think
> > of:
> > > - everyone has same env to compare results with you
> > > - it is all ready to go
> > >
> > > See http://community.jboss.org/wiki/InfinispanandMaven
> > >
> > > On 11-01-17 11:38 AM, Dave wrote:
> > > All,
> > >
> > > I am trying to test some changes that I made using Eclipse and the TestNG
> > plugin. I am not familiar with TestNG. Are there any settings that I should
> > make? Any ideas?
> > >
> > > java.lang.OutOfMemoryError: Java heap space
> > > at
> > org.infinispan.util.concurrent.BoundedConcurrentHashMap$HashEntry.newArray(B
> > oundedConcurrentHashMap.java:295)
> > > at
> >
org.infinispan.util.concurrent.BoundedConcurrentHashMap$Segment.<init>(Bound
> > edConcurrentHashMap.java:898)
> > > at
> >
org.infinispan.util.concurrent.BoundedConcurrentHashMap.<init>(BoundedConcur
> > rentHashMap.java:1367)
> > > at
> >
org.infinispan.container.DefaultDataContainer.<init>(DefaultDataContainer.ja
> > va:78)
> > > at
> >
org.infinispan.container.DefaultDataContainer.<init>(DefaultDataContainer.ja
> > va:47)
> > > at
> > org.infinispan.container.DefaultDataContainer.unBoundedDataContainer(Default
> > DataContainer.java:93)
> > > at
> > org.infinispan.factories.DataContainerFactory.construct(DataContainerFactory
> > .java:53)
> > > at
> > org.infinispan.factories.AbstractComponentRegistry.getOrCreateComponent(Abst
> > ractComponentRegistry.java:315)
> > > at
> > org.infinispan.factories.AbstractComponentRegistry.invokeInjectionMethod(Abs
> > tractComponentRegistry.java:251)
> > > at
> > org.infinispan.factories.AbstractComponentRegistry$Component.injectDependenc
> > ies(AbstractComponentRegistry.java:840)
> > > at
> > org.infinispan.factories.AbstractComponentRegistry.registerComponent(Abstrac
> > tComponentRegistry.java:225)
> > > at
> > org.infinispan.factories.ComponentRegistry.registerComponent(ComponentRegist
> > ry.java:120)
> > > at
> > org.infinispan.factories.AbstractComponentRegistry.registerComponent(Abstrac
> > tComponentRegistry.java:192)
> > > at
> > org.infinispan.factories.AbstractComponentRegistry.getOrCreateComponent(Abst
> > ractComponentRegistry.java:323)
> > > at
> > org.infinispan.factories.AbstractComponentRegistry.invokeInjectionMethod(Abs
> > tractComponentRegistry.java:251)
> > > at
> > org.infinispan.factories.AbstractComponentRegistry$Component.injectDependenc
> > ies(AbstractComponentRegistry.java:840)
> > > at
> > org.infinispan.factories.AbstractComponentRegistry.registerComponent(Abstrac
> > tComponentRegistry.java:225)
> > > at
> > org.infinispan.factories.ComponentRegistry.registerComponent(ComponentRegist
> > ry.java:120)
> > > at
> > org.infinispan.factories.AbstractComponentRegistry.registerComponent(Abstrac
> > tComponentRegistry.java:192)
> > > at
> > org.infinispan.factories.AbstractComponentRegistry.getOrCreateComponent(Abst
> > ractComponentRegistry.java:323)
> > > at
> > org.infinispan.factories.AbstractComponentRegistry.invokeInjectionMethod(Abs
> > tractComponentRegistry.java:251)
> > > at
> > org.infinispan.factories.AbstractComponentRegistry$Component.injectDependenc
> > ies(AbstractComponentRegistry.java:840)
> > > at
> > org.infinispan.factories.AbstractComponentRegistry.registerComponent(Abstrac
> > tComponentRegistry.java:225)
> > > at
> > org.infinispan.factories.ComponentRegistry.registerComponent(ComponentRegist
> > ry.java:120)
> > > at
> > org.infinispan.factories.AbstractComponentRegistry.registerComponent(Abstrac
> > tComponentRegistry.java:192)
> > > at
> > org.infinispan.factories.InternalCacheFactory.bootstrap(InternalCacheFactory
> > .java:92)
> > > at
> > org.infinispan.factories.InternalCacheFactory.createAndWire(InternalCacheFac
> > tory.java:78)
> > > at
> > org.infinispan.factories.InternalCacheFactory.createCache(InternalCacheFacto
> > ry.java:62)
> > > at
> > org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.j
> > ava:510)
> > > at
> > org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java
> > :440)
> > > at
> > org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java
> > :409)
> > > at
> > org.infinispan.config.ConfigurationValidationTest.testDefaultMemoryGuardConf
> > iguration(ConfigurationValidationTest.java:100)
> > >
> > >
> > >
> > > -----Original Message-----
> > > From: infinispan-dev-bounces(a)lists.jboss.org
> > [mailto:infinispan-dev-bounces@lists.jboss.org] On Behalf Of Vladimir
> > Blagojevic
> > > Sent: Friday, January 14, 2011 11:05 AM
> > > To: infinispan -Dev List
> > > Subject: Re: [infinispan-dev] ISPN-863 - Thoughts / Questions
> > >
> > > Hey David,
> > >
> > > First of all kudos for taking on such a non-trivial task!
> > >
> > > On 11-01-14 12:22 AM, david marion wrote:
> > > 1. Modify configuration in some way so that the following can be
> > specified:
> > > a. The percentage value of used JVM memory (i.e. 95) at which
> > entries should be evicted to try and avoid an OOM error.
> > > b. The number of items that should be evicted when memory reaches
> > this threshold
> > > 2. Modify LRU and LIRS Eviction class so that the accessQueue member
> > can be accessed by the new Eviction class so that two access queues don’t
> > have to be maintained.
> > > 3. Create a new Eviction class, a subclass of LIRS, where the
> > accessQueue is used from the Eviction strategy the user specifies and the
> > for loop in the execute method is exited when the evicted set equals value
> > from 1.b above.
> > > 4. Modify DataContainerFactory.construct() to call
> > DefaultDataContainer.boundedDataContainer() regardless of eviction policy.
> > This will always create a BoundedConcurrentHashMap
> > >
> > > I don't think you have to extend LRU and LIRS, as long as you
implement
> > EvictionPolicy you are fullfilling the contract. What extending LRU and LIRS
> > can do is give you some order in selecting proper elements for eviction. I
> > think the precise technical term in research literature is "eviction
> > precision".
> > >
> > >
> > > 1.
> > > 2. Create an instance of the new Eviction class in each segment.
> > > 3. Modify BoundedConcurrentHashMap.Segment put and replace methods
> > such that when new values are going to be put into the Segment, the memory
> > usage is checked and the execute method is called on the new Eviction class.
> > >
> > > Sounds right!
> > >
> > >
> > > 1.
> > >
> > > Questions:
> > >
> > > 1. What are the implications of using a BoundedConcurrentHashMap
> > instead of a ConcurrentHashMap when maxEntries is set to -1?
> > >
> > > I think none except you turn on eviction by using
> > BoundedConcurrentHashMap.
> > >
> > >
> > > 1.
> > >
> > > Thoughts
> > >
> > > 1. This will not guarantee that an OOM error does not occur. It will
> > attempt to guard against an OOM caused by putting new values into the cache.
> > This will probably be more effective when the cache is being used in
> > client/server mode, and less effective when used in embedded mode as to
> > other code running in the JVM.
> > >
> > >
> > > -- Dave Marion
> > >
> > >
> > > Cheers,
> > > Vladimir
> > >
> > > _______________________________________________
> > > infinispan-dev mailing list
> > >
> > > infinispan-dev(a)lists.jboss.org
> > > https://lists.jboss.org/mailman/listinfo/infinispan-dev
> > >
> > >
> > > _______________________________________________ infinispan-dev mailing
> > list infinispan-dev(a)lists.jboss.org
> > https://lists.jboss.org/mailman/listinfo/infinispan-dev_____________________
> > __________________________
> > > infinispan-dev mailing list
> > > infinispan-dev(a)lists.jboss.org
> > > https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >
> > --
> > Galder Zamarreño
> > Sr. Software Engineer
> > Infinispan, JBoss Cache
> >
> >
> > _______________________________________________
> > infinispan-dev mailing list
> > infinispan-dev(a)lists.jboss.org
> > https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >
> >
> > _______________________________________________
> > infinispan-dev mailing list
> > infinispan-dev(a)lists.jboss.org
> > https://lists.jboss.org/mailman/listinfo/infinispan-dev
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
--
Galder Zamarreño
Sr. Software Engineer
Infinispan, JBoss Cache
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
9:38 a.m.
New subject: ISPN-863 - Thoughts / Questions
On Jan 19, 2011, at 1:47 PM, david marion wrote:
It was my problem. If -1 is set for maxEntries, then I am creating a
BoundedConcurrentHashMap with a size of Integer.MAX_VALUE. Then when I ran the tests all
of them ran out of memory. I should have looked at the stack trace a little more closely.
I am going to have to put logic in so that the maximum capacity is determined by the
amount of free memory when the cache is created. Previously, if -1 was set for maxEntries,
a ConcurrentHashMap was created with maximum capacity of Integer.MAX_VALUE. I have not
looked at the code, but my guess is that it works because it grows over time and does not
pre-allocate all structures ahead of time.
Hmmm, that sounds odd. Maybe you wanna build a test to show the exact differences and see
if too much is being allocated on startup?
Any thoughts on logic for determining maximum capacity at creation
time?
A percentage of the available free memory? 40% by default? You have to accomodate for the
other apps running on the same JVM, so we shouldn't be too aggressive by default.
> From: galder(a)redhat.com
> Date: Wed, 19 Jan 2011 10:32:42 +0100
> To: infinispan-dev(a)lists.jboss.org
> Subject: Re: [infinispan-dev] ISPN-863 - Thoughts / Questions
>
> Out of curiosity, what is a code problem on your side? Or a set up issue?
>
> On Jan 19, 2011, at 4:29 AM, david marion wrote:
>
> > Disregard, I found the issue. Sorry for the spam.
> >
> > > From: dlmarion(a)hotmail.com
> > > To: infinispan-dev(a)lists.jboss.org
> > > Date: Tue, 18 Jan 2011 20:39:21 -0500
> > > Subject: Re: [infinispan-dev] ISPN-863 - Thoughts / Questions
> > >
> > > Hah, might be because the POM is configured to run tests in parallel with
a
> > > max heap size of 1GB. Can someone update
> > > http://community.jboss.org/wiki/ParallelTestSuite or
> > >
http://community.jboss.org/wiki/InfinispanQuickStartGuideforDevelopers#Testi
> > > ng with instructions on how to run test serially (I didn’t find anything
wrt
> > > Maven surefire plugin)? Thanks
> > >
> > > -----Original Message-----
> > > From: infinispan-dev-bounces(a)lists.jboss.org
> > > [mailto:infinispan-dev-bounces@lists.jboss.org] On Behalf Of Galder
> > > Zamarreño
> > > Sent: Tuesday, January 18, 2011 6:12 AM
> > > To: infinispan -Dev List
> > > Subject: Re: [infinispan-dev] ISPN-863 - Thoughts / Questions
> > >
> > > Hmmm, try increasing -Xmx? Start with -XX:+HeapDumpOnOutOfMemoryError
> > > -XX:HeapDumpPath=/tmp/java_heap and inspect the dump with Eclipse MAT?...
> > >
> > > On Jan 17, 2011, at 4:51 PM, david marion wrote:
> > >
> > > > same result.
> > > >
> > > > Date: Mon, 17 Jan 2011 11:54:15 -0300
> > > > From: vblagoje(a)redhat.com
> > > > To: infinispan-dev(a)lists.jboss.org
> > > > Subject: Re: [infinispan-dev] ISPN-863 - Thoughts / Questions
> > > >
> > > > Its easier to run everything from command line for two reason I can
think
> > > of:
> > > > - everyone has same env to compare results with you
> > > > - it is all ready to go
> > > >
> > > > See http://community.jboss.org/wiki/InfinispanandMaven
> > > >
> > > > On 11-01-17 11:38 AM, Dave wrote:
> > > > All,
> > > >
> > > > I am trying to test some changes that I made using Eclipse and the
TestNG
> > > plugin. I am not familiar with TestNG. Are there any settings that I
should
> > > make? Any ideas?
> > > >
> > > > java.lang.OutOfMemoryError: Java heap space
> > > > at
> > >
org.infinispan.util.concurrent.BoundedConcurrentHashMap$HashEntry.newArray(B
> > > oundedConcurrentHashMap.java:295)
> > > > at
> > >
org.infinispan.util.concurrent.BoundedConcurrentHashMap$Segment.<init>(Bound
> > > edConcurrentHashMap.java:898)
> > > > at
> > >
org.infinispan.util.concurrent.BoundedConcurrentHashMap.<init>(BoundedConcur
> > > rentHashMap.java:1367)
> > > > at
> > >
org.infinispan.container.DefaultDataContainer.<init>(DefaultDataContainer.ja
> > > va:78)
> > > > at
> > >
org.infinispan.container.DefaultDataContainer.<init>(DefaultDataContainer.ja
> > > va:47)
> > > > at
> > >
org.infinispan.container.DefaultDataContainer.unBoundedDataContainer(Default
> > > DataContainer.java:93)
> > > > at
> > >
org.infinispan.factories.DataContainerFactory.construct(DataContainerFactory
> > > .java:53)
> > > > at
> > >
org.infinispan.factories.AbstractComponentRegistry.getOrCreateComponent(Abst
> > > ractComponentRegistry.java:315)
> > > > at
> > >
org.infinispan.factories.AbstractComponentRegistry.invokeInjectionMethod(Abs
> > > tractComponentRegistry.java:251)
> > > > at
> > >
org.infinispan.factories.AbstractComponentRegistry$Component.injectDependenc
> > > ies(AbstractComponentRegistry.java:840)
> > > > at
> > >
org.infinispan.factories.AbstractComponentRegistry.registerComponent(Abstrac
> > > tComponentRegistry.java:225)
> > > > at
> > >
org.infinispan.factories.ComponentRegistry.registerComponent(ComponentRegist
> > > ry.java:120)
> > > > at
> > >
org.infinispan.factories.AbstractComponentRegistry.registerComponent(Abstrac
> > > tComponentRegistry.java:192)
> > > > at
> > >
org.infinispan.factories.AbstractComponentRegistry.getOrCreateComponent(Abst
> > > ractComponentRegistry.java:323)
> > > > at
> > >
org.infinispan.factories.AbstractComponentRegistry.invokeInjectionMethod(Abs
> > > tractComponentRegistry.java:251)
> > > > at
> > >
org.infinispan.factories.AbstractComponentRegistry$Component.injectDependenc
> > > ies(AbstractComponentRegistry.java:840)
> > > > at
> > >
org.infinispan.factories.AbstractComponentRegistry.registerComponent(Abstrac
> > > tComponentRegistry.java:225)
> > > > at
> > >
org.infinispan.factories.ComponentRegistry.registerComponent(ComponentRegist
> > > ry.java:120)
> > > > at
> > >
org.infinispan.factories.AbstractComponentRegistry.registerComponent(Abstrac
> > > tComponentRegistry.java:192)
> > > > at
> > >
org.infinispan.factories.AbstractComponentRegistry.getOrCreateComponent(Abst
> > > ractComponentRegistry.java:323)
> > > > at
> > >
org.infinispan.factories.AbstractComponentRegistry.invokeInjectionMethod(Abs
> > > tractComponentRegistry.java:251)
> > > > at
> > >
org.infinispan.factories.AbstractComponentRegistry$Component.injectDependenc
> > > ies(AbstractComponentRegistry.java:840)
> > > > at
> > >
org.infinispan.factories.AbstractComponentRegistry.registerComponent(Abstrac
> > > tComponentRegistry.java:225)
> > > > at
> > >
org.infinispan.factories.ComponentRegistry.registerComponent(ComponentRegist
> > > ry.java:120)
> > > > at
> > >
org.infinispan.factories.AbstractComponentRegistry.registerComponent(Abstrac
> > > tComponentRegistry.java:192)
> > > > at
> > >
org.infinispan.factories.InternalCacheFactory.bootstrap(InternalCacheFactory
> > > .java:92)
> > > > at
> > >
org.infinispan.factories.InternalCacheFactory.createAndWire(InternalCacheFac
> > > tory.java:78)
> > > > at
> > >
org.infinispan.factories.InternalCacheFactory.createCache(InternalCacheFacto
> > > ry.java:62)
> > > > at
> > >
org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.j
> > > ava:510)
> > > > at
> > >
org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java
> > > :440)
> > > > at
> > >
org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java
> > > :409)
> > > > at
> > >
org.infinispan.config.ConfigurationValidationTest.testDefaultMemoryGuardConf
> > > iguration(ConfigurationValidationTest.java:100)
> > > >
> > > >
> > > >
> > > > -----Original Message-----
> > > > From: infinispan-dev-bounces(a)lists.jboss.org
> > > [mailto:infinispan-dev-bounces@lists.jboss.org] On Behalf Of Vladimir
> > > Blagojevic
> > > > Sent: Friday, January 14, 2011 11:05 AM
> > > > To: infinispan -Dev List
> > > > Subject: Re: [infinispan-dev] ISPN-863 - Thoughts / Questions
> > > >
> > > > Hey David,
> > > >
> > > > First of all kudos for taking on such a non-trivial task!
> > > >
> > > > On 11-01-14 12:22 AM, david marion wrote:
> > > > 1. Modify configuration in some way so that the following can be
> > > specified:
> > > > a. The percentage value of used JVM memory (i.e. 95) at which
> > > entries should be evicted to try and avoid an OOM error.
> > > > b. The number of items that should be evicted when memory reaches
> > > this threshold
> > > > 2. Modify LRU and LIRS Eviction class so that the accessQueue member
> > > can be accessed by the new Eviction class so that two access queues don’t
> > > have to be maintained.
> > > > 3. Create a new Eviction class, a subclass of LIRS, where the
> > > accessQueue is used from the Eviction strategy the user specifies and the
> > > for loop in the execute method is exited when the evicted set equals
value
> > > from 1.b above.
> > > > 4. Modify DataContainerFactory.construct() to call
> > > DefaultDataContainer.boundedDataContainer() regardless of eviction
policy.
> > > This will always create a BoundedConcurrentHashMap
> > > >
> > > > I don't think you have to extend LRU and LIRS, as long as you
implement
> > > EvictionPolicy you are fullfilling the contract. What extending LRU and
LIRS
> > > can do is give you some order in selecting proper elements for eviction.
I
> > > think the precise technical term in research literature is "eviction
> > > precision".
> > > >
> > > >
> > > > 1.
> > > > 2. Create an instance of the new Eviction class in each segment.
> > > > 3. Modify BoundedConcurrentHashMap.Segment put and replace methods
> > > such that when new values are going to be put into the Segment, the
memory
> > > usage is checked and the execute method is called on the new Eviction
class.
> > > >
> > > > Sounds right!
> > > >
> > > >
> > > > 1.
> > > >
> > > > Questions:
> > > >
> > > > 1. What are the implications of using a BoundedConcurrentHashMap
> > > instead of a ConcurrentHashMap when maxEntries is set to -1?
> > > >
> > > > I think none except you turn on eviction by using
> > > BoundedConcurrentHashMap.
> > > >
> > > >
> > > > 1.
> > > >
> > > > Thoughts
> > > >
> > > > 1. This will not guarantee that an OOM error does not occur. It will
> > > attempt to guard against an OOM caused by putting new values into the
cache.
> > > This will probably be more effective when the cache is being used in
> > > client/server mode, and less effective when used in embedded mode as to
> > > other code running in the JVM.
> > > >
> > > >
> > > > -- Dave Marion
> > > >
> > > >
> > > > Cheers,
> > > > Vladimir
> > > >
> > > > _______________________________________________
> > > > infinispan-dev mailing list
> > > >
> > > > infinispan-dev(a)lists.jboss.org
> > > > https://lists.jboss.org/mailman/listinfo/infinispan-dev
> > > >
> > > >
> > > > _______________________________________________ infinispan-dev
mailing
> > > list infinispan-dev(a)lists.jboss.org
> > >
https://lists.jboss.org/mailman/listinfo/infinispan-dev_____________________
> > > __________________________
> > > > infinispan-dev mailing list
> > > > infinispan-dev(a)lists.jboss.org
> > > > https://lists.jboss.org/mailman/listinfo/infinispan-dev
> > >
> > > --
> > > Galder Zamarreño
> > > Sr. Software Engineer
> > > Infinispan, JBoss Cache
> > >
> > >
> > > _______________________________________________
> > > infinispan-dev mailing list
> > > infinispan-dev(a)lists.jboss.org
> > > https://lists.jboss.org/mailman/listinfo/infinispan-dev
> > >
> > >
> > > _______________________________________________
> > > infinispan-dev mailing list
> > > infinispan-dev(a)lists.jboss.org
> > > https://lists.jboss.org/mailman/listinfo/infinispan-dev
> > _______________________________________________
> > infinispan-dev mailing list
> > infinispan-dev(a)lists.jboss.org
> > https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
> --
> Galder Zamarreño
> Sr. Software Engineer
> Infinispan, JBoss Cache
>
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
--
Galder Zamarreño
Sr. Software Engineer
Infinispan, JBoss Cache
1:52 p.m.
New subject: ISPN-863 - Thoughts / Questions
I did exactly that last night. I built a test that allocated HashEntry<K,V>[] just
like it does in the Segment constructor. I don't have the numbers in front of me right
now, but at the maximum size (2^30) and without specifying concurrency it was trying to
allocate 32 Segment objects each having a HashEntry<K,V>[] with 2^25 elements. Each
one of these HashEntry<K,V>[] takes up around 128MB of memory.
From: galder(a)redhat.com
Date: Thu, 20 Jan 2011 09:38:43 +0100
To: infinispan-dev(a)lists.jboss.org
Subject: Re: [infinispan-dev] ISPN-863 - Thoughts / Questions
On Jan 19, 2011, at 1:47 PM, david marion wrote:
>
> It was my problem. If -1 is set for maxEntries, then I am creating a
BoundedConcurrentHashMap with a size of Integer.MAX_VALUE. Then when I ran the tests all
of them ran out of memory. I should have looked at the stack trace a little more closely.
I am going to have to put logic in so that the maximum capacity is determined by the
amount of free memory when the cache is created. Previously, if -1 was set for maxEntries,
a ConcurrentHashMap was created with maximum capacity of Integer.MAX_VALUE. I have not
looked at the code, but my guess is that it works because it grows over time and does not
pre-allocate all structures ahead of time.
Hmmm, that sounds odd. Maybe you wanna build a test to show the exact differences and see
if too much is being allocated on startup?
> Any thoughts on logic for determining maximum capacity at creation time?
A percentage of the available free memory? 40% by default? You have to accomodate for the
other apps running on the same JVM, so we shouldn't be too aggressive by default.
>
> > From: galder(a)redhat.com
> > Date: Wed, 19 Jan 2011 10:32:42 +0100
> > To: infinispan-dev(a)lists.jboss.org
> > Subject: Re: [infinispan-dev] ISPN-863 - Thoughts / Questions
> >
> > Out of curiosity, what is a code problem on your side? Or a set up issue?
> >
> > On Jan 19, 2011, at 4:29 AM, david marion wrote:
> >
> > > Disregard, I found the issue. Sorry for the spam.
> > >
> > > > From: dlmarion(a)hotmail.com
> > > > To: infinispan-dev(a)lists.jboss.org
> > > > Date: Tue, 18 Jan 2011 20:39:21 -0500
> > > > Subject: Re: [infinispan-dev] ISPN-863 - Thoughts / Questions
> > > >
> > > > Hah, might be because the POM is configured to run tests in parallel
with a
> > > > max heap size of 1GB. Can someone update
> > > > http://community.jboss.org/wiki/ParallelTestSuite or
> > > >
http://community.jboss.org/wiki/InfinispanQuickStartGuideforDevelopers#Testi
> > > > ng with instructions on how to run test serially (I didn’t find
anything wrt
> > > > Maven surefire plugin)? Thanks
> > > >
> > > > -----Original Message-----
> > > > From: infinispan-dev-bounces(a)lists.jboss.org
> > > > [mailto:infinispan-dev-bounces@lists.jboss.org] On Behalf Of Galder
> > > > Zamarreño
> > > > Sent: Tuesday, January 18, 2011 6:12 AM
> > > > To: infinispan -Dev List
> > > > Subject: Re: [infinispan-dev] ISPN-863 - Thoughts / Questions
> > > >
> > > > Hmmm, try increasing -Xmx? Start with
-XX:+HeapDumpOnOutOfMemoryError
> > > > -XX:HeapDumpPath=/tmp/java_heap and inspect the dump with Eclipse
MAT?...
> > > >
> > > > On Jan 17, 2011, at 4:51 PM, david marion wrote:
> > > >
> > > > > same result.
> > > > >
> > > > > Date: Mon, 17 Jan 2011 11:54:15 -0300
> > > > > From: vblagoje(a)redhat.com
> > > > > To: infinispan-dev(a)lists.jboss.org
> > > > > Subject: Re: [infinispan-dev] ISPN-863 - Thoughts / Questions
> > > > >
> > > > > Its easier to run everything from command line for two reason I
can think
> > > > of:
> > > > > - everyone has same env to compare results with you
> > > > > - it is all ready to go
> > > > >
> > > > > See http://community.jboss.org/wiki/InfinispanandMaven
> > > > >
> > > > > On 11-01-17 11:38 AM, Dave wrote:
> > > > > All,
> > > > >
> > > > > I am trying to test some changes that I made using Eclipse and
the TestNG
> > > > plugin. I am not familiar with TestNG. Are there any settings that I
should
> > > > make? Any ideas?
> > > > >
> > > > > java.lang.OutOfMemoryError: Java heap space
> > > > > at
> > > >
org.infinispan.util.concurrent.BoundedConcurrentHashMap$HashEntry.newArray(B
> > > > oundedConcurrentHashMap.java:295)
> > > > > at
> > > >
org.infinispan.util.concurrent.BoundedConcurrentHashMap$Segment.<init>(Bound
> > > > edConcurrentHashMap.java:898)
> > > > > at
> > > >
org.infinispan.util.concurrent.BoundedConcurrentHashMap.<init>(BoundedConcur
> > > > rentHashMap.java:1367)
> > > > > at
> > > >
org.infinispan.container.DefaultDataContainer.<init>(DefaultDataContainer.ja
> > > > va:78)
> > > > > at
> > > >
org.infinispan.container.DefaultDataContainer.<init>(DefaultDataContainer.ja
> > > > va:47)
> > > > > at
> > > >
org.infinispan.container.DefaultDataContainer.unBoundedDataContainer(Default
> > > > DataContainer.java:93)
> > > > > at
> > > >
org.infinispan.factories.DataContainerFactory.construct(DataContainerFactory
> > > > .java:53)
> > > > > at
> > > >
org.infinispan.factories.AbstractComponentRegistry.getOrCreateComponent(Abst
> > > > ractComponentRegistry.java:315)
> > > > > at
> > > >
org.infinispan.factories.AbstractComponentRegistry.invokeInjectionMethod(Abs
> > > > tractComponentRegistry.java:251)
> > > > > at
> > > >
org.infinispan.factories.AbstractComponentRegistry$Component.injectDependenc
> > > > ies(AbstractComponentRegistry.java:840)
> > > > > at
> > > >
org.infinispan.factories.AbstractComponentRegistry.registerComponent(Abstrac
> > > > tComponentRegistry.java:225)
> > > > > at
> > > >
org.infinispan.factories.ComponentRegistry.registerComponent(ComponentRegist
> > > > ry.java:120)
> > > > > at
> > > >
org.infinispan.factories.AbstractComponentRegistry.registerComponent(Abstrac
> > > > tComponentRegistry.java:192)
> > > > > at
> > > >
org.infinispan.factories.AbstractComponentRegistry.getOrCreateComponent(Abst
> > > > ractComponentRegistry.java:323)
> > > > > at
> > > >
org.infinispan.factories.AbstractComponentRegistry.invokeInjectionMethod(Abs
> > > > tractComponentRegistry.java:251)
> > > > > at
> > > >
org.infinispan.factories.AbstractComponentRegistry$Component.injectDependenc
> > > > ies(AbstractComponentRegistry.java:840)
> > > > > at
> > > >
org.infinispan.factories.AbstractComponentRegistry.registerComponent(Abstrac
> > > > tComponentRegistry.java:225)
> > > > > at
> > > >
org.infinispan.factories.ComponentRegistry.registerComponent(ComponentRegist
> > > > ry.java:120)
> > > > > at
> > > >
org.infinispan.factories.AbstractComponentRegistry.registerComponent(Abstrac
> > > > tComponentRegistry.java:192)
> > > > > at
> > > >
org.infinispan.factories.AbstractComponentRegistry.getOrCreateComponent(Abst
> > > > ractComponentRegistry.java:323)
> > > > > at
> > > >
org.infinispan.factories.AbstractComponentRegistry.invokeInjectionMethod(Abs
> > > > tractComponentRegistry.java:251)
> > > > > at
> > > >
org.infinispan.factories.AbstractComponentRegistry$Component.injectDependenc
> > > > ies(AbstractComponentRegistry.java:840)
> > > > > at
> > > >
org.infinispan.factories.AbstractComponentRegistry.registerComponent(Abstrac
> > > > tComponentRegistry.java:225)
> > > > > at
> > > >
org.infinispan.factories.ComponentRegistry.registerComponent(ComponentRegist
> > > > ry.java:120)
> > > > > at
> > > >
org.infinispan.factories.AbstractComponentRegistry.registerComponent(Abstrac
> > > > tComponentRegistry.java:192)
> > > > > at
> > > >
org.infinispan.factories.InternalCacheFactory.bootstrap(InternalCacheFactory
> > > > .java:92)
> > > > > at
> > > >
org.infinispan.factories.InternalCacheFactory.createAndWire(InternalCacheFac
> > > > tory.java:78)
> > > > > at
> > > >
org.infinispan.factories.InternalCacheFactory.createCache(InternalCacheFacto
> > > > ry.java:62)
> > > > > at
> > > >
org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.j
> > > > ava:510)
> > > > > at
> > > >
org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java
> > > > :440)
> > > > > at
> > > >
org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java
> > > > :409)
> > > > > at
> > > >
org.infinispan.config.ConfigurationValidationTest.testDefaultMemoryGuardConf
> > > > iguration(ConfigurationValidationTest.java:100)
> > > > >
> > > > >
> > > > >
> > > > > -----Original Message-----
> > > > > From: infinispan-dev-bounces(a)lists.jboss.org
> > > > [mailto:infinispan-dev-bounces@lists.jboss.org] On Behalf Of
Vladimir
> > > > Blagojevic
> > > > > Sent: Friday, January 14, 2011 11:05 AM
> > > > > To: infinispan -Dev List
> > > > > Subject: Re: [infinispan-dev] ISPN-863 - Thoughts / Questions
> > > > >
> > > > > Hey David,
> > > > >
> > > > > First of all kudos for taking on such a non-trivial task!
> > > > >
> > > > > On 11-01-14 12:22 AM, david marion wrote:
> > > > > 1. Modify configuration in some way so that the following can
be
> > > > specified:
> > > > > a. The percentage value of used JVM memory (i.e. 95) at which
> > > > entries should be evicted to try and avoid an OOM error.
> > > > > b. The number of items that should be evicted when memory
reaches
> > > > this threshold
> > > > > 2. Modify LRU and LIRS Eviction class so that the accessQueue
member
> > > > can be accessed by the new Eviction class so that two access queues
don’t
> > > > have to be maintained.
> > > > > 3. Create a new Eviction class, a subclass of LIRS, where the
> > > > accessQueue is used from the Eviction strategy the user specifies and
the
> > > > for loop in the execute method is exited when the evicted set equals
value
> > > > from 1.b above.
> > > > > 4. Modify DataContainerFactory.construct() to call
> > > > DefaultDataContainer.boundedDataContainer() regardless of eviction
policy.
> > > > This will always create a BoundedConcurrentHashMap
> > > > >
> > > > > I don't think you have to extend LRU and LIRS, as long as
you implement
> > > > EvictionPolicy you are fullfilling the contract. What extending LRU
and LIRS
> > > > can do is give you some order in selecting proper elements for
eviction. I
> > > > think the precise technical term in research literature is
"eviction
> > > > precision".
> > > > >
> > > > >
> > > > > 1.
> > > > > 2. Create an instance of the new Eviction class in each
segment.
> > > > > 3. Modify BoundedConcurrentHashMap.Segment put and replace
methods
> > > > such that when new values are going to be put into the Segment, the
memory
> > > > usage is checked and the execute method is called on the new Eviction
class.
> > > > >
> > > > > Sounds right!
> > > > >
> > > > >
> > > > > 1.
> > > > >
> > > > > Questions:
> > > > >
> > > > > 1. What are the implications of using a
BoundedConcurrentHashMap
> > > > instead of a ConcurrentHashMap when maxEntries is set to -1?
> > > > >
> > > > > I think none except you turn on eviction by using
> > > > BoundedConcurrentHashMap.
> > > > >
> > > > >
> > > > > 1.
> > > > >
> > > > > Thoughts
> > > > >
> > > > > 1. This will not guarantee that an OOM error does not occur. It
will
> > > > attempt to guard against an OOM caused by putting new values into the
cache.
> > > > This will probably be more effective when the cache is being used in
> > > > client/server mode, and less effective when used in embedded mode as
to
> > > > other code running in the JVM.
> > > > >
> > > > >
> > > > > -- Dave Marion
> > > > >
> > > > >
> > > > > Cheers,
> > > > > Vladimir
> > > > >
> > > > > _______________________________________________
> > > > > infinispan-dev mailing list
> > > > >
> > > > > infinispan-dev(a)lists.jboss.org
> > > > > https://lists.jboss.org/mailman/listinfo/infinispan-dev
> > > > >
> > > > >
> > > > > _______________________________________________ infinispan-dev
mailing
> > > > list infinispan-dev(a)lists.jboss.org
> > > >
https://lists.jboss.org/mailman/listinfo/infinispan-dev_____________________
> > > > __________________________
> > > > > infinispan-dev mailing list
> > > > > infinispan-dev(a)lists.jboss.org
> > > > > https://lists.jboss.org/mailman/listinfo/infinispan-dev
> > > >
> > > > --
> > > > Galder Zamarreño
> > > > Sr. Software Engineer
> > > > Infinispan, JBoss Cache
> > > >
> > > >
> > > > _______________________________________________
> > > > infinispan-dev mailing list
> > > > infinispan-dev(a)lists.jboss.org
> > > > https://lists.jboss.org/mailman/listinfo/infinispan-dev
> > > >
> > > >
> > > > _______________________________________________
> > > > infinispan-dev mailing list
> > > > infinispan-dev(a)lists.jboss.org
> > > > https://lists.jboss.org/mailman/listinfo/infinispan-dev
> > > _______________________________________________
> > > infinispan-dev mailing list
> > > infinispan-dev(a)lists.jboss.org
> > > https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >
> > --
> > Galder Zamarreño
> > Sr. Software Engineer
> > Infinispan, JBoss Cache
> >
> >
> > _______________________________________________
> > infinispan-dev mailing list
> > infinispan-dev(a)lists.jboss.org
> > https://lists.jboss.org/mailman/listinfo/infinispan-dev
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
--
Galder Zamarreño
Sr. Software Engineer
Infinispan, JBoss Cache
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
4:34 p.m.
New subject: ISPN-863 - Thoughts / Questions
On 11-01-20 9:52 AM, david marion wrote:
I did exactly that last night. I built a test that allocated
HashEntry<K,V>[] just like it does in the Segment constructor. I don't
have the numbers in front of me right now, but at the maximum size
(2^30) and without specifying concurrency it was trying to allocate 32
Segment objects each having a HashEntry<K,V>[] with 2^25 elements.
Each one of these HashEntry<K,V>[] takes up around 128MB of memory.
Ok, let
me just make sure I understand you. You are saying that BCHM is
allocating too much memory if a certain constructor is used?
5:01 p.m.
New subject: ISPN-863 - Thoughts / Questions
Yes, but its because I changed the meaning of maxEntries == -1. It used to create a
ConcurrentHashMap that has a max size of 2^30. In my fork, when -1 is used it creates a
BoundedConcurrentHashMap with a max size of 2^30. My guess is that a ConcurrentHashMap
grows as items are placed into the map, whereas the BoundedConcurrentHashMap pre-allocates
structures for that designated size. So, I am thinking that I need to account for the size
of the JVM to determine the maximum entries for the BoundedConcurrentHashMap.
Date: Thu, 20 Jan 2011 12:34:22 -0300
From: vblagoje(a)redhat.com
To: infinispan-dev(a)lists.jboss.org
Subject: Re: [infinispan-dev] ISPN-863 - Thoughts / Questions
On 11-01-20 9:52 AM, david marion wrote:
I did exactly that last night. I built a test that allocated HashEntry<K,V>[] just
like it does in the Segment constructor. I don't have the numbers in front of me right
now, but at the maximum size (2^30) and without specifying concurrency it was trying to
allocate 32 Segment objects each having a HashEntry<K,V>[] with 2^25 elements. Each
one of these HashEntry<K,V>[] takes up around 128MB of memory.
Ok, let me just make sure I understand you. You are saying that BCHM is allocating too
much memory if a certain constructor is used?
_______________________________________________ infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev
5:45 p.m.
New subject: ISPN-863 - Thoughts / Questions
BCHM creates segment array and for each segment it creates HashEntry
array just like the ConcurrentHashMap. The difference is that BCHM has
EvictionPolicy instance per segment.
On 11-01-20 1:01 PM, david marion wrote:
Yes, but its because I changed the meaning of maxEntries == -1. It
used to create a ConcurrentHashMap that has a max size of 2^30. In my
fork, when -1 is used it creates a BoundedConcurrentHashMap with a max
size of 2^30. My guess is that a ConcurrentHashMap grows as items are
placed into the map, whereas the
BoundedConcurrentHashMap pre-allocates structures for that designated
size. So, I am thinking that I need to account for the size of the JVM
to determine the maximum entries for the BoundedConcurrentHashMap.
------------------------------------------------------------------------
Date: Thu, 20 Jan 2011 12:34:22 -0300
From: vblagoje(a)redhat.com
To: infinispan-dev(a)lists.jboss.org
Subject: Re: [infinispan-dev] ISPN-863 - Thoughts / Questions
On 11-01-20 9:52 AM, david marion wrote:
I did exactly that last night. I built a test that allocated
HashEntry<K,V>[] just like it does in the Segment constructor. I
don't have the numbers in front of me right now, but at the
maximum size (2^30) and without specifying concurrency it
was trying to allocate 32 Segment objects each having a
HashEntry<K,V>[] with 2^25 elements. Each one of these
HashEntry<K,V>[] takes up around 128MB of memory.
Ok, let me just make sure I understand you. You are saying that BCHM
is allocating too much memory if a certain constructor is used?
_______________________________________________ infinispan-dev mailing
list infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
3:03 a.m.
New subject: ISPN-863 - Thoughts / Questions
I believe that I have a working first draft. Whats the usual approach for
putting it out there so folks can comment? Shall I upload a patch to the
JIRA issue or do most of you look at the clone/fork on GitHub?
-- Dave Marion
-----Original Message-----
From: infinispan-dev-bounces(a)lists.jboss.org
[mailto:infinispan-dev-bounces@lists.jboss.org] On Behalf Of david marion
Sent: Thursday, January 20, 2011 7:53 AM
To: infinispan-dev(a)lists.jboss.org
Subject: Re: [infinispan-dev] ISPN-863 - Thoughts / Questions
I did exactly that last night. I built a test that allocated
HashEntry<K,V>[] just like it does in the Segment constructor. I don't have
the numbers in front of me right now, but at the maximum size (2^30) and
without specifying concurrency it was trying to allocate 32 Segment objects
each having a HashEntry<K,V>[] with 2^25 elements. Each one of these
HashEntry<K,V>[] takes up around 128MB of memory.
From: galder(a)redhat.com
Date: Thu, 20 Jan 2011 09:38:43 +0100
To: infinispan-dev(a)lists.jboss.org
Subject: Re: [infinispan-dev] ISPN-863 - Thoughts / Questions
On Jan 19, 2011, at 1:47 PM, david marion wrote:
>
> It was my problem. If -1 is set for maxEntries, then I am creating a
BoundedConcurrentHashMap with a size of Integer.MAX_VALUE. Then when I ran
the tests all of them ran out of memory. I should have looked at the stack
trace a little more closely. I am going to have to put logic in so that the
maximum capacity is determined by the amount of free memory when the cache
is created. Previously, if -1 was set for maxEntries, a ConcurrentHashMap
was created with maximum capacity of Integer.MAX_VALUE. I have not looked at
the code, but my guess is that it works because it grows over time and does
not pre-allocate all structures ahead of time.
Hmmm, that sounds odd. Maybe you wanna build a test to show the exact
differences
and see if too much is being allocated on startup?
> Any thoughts on logic for determining maximum capacity at creation time?
A percentage of the available free memory? 40% by default? You have to
accomodate
for the other apps running on the same JVM, so we shouldn't be
too aggressive by default.
>
> > From: galder(a)redhat.com
> > Date: Wed, 19 Jan 2011 10:32:42 +0100
> > To: infinispan-dev(a)lists.jboss.org
> > Subject: Re: [infinispan-dev] ISPN-863 - Thoughts / Questions
> >
> > Out of curiosity, what is a code problem on your side? Or a set up
issue?
> >
> > On Jan 19, 2011, at 4:29 AM, david marion wrote:
> >
> > > Disregard, I found the issue. Sorry for the spam.
> > >
> > > > From: dlmarion(a)hotmail.com
> > > > To: infinispan-dev(a)lists.jboss.org
> > > > Date: Tue, 18 Jan 2011 20:39:21 -0500
> > > > Subject: Re: [infinispan-dev] ISPN-863 - Thoughts / Questions
> > > >
> > > > Hah, might be because the POM is configured to run tests in
parallel with a
> > > > max heap size of 1GB. Can someone update
> > > > http://community.jboss.org/wiki/ParallelTestSuite or
> > > >
http://community.jboss.org/wiki/InfinispanQuickStartGuideforDevelopers#Testi
> > > > ng with instructions on how to run test serially
(I didnt find
anything wrt
> > > > Maven surefire plugin)? Thanks
> > > >
> > > > -----Original Message-----
> > > > From: infinispan-dev-bounces(a)lists.jboss.org
> > > > [mailto:infinispan-dev-bounces@lists.jboss.org] On Behalf Of
Galder
> > > > Zamarreño
> > > > Sent: Tuesday, January 18, 2011 6:12 AM
> > > > To: infinispan -Dev List
> > > > Subject: Re: [infinispan-dev] ISPN-863 - Thoughts / Questions
> > > >
> > > > Hmmm, try increasing -Xmx? Start with
-XX:+HeapDumpOnOutOfMemoryError
> > > > -XX:HeapDumpPath=/tmp/java_heap and inspect the
dump with Eclipse
MAT?...
> > > >
> > > > On Jan 17, 2011, at 4:51 PM, david marion wrote:
> > > >
> > > > > same result.
> > > > >
> > > > > Date: Mon, 17 Jan 2011 11:54:15 -0300
> > > > > From: vblagoje(a)redhat.com
> > > > > To: infinispan-dev(a)lists.jboss.org
> > > > > Subject: Re: [infinispan-dev] ISPN-863 - Thoughts / Questions
> > > > >
> > > > > Its easier to run everything from command line for two reason I
can think
> > > > of:
> > > > > - everyone has same env to compare results with you
> > > > > - it is all ready to go
> > > > >
> > > > > See http://community.jboss.org/wiki/InfinispanandMaven
> > > > >
> > > > > On 11-01-17 11:38 AM, Dave wrote:
> > > > > All,
> > > > >
> > > > > I am trying to test some changes that I made using Eclipse and
the TestNG
> > > > plugin. I am not familiar with TestNG. Are there
any settings that
I should
> > > > make? Any ideas?
> > > > >
> > > > > java.lang.OutOfMemoryError: Java heap space
> > > > > at
> > > >
org.infinispan.util.concurrent.BoundedConcurrentHashMap$HashEntry.newArray(B
> > > > oundedConcurrentHashMap.java:295)
> > > > > at
> > > >
org.infinispan.util.concurrent.BoundedConcurrentHashMap$Segment.<init>(Bound
> > > > edConcurrentHashMap.java:898)
> > > > > at
> > > >
org.infinispan.util.concurrent.BoundedConcurrentHashMap.<init>(BoundedConcur
> > > > rentHashMap.java:1367)
> > > > > at
> > > >
org.infinispan.container.DefaultDataContainer.<init>(DefaultDataContainer.ja
> > > > va:78)
> > > > > at
> > > >
org.infinispan.container.DefaultDataContainer.<init>(DefaultDataContainer.ja
> > > > va:47)
> > > > > at
> > > >
org.infinispan.container.DefaultDataContainer.unBoundedDataContainer(Default
> > > > DataContainer.java:93)
> > > > > at
> > > >
org.infinispan.factories.DataContainerFactory.construct(DataContainerFactory
> > > > .java:53)
> > > > > at
> > > >
org.infinispan.factories.AbstractComponentRegistry.getOrCreateComponent(Abst
> > > > ractComponentRegistry.java:315)
> > > > > at
> > > >
org.infinispan.factories.AbstractComponentRegistry.invokeInjectionMethod(Abs
> > > > tractComponentRegistry.java:251)
> > > > > at
> > > >
org.infinispan.factories.AbstractComponentRegistry$Component.injectDependenc
> > > > ies(AbstractComponentRegistry.java:840)
> > > > > at
> > > >
org.infinispan.factories.AbstractComponentRegistry.registerComponent(Abstrac
> > > > tComponentRegistry.java:225)
> > > > > at
> > > >
org.infinispan.factories.ComponentRegistry.registerComponent(ComponentRegist
> > > > ry.java:120)
> > > > > at
> > > >
org.infinispan.factories.AbstractComponentRegistry.registerComponent(Abstrac
> > > > tComponentRegistry.java:192)
> > > > > at
> > > >
org.infinispan.factories.AbstractComponentRegistry.getOrCreateComponent(Abst
> > > > ractComponentRegistry.java:323)
> > > > > at
> > > >
org.infinispan.factories.AbstractComponentRegistry.invokeInjectionMethod(Abs
> > > > tractComponentRegistry.java:251)
> > > > > at
> > > >
org.infinispan.factories.AbstractComponentRegistry$Component.injectDependenc
> > > > ies(AbstractComponentRegistry.java:840)
> > > > > at
> > > >
org.infinispan.factories.AbstractComponentRegistry.registerComponent(Abstrac
> > > > tComponentRegistry.java:225)
> > > > > at
> > > >
org.infinispan.factories.ComponentRegistry.registerComponent(ComponentRegist
> > > > ry.java:120)
> > > > > at
> > > >
org.infinispan.factories.AbstractComponentRegistry.registerComponent(Abstrac
> > > > tComponentRegistry.java:192)
> > > > > at
> > > >
org.infinispan.factories.AbstractComponentRegistry.getOrCreateComponent(Abst
> > > > ractComponentRegistry.java:323)
> > > > > at
> > > >
org.infinispan.factories.AbstractComponentRegistry.invokeInjectionMethod(Abs
> > > > tractComponentRegistry.java:251)
> > > > > at
> > > >
org.infinispan.factories.AbstractComponentRegistry$Component.injectDependenc
> > > > ies(AbstractComponentRegistry.java:840)
> > > > > at
> > > >
org.infinispan.factories.AbstractComponentRegistry.registerComponent(Abstrac
> > > > tComponentRegistry.java:225)
> > > > > at
> > > >
org.infinispan.factories.ComponentRegistry.registerComponent(ComponentRegist
> > > > ry.java:120)
> > > > > at
> > > >
org.infinispan.factories.AbstractComponentRegistry.registerComponent(Abstrac
> > > > tComponentRegistry.java:192)
> > > > > at
> > > >
org.infinispan.factories.InternalCacheFactory.bootstrap(InternalCacheFactory
> > > > .java:92)
> > > > > at
> > > >
org.infinispan.factories.InternalCacheFactory.createAndWire(InternalCacheFac
> > > > tory.java:78)
> > > > > at
> > > >
org.infinispan.factories.InternalCacheFactory.createCache(InternalCacheFacto
> > > > ry.java:62)
> > > > > at
> > > >
org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.j
> > > > ava:510)
> > > > > at
> > > >
org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java
> > > > :440)
> > > > > at
> > > >
org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java
> > > > :409)
> > > > > at
> > > >
org.infinispan.config.ConfigurationValidationTest.testDefaultMemoryGuardConf
> > > > iguration(ConfigurationValidationTest.java:100)
> > > > >
> > > > >
> > > > >
> > > > > -----Original Message-----
> > > > > From: infinispan-dev-bounces(a)lists.jboss.org
> > > > [mailto:infinispan-dev-bounces@lists.jboss.org] On Behalf Of
Vladimir
> > > > Blagojevic
> > > > > Sent: Friday, January 14, 2011 11:05 AM
> > > > > To: infinispan -Dev List
> > > > > Subject: Re: [infinispan-dev] ISPN-863 - Thoughts / Questions
> > > > >
> > > > > Hey David,
> > > > >
> > > > > First of all kudos for taking on such a non-trivial task!
> > > > >
> > > > > On 11-01-14 12:22 AM, david marion wrote:
> > > > > 1. Modify configuration in some way so that the following can
be
> > > > specified:
> > > > > a. The percentage value of used JVM memory (i.e. 95) at which
> > > > entries should be evicted to try and avoid an OOM error.
> > > > > b. The number of items that should be evicted when memory
reaches
> > > > this threshold
> > > > > 2. Modify LRU and LIRS Eviction class so that the accessQueue
member
> > > > can be accessed by the new Eviction class so that
two access
queues dont
> > > > have to be maintained.
> > > > > 3. Create a new Eviction class, a subclass of LIRS, where the
> > > > accessQueue is used from the Eviction strategy the user specifies
and the
> > > > for loop in the execute method is exited when the
evicted set
equals value
> > > > from 1.b above.
> > > > > 4. Modify DataContainerFactory.construct() to call
> > > > DefaultDataContainer.boundedDataContainer() regardless of eviction
policy.
> > > > This will always create a
BoundedConcurrentHashMap
> > > > >
> > > > > I don't think you have to extend LRU and LIRS, as long as
you
implement
> > > > EvictionPolicy you are fullfilling the contract.
What extending
LRU and LIRS
> > > > can do is give you some order in selecting proper
elements for
eviction. I
> > > > think the precise technical term in research
literature is
"eviction
> > > > precision".
> > > > >
> > > > >
> > > > > 1.
> > > > > 2. Create an instance of the new Eviction class in each
segment.
> > > > > 3. Modify BoundedConcurrentHashMap.Segment put and replace
methods
> > > > such that when new values are going to be put
into the Segment,
the memory
> > > > usage is checked and the execute method is called
on the new
Eviction class.
> > > > >
> > > > > Sounds right!
> > > > >
> > > > >
> > > > > 1.
> > > > >
> > > > > Questions:
> > > > >
> > > > > 1. What are the implications of using a
BoundedConcurrentHashMap
> > > > instead of a ConcurrentHashMap when maxEntries is set to -1?
> > > > >
> > > > > I think none except you turn on eviction by using
> > > > BoundedConcurrentHashMap.
> > > > >
> > > > >
> > > > > 1.
> > > > >
> > > > > Thoughts
> > > > >
> > > > > 1. This will not guarantee that an OOM error does not occur. It
will
> > > > attempt to guard against an OOM caused by putting
new values into
the cache.
> > > > This will probably be more effective when the
cache is being used
in
> > > > client/server mode, and less effective when used
in embedded mode
as to
> > > > other code running in the JVM.
> > > > >
> > > > >
> > > > > -- Dave Marion
> > > > >
> > > > >
> > > > > Cheers,
> > > > > Vladimir
> > > > >
> > > > > _______________________________________________
> > > > > infinispan-dev mailing list
> > > > >
> > > > > infinispan-dev(a)lists.jboss.org
> > > > > https://lists.jboss.org/mailman/listinfo/infinispan-dev
> > > > >
> > > > >
> > > > > _______________________________________________ infinispan-dev
mailing
> > > > list infinispan-dev(a)lists.jboss.org
> > > >
https://lists.jboss.org/mailman/listinfo/infinispan-dev_____________________
> > > > __________________________
> > > > > infinispan-dev mailing list
> > > > > infinispan-dev(a)lists.jboss.org
> > > > > https://lists.jboss.org/mailman/listinfo/infinispan-dev
> > > >
> > > > --
> > > > Galder Zamarreño
> > > > Sr. Software Engineer
> > > > Infinispan, JBoss Cache
> > > >
> > > >
> > > > _______________________________________________
> > > > infinispan-dev mailing list
> > > > infinispan-dev(a)lists.jboss.org
> > > > https://lists.jboss.org/mailman/listinfo/infinispan-dev
> > > >
> > > >
> > > > _______________________________________________
> > > > infinispan-dev mailing list
> > > > infinispan-dev(a)lists.jboss.org
> > > > https://lists.jboss.org/mailman/listinfo/infinispan-dev
> > > _______________________________________________
> > > infinispan-dev mailing list
> > > infinispan-dev(a)lists.jboss.org
> > > https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >
> > --
> > Galder Zamarreño
> > Sr. Software Engineer
> > Infinispan, JBoss Cache
> >
> >
> > _______________________________________________
> > infinispan-dev mailing list
> > infinispan-dev(a)lists.jboss.org
> > https://lists.jboss.org/mailman/listinfo/infinispan-dev
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
--
Galder Zamarreño
Sr. Software Engineer
Infinispan, JBoss Cache
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
12:11 p.m.
New subject: ISPN-863 - Thoughts / Questions
Clone the repo, make your changes in a topic branch and the send a pull request:
http://community.jboss.org/docs/DOC-16089
On Jan 25, 2011, at 3:03 AM, Dave wrote:
I believe that I have a working first draft. What’s the usual
approach for putting it out there so folks can comment? Shall I upload a patch to the JIRA
issue or do most of you look at the clone/fork on GitHub?
-- Dave Marion
-----Original Message-----
From: infinispan-dev-bounces(a)lists.jboss.org
[mailto:infinispan-dev-bounces@lists.jboss.org] On Behalf Of david marion
Sent: Thursday, January 20, 2011 7:53 AM
To: infinispan-dev(a)lists.jboss.org
Subject: Re: [infinispan-dev] ISPN-863 - Thoughts / Questions
I did exactly that last night. I built a test that allocated HashEntry<K,V>[] just
like it does in the Segment constructor. I don't have the numbers in front of me right
now, but at the maximum size (2^30) and without specifying concurrency it was trying to
allocate 32 Segment objects each having a HashEntry<K,V>[] with 2^25 elements. Each
one of these HashEntry<K,V>[] takes up around 128MB of memory.
> From: galder(a)redhat.com
> Date: Thu, 20 Jan 2011 09:38:43 +0100
> To: infinispan-dev(a)lists.jboss.org
> Subject: Re: [infinispan-dev] ISPN-863 - Thoughts / Questions
>
>
> On Jan 19, 2011, at 1:47 PM, david marion wrote:
>
> >
> > It was my problem. If -1 is set for maxEntries, then I am creating a
BoundedConcurrentHashMap with a size of Integer.MAX_VALUE. Then when I ran the tests all
of them ran out of memory. I should have looked at the stack trace a little more closely.
I am going to have to put logic in so that the maximum capacity is determined by the
amount of free memory when the cache is created. Previously, if -1 was set for maxEntries,
a ConcurrentHashMap was created with maximum capacity of Integer.MAX_VALUE. I have not
looked at the code, but my guess is that it works because it grows over time and does not
pre-allocate all structures ahead of time.
>
> Hmmm, that sounds odd. Maybe you wanna build a test to show the exact differences
and see if too much is being allocated on startup?
>
> > Any thoughts on logic for determining maximum capacity at creation time?
>
> A percentage of the available free memory? 40% by default? You have to accomodate
for the other apps running on the same JVM, so we shouldn't be too aggressive by
default.
>
> >
> > > From: galder(a)redhat.com
> > > Date: Wed, 19 Jan 2011 10:32:42 +0100
> > > To: infinispan-dev(a)lists.jboss.org
> > > Subject: Re: [infinispan-dev] ISPN-863 - Thoughts / Questions
> > >
> > > Out of curiosity, what is a code problem on your side? Or a set up issue?
> > >
> > > On Jan 19, 2011, at 4:29 AM, david marion wrote:
> > >
> > > > Disregard, I found the issue. Sorry for the spam.
> > > >
> > > > > From: dlmarion(a)hotmail.com
> > > > > To: infinispan-dev(a)lists.jboss.org
> > > > > Date: Tue, 18 Jan 2011 20:39:21 -0500
> > > > > Subject: Re: [infinispan-dev] ISPN-863 - Thoughts / Questions
> > > > >
> > > > > Hah, might be because the POM is configured to run tests in
parallel with a
> > > > > max heap size of 1GB. Can someone update
> > > > > http://community.jboss.org/wiki/ParallelTestSuite or
> > > > >
http://community.jboss.org/wiki/InfinispanQuickStartGuideforDevelopers#Testi
> > > > > ng with instructions on how to run test serially (I didn’t find
anything wrt
> > > > > Maven surefire plugin)? Thanks
> > > > >
> > > > > -----Original Message-----
> > > > > From: infinispan-dev-bounces(a)lists.jboss.org
> > > > > [mailto:infinispan-dev-bounces@lists.jboss.org] On Behalf Of
Galder
> > > > > Zamarreño
> > > > > Sent: Tuesday, January 18, 2011 6:12 AM
> > > > > To: infinispan -Dev List
> > > > > Subject: Re: [infinispan-dev] ISPN-863 - Thoughts / Questions
> > > > >
> > > > > Hmmm, try increasing -Xmx? Start with
-XX:+HeapDumpOnOutOfMemoryError
> > > > > -XX:HeapDumpPath=/tmp/java_heap and inspect the dump with
Eclipse MAT?...
> > > > >
> > > > > On Jan 17, 2011, at 4:51 PM, david marion wrote:
> > > > >
> > > > > > same result.
> > > > > >
> > > > > > Date: Mon, 17 Jan 2011 11:54:15 -0300
> > > > > > From: vblagoje(a)redhat.com
> > > > > > To: infinispan-dev(a)lists.jboss.org
> > > > > > Subject: Re: [infinispan-dev] ISPN-863 - Thoughts /
Questions
> > > > > >
> > > > > > Its easier to run everything from command line for two
reason I can think
> > > > > of:
> > > > > > - everyone has same env to compare results with you
> > > > > > - it is all ready to go
> > > > > >
> > > > > > See http://community.jboss.org/wiki/InfinispanandMaven
> > > > > >
> > > > > > On 11-01-17 11:38 AM, Dave wrote:
> > > > > > All,
> > > > > >
> > > > > > I am trying to test some changes that I made using Eclipse
and the TestNG
> > > > > plugin. I am not familiar with TestNG. Are there any settings
that I should
> > > > > make? Any ideas?
> > > > > >
> > > > > > java.lang.OutOfMemoryError: Java heap space
> > > > > > at
> > > > >
org.infinispan.util.concurrent.BoundedConcurrentHashMap$HashEntry.newArray(B
> > > > > oundedConcurrentHashMap.java:295)
> > > > > > at
> > > > >
org.infinispan.util.concurrent.BoundedConcurrentHashMap$Segment.<init>(Bound
> > > > > edConcurrentHashMap.java:898)
> > > > > > at
> > > > >
org.infinispan.util.concurrent.BoundedConcurrentHashMap.<init>(BoundedConcur
> > > > > rentHashMap.java:1367)
> > > > > > at
> > > > >
org.infinispan.container.DefaultDataContainer.<init>(DefaultDataContainer.ja
> > > > > va:78)
> > > > > > at
> > > > >
org.infinispan.container.DefaultDataContainer.<init>(DefaultDataContainer.ja
> > > > > va:47)
> > > > > > at
> > > > >
org.infinispan.container.DefaultDataContainer.unBoundedDataContainer(Default
> > > > > DataContainer.java:93)
> > > > > > at
> > > > >
org.infinispan.factories.DataContainerFactory.construct(DataContainerFactory
> > > > > .java:53)
> > > > > > at
> > > > >
org.infinispan.factories.AbstractComponentRegistry.getOrCreateComponent(Abst
> > > > > ractComponentRegistry.java:315)
> > > > > > at
> > > > >
org.infinispan.factories.AbstractComponentRegistry.invokeInjectionMethod(Abs
> > > > > tractComponentRegistry.java:251)
> > > > > > at
> > > > >
org.infinispan.factories.AbstractComponentRegistry$Component.injectDependenc
> > > > > ies(AbstractComponentRegistry.java:840)
> > > > > > at
> > > > >
org.infinispan.factories.AbstractComponentRegistry.registerComponent(Abstrac
> > > > > tComponentRegistry.java:225)
> > > > > > at
> > > > >
org.infinispan.factories.ComponentRegistry.registerComponent(ComponentRegist
> > > > > ry.java:120)
> > > > > > at
> > > > >
org.infinispan.factories.AbstractComponentRegistry.registerComponent(Abstrac
> > > > > tComponentRegistry.java:192)
> > > > > > at
> > > > >
org.infinispan.factories.AbstractComponentRegistry.getOrCreateComponent(Abst
> > > > > ractComponentRegistry.java:323)
> > > > > > at
> > > > >
org.infinispan.factories.AbstractComponentRegistry.invokeInjectionMethod(Abs
> > > > > tractComponentRegistry.java:251)
> > > > > > at
> > > > >
org.infinispan.factories.AbstractComponentRegistry$Component.injectDependenc
> > > > > ies(AbstractComponentRegistry.java:840)
> > > > > > at
> > > > >
org.infinispan.factories.AbstractComponentRegistry.registerComponent(Abstrac
> > > > > tComponentRegistry.java:225)
> > > > > > at
> > > > >
org.infinispan.factories.ComponentRegistry.registerComponent(ComponentRegist
> > > > > ry.java:120)
> > > > > > at
> > > > >
org.infinispan.factories.AbstractComponentRegistry.registerComponent(Abstrac
> > > > > tComponentRegistry.java:192)
> > > > > > at
> > > > >
org.infinispan.factories.AbstractComponentRegistry.getOrCreateComponent(Abst
> > > > > ractComponentRegistry.java:323)
> > > > > > at
> > > > >
org.infinispan.factories.AbstractComponentRegistry.invokeInjectionMethod(Abs
> > > > > tractComponentRegistry.java:251)
> > > > > > at
> > > > >
org.infinispan.factories.AbstractComponentRegistry$Component.injectDependenc
> > > > > ies(AbstractComponentRegistry.java:840)
> > > > > > at
> > > > >
org.infinispan.factories.AbstractComponentRegistry.registerComponent(Abstrac
> > > > > tComponentRegistry.java:225)
> > > > > > at
> > > > >
org.infinispan.factories.ComponentRegistry.registerComponent(ComponentRegist
> > > > > ry.java:120)
> > > > > > at
> > > > >
org.infinispan.factories.AbstractComponentRegistry.registerComponent(Abstrac
> > > > > tComponentRegistry.java:192)
> > > > > > at
> > > > >
org.infinispan.factories.InternalCacheFactory.bootstrap(InternalCacheFactory
> > > > > .java:92)
> > > > > > at
> > > > >
org.infinispan.factories.InternalCacheFactory.createAndWire(InternalCacheFac
> > > > > tory.java:78)
> > > > > > at
> > > > >
org.infinispan.factories.InternalCacheFactory.createCache(InternalCacheFacto
> > > > > ry.java:62)
> > > > > > at
> > > > >
org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.j
> > > > > ava:510)
> > > > > > at
> > > > >
org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java
> > > > > :440)
> > > > > > at
> > > > >
org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java
> > > > > :409)
> > > > > > at
> > > > >
org.infinispan.config.ConfigurationValidationTest.testDefaultMemoryGuardConf
> > > > > iguration(ConfigurationValidationTest.java:100)
> > > > > >
> > > > > >
> > > > > >
> > > > > > -----Original Message-----
> > > > > > From: infinispan-dev-bounces(a)lists.jboss.org
> > > > > [mailto:infinispan-dev-bounces@lists.jboss.org] On Behalf Of
Vladimir
> > > > > Blagojevic
> > > > > > Sent: Friday, January 14, 2011 11:05 AM
> > > > > > To: infinispan -Dev List
> > > > > > Subject: Re: [infinispan-dev] ISPN-863 - Thoughts /
Questions
> > > > > >
> > > > > > Hey David,
> > > > > >
> > > > > > First of all kudos for taking on such a non-trivial task!
> > > > > >
> > > > > > On 11-01-14 12:22 AM, david marion wrote:
> > > > > > 1. Modify configuration in some way so that the following
can be
> > > > > specified:
> > > > > > a. The percentage value of used JVM memory (i.e. 95) at
which
> > > > > entries should be evicted to try and avoid an OOM error.
> > > > > > b. The number of items that should be evicted when memory
reaches
> > > > > this threshold
> > > > > > 2. Modify LRU and LIRS Eviction class so that the
accessQueue member
> > > > > can be accessed by the new Eviction class so that two access
queues don’t
> > > > > have to be maintained.
> > > > > > 3. Create a new Eviction class, a subclass of LIRS, where
the
> > > > > accessQueue is used from the Eviction strategy the user
specifies and the
> > > > > for loop in the execute method is exited when the evicted set
equals value
> > > > > from 1.b above.
> > > > > > 4. Modify DataContainerFactory.construct() to call
> > > > > DefaultDataContainer.boundedDataContainer() regardless of
eviction policy.
> > > > > This will always create a BoundedConcurrentHashMap
> > > > > >
> > > > > > I don't think you have to extend LRU and LIRS, as long
as you implement
> > > > > EvictionPolicy you are fullfilling the contract. What extending
LRU and LIRS
> > > > > can do is give you some order in selecting proper elements for
eviction. I
> > > > > think the precise technical term in research literature is
"eviction
> > > > > precision".
> > > > > >
> > > > > >
> > > > > > 1.
> > > > > > 2. Create an instance of the new Eviction class in each
segment.
> > > > > > 3. Modify BoundedConcurrentHashMap.Segment put and replace
methods
> > > > > such that when new values are going to be put into the Segment,
the memory
> > > > > usage is checked and the execute method is called on the new
Eviction class.
> > > > > >
> > > > > > Sounds right!
> > > > > >
> > > > > >
> > > > > > 1.
> > > > > >
> > > > > > Questions:
> > > > > >
> > > > > > 1. What are the implications of using a
BoundedConcurrentHashMap
> > > > > instead of a ConcurrentHashMap when maxEntries is set to -1?
> > > > > >
> > > > > > I think none except you turn on eviction by using
> > > > > BoundedConcurrentHashMap.
> > > > > >
> > > > > >
> > > > > > 1.
> > > > > >
> > > > > > Thoughts
> > > > > >
> > > > > > 1. This will not guarantee that an OOM error does not
occur. It will
> > > > > attempt to guard against an OOM caused by putting new values
into the cache.
> > > > > This will probably be more effective when the cache is being
used in
> > > > > client/server mode, and less effective when used in embedded
mode as to
> > > > > other code running in the JVM.
> > > > > >
> > > > > >
> > > > > > -- Dave Marion
> > > > > >
> > > > > >
> > > > > > Cheers,
> > > > > > Vladimir
> > > > > >
> > > > > > _______________________________________________
> > > > > > infinispan-dev mailing list
> > > > > >
> > > > > > infinispan-dev(a)lists.jboss.org
> > > > > > https://lists.jboss.org/mailman/listinfo/infinispan-dev
> > > > > >
> > > > > >
> > > > > > _______________________________________________
infinispan-dev mailing
> > > > > list infinispan-dev(a)lists.jboss.org
> > > > >
https://lists.jboss.org/mailman/listinfo/infinispan-dev_____________________
> > > > > __________________________
> > > > > > infinispan-dev mailing list
> > > > > > infinispan-dev(a)lists.jboss.org
> > > > > > https://lists.jboss.org/mailman/listinfo/infinispan-dev
> > > > >
> > > > > --
> > > > > Galder Zamarreño
> > > > > Sr. Software Engineer
> > > > > Infinispan, JBoss Cache
> > > > >
> > > > >
> > > > > _______________________________________________
> > > > > infinispan-dev mailing list
> > > > > infinispan-dev(a)lists.jboss.org
> > > > > https://lists.jboss.org/mailman/listinfo/infinispan-dev
> > > > >
> > > > >
> > > > > _______________________________________________
> > > > > infinispan-dev mailing list
> > > > > infinispan-dev(a)lists.jboss.org
> > > > > https://lists.jboss.org/mailman/listinfo/infinispan-dev
> > > > _______________________________________________
> > > > infinispan-dev mailing list
> > > > infinispan-dev(a)lists.jboss.org
> > > > https://lists.jboss.org/mailman/listinfo/infinispan-dev
> > >
> > > --
> > > Galder Zamarreño
> > > Sr. Software Engineer
> > > Infinispan, JBoss Cache
> > >
> > >
> > > _______________________________________________
> > > infinispan-dev mailing list
> > > infinispan-dev(a)lists.jboss.org
> > > https://lists.jboss.org/mailman/listinfo/infinispan-dev
> > _______________________________________________
> > infinispan-dev mailing list
> > infinispan-dev(a)lists.jboss.org
> > https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
> --
> Galder Zamarreño
> Sr. Software Engineer
> Infinispan, JBoss Cache
>
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
--
Galder Zamarreño
Sr. Software Engineer
Infinispan, JBoss Cache
6:16 p.m.
New subject: ISPN-863 - Thoughts / Questions
It took me some time to figure out how to use Git and add some more tests. I have pushed
my changes to a topic branch located at
https://github.com/dlmarion/infinispan/tree/ISPN-863-master.
Please let me know if you have any questions or concerns. I have put up some documentation
at https://github.com/dlmarion/infinispan/wiki/ISPN-863-Implementation
-- Dave Marion
From: galder(a)redhat.com
Date: Tue, 25 Jan 2011 12:11:28 +0100
To: infinispan-dev(a)lists.jboss.org
Subject: Re: [infinispan-dev] ISPN-863 - Thoughts / Questions
Clone the repo, make your changes in a topic branch and the send a pull request:
http://community.jboss.org/docs/DOC-16089
On Jan 25, 2011, at 3:03 AM, Dave wrote:
> I believe that I have a working first draft. What’s the usual approach for putting
it out there so folks can comment? Shall I upload a patch to the JIRA issue or do most of
you look at the clone/fork on GitHub?
>
> -- Dave Marion
>
> -----Original Message-----
> From: infinispan-dev-bounces(a)lists.jboss.org
[mailto:infinispan-dev-bounces@lists.jboss.org] On Behalf Of david marion
> Sent: Thursday, January 20, 2011 7:53 AM
> To: infinispan-dev(a)lists.jboss.org
> Subject: Re: [infinispan-dev] ISPN-863 - Thoughts / Questions
>
> I did exactly that last night. I built a test that allocated HashEntry<K,V>[]
just like it does in the Segment constructor. I don't have the numbers in front of me
right now, but at the maximum size (2^30) and without specifying concurrency it was trying
to allocate 32 Segment objects each having a HashEntry<K,V>[] with 2^25 elements.
Each one of these HashEntry<K,V>[] takes up around 128MB of memory.
>
> > From: galder(a)redhat.com
> > Date: Thu, 20 Jan 2011 09:38:43 +0100
> > To: infinispan-dev(a)lists.jboss.org
> > Subject: Re: [infinispan-dev] ISPN-863 - Thoughts / Questions
> >
> >
> > On Jan 19, 2011, at 1:47 PM, david marion wrote:
> >
> > >
> > > It was my problem. If -1 is set for maxEntries, then I am creating a
BoundedConcurrentHashMap with a size of Integer.MAX_VALUE. Then when I ran the tests all
of them ran out of memory. I should have looked at the stack trace a little more closely.
I am going to have to put logic in so that the maximum capacity is determined by the
amount of free memory when the cache is created. Previously, if -1 was set for maxEntries,
a ConcurrentHashMap was created with maximum capacity of Integer.MAX_VALUE. I have not
looked at the code, but my guess is that it works because it grows over time and does not
pre-allocate all structures ahead of time.
> >
> > Hmmm, that sounds odd. Maybe you wanna build a test to show the exact
differences and see if too much is being allocated on startup?
> >
> > > Any thoughts on logic for determining maximum capacity at creation time?
> >
> > A percentage of the available free memory? 40% by default? You have to
accomodate for the other apps running on the same JVM, so we shouldn't be too
aggressive by default.
> >
> > >
> > > > From: galder(a)redhat.com
> > > > Date: Wed, 19 Jan 2011 10:32:42 +0100
> > > > To: infinispan-dev(a)lists.jboss.org
> > > > Subject: Re: [infinispan-dev] ISPN-863 - Thoughts / Questions
> > > >
> > > > Out of curiosity, what is a code problem on your side? Or a set up
issue?
> > > >
> > > > On Jan 19, 2011, at 4:29 AM, david marion wrote:
> > > >
> > > > > Disregard, I found the issue. Sorry for the spam.
> > > > >
> > > > > > From: dlmarion(a)hotmail.com
> > > > > > To: infinispan-dev(a)lists.jboss.org
> > > > > > Date: Tue, 18 Jan 2011 20:39:21 -0500
> > > > > > Subject: Re: [infinispan-dev] ISPN-863 - Thoughts /
Questions
> > > > > >
> > > > > > Hah, might be because the POM is configured to run tests in
parallel with a
> > > > > > max heap size of 1GB. Can someone update
> > > > > > http://community.jboss.org/wiki/ParallelTestSuite or
> > > > > >
http://community.jboss.org/wiki/InfinispanQuickStartGuideforDevelopers#Testi
> > > > > > ng with instructions on how to run test serially (I didn’t
find anything wrt
> > > > > > Maven surefire plugin)? Thanks
> > > > > >
> > > > > > -----Original Message-----
> > > > > > From: infinispan-dev-bounces(a)lists.jboss.org
> > > > > > [mailto:infinispan-dev-bounces@lists.jboss.org] On Behalf
Of Galder
> > > > > > Zamarreño
> > > > > > Sent: Tuesday, January 18, 2011 6:12 AM
> > > > > > To: infinispan -Dev List
> > > > > > Subject: Re: [infinispan-dev] ISPN-863 - Thoughts /
Questions
> > > > > >
> > > > > > Hmmm, try increasing -Xmx? Start with
-XX:+HeapDumpOnOutOfMemoryError
> > > > > > -XX:HeapDumpPath=/tmp/java_heap and inspect the dump with
Eclipse MAT?...
> > > > > >
> > > > > > On Jan 17, 2011, at 4:51 PM, david marion wrote:
> > > > > >
> > > > > > > same result.
> > > > > > >
> > > > > > > Date: Mon, 17 Jan 2011 11:54:15 -0300
> > > > > > > From: vblagoje(a)redhat.com
> > > > > > > To: infinispan-dev(a)lists.jboss.org
> > > > > > > Subject: Re: [infinispan-dev] ISPN-863 - Thoughts /
Questions
> > > > > > >
> > > > > > > Its easier to run everything from command line for two
reason I can think
> > > > > > of:
> > > > > > > - everyone has same env to compare results with you
> > > > > > > - it is all ready to go
> > > > > > >
> > > > > > > See
http://community.jboss.org/wiki/InfinispanandMaven
> > > > > > >
> > > > > > > On 11-01-17 11:38 AM, Dave wrote:
> > > > > > > All,
> > > > > > >
> > > > > > > I am trying to test some changes that I made using
Eclipse and the TestNG
> > > > > > plugin. I am not familiar with TestNG. Are there any
settings that I should
> > > > > > make? Any ideas?
> > > > > > >
> > > > > > > java.lang.OutOfMemoryError: Java heap space
> > > > > > > at
> > > > > >
org.infinispan.util.concurrent.BoundedConcurrentHashMap$HashEntry.newArray(B
> > > > > > oundedConcurrentHashMap.java:295)
> > > > > > > at
> > > > > >
org.infinispan.util.concurrent.BoundedConcurrentHashMap$Segment.<init>(Bound
> > > > > > edConcurrentHashMap.java:898)
> > > > > > > at
> > > > > >
org.infinispan.util.concurrent.BoundedConcurrentHashMap.<init>(BoundedConcur
> > > > > > rentHashMap.java:1367)
> > > > > > > at
> > > > > >
org.infinispan.container.DefaultDataContainer.<init>(DefaultDataContainer.ja
> > > > > > va:78)
> > > > > > > at
> > > > > >
org.infinispan.container.DefaultDataContainer.<init>(DefaultDataContainer.ja
> > > > > > va:47)
> > > > > > > at
> > > > > >
org.infinispan.container.DefaultDataContainer.unBoundedDataContainer(Default
> > > > > > DataContainer.java:93)
> > > > > > > at
> > > > > >
org.infinispan.factories.DataContainerFactory.construct(DataContainerFactory
> > > > > > .java:53)
> > > > > > > at
> > > > > >
org.infinispan.factories.AbstractComponentRegistry.getOrCreateComponent(Abst
> > > > > > ractComponentRegistry.java:315)
> > > > > > > at
> > > > > >
org.infinispan.factories.AbstractComponentRegistry.invokeInjectionMethod(Abs
> > > > > > tractComponentRegistry.java:251)
> > > > > > > at
> > > > > >
org.infinispan.factories.AbstractComponentRegistry$Component.injectDependenc
> > > > > > ies(AbstractComponentRegistry.java:840)
> > > > > > > at
> > > > > >
org.infinispan.factories.AbstractComponentRegistry.registerComponent(Abstrac
> > > > > > tComponentRegistry.java:225)
> > > > > > > at
> > > > > >
org.infinispan.factories.ComponentRegistry.registerComponent(ComponentRegist
> > > > > > ry.java:120)
> > > > > > > at
> > > > > >
org.infinispan.factories.AbstractComponentRegistry.registerComponent(Abstrac
> > > > > > tComponentRegistry.java:192)
> > > > > > > at
> > > > > >
org.infinispan.factories.AbstractComponentRegistry.getOrCreateComponent(Abst
> > > > > > ractComponentRegistry.java:323)
> > > > > > > at
> > > > > >
org.infinispan.factories.AbstractComponentRegistry.invokeInjectionMethod(Abs
> > > > > > tractComponentRegistry.java:251)
> > > > > > > at
> > > > > >
org.infinispan.factories.AbstractComponentRegistry$Component.injectDependenc
> > > > > > ies(AbstractComponentRegistry.java:840)
> > > > > > > at
> > > > > >
org.infinispan.factories.AbstractComponentRegistry.registerComponent(Abstrac
> > > > > > tComponentRegistry.java:225)
> > > > > > > at
> > > > > >
org.infinispan.factories.ComponentRegistry.registerComponent(ComponentRegist
> > > > > > ry.java:120)
> > > > > > > at
> > > > > >
org.infinispan.factories.AbstractComponentRegistry.registerComponent(Abstrac
> > > > > > tComponentRegistry.java:192)
> > > > > > > at
> > > > > >
org.infinispan.factories.AbstractComponentRegistry.getOrCreateComponent(Abst
> > > > > > ractComponentRegistry.java:323)
> > > > > > > at
> > > > > >
org.infinispan.factories.AbstractComponentRegistry.invokeInjectionMethod(Abs
> > > > > > tractComponentRegistry.java:251)
> > > > > > > at
> > > > > >
org.infinispan.factories.AbstractComponentRegistry$Component.injectDependenc
> > > > > > ies(AbstractComponentRegistry.java:840)
> > > > > > > at
> > > > > >
org.infinispan.factories.AbstractComponentRegistry.registerComponent(Abstrac
> > > > > > tComponentRegistry.java:225)
> > > > > > > at
> > > > > >
org.infinispan.factories.ComponentRegistry.registerComponent(ComponentRegist
> > > > > > ry.java:120)
> > > > > > > at
> > > > > >
org.infinispan.factories.AbstractComponentRegistry.registerComponent(Abstrac
> > > > > > tComponentRegistry.java:192)
> > > > > > > at
> > > > > >
org.infinispan.factories.InternalCacheFactory.bootstrap(InternalCacheFactory
> > > > > > .java:92)
> > > > > > > at
> > > > > >
org.infinispan.factories.InternalCacheFactory.createAndWire(InternalCacheFac
> > > > > > tory.java:78)
> > > > > > > at
> > > > > >
org.infinispan.factories.InternalCacheFactory.createCache(InternalCacheFacto
> > > > > > ry.java:62)
> > > > > > > at
> > > > > >
org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.j
> > > > > > ava:510)
> > > > > > > at
> > > > > >
org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java
> > > > > > :440)
> > > > > > > at
> > > > > >
org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java
> > > > > > :409)
> > > > > > > at
> > > > > >
org.infinispan.config.ConfigurationValidationTest.testDefaultMemoryGuardConf
> > > > > > iguration(ConfigurationValidationTest.java:100)
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > -----Original Message-----
> > > > > > > From: infinispan-dev-bounces(a)lists.jboss.org
> > > > > > [mailto:infinispan-dev-bounces@lists.jboss.org] On Behalf
Of Vladimir
> > > > > > Blagojevic
> > > > > > > Sent: Friday, January 14, 2011 11:05 AM
> > > > > > > To: infinispan -Dev List
> > > > > > > Subject: Re: [infinispan-dev] ISPN-863 - Thoughts /
Questions
> > > > > > >
> > > > > > > Hey David,
> > > > > > >
> > > > > > > First of all kudos for taking on such a non-trivial
task!
> > > > > > >
> > > > > > > On 11-01-14 12:22 AM, david marion wrote:
> > > > > > > 1. Modify configuration in some way so that the
following can be
> > > > > > specified:
> > > > > > > a. The percentage value of used JVM memory (i.e. 95)
at which
> > > > > > entries should be evicted to try and avoid an OOM error.
> > > > > > > b. The number of items that should be evicted when
memory reaches
> > > > > > this threshold
> > > > > > > 2. Modify LRU and LIRS Eviction class so that the
accessQueue member
> > > > > > can be accessed by the new Eviction class so that two
access queues don’t
> > > > > > have to be maintained.
> > > > > > > 3. Create a new Eviction class, a subclass of LIRS,
where the
> > > > > > accessQueue is used from the Eviction strategy the user
specifies and the
> > > > > > for loop in the execute method is exited when the evicted
set equals value
> > > > > > from 1.b above.
> > > > > > > 4. Modify DataContainerFactory.construct() to call
> > > > > > DefaultDataContainer.boundedDataContainer() regardless of
eviction policy.
> > > > > > This will always create a BoundedConcurrentHashMap
> > > > > > >
> > > > > > > I don't think you have to extend LRU and LIRS, as
long as you implement
> > > > > > EvictionPolicy you are fullfilling the contract. What
extending LRU and LIRS
> > > > > > can do is give you some order in selecting proper elements
for eviction. I
> > > > > > think the precise technical term in research literature is
"eviction
> > > > > > precision".
> > > > > > >
> > > > > > >
> > > > > > > 1.
> > > > > > > 2. Create an instance of the new Eviction class in
each segment.
> > > > > > > 3. Modify BoundedConcurrentHashMap.Segment put and
replace methods
> > > > > > such that when new values are going to be put into the
Segment, the memory
> > > > > > usage is checked and the execute method is called on the
new Eviction class.
> > > > > > >
> > > > > > > Sounds right!
> > > > > > >
> > > > > > >
> > > > > > > 1.
> > > > > > >
> > > > > > > Questions:
> > > > > > >
> > > > > > > 1. What are the implications of using a
BoundedConcurrentHashMap
> > > > > > instead of a ConcurrentHashMap when maxEntries is set to
-1?
> > > > > > >
> > > > > > > I think none except you turn on eviction by using
> > > > > > BoundedConcurrentHashMap.
> > > > > > >
> > > > > > >
> > > > > > > 1.
> > > > > > >
> > > > > > > Thoughts
> > > > > > >
> > > > > > > 1. This will not guarantee that an OOM error does not
occur. It will
> > > > > > attempt to guard against an OOM caused by putting new
values into the cache.
> > > > > > This will probably be more effective when the cache is
being used in
> > > > > > client/server mode, and less effective when used in
embedded mode as to
> > > > > > other code running in the JVM.
> > > > > > >
> > > > > > >
> > > > > > > -- Dave Marion
> > > > > > >
> > > > > > >
> > > > > > > Cheers,
> > > > > > > Vladimir
> > > > > > >
> > > > > > > _______________________________________________
> > > > > > > infinispan-dev mailing list
> > > > > > >
> > > > > > > infinispan-dev(a)lists.jboss.org
> > > > > > >
https://lists.jboss.org/mailman/listinfo/infinispan-dev
> > > > > > >
> > > > > > >
> > > > > > > _______________________________________________
infinispan-dev mailing
> > > > > > list infinispan-dev(a)lists.jboss.org
> > > > > >
https://lists.jboss.org/mailman/listinfo/infinispan-dev_____________________
> > > > > > __________________________
> > > > > > > infinispan-dev mailing list
> > > > > > > infinispan-dev(a)lists.jboss.org
> > > > > > >
https://lists.jboss.org/mailman/listinfo/infinispan-dev
> > > > > >
> > > > > > --
> > > > > > Galder Zamarreño
> > > > > > Sr. Software Engineer
> > > > > > Infinispan, JBoss Cache
> > > > > >
> > > > > >
> > > > > > _______________________________________________
> > > > > > infinispan-dev mailing list
> > > > > > infinispan-dev(a)lists.jboss.org
> > > > > > https://lists.jboss.org/mailman/listinfo/infinispan-dev
> > > > > >
> > > > > >
> > > > > > _______________________________________________
> > > > > > infinispan-dev mailing list
> > > > > > infinispan-dev(a)lists.jboss.org
> > > > > > https://lists.jboss.org/mailman/listinfo/infinispan-dev
> > > > > _______________________________________________
> > > > > infinispan-dev mailing list
> > > > > infinispan-dev(a)lists.jboss.org
> > > > > https://lists.jboss.org/mailman/listinfo/infinispan-dev
> > > >
> > > > --
> > > > Galder Zamarreño
> > > > Sr. Software Engineer
> > > > Infinispan, JBoss Cache
> > > >
> > > >
> > > > _______________________________________________
> > > > infinispan-dev mailing list
> > > > infinispan-dev(a)lists.jboss.org
> > > > https://lists.jboss.org/mailman/listinfo/infinispan-dev
> > > _______________________________________________
> > > infinispan-dev mailing list
> > > infinispan-dev(a)lists.jboss.org
> > > https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >
> > --
> > Galder Zamarreño
> > Sr. Software Engineer
> > Infinispan, JBoss Cache
> >
> >
> > _______________________________________________
> > infinispan-dev mailing list
> > infinispan-dev(a)lists.jboss.org
> > https://lists.jboss.org/mailman/listinfo/infinispan-dev
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
--
Galder Zamarreño
Sr. Software Engineer
Infinispan, JBoss Cache
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
1:24 p.m.
New subject: ISPN-863 - Thoughts / Questions
On 11-01-29 2:16 PM, david marion wrote:
It took me some time to figure out how to use Git and add some more
tests. I have pushed my changes to a topic branch located at
https://github.com/dlmarion/infinispan/tree/ISPN-863-master.
Please let me know if you have any questions or concerns. I have put
up some documentation at
https://github.com/dlmarion/infinispan/wiki/ISPN-863-Implementation
David,
On initial glance it looks solid. Let me pick up your tree branch
locally and I'll have a detailed feedback by the end of the day.
Regards,
Vladimir
4840
days inactive
4863
days old
infinispan-dev@lists.jboss.org
36 comments
6 participants
participants (6)
-
Bela Ban -
Dave -
david marion
-
Galder Zamarreño -
Manik Surtani -
Vladimir Blagojevic