7:21 a.m.
Hi all,
A heads up on what is going on with https://issues.jboss.org/browse/ISPN-2281
While discussing this, Tristan and I came to the conclusion that we could avoid the need
to create some wrappers required to fulfill requirements in this JIRA, and as a side
effect, reduce the memory consumption of Infinispan servers, if we could have internal
data containers based on concurrent hash maps that took a custom function for
equals/hashCode…etc. By doing that, you could effectively have **byte[] keys and values
for maps**.
By doing that, you avoid creating wrappers (yippee!) for keys (bye bye ByteArrayKey), and
combined with a better way to pass metadata into Infinispan Caches (i.e. version) that is
stored within the internal cache entries, you avoid wrapper values too! (bye bye
CacheValue).
Doing the latter was relatively simple (I have this stashed), but having a CHM that could
take a byte[] as key wasn't that easy, since we can't change JDK CHM.
This is why, I've created a new CHM, based on the CHMv8, called
ComparingConcurrentHashMapv8 (thx Tristan for the name!). The work for this can be seen
in: https://github.com/galderz/infinispan/commit/351e29d327d163ca8e941edf873f...
I'm sending it here so that I can get feedback early on. I've added some tests as
well that verify that ComparingConcurrentHashMapv8, with byte[] keys and values, works as
expected, and checks that the expectations are opposite with JDK CHM. It also tests new
function-based methods.
To make it easier to track changes as original CHMv8 evolves, I've marked all changes
with a marker comment that should make it easy to apply same changes in new CHMv8
versions. Plus, with the tests I've added, it can easily be seen if it works as
expected or not.
Two important TODOs, which will be most likely separated into separate JIRAs:
1. Note that TreeBin has not been modified to use custom equals/hashCode functions. That
is cos I need to implement a way to compare byte arrays, i.e. provide equivalent logic for
Comparable.compare().
2. Compare memory consumption of a CHMv8 with wrapper classes for byte arrays versus
ComparingConcurrentHashMapv8<byte[], byte[]>. I'll do that once it's closer
to CR stages. Right now fulfilling requirements in ISPN-2281 is more priority.
Finally, I need to do the same thing with out BoundedConcurrentHashMap, iow, provide a way
to do comparison based on custom equals/hashCode. That's gonna be my next task, before
I get to transform Infinispan Servers to take a type directly, and avoid relying on
ByteArrayKey or CacheValue wrappers.
IOW, you'll be able to say: create an Infinsipan Server that has String as key and
value of type X, where X is the actual data type, no metadata!! The metadata (version,
encoding, whatever is requried to fulfill the compatibility reqs in ISPN-2281) will be
passed as part of the put/replace…etc (I will email this around when in place).
Cheers,
--
Galder Zamarreño
galder(a)redhat.com
twitter.com/galderz
Project Lead, Escalante
http://escalante.io
Engineer, Infinispan
http://infinispan.org
9:39 a.m.
On 03/18/2013 01:21 PM, Galder Zamarreño wrote:
Hi all,
A heads up on what is going on with https://issues.jboss.org/browse/ISPN-2281
While discussing this, Tristan and I came to the conclusion that we could avoid the need
to create some wrappers required to fulfill requirements in this JIRA, and as a side
effect, reduce the memory consumption of Infinispan servers, if we could have internal
data containers based on concurrent hash maps that took a custom function for
equals/hashCode…etc. By doing that, you could effectively have **byte[] keys and values
for maps**.
By doing that, you avoid creating wrappers (yippee!) for keys (bye bye ByteArrayKey), and
combined with a better way to pass metadata into Infinispan Caches (i.e. version) that is
stored within the internal cache entries, you avoid wrapper values too! (bye bye
CacheValue).
Pushing this even further would be to keep data and metadata separate in
the store representation too as we currently write/read marshalled
InternalCacheEntries to the stores. Probably this will have to wait for
6.0 when we overhaul the stores.
Tristan
11:47 a.m.
And will conditional operations on byte[] work as well within this fix? I have just
noticed that in library mode I can't use remove with byte[] value (should I file a
feature request JIRA for that?)
Radim
----- Original Message -----
| From: "Tristan Tarrant" <ttarrant(a)redhat.com>
| To: "infinispan -Dev List" <infinispan-dev(a)lists.jboss.org>
| Sent: Monday, March 18, 2013 3:39:05 PM
| Subject: Re: [infinispan-dev] Bye bye wrappers, ComparingConcurrentHashMapv8 is here
(ISPN-2281)
|
| On 03/18/2013 01:21 PM, Galder Zamarreño wrote:
| > Hi all,
| >
| > A heads up on what is going on with
| > https://issues.jboss.org/browse/ISPN-2281
| >
| > While discussing this, Tristan and I came to the conclusion that we
| > could avoid the need to create some wrappers required to fulfill
| > requirements in this JIRA, and as a side effect, reduce the memory
| > consumption of Infinispan servers, if we could have internal data
| > containers based on concurrent hash maps that took a custom
| > function for equals/hashCode…etc. By doing that, you could
| > effectively have **byte[] keys and values for maps**.
| >
| > By doing that, you avoid creating wrappers (yippee!) for keys (bye
| > bye ByteArrayKey), and combined with a better way to pass metadata
| > into Infinispan Caches (i.e. version) that is stored within the
| > internal cache entries, you avoid wrapper values too! (bye bye
| > CacheValue).
| >
| Pushing this even further would be to keep data and metadata separate
| in
| the store representation too as we currently write/read marshalled
| InternalCacheEntries to the stores. Probably this will have to wait
| for
| 6.0 when we overhaul the stores.
|
| Tristan
|
| _______________________________________________
| infinispan-dev mailing list
| infinispan-dev(a)lists.jboss.org
| https://lists.jboss.org/mailman/listinfo/infinispan-dev
|
5:58 a.m.
On Mar 19, 2013, at 5:47 PM, Radim Vansa <rvansa(a)redhat.com> wrote:
And will conditional operations on byte[] work as well within this
fix?
^ Of course. Ssee the tests I added in that commit for traditional JDK CHM conditional
operations, and new conditional operations using functions coming up in JDK8. I've
added some more since then, but *all methods* that can lead to a key.equals() or
value.equals should be covered, no exceptions.
I have just noticed that in library mode I can't use remove with
byte[] value (should I file a feature request JIRA for that?)
^ You can, but not if you're relying on conditional operations based on value :)
No need for JIRA since I'm already doing it as part of ISPN-2281.
Radim
----- Original Message -----
| From: "Tristan Tarrant" <ttarrant(a)redhat.com>
| To: "infinispan -Dev List" <infinispan-dev(a)lists.jboss.org>
| Sent: Monday, March 18, 2013 3:39:05 PM
| Subject: Re: [infinispan-dev] Bye bye wrappers, ComparingConcurrentHashMapv8 is here
(ISPN-2281)
|
| On 03/18/2013 01:21 PM, Galder Zamarreño wrote:
| > Hi all,
| >
| > A heads up on what is going on with
| > https://issues.jboss.org/browse/ISPN-2281
| >
| > While discussing this, Tristan and I came to the conclusion that we
| > could avoid the need to create some wrappers required to fulfill
| > requirements in this JIRA, and as a side effect, reduce the memory
| > consumption of Infinispan servers, if we could have internal data
| > containers based on concurrent hash maps that took a custom
| > function for equals/hashCode…etc. By doing that, you could
| > effectively have **byte[] keys and values for maps**.
| >
| > By doing that, you avoid creating wrappers (yippee!) for keys (bye
| > bye ByteArrayKey), and combined with a better way to pass metadata
| > into Infinispan Caches (i.e. version) that is stored within the
| > internal cache entries, you avoid wrapper values too! (bye bye
| > CacheValue).
| >
| Pushing this even further would be to keep data and metadata separate
| in
| the store representation too as we currently write/read marshalled
| InternalCacheEntries to the stores. Probably this will have to wait
| for
| 6.0 when we overhaul the stores.
|
| Tristan
|
| _______________________________________________
| infinispan-dev mailing list
| infinispan-dev(a)lists.jboss.org
| https://lists.jboss.org/mailman/listinfo/infinispan-dev
|
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
--
Galder Zamarreño
galder(a)redhat.com
twitter.com/galderz
Project Lead, Escalante
http://escalante.io
Engineer, Infinispan
http://infinispan.org
2:56 p.m.
On 18 Mar 2013, at 12:21, Galder Zamarreño wrote:
Hi all,
A heads up on what is going on with https://issues.jboss.org/browse/ISPN-2281
While discussing this, Tristan and I came to the conclusion that we could avoid the need
to create some wrappers required to fulfill requirements in this JIRA, and as a side
effect, reduce the memory consumption of Infinispan servers, if we could have internal
data containers based on concurrent hash maps that took a custom function for
equals/hashCode…etc. By doing that, you could effectively have **byte[] keys and values
for maps**.
By doing that, you avoid creating wrappers (yippee!) for keys (bye bye ByteArrayKey), and
combined with a better way to pass metadata into Infinispan Caches (i.e. version) that is
stored within the internal cache entries, you avoid wrapper values too! (bye bye
CacheValue).
Do you plan to use Versioned*CacheValue for storing values? if so
you'd still create a version object to be aggregated in Versioned*CacheValue.
Doing the latter was relatively simple (I have this stashed), but having a CHM that could
take a byte[] as key wasn't that easy, since we can't change JDK CHM.
why's that? copyright?
This is why, I've created a new CHM, based on the CHMv8, called
ComparingConcurrentHashMapv8 (thx Tristan for the name!). The work for this can be seen
in: https://github.com/galderz/infinispan/commit/351e29d327d163ca8e941edf873f...
this relies on sun.misc.Unsafe, so won't work with a non-oracle JVM.
This class would be aggregated in the DataContainer in the case we use infinispan in
server mode, right?
I'm sending it here so that I can get feedback early on. I've added some tests as
well that verify that ComparingConcurrentHashMapv8, with byte[] keys and values, works as
expected, and checks that the expectations are opposite with JDK CHM. It also tests new
function-based methods.
To make it easier to track changes as original CHMv8 evolves, I've marked all changes
with a marker comment that should make it easy to apply same changes in new CHMv8
versions. Plus, with the tests I've added, it can easily be seen if it works as
expected or not.
Two important TODOs, which will be most likely separated into separate JIRAs:
1. Note that TreeBin has not been modified to use custom equals/hashCode functions. That
is cos I need to implement a way to compare byte arrays, i.e. provide equivalent logic for
Comparable.compare().
2. Compare memory consumption of a CHMv8 with wrapper classes for byte arrays versus
ComparingConcurrentHashMapv8<byte[], byte[]>. I'll do that once it's closer
to CR stages. Right now fulfilling requirements in ISPN-2281 is more priority.
Finally, I need to do the same thing with out BoundedConcurrentHashMap, iow, provide a
way to do comparison based on custom equals/hashCode. That's gonna be my next task,
before I get to transform Infinispan Servers to take a type directly, and avoid relying on
ByteArrayKey or CacheValue wrappers.
IOW, you'll be able to say: create an Infinsipan Server that has String as key and
value of type X, where X is the actual data type, no metadata!! The metadata (version,
encoding, whatever is requried to fulfill the compatibility reqs in ISPN-2281) will be
passed as part of the put/replace…etc (I will email this around when in place).
Cheers,
--
Galder Zamarreño
galder(a)redhat.com
twitter.com/galderz
Project Lead, Escalante
http://escalante.io
Engineer, Infinispan
http://infinispan.org
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Cheers,
--
Mircea Markus
Infinispan lead (www.infinispan.org)
4:49 a.m.
Great stuff. Another benefit to keep in mind is that this can make it
easy to use equality defined as identity of objects (the "=="):
something I'm looking forward for, as with all use cases I'm familiar
with (Lucene Directory + Hibernate OGM) we should be able to maintain
uniqueness of instances per key.
You could take this a step further (maybe as a second step in future)
and break out of the ConcurrentMap contract.. we don't strictly need
that; as from previous brainstorming we can optimise certain
operations - although I don't remember if any is left as Manik already
applied some great improvements - so just to remind it doesn't have to
strictly guarantee the ConcurrentMap contract.
Sanne
On 19 March 2013 19:56, Mircea Markus <mmarkus(a)redhat.com> wrote:
On 18 Mar 2013, at 12:21, Galder Zamarreño wrote:
> Hi all,
>
> A heads up on what is going on with https://issues.jboss.org/browse/ISPN-2281
>
> While discussing this, Tristan and I came to the conclusion that we could avoid the
need to create some wrappers required to fulfill requirements in this JIRA, and as a side
effect, reduce the memory consumption of Infinispan servers, if we could have internal
data containers based on concurrent hash maps that took a custom function for
equals/hashCode…etc. By doing that, you could effectively have **byte[] keys and values
for maps**.
>
> By doing that, you avoid creating wrappers (yippee!) for keys (bye bye ByteArrayKey),
and combined with a better way to pass metadata into Infinispan Caches (i.e. version) that
is stored within the internal cache entries, you avoid wrapper values too! (bye bye
CacheValue).
Do you plan to use Versioned*CacheValue for storing values? if so you'd still create
a version object to be aggregated in Versioned*CacheValue.
>
> Doing the latter was relatively simple (I have this stashed), but having a CHM that
could take a byte[] as key wasn't that easy, since we can't change JDK CHM.
why's that? copyright?
>
> This is why, I've created a new CHM, based on the CHMv8, called
ComparingConcurrentHashMapv8 (thx Tristan for the name!). The work for this can be seen
in: https://github.com/galderz/infinispan/commit/351e29d327d163ca8e941edf873f...
this relies on sun.misc.Unsafe, so won't work with a non-oracle JVM.
This class would be aggregated in the DataContainer in the case we use infinispan in
server mode, right?
>
> I'm sending it here so that I can get feedback early on. I've added some
tests as well that verify that ComparingConcurrentHashMapv8, with byte[] keys and values,
works as expected, and checks that the expectations are opposite with JDK CHM. It also
tests new function-based methods.
>
> To make it easier to track changes as original CHMv8 evolves, I've marked all
changes with a marker comment that should make it easy to apply same changes in new CHMv8
versions. Plus, with the tests I've added, it can easily be seen if it works as
expected or not.
>
> Two important TODOs, which will be most likely separated into separate JIRAs:
>
> 1. Note that TreeBin has not been modified to use custom equals/hashCode functions.
That is cos I need to implement a way to compare byte arrays, i.e. provide equivalent
logic for Comparable.compare().
>
> 2. Compare memory consumption of a CHMv8 with wrapper classes for byte arrays versus
ComparingConcurrentHashMapv8<byte[], byte[]>. I'll do that once it's closer
to CR stages. Right now fulfilling requirements in ISPN-2281 is more priority.
>
> Finally, I need to do the same thing with out BoundedConcurrentHashMap, iow, provide
a way to do comparison based on custom equals/hashCode. That's gonna be my next task,
before I get to transform Infinispan Servers to take a type directly, and avoid relying on
ByteArrayKey or CacheValue wrappers.
>
> IOW, you'll be able to say: create an Infinsipan Server that has String as key
and value of type X, where X is the actual data type, no metadata!! The metadata (version,
encoding, whatever is requried to fulfill the compatibility reqs in ISPN-2281) will be
passed as part of the put/replace…etc (I will email this around when in place).
>
> Cheers,
> --
> Galder Zamarreño
> galder(a)redhat.com
> twitter.com/galderz
>
> Project Lead, Escalante
> http://escalante.io
>
> Engineer, Infinispan
> http://infinispan.org
>
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
Cheers,
--
Mircea Markus
Infinispan lead (www.infinispan.org)
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
6:12 a.m.
On Mar 20, 2013, at 10:49 AM, Sanne Grinovero <sanne(a)infinispan.org> wrote:
Great stuff. Another benefit to keep in mind is that this can make
it
easy to use equality defined as identity of objects (the "=="):
something I'm looking forward for, as with all use cases I'm familiar
with (Lucene Directory + Hibernate OGM) we should be able to maintain
uniqueness of instances per key.
^ That'd be dead easy to do. Just create an implementation of Comparing, i.e.
ComparingByIdentity or something like, and pass it to as key/value Comparing function,
depending at which level you wanna do the identitiy comparison, whether key or value only,
or both.
You could take this a step further (maybe as a second step in
future)
and break out of the ConcurrentMap contract.. we don't strictly need
that; as from previous brainstorming we can optimise certain
operations - although I don't remember if any is left as Manik already
applied some great improvements - so just to remind it doesn't have to
strictly guarantee the ConcurrentMap contract.
Good point. We're certainly not obliged to keep a "ConcurrentMap" per se
internally.
What optimisations did you brainstorm? Would you mind creating a JIRA about this last part
and assigning it to me?
Cheers,
Sanne
On 19 March 2013 19:56, Mircea Markus <mmarkus(a)redhat.com> wrote:
>
> On 18 Mar 2013, at 12:21, Galder Zamarreño wrote:
>
>> Hi all,
>>
>> A heads up on what is going on with https://issues.jboss.org/browse/ISPN-2281
>>
>> While discussing this, Tristan and I came to the conclusion that we could avoid
the need to create some wrappers required to fulfill requirements in this JIRA, and as a
side effect, reduce the memory consumption of Infinispan servers, if we could have
internal data containers based on concurrent hash maps that took a custom function for
equals/hashCode…etc. By doing that, you could effectively have **byte[] keys and values
for maps**.
>>
>> By doing that, you avoid creating wrappers (yippee!) for keys (bye bye
ByteArrayKey), and combined with a better way to pass metadata into Infinispan Caches
(i.e. version) that is stored within the internal cache entries, you avoid wrapper values
too! (bye bye CacheValue).
> Do you plan to use Versioned*CacheValue for storing values? if so you'd still
create a version object to be aggregated in Versioned*CacheValue.
>>
>> Doing the latter was relatively simple (I have this stashed), but having a CHM
that could take a byte[] as key wasn't that easy, since we can't change JDK CHM.
> why's that? copyright?
>>
>> This is why, I've created a new CHM, based on the CHMv8, called
ComparingConcurrentHashMapv8 (thx Tristan for the name!). The work for this can be seen
in: https://github.com/galderz/infinispan/commit/351e29d327d163ca8e941edf873f...
> this relies on sun.misc.Unsafe, so won't work with a non-oracle JVM.
> This class would be aggregated in the DataContainer in the case we use infinispan in
server mode, right?
>>
>> I'm sending it here so that I can get feedback early on. I've added some
tests as well that verify that ComparingConcurrentHashMapv8, with byte[] keys and values,
works as expected, and checks that the expectations are opposite with JDK CHM. It also
tests new function-based methods.
>>
>> To make it easier to track changes as original CHMv8 evolves, I've marked all
changes with a marker comment that should make it easy to apply same changes in new CHMv8
versions. Plus, with the tests I've added, it can easily be seen if it works as
expected or not.
>>
>> Two important TODOs, which will be most likely separated into separate JIRAs:
>>
>> 1. Note that TreeBin has not been modified to use custom equals/hashCode
functions. That is cos I need to implement a way to compare byte arrays, i.e. provide
equivalent logic for Comparable.compare().
>>
>> 2. Compare memory consumption of a CHMv8 with wrapper classes for byte arrays
versus ComparingConcurrentHashMapv8<byte[], byte[]>. I'll do that once it's
closer to CR stages. Right now fulfilling requirements in ISPN-2281 is more priority.
>>
>> Finally, I need to do the same thing with out BoundedConcurrentHashMap, iow,
provide a way to do comparison based on custom equals/hashCode. That's gonna be my
next task, before I get to transform Infinispan Servers to take a type directly, and avoid
relying on ByteArrayKey or CacheValue wrappers.
>>
>> IOW, you'll be able to say: create an Infinsipan Server that has String as
key and value of type X, where X is the actual data type, no metadata!! The metadata
(version, encoding, whatever is requried to fulfill the compatibility reqs in ISPN-2281)
will be passed as part of the put/replace…etc (I will email this around when in place).
>>
>> Cheers,
>> --
>> Galder Zamarreño
>> galder(a)redhat.com
>> twitter.com/galderz
>>
>> Project Lead, Escalante
>> http://escalante.io
>>
>> Engineer, Infinispan
>> http://infinispan.org
>>
>>
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev(a)lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
> Cheers,
> --
> Mircea Markus
> Infinispan lead (www.infinispan.org)
>
>
>
>
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
--
Galder Zamarreño
galder(a)redhat.com
twitter.com/galderz
Project Lead, Escalante
http://escalante.io
Engineer, Infinispan
http://infinispan.org
5:50 a.m.
On 19 Mar 2013, at 19:56, Mircea Markus <mmarkus(a)redhat.com> wrote:
>
>
> Doing the latter was relatively simple (I have this stashed), but having a CHM that
could take a byte[] as key wasn't that easy, since we can't change JDK CHM.
why's that? copyright?
A lot of JDK CHM code is private or package-protected so cannot be extended/overridden.
>
> This is why, I've created a new CHM, based on the CHMv8, called
ComparingConcurrentHashMapv8 (thx Tristan for the name!). The work for this can be seen
in:https://github.com/galderz/infinispan/commit/351e29d327d163ca8e941edf8...
this relies on sun.misc.Unsafe, so won't work with a non-oracle JVM.
Not true - see this comment on the Disruptor forums, which also use sun.misc.Unsafe:
https://groups.google.com/forum/?fromgroups=#!topic/lmax-disruptor/t7Hr8q...
This class would be aggregated in the DataContainer in the case we
use infinispan in server mode, right?
--
Manik Surtani
manik(a)jboss.org
twitter.com/maniksurtani
Platform Architect, JBoss Data Grid
http://red.ht/data-grid
6:28 a.m.
On 20 Mar 2013, at 10:50, Manik Surtani wrote:
>>
>> This is why, I've created a new CHM, based on the CHMv8, called
ComparingConcurrentHashMapv8 (thx Tristan for the name!). The work for this can be seen
in:https://github.com/galderz/infinispan/commit/351e29d327d163ca8e941edf8...
> this relies on sun.misc.Unsafe, so won't work with a non-oracle JVM.
Not true - see this comment on the Disruptor forums, which also use sun.misc.Unsafe:
https://groups.google.com/forum/?fromgroups=#!topic/lmax-disruptor/t7Hr8q...
thanks, good to know :-)
Cheers,
--
Mircea Markus
Infinispan lead (www.infinispan.org)
6:08 a.m.
On Mar 19, 2013, at 8:56 PM, Mircea Markus <mmarkus(a)redhat.com> wrote:
On 18 Mar 2013, at 12:21, Galder Zamarreño wrote:
> Hi all,
>
> A heads up on what is going on with https://issues.jboss.org/browse/ISPN-2281
>
> While discussing this, Tristan and I came to the conclusion that we could avoid the
need to create some wrappers required to fulfill requirements in this JIRA, and as a side
effect, reduce the memory consumption of Infinispan servers, if we could have internal
data containers based on concurrent hash maps that took a custom function for
equals/hashCode…etc. By doing that, you could effectively have **byte[] keys and values
for maps**.
>
> By doing that, you avoid creating wrappers (yippee!) for keys (bye bye ByteArrayKey),
and combined with a better way to pass metadata into Infinispan Caches (i.e. version) that
is stored within the internal cache entries, you avoid wrapper values too! (bye bye
CacheValue).
Do you plan to use Versioned*CacheValue for storing values? if so you'd still create
a version object to be aggregated in Versioned*CacheValue.
^ Yes, precisely! This was very easy to do actually. Once I had the API to feed in version
(and other metadata down the line…), it was very easy to change the core to use internal
cache entry classes to store. Hence, no extra wrapper value other than what Infinispan
uses internally.
>
> Doing the latter was relatively simple (I have this stashed), but having a CHM that
could take a byte[] as key wasn't that easy, since we can't change JDK CHM.
why's that? copyright?
^ Well, I cannot modify the JDK source code directly. I'd have to fork the JDK CHM,
but I prefer to take the CHMv8 version since it gets rid of segments.
>
> This is why, I've created a new CHM, based on the CHMv8, called
ComparingConcurrentHashMapv8 (thx Tristan for the name!). The work for this can be seen
in: https://github.com/galderz/infinispan/commit/351e29d327d163ca8e941edf873f...
this relies on sun.misc.Unsafe, so won't work with a non-oracle JVM.
^ Hmmm, possibly. So we might need a JDK CHM version for it. I've already transformed
BoundedCHM to it, so woud be easy to achieve.
This class would be aggregated in the DataContainer in the case we
use infinispan in server mode, right?
^ Right now, I've added two configuration options to dataContainer element, called
comparingKey and comparingValue, both of which are class names to implementations of
Comparing. If either is defined, a Comparing*CHM would be used. So, Hot Rod servers would
need to have ComparingByteArray defined in comparingKey and comparingValue in their
configuration.
I think this is something we can control in both the default Infinispan and JDG servers.
Thanks for the excellent feedback! :)
>
> I'm sending it here so that I can get feedback early on. I've added some
tests as well that verify that ComparingConcurrentHashMapv8, with byte[] keys and values,
works as expected, and checks that the expectations are opposite with JDK CHM. It also
tests new function-based methods.
>
> To make it easier to track changes as original CHMv8 evolves, I've marked all
changes with a marker comment that should make it easy to apply same changes in new CHMv8
versions. Plus, with the tests I've added, it can easily be seen if it works as
expected or not.
>
> Two important TODOs, which will be most likely separated into separate JIRAs:
>
> 1. Note that TreeBin has not been modified to use custom equals/hashCode functions.
That is cos I need to implement a way to compare byte arrays, i.e. provide equivalent
logic for Comparable.compare().
>
> 2. Compare memory consumption of a CHMv8 with wrapper classes for byte arrays versus
ComparingConcurrentHashMapv8<byte[], byte[]>. I'll do that once it's closer
to CR stages. Right now fulfilling requirements in ISPN-2281 is more priority.
>
> Finally, I need to do the same thing with out BoundedConcurrentHashMap, iow, provide
a way to do comparison based on custom equals/hashCode. That's gonna be my next task,
before I get to transform Infinispan Servers to take a type directly, and avoid relying on
ByteArrayKey or CacheValue wrappers.
>
> IOW, you'll be able to say: create an Infinsipan Server that has String as key
and value of type X, where X is the actual data type, no metadata!! The metadata (version,
encoding, whatever is requried to fulfill the compatibility reqs in ISPN-2281) will be
passed as part of the put/replace…etc (I will email this around when in place).
>
> Cheers,
> --
> Galder Zamarreño
> galder(a)redhat.com
> twitter.com/galderz
>
> Project Lead, Escalante
> http://escalante.io
>
> Engineer, Infinispan
> http://infinispan.org
>
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
Cheers,
--
Mircea Markus
Infinispan lead (www.infinispan.org)
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
--
Galder Zamarreño
galder(a)redhat.com
twitter.com/galderz
Project Lead, Escalante
http://escalante.io
Engineer, Infinispan
http://infinispan.org
5:51 a.m.
On 18 Mar 2013, at 12:21, Galder Zamarreño <galder(a)redhat.com> wrote:
IOW, you'll be able to say: create an Infinsipan Server that has
String as key and value of type X, where X is the actual data type, no metadata!! The
metadata (version, encoding, whatever is requried to fulfill the compatibility reqs in
ISPN-2281) will be passed as part of the put/replace…etc (I will email this around when in
place).
Not sure I get this. Where will this metadata be stored?
--
Manik Surtani
manik(a)jboss.org
twitter.com/maniksurtani
Platform Architect, JBoss Data Grid
http://red.ht/data-grid
6:04 a.m.
On 03/20/2013 11:51 AM, Manik Surtani wrote:
On 18 Mar 2013, at 12:21, Galder Zamarreño <galder(a)redhat.com>
wrote:
> IOW, you'll be able to say: create an Infinsipan Server that has String as key
and value of type X, where X is the actual data type, no metadata!! The metadata (version,
encoding, whatever is requried to fulfill the compatibility reqs in ISPN-2281) will be
passed as part of the put/replace…etc (I will email this around when in place).
Not sure I get this. Where will this metadata be stored?
Metadata should be moved
to the ICE-level, or the Cache-level (for
metadata common to all entries in the cache, to save space: e.g. if all
entries in my cache are text/plain)
Tristan
6:25 a.m.
Why would you need to *store* anything which is equal for all entries?
I'd have an interceptor "decorate" stuff with some constant additions;
otherwise you could also think of reusing instances, like I had
suggested a while back to change the Externalizer contract to allow it
to be passed some context to make unique objects taking them from a
pool.
At least it looks like you would have to split such metadata: version
for example needs to be maintained per entry, as will be vector
clocks.
Sanne
On 20 March 2013 11:04, Tristan Tarrant <ttarrant(a)redhat.com> wrote:
On 03/20/2013 11:51 AM, Manik Surtani wrote:
> On 18 Mar 2013, at 12:21, Galder Zamarreño <galder(a)redhat.com> wrote:
>
>> IOW, you'll be able to say: create an Infinsipan Server that has String as
key and value of type X, where X is the actual data type, no metadata!! The metadata
(version, encoding, whatever is requried to fulfill the compatibility reqs in ISPN-2281)
will be passed as part of the put/replace…etc (I will email this around when in place).
> Not sure I get this. Where will this metadata be stored?
Metadata should be moved to the ICE-level, or the Cache-level (for
metadata common to all entries in the cache, to save space: e.g. if all
entries in my cache are text/plain)
Tristan
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
9:07 a.m.
Didn't I just say exactly that ? I don't understand where we differ.
Tristan
On 03/20/2013 12:25 PM, Sanne Grinovero wrote:
Why would you need to *store* anything which is equal for all
entries?
I'd have an interceptor "decorate" stuff with some constant additions;
otherwise you could also think of reusing instances, like I had
suggested a while back to change the Externalizer contract to allow it
to be passed some context to make unique objects taking them from a
pool.
At least it looks like you would have to split such metadata: version
for example needs to be maintained per entry, as will be vector
clocks.
Sanne
On 20 March 2013 11:04, Tristan Tarrant <ttarrant(a)redhat.com> wrote:
> On 03/20/2013 11:51 AM, Manik Surtani wrote:
>> On 18 Mar 2013, at 12:21, Galder Zamarreño <galder(a)redhat.com> wrote:
>>
>>> IOW, you'll be able to say: create an Infinsipan Server that has String
as key and value of type X, where X is the actual data type, no metadata!! The metadata
(version, encoding, whatever is requried to fulfill the compatibility reqs in ISPN-2281)
will be passed as part of the put/replace…etc (I will email this around when in place).
>> Not sure I get this. Where will this metadata be stored?
> Metadata should be moved to the ICE-level, or the Cache-level (for
> metadata common to all entries in the cache, to save space: e.g. if all
> entries in my cache are text/plain)
>
> Tristan
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
7:20 a.m.
I wrote some of these replies while offline… before I had seen your email :)
On Mar 21, 2013, at 2:07 PM, Tristan Tarrant <ttarrant(a)redhat.com> wrote:
Didn't I just say exactly that ? I don't understand where we
differ.
Tristan
On 03/20/2013 12:25 PM, Sanne Grinovero wrote:
> Why would you need to *store* anything which is equal for all entries?
> I'd have an interceptor "decorate" stuff with some constant additions;
> otherwise you could also think of reusing instances, like I had
> suggested a while back to change the Externalizer contract to allow it
> to be passed some context to make unique objects taking them from a
> pool.
>
> At least it looks like you would have to split such metadata: version
> for example needs to be maintained per entry, as will be vector
> clocks.
>
> Sanne
>
> On 20 March 2013 11:04, Tristan Tarrant <ttarrant(a)redhat.com> wrote:
>> On 03/20/2013 11:51 AM, Manik Surtani wrote:
>>> On 18 Mar 2013, at 12:21, Galder Zamarreño <galder(a)redhat.com> wrote:
>>>
>>>> IOW, you'll be able to say: create an Infinsipan Server that has
String as key and value of type X, where X is the actual data type, no metadata!! The
metadata (version, encoding, whatever is requried to fulfill the compatibility reqs in
ISPN-2281) will be passed as part of the put/replace…etc (I will email this around when in
place).
>>> Not sure I get this. Where will this metadata be stored?
>> Metadata should be moved to the ICE-level, or the Cache-level (for
>> metadata common to all entries in the cache, to save space: e.g. if all
>> entries in my cache are text/plain)
>>
>> Tristan
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev(a)lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
--
Galder Zamarreño
galder(a)redhat.com
twitter.com/galderz
Project Lead, Escalante
http://escalante.io
Engineer, Infinispan
http://infinispan.org
9:18 a.m.
On Mar 20, 2013, at 11:25 AM, Sanne Grinovero <sanne(a)infinispan.org> wrote:
Why would you need to *store* anything which is equal for all
entries?
^ It's not, necessarily. lifespan/maxIdle can be different for each entry. Version is
most likely different per entry.
I'd have an interceptor "decorate" stuff with some
constant additions;
otherwise you could also think of reusing instances, like I had
suggested a while back to change the Externalizer contract to allow it
to be passed some context to make unique objects taking them from a
pool.
At least it looks like you would have to split such metadata: version
for example needs to be maintained per entry, as will be vector
clocks.
^ I think Tristan confused things a little bit here. It's at the ICE level where
metadata will be stored, as is already currently.
Sanne
On 20 March 2013 11:04, Tristan Tarrant <ttarrant(a)redhat.com> wrote:
> On 03/20/2013 11:51 AM, Manik Surtani wrote:
>> On 18 Mar 2013, at 12:21, Galder Zamarreño <galder(a)redhat.com> wrote:
>>
>>> IOW, you'll be able to say: create an Infinsipan Server that has String
as key and value of type X, where X is the actual data type, no metadata!! The metadata
(version, encoding, whatever is requried to fulfill the compatibility reqs in ISPN-2281)
will be passed as part of the put/replace…etc (I will email this around when in place).
>> Not sure I get this. Where will this metadata be stored?
> Metadata should be moved to the ICE-level, or the Cache-level (for
> metadata common to all entries in the cache, to save space: e.g. if all
> entries in my cache are text/plain)
>
> Tristan
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
--
Galder Zamarreño
galder(a)redhat.com
twitter.com/galderz
Project Lead, Escalante
http://escalante.io
Engineer, Infinispan
http://infinispan.org
9:16 a.m.
On Mar 20, 2013, at 11:04 AM, Tristan Tarrant <ttarrant(a)redhat.com> wrote:
On 03/20/2013 11:51 AM, Manik Surtani wrote:
> On 18 Mar 2013, at 12:21, Galder Zamarreño <galder(a)redhat.com> wrote:
>
>> IOW, you'll be able to say: create an Infinsipan Server that has String as
key and value of type X, where X is the actual data type, no metadata!! The metadata
(version, encoding, whatever is requried to fulfill the compatibility reqs in ISPN-2281)
will be passed as part of the put/replace…etc (I will email this around when in place).
> Not sure I get this. Where will this metadata be stored?
Metadata should be moved to the ICE-level, or the Cache-level (for
metadata common to all entries in the cache, to save space: e.g. if all
entries in my cache are text/plain)
^ Indeed, that's the plan. ICE's already have metadata: maxIdle, lifespan,
version…, so as mentioned in another email, a more extensible way to add metadata at the
ICE level.
Tristan
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
--
Galder Zamarreño
galder(a)redhat.com
twitter.com/galderz
Project Lead, Escalante
http://escalante.io
Engineer, Infinispan
http://infinispan.org
6:23 a.m.
On Mar 20, 2013, at 11:51 AM, Manik Surtani <msurtani(a)redhat.com> wrote:
On 18 Mar 2013, at 12:21, Galder Zamarreño <galder(a)redhat.com> wrote:
> IOW, you'll be able to say: create an Infinsipan Server that has String as key
and value of type X, where X is the actual data type, no metadata!! The metadata (version,
encoding, whatever is requried to fulfill the compatibility reqs in ISPN-2281) will be
passed as part of the put/replace…etc (I will email this around when in place).
Not sure I get this.
Understandable.
Where will this metadata be stored?
^ Right now, lifespan/maxIdle/version are stored in InternalCacheEntry. My plan is to wrap
all this Metadata mentioned (lifespan/maxidle/version) into a a class that's stored in
InternalCacheEntry.
The way Hot Rod, and others would pass more metadata if needed, would be by
extending/implementing that Metadata class and adding extra information there.
In some stashed changes I have I already added putWithMetadata(K, V, Metadata) which
includes lifespan/maxIdle/version. This would work nicely towards avoiding all the mayhem
of overloaded methods we have which I hate with passion…
This will also work nicely towards having Hibernate provide a version to the data stored
in Infinispan
The API is not finished by any means and this is still in flux… but as soon as I have
something that I'm happy with and I've confirmed that it works nicely, I'll
post for review of course.
What I mean by nice is: I can pass standard metadata from embedded, but servers can extend
it nicely to put more metadata into the
internal container…
Cheers,
--
Manik Surtani
manik(a)jboss.org
twitter.com/maniksurtani
Platform Architect, JBoss Data Grid
http://red.ht/data-grid
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
--
Galder Zamarreño
galder(a)redhat.com
twitter.com/galderz
Project Lead, Escalante
http://escalante.io
Engineer, Infinispan
http://infinispan.org
5:52 a.m.
On 18 Mar 2013, at 12:21, Galder Zamarreño <galder(a)redhat.com> wrote:
This is why, I've created a new CHM, based on the CHMv8, called
ComparingConcurrentHashMapv8 (thx Tristan for the name!). The work for this can be seen
in:https://github.com/galderz/infinispan/commit/351e29d327d163ca8e941edf8...
Sounds good, but why not extend
org.infinispan.util.concurrent.jdk8backported.ConcurrentHashMapV8?
--
Manik Surtani
manik(a)jboss.org
twitter.com/maniksurtani
Platform Architect, JBoss Data Grid
http://red.ht/data-grid
6:17 a.m.
On Mar 20, 2013, at 11:52 AM, Manik Surtani <msurtani(a)redhat.com> wrote:
On 18 Mar 2013, at 12:21, Galder Zamarreño <galder(a)redhat.com> wrote:
> This is why, I've created a new CHM, based on the CHMv8, called
ComparingConcurrentHashMapv8 (thx Tristan for the name!). The work for this can be seen
in:https://github.com/galderz/infinispan/commit/351e29d327d163ca8e941edf8...
Sounds good, but why not extend
org.infinispan.util.concurrent.jdk8backported.ConcurrentHashMapV8?
To be honest, I'm considering keeping only one ConcurrentHashMapV8 around, which had
the Comparing functions pluggable…, and I might end up doing that. IOW, a
ComparingConcurrentHashMapV8 instance created with ComparingObject function for both keys
and values is functionality equivalent ConcurrentHashMapV8 with little/hardly impact.
I originally wanted to keep two versions so that I could more easily port over changes in
JSR-166 to org.infinispan.util.concurrent.jdk8backported.ConcurrentHashMapV8, and then
pass this on to ComparingCHMv8… but I don't think it's worth it.
Cheers,
--
Manik Surtani
manik(a)jboss.org
twitter.com/maniksurtani
Platform Architect, JBoss Data Grid
http://red.ht/data-grid
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
--
Galder Zamarreño
galder(a)redhat.com
twitter.com/galderz
Project Lead, Escalante
http://escalante.io
Engineer, Infinispan
http://infinispan.org
10:10 a.m.
On Thu, Mar 21, 2013 at 1:17 PM, Galder Zamarreño <galder(a)redhat.com> wrote:
On Mar 20, 2013, at 11:52 AM, Manik Surtani <msurtani(a)redhat.com> wrote:
>
> On 18 Mar 2013, at 12:21, Galder Zamarreño <galder(a)redhat.com> wrote:
>
>> This is why, I've created a new CHM, based on the CHMv8, called
ComparingConcurrentHashMapv8 (thx Tristan for the name!). The work for this
can be seen in:
https://github.com/galderz/infinispan/commit/351e29d327d163ca8e941edf873f...
>
> Sounds good, but why not extend
org.infinispan.util.concurrent.jdk8backported.ConcurrentHashMapV8?
To be honest, I'm considering keeping only one ConcurrentHashMapV8 around,
which had the Comparing functions pluggable…, and I might end up doing
that. IOW, a ComparingConcurrentHashMapV8 instance created with
ComparingObject function for both keys and values is functionality
equivalent ConcurrentHashMapV8 with little/hardly impact.
I originally wanted to keep two versions so that I could more easily port
over changes in JSR-166 to
org.infinispan.util.concurrent.jdk8backported.ConcurrentHashMapV8, and then
pass this on to ComparingCHMv8… but I don't think it's worth it.
Interesting, the extra166y package had a CustomConcurrentHashMap [1] that
did exactly what you want, but Doug Lea didn't port it over to
ConcurrentHashMapV8. Guava too has a CustomConcurrentHashMap class, but
it's also based on the old ConcurrentHashMap code.
I think maintaining one ConcurrentHashMapV8 class is hard enough, and I
doubt there would be any performance difference between the two versions.
So I vote to keep only the Comparing version.
[1]
http://gee.cs.oswego.edu/dl/jsr166/dist/extra166ydocs/extra166y/CustomCon...
7:26 a.m.
On Mar 21, 2013, at 3:10 PM, Dan Berindei <dan.berindei(a)gmail.com> wrote:
On Thu, Mar 21, 2013 at 1:17 PM, Galder Zamarreño <galder(a)redhat.com> wrote:
On Mar 20, 2013, at 11:52 AM, Manik Surtani <msurtani(a)redhat.com> wrote:
>
> On 18 Mar 2013, at 12:21, Galder Zamarreño <galder(a)redhat.com> wrote:
>
>> This is why, I've created a new CHM, based on the CHMv8, called
ComparingConcurrentHashMapv8 (thx Tristan for the name!). The work for this can be seen
in:https://github.com/galderz/infinispan/commit/351e29d327d163ca8e941edf8...
>
> Sounds good, but why not extend
org.infinispan.util.concurrent.jdk8backported.ConcurrentHashMapV8?
To be honest, I'm considering keeping only one ConcurrentHashMapV8 around, which had
the Comparing functions pluggable…, and I might end up doing that. IOW, a
ComparingConcurrentHashMapV8 instance created with ComparingObject function for both keys
and values is functionality equivalent ConcurrentHashMapV8 with little/hardly impact.
I originally wanted to keep two versions so that I could more easily port over changes in
JSR-166 to org.infinispan.util.concurrent.jdk8backported.ConcurrentHashMapV8, and then
pass this on to ComparingCHMv8… but I don't think it's worth it.
Interesting, the extra166y package had a CustomConcurrentHashMap [1] that did exactly
what you want, but Doug Lea didn't port it over to ConcurrentHashMapV8. Guava too has
a CustomConcurrentHashMap class, but it's also based on the old ConcurrentHashMap
code.
I think maintaining one ConcurrentHashMapV8 class is hard enough, and I doubt there would
be any performance difference between the two versions. So I vote to keep only the
Comparing version.
[1]
http://gee.cs.oswego.edu/dl/jsr166/dist/extra166ydocs/extra166y/CustomCon...
^ Well spotted!! I hadn't seen it :)
It goes a little beyond with strong/weak map selection which is not important for us in
the internal container right now (it could be if some crazy person decided to cache things
crazy stuff like classloaders in Infinispan :O…)
It's interesting what he's done with Equivalence interface (better name than
Comparing TBH), but can't see how he can make equals() typed on one of the elements… I
need to revise my implementation to see if that's possible…
Dan, excellent feedback! As always :))
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
--
Galder Zamarreño
galder(a)redhat.com
twitter.com/galderz
Project Lead, Escalante
http://escalante.io
Engineer, Infinispan
http://infinispan.org
4241
days inactive
4245
days old
infinispan-dev@lists.jboss.org
21 comments
7 participants
participants (7)
-
Dan Berindei -
Galder Zamarreño -
Manik Surtani -
Mircea Markus -
Radim Vansa -
Sanne Grinovero -
Tristan Tarrant