7:05 a.m.
http://infinispan.blogspot.co.uk/2013/09/new-persistence-api-in-infinispa...
Cheers,
--
Mircea Markus
Infinispan lead (www.infinispan.org)
12:15 p.m.
Thanks for the write-up! Good to have a recap.
I'm very happy with these improvements; I've been tinkering around the
new API - just playing so far - but it definitely is very powerful:
all SPI restrictions are taken away while still not exposing too much
complexity; there is a lot I can improve in my implementations now and
also it looks like it will be quite easy.
Cheers,
Sanne
On 16 September 2013 14:05, Mircea Markus <mmarkus(a)redhat.com> wrote:
http://infinispan.blogspot.co.uk/2013/09/new-persistence-api-in-infinispa...
Cheers,
--
Mircea Markus
Infinispan lead (www.infinispan.org)
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
3:29 p.m.
Thanks for the feedback Sanne!
And thanks for the API as well - after all the parallel iteration - the most feature I
think - was your idea ;)
On Sep 16, 2013, at 6:15 PM, Sanne Grinovero <sanne(a)infinispan.org> wrote:
Thanks for the write-up! Good to have a recap.
I'm very happy with these improvements; I've been tinkering around the
new API - just playing so far - but it definitely is very powerful:
all SPI restrictions are taken away while still not exposing too much
complexity; there is a lot I can improve in my implementations now and
also it looks like it will be quite easy.
Cheers,
--
Mircea Markus
Infinispan lead (www.infinispan.org)
1:11 p.m.
"parallel iteration: it is now possible to iterate over entries in the store with
multiple threads in parallel. Map/Reduce tasks immediately benefit from this, as the
map/reduce tasks now run in parallel over both the nodes in the cluster and within the
same node (multiple threads)"
Does this apply to entries in the cache as well (ISPN-2284)?
Shane
----- Original Message -----
From: "Mircea Markus" <mmarkus(a)redhat.com>
To: "infinispan -Dev List" <infinispan-dev(a)lists.jboss.org>
Sent: Monday, September 16, 2013 7:05:11 AM
Subject: [infinispan-dev] blog on new cache store API
http://infinispan.blogspot.co.uk/2013/09/new-persistence-api-in-infinispa...
Cheers,
--
Mircea Markus
Infinispan lead (www.infinispan.org)
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
3:31 p.m.
On Sep 16, 2013, at 7:11 PM, Shane Johnson <shjohnso(a)redhat.com> wrote:
"parallel iteration: it is now possible to iterate over entries
in the store with multiple threads in parallel. Map/Reduce tasks immediately benefit from
this, as the map/reduce tasks now run in parallel over both the nodes in the cluster and
within the same node (multiple threads)"
Does this apply to entries in the cache as well (ISPN-2284)?
no, only the entries in the store are iterated in parallel.
Cheers,
--
Mircea Markus
Infinispan lead (www.infinispan.org)
3:59 p.m.
But there are now multiple map threads per node? Or, is there one map thread and multiple
cache store threads? I'm not sure how a single map thread could benefit from multiple
cache store threads.
Shane
----- Original Message -----
From: "Mircea Markus" <mmarkus(a)redhat.com>
To: "infinispan -Dev List" <infinispan-dev(a)lists.jboss.org>
Sent: Monday, September 16, 2013 3:31:01 PM
Subject: Re: [infinispan-dev] blog on new cache store API
On Sep 16, 2013, at 7:11 PM, Shane Johnson <shjohnso(a)redhat.com> wrote:
"parallel iteration: it is now possible to iterate over entries
in the store with multiple threads in parallel. Map/Reduce tasks immediately benefit from
this, as the map/reduce tasks now run in parallel over both the nodes in the cluster and
within the same node (multiple threads)"
Does this apply to entries in the cache as well (ISPN-2284)?
no, only the entries in the store are iterated in parallel.
Cheers,
--
Mircea Markus
Infinispan lead (www.infinispan.org)
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
10:32 a.m.
Shane,
When MapReduce command arrives on the Infinispan node it is execute on a
single thread that carries the incoming message. I have done preliminary
work on multithreaded execution [1] but I have not get around to
complete it. The main idea is that incoming thread submits a task to
executor that in turns splits map/reduce work on multiple threads and
executes work. Once work is completed incoming thread is given result to
return response back.
I am not sure how Mircea implemented parallel iteration in stores but it
is definitely a different beast. Although I agree with him that parallel
reading from stores definitely helps. The above thread we mentioned will
wait for reading from stores much less.
Hope it all makes more sense now!
Regards,
Vladimir
[1] https://github.com/vblagoje/infinispan/tree/t_2284
On 13-09-16 4:59 PM, Shane Johnson wrote:
But there are now multiple map threads per node? Or, is there one map
thread and multiple cache store threads? I'm not sure how a single map thread could
benefit from multiple cache store threads.
Shane
----- Original Message -----
From: "Mircea Markus" <mmarkus(a)redhat.com>
To: "infinispan -Dev List" <infinispan-dev(a)lists.jboss.org>
Sent: Monday, September 16, 2013 3:31:01 PM
Subject: Re: [infinispan-dev] blog on new cache store API
On Sep 16, 2013, at 7:11 PM, Shane Johnson <shjohnso(a)redhat.com> wrote:
> "parallel iteration: it is now possible to iterate over entries in the store
with multiple threads in parallel. Map/Reduce tasks immediately benefit from this, as the
map/reduce tasks now run in parallel over both the nodes in the cluster and within the
same node (multiple threads)"
>
> Does this apply to entries in the cache as well (ISPN-2284)?
no, only the entries in the store are iterated in parallel.
Cheers,
10:50 a.m.
Right. I'm familiar with the map/reduce process and the proposed improvements.
This part of the blog threw me off:
"as the map/reduce tasks now run in parallel over both the nodes in the cluster and
within the same node (multiple threads)"
To me, it implies that there are now multiple map threads per node. Further, I thought
that the map / reduce 'working set' was limited to what was in memory. I did not
realize that map / reduce would iterate over all of the data both in memory and on disk.
That is good to hear, though I'm curious if it will apply to all cache stores (e.g.
LevelDB) and how ISPN map / reduce handles a data set that is greater than the available
memory. A lot in-memory stores face this limitation when backed by on-disk stores. If the
data is retrieved one entry at a time, I don't see how multiple threads will help.
However, if it is retrieved in bulk I can see how it might. Not entirely sure.
Shane
----- Original Message -----
From: "Vladimir Blagojevic" <vblagoje(a)redhat.com>
To: "infinispan -Dev List" <infinispan-dev(a)lists.jboss.org>
Cc: "Shane Johnson" <shjohnso(a)redhat.com>
Sent: Tuesday, September 17, 2013 10:32:39 AM
Subject: Re: [infinispan-dev] blog on new cache store API
Shane,
When MapReduce command arrives on the Infinispan node it is execute on a
single thread that carries the incoming message. I have done preliminary
work on multithreaded execution [1] but I have not get around to
complete it. The main idea is that incoming thread submits a task to
executor that in turns splits map/reduce work on multiple threads and
executes work. Once work is completed incoming thread is given result to
return response back.
I am not sure how Mircea implemented parallel iteration in stores but it
is definitely a different beast. Although I agree with him that parallel
reading from stores definitely helps. The above thread we mentioned will
wait for reading from stores much less.
Hope it all makes more sense now!
Regards,
Vladimir
[1] https://github.com/vblagoje/infinispan/tree/t_2284
On 13-09-16 4:59 PM, Shane Johnson wrote:
But there are now multiple map threads per node? Or, is there one map
thread and multiple cache store threads? I'm not sure how a single map thread could
benefit from multiple cache store threads.
Shane
----- Original Message -----
From: "Mircea Markus" <mmarkus(a)redhat.com>
To: "infinispan -Dev List" <infinispan-dev(a)lists.jboss.org>
Sent: Monday, September 16, 2013 3:31:01 PM
Subject: Re: [infinispan-dev] blog on new cache store API
On Sep 16, 2013, at 7:11 PM, Shane Johnson <shjohnso(a)redhat.com> wrote:
> "parallel iteration: it is now possible to iterate over entries in the store
with multiple threads in parallel. Map/Reduce tasks immediately benefit from this, as the
map/reduce tasks now run in parallel over both the nodes in the cluster and within the
same node (multiple threads)"
>
> Does this apply to entries in the cache as well (ISPN-2284)?
no, only the entries in the store are iterated in parallel.
Cheers,
10:59 a.m.
On 13-09-17 11:50 AM, Shane Johnson wrote:
Right. I'm familiar with the map/reduce process and the proposed
improvements.
This part of the blog threw me off:
"as the map/reduce tasks now run in parallel over both the nodes in the cluster and
within the same node (multiple threads)"
To me, it implies that there are now multiple map threads per node. Further, I thought
that the map / reduce 'working set' was limited to what was in memory. I did not
realize that map / reduce would iterate over all of the data both in memory and on disk.
That is good to hear, though I'm curious if it will apply to all cache stores (e.g.
LevelDB) and how ISPN map / reduce handles a data set that is greater than the available
memory. A lot in-memory stores face this limitation when backed by on-disk stores. If the
data is retrieved one entry at a time, I don't see how multiple threads will help.
However, if it is retrieved in bulk I can see how it might. Not entirely sure.
The implementation in MapReduceManagerImpl.java is cache store agnostic.
Algorithm loads all keys (pinned to that owner node) and iterates over
all values one value at at time.
Now that we are breaking this down into details I am not sure how
multiple threads in cache store would help either. Mircea?
Regards,
Vladimir
11:07 a.m.
Right.
Regarding the cache store, while I'm not fully aware of its implementation, one of the
primary reasons to use LevelDB is that the entire key set no longer has to be in memory.
That's why I asked. Because if it was designed like that, and I have no idea if it is,
m/r would not have the entire key set beforehand.
Shane
----- Original Message -----
From: "Vladimir Blagojevic" <vblagoje(a)redhat.com>
To: "Shane Johnson" <shjohnso(a)redhat.com>
Cc: "infinispan -Dev List" <infinispan-dev(a)lists.jboss.org>
Sent: Tuesday, September 17, 2013 10:59:18 AM
Subject: Re: [infinispan-dev] blog on new cache store API
On 13-09-17 11:50 AM, Shane Johnson wrote:
Right. I'm familiar with the map/reduce process and the proposed
improvements.
This part of the blog threw me off:
"as the map/reduce tasks now run in parallel over both the nodes in the cluster and
within the same node (multiple threads)"
To me, it implies that there are now multiple map threads per node. Further, I thought
that the map / reduce 'working set' was limited to what was in memory. I did not
realize that map / reduce would iterate over all of the data both in memory and on disk.
That is good to hear, though I'm curious if it will apply to all cache stores (e.g.
LevelDB) and how ISPN map / reduce handles a data set that is greater than the available
memory. A lot in-memory stores face this limitation when backed by on-disk stores. If the
data is retrieved one entry at a time, I don't see how multiple threads will help.
However, if it is retrieved in bulk I can see how it might. Not entirely sure.
The implementation in MapReduceManagerImpl.java is cache store agnostic.
Algorithm loads all keys (pinned to that owner node) and iterates over
all values one value at at time.
Now that we are breaking this down into details I am not sure how
multiple threads in cache store would help either. Mircea?
Regards,
Vladimir
4639
days inactive
4640
days old
infinispan-dev@lists.jboss.org
9 comments
4 participants
participants (4)
-
Mircea Markus -
Sanne Grinovero -
Shane Johnson -
Vladimir Blagojevic