5:09 a.m.
FYI,
I've uploaded a 10-minute video [1] to YouTube showing how I setup and
run a perf test for Infinispan xsite replication [2]. The test
(including all configuration) is available at [3]. This was run with a
snapshot of Infinispan (roughly 5.2.0.Beta6) and JGroups 3.3.0.Alpha1.
Enjoy !
[1] http://www.youtube.com/watch?v=owOs430vLZo
[2] https://docs.jboss.org/author/display/ISPN/Cross+site+replication
[3] https://github.com/belaban/IspnPerfTest
--
Bela Ban, JGroups lead (http://www.jgroups.org)
5:47 a.m.
Thanks Bela, I enjoyed the demo!
I suggest adding some sort of visualization (maybe just a few diagrams
within the demo) of what is actually happening with the data would help
understanding for users who are just starting with xsite.
Rado
On 14/12/12 12:09, Bela Ban wrote:
FYI,
I've uploaded a 10-minute video [1] to YouTube showing how I setup and
run a perf test for Infinispan xsite replication [2]. The test
(including all configuration) is available at [3]. This was run with a
snapshot of Infinispan (roughly 5.2.0.Beta6) and JGroups 3.3.0.Alpha1.
Enjoy !
[1] http://www.youtube.com/watch?v=owOs430vLZo
[2] https://docs.jboss.org/author/display/ISPN/Cross+site+replication
[3] https://github.com/belaban/IspnPerfTest
7:01 p.m.
That was very very nice to see.
Assuming you also asked for feedback to improve this as a talk:
1# you stress several times that reads are going to be local, so very
fast. I think you meant "local to the site" ? as some ~33% of entries
will need to be fetched from peers on the same site.
2# you aren't actually running this on multiple sites are you? When
pointing out the different IP addresses you say something about
needing them to be different, but I didn't understand if you needed
them different because they are in different places, or to separate
otherwise local machines to have them appear as in different places.
3# Since get operations are always local (site), they are as you say
not meaningful for the benchmark; now since put operations are also
not meaningful as it's async .. what is the benchmark measuring?
4# There seems to be some degree of redundancy when explaining
LON/SFO/NYC setting as the local site vs the backup sites. Wouldn't it
make more sense to be able to configure all backup sites the same and
have it automatically ignore the "self" element as a backup site? So
your script would only need to specify what the local site is. If that
makes any sense it would even be nice to extend this to the IP
addresses being defined in the zones area, so that they are applied
both to the JGroups configuration for the local cluster and to the
bridge configuration.
5# I was initially surprised to see x-site configuration as part of a
cache configuration; I understand the reasons for options like
"strategy" which one might want to specify differently on each cache,
but what about "take offline" ? that sounds more something which
should be globally managed at the channel level - not sure if in
JGroups directly but if it's to be handled in Infinispan I would
expect to have all caches use the same policy, consistent with FD.
Also it doesn't looks like you have much of a choice in to which sites
you want to replicate, as relay is setup at the jgroups level so
affecting all caches: is relay going to be ignored by caches having no
x-site enabled? And is it going to be relayed only to one site if the
Infinispan configuration lists a single site?
Not sure if this makes any sense, I just found it contrasting with my
naive expectations of how such a configuration would look like.
thanks a lot, I hope this is proof enough that your video was pretty catchy :)
Cheers,
Sanne
On 14 December 2012 11:47, Radoslav Husar <rhusar(a)redhat.com> wrote:
Thanks Bela, I enjoyed the demo!
I suggest adding some sort of visualization (maybe just a few diagrams
within the demo) of what is actually happening with the data would help
understanding for users who are just starting with xsite.
Rado
On 14/12/12 12:09, Bela Ban wrote:
> FYI,
>
> I've uploaded a 10-minute video [1] to YouTube showing how I setup and
> run a perf test for Infinispan xsite replication [2]. The test
> (including all configuration) is available at [3]. This was run with a
> snapshot of Infinispan (roughly 5.2.0.Beta6) and JGroups 3.3.0.Alpha1.
> Enjoy !
>
> [1] http://www.youtube.com/watch?v=owOs430vLZo
> [2] https://docs.jboss.org/author/display/ISPN/Cross+site+replication
> [3] https://github.com/belaban/IspnPerfTest
>
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
9:41 p.m.
Great stuff Bela!
Btw, is xsite replication bidirectional? LON has SFO as backup, and SFO
also has LON as backup.
On Fri, Dec 14, 2012 at 8:01 PM, Sanne Grinovero <sanne(a)infinispan.org>wrote:
That was very very nice to see.
Assuming you also asked for feedback to improve this as a talk:
1# you stress several times that reads are going to be local, so very
fast. I think you meant "local to the site" ? as some ~33% of entries
will need to be fetched from peers on the same site.
2# you aren't actually running this on multiple sites are you? When
pointing out the different IP addresses you say something about
needing them to be different, but I didn't understand if you needed
them different because they are in different places, or to separate
otherwise local machines to have them appear as in different places.
3# Since get operations are always local (site), they are as you say
not meaningful for the benchmark; now since put operations are also
not meaningful as it's async .. what is the benchmark measuring?
4# There seems to be some degree of redundancy when explaining
LON/SFO/NYC setting as the local site vs the backup sites. Wouldn't it
make more sense to be able to configure all backup sites the same and
have it automatically ignore the "self" element as a backup site? So
your script would only need to specify what the local site is. If that
makes any sense it would even be nice to extend this to the IP
addresses being defined in the zones area, so that they are applied
both to the JGroups configuration for the local cluster and to the
bridge configuration.
5# I was initially surprised to see x-site configuration as part of a
cache configuration; I understand the reasons for options like
"strategy" which one might want to specify differently on each cache,
but what about "take offline" ? that sounds more something which
should be globally managed at the channel level - not sure if in
JGroups directly but if it's to be handled in Infinispan I would
expect to have all caches use the same policy, consistent with FD.
Also it doesn't looks like you have much of a choice in to which sites
you want to replicate, as relay is setup at the jgroups level so
affecting all caches: is relay going to be ignored by caches having no
x-site enabled? And is it going to be relayed only to one site if the
Infinispan configuration lists a single site?
Not sure if this makes any sense, I just found it contrasting with my
naive expectations of how such a configuration would look like.
thanks a lot, I hope this is proof enough that your video was pretty
catchy :)
Cheers,
Sanne
On 14 December 2012 11:47, Radoslav Husar <rhusar(a)redhat.com> wrote:
> Thanks Bela, I enjoyed the demo!
>
> I suggest adding some sort of visualization (maybe just a few diagrams
> within the demo) of what is actually happening with the data would help
> understanding for users who are just starting with xsite.
>
> Rado
>
> On 14/12/12 12:09, Bela Ban wrote:
>> FYI,
>>
>> I've uploaded a 10-minute video [1] to YouTube showing how I setup and
>> run a perf test for Infinispan xsite replication [2]. The test
>> (including all configuration) is available at [3]. This was run with a
>> snapshot of Infinispan (roughly 5.2.0.Beta6) and JGroups 3.3.0.Alpha1.
>> Enjoy !
>>
>> [1] http://www.youtube.com/watch?v=owOs430vLZo
>> [2] https://docs.jboss.org/author/display/ISPN/Cross+site+replication
>> [3] https://github.com/belaban/IspnPerfTest
>>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
3:16 a.m.
Thanks Ray,
yes, each site has the other 2as backup sites. The current assumption
though is for xsite replication to only be unidirectional, e.g. LON -->
SFO/NYC, or if it is bidirectional, for the data that's concurrently
accessed to be disjoint.
This setup allows LON to be the active site and SFO/NYC to be the backup
sites. One could imagine a follow-the-sun implementation where LON
becomes inactive and NYC becomes active when it is EOB in LON. Later,
SFO becomes active, then LON again and so on.
In IRAC [1], we'll handle the problem of concurrent updates from
different sites to the same data set.
[1] https://community.jboss.org/wiki/RAC
On 12/15/12 4:41 AM, Ray Tsang wrote:
Great stuff Bela!
Btw, is xsite replication bidirectional? LON has SFO as backup, and
SFO also has LON as backup.
On Fri, Dec 14, 2012 at 8:01 PM, Sanne Grinovero <sanne(a)infinispan.org
<mailto:sanne@infinispan.org>> wrote:
That was very very nice to see.
Assuming you also asked for feedback to improve this as a talk:
1# you stress several times that reads are going to be local, so very
fast. I think you meant "local to the site" ? as some ~33% of entries
will need to be fetched from peers on the same site.
2# you aren't actually running this on multiple sites are you? When
pointing out the different IP addresses you say something about
needing them to be different, but I didn't understand if you needed
them different because they are in different places, or to separate
otherwise local machines to have them appear as in different places.
3# Since get operations are always local (site), they are as you say
not meaningful for the benchmark; now since put operations are also
not meaningful as it's async .. what is the benchmark measuring?
4# There seems to be some degree of redundancy when explaining
LON/SFO/NYC setting as the local site vs the backup sites. Wouldn't it
make more sense to be able to configure all backup sites the same and
have it automatically ignore the "self" element as a backup site? So
your script would only need to specify what the local site is. If that
makes any sense it would even be nice to extend this to the IP
addresses being defined in the zones area, so that they are applied
both to the JGroups configuration for the local cluster and to the
bridge configuration.
5# I was initially surprised to see x-site configuration as part of a
cache configuration; I understand the reasons for options like
"strategy" which one might want to specify differently on each cache,
but what about "take offline" ? that sounds more something which
should be globally managed at the channel level - not sure if in
JGroups directly but if it's to be handled in Infinispan I would
expect to have all caches use the same policy, consistent with FD.
Also it doesn't looks like you have much of a choice in to which sites
you want to replicate, as relay is setup at the jgroups level so
affecting all caches: is relay going to be ignored by caches having no
x-site enabled? And is it going to be relayed only to one site if the
Infinispan configuration lists a single site?
Not sure if this makes any sense, I just found it contrasting with my
naive expectations of how such a configuration would look like.
thanks a lot, I hope this is proof enough that your video was
pretty catchy :)
Cheers,
Sanne
On 14 December 2012 11:47, Radoslav Husar <rhusar(a)redhat.com
<mailto:rhusar@redhat.com>> wrote:
> Thanks Bela, I enjoyed the demo!
>
> I suggest adding some sort of visualization (maybe just a few
diagrams
> within the demo) of what is actually happening with the data
would help
> understanding for users who are just starting with xsite.
>
> Rado
>
> On 14/12/12 12:09, Bela Ban wrote:
>> FYI,
>>
>> I've uploaded a 10-minute video [1] to YouTube showing how I
setup and
>> run a perf test for Infinispan xsite replication [2]. The test
>> (including all configuration) is available at [3]. This was run
with a
>> snapshot of Infinispan (roughly 5.2.0.Beta6) and JGroups
3.3.0.Alpha1.
>> Enjoy !
>>
>> [1] http://www.youtube.com/watch?v=owOs430vLZo
>> [2]
https://docs.jboss.org/author/display/ISPN/Cross+site+replication
>> [3] https://github.com/belaban/IspnPerfTest
>>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev(a)lists.jboss.org
<mailto:infinispan-dev@lists.jboss.org>
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org <mailto:infinispan-dev@lists.jboss.org>
https://lists.jboss.org/mailman/listinfo/infinispan-dev
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
--
Bela Ban, JGroups lead (http://www.jgroups.org)
3:54 a.m.
Hi Sanne,
thanks ! My first YouTube video, so next time I might venture into
editing the video... :-)
The video was initially meant to show the perf when adding 2 backup
sites, but then (because I had to include config info) 'degenerated'
into a tutorial on xsite. I know, bad job on that, but we should come up
with a tutorial that does this in a much more comprehensive way...
Comments inline
On 12/15/12 2:01 AM, Sanne Grinovero wrote:
That was very very nice to see.
Assuming you also asked for feedback to improve this as a talk:
1# you stress several times that reads are going to be local, so very
fast. I think you meant "local to the site" ? as some ~33% of entries
will need to be fetched from peers on the same site.
Yes, 'local' always refers to the local site (cluster). The scenario
we're looking at is dist-sync within the site and async xsite repl
between sites. I should also have mentioned that latency within a site
is very small (e.g. 0.05ms) whereas we might have up to 60 *ms* between
sites.
2# you aren't actually running this on multiple sites are you?
Correct. This was all within our Boston lab, every node was running on a
different box though. The ultimate goal is to inject latency into the
system, e.g. using the DELAY protocol in the global cluster.
But as a first step, I wanted to get the base performance for xsite repl.
When pointing out the different IP addresses you say something
about
needing them to be different, but I didn't understand if you needed
them different because they are in different places, or to separate
otherwise local machines to have them appear as in different places.
The reason to separate mcast_addr/mcast_port was to simulate 3 sites on
the same box. Had I not used different addresses/ports for the 3 sites,
all nodes of the 3 sites would have found each other and formed a
cluster of 9.
3# Since get operations are always local (site), they are as you say
not meaningful for the benchmark; now since put operations are also
not meaningful as it's async .. what is the benchmark measuring?
Well, in the meantime I modified the test and now we do support reads;
you can define a read/write ratio.
This scenario mimics how a large prospective xsite customer will use
xsite repl: dist-sync for intra-site and xsite async repl for inter-site
communication. One thing we ran into was that there was a 20% perf
degradation if we added async xsite repl *per site* even if that site
was down ! The root cause was that async xsite repl does not mark a site
as offline in Infinispan even if it is down in JGroups. This will get
fixed in Infinispan and should increase async xsite repl for down sites,
see [1] for details.
Note that the test can also measure sync xsite replication between
sites; this is just a matter of configuring the cache differently. But
as the scenario this will be used initially is async xseit repl, that's
what we're focusing on for now.
4# There seems to be some degree of redundancy when explaining
LON/SFO/NYC setting as the local site vs the backup sites. Wouldn't it
make more sense to be able to configure all backup sites the same and
have it automatically ignore the "self" element as a backup site? So
your script would only need to specify what the local site is. If that
makes any sense it would even be nice to extend this to the IP
addresses being defined in the zones area, so that they are applied
both to the JGroups configuration for the local cluster and to the
bridge configuration.
Regarding the mcast_addr/mcast_port settings: yes, I could have used
only 1 config file (local.xml) and set these properties as variables.
I've already changed and committed this.
Regarding the setup of the self and backup sites: yes, this could have
been done. Again, this is just a matter of setup and lazyness on my
behalf :-)
5# I was initially surprised to see x-site configuration as part of a
cache configuration; I understand the reasons for options like
"strategy" which one might want to specify differently on each cache,
but what about "take offline" ?
Take offline/online is currently available via a JMX operations. Taking
a site offline is also done automatically, but currently only when xsite
repl is *sync*. There's a JIRA that'll fix this for async xsite repl.
that sounds more something which
should be globally managed at the channel level - not sure if in
JGroups directly but if it's to be handled in Infinispan I would
expect to have all caches use the same policy, consistent with FD.
Actually this doesn't use JGroups failure detection as we can't use it
across sites. This is where the <takeOffline ...> element comes in (as I
mentioned, currently only for async)
Also it doesn't looks like you have much of a choice in to which
sites
you want to replicate, as relay is setup at the jgroups level so
affecting all caches: is relay going to be ignored by caches having no
x-site enabled?
Actually, you can define the backupSites, so LON could choose *not* to
replicate to a backup site at all, and NYC could pick only SFO as backup
site.
Yes, RELAY2 can be ignored on a per-message basis: we have a NO_RELAY
flag that AFAIR Mircea uses to exclude certain messages from getting
relayed.
Note that in the demo I defined xsite repl in the defaults section and
clusteredCache simply used it. I can define an empty <sites/> inside
clusteredCache if I don't want xsite repl for that particular cache. Or
it could be done the other way round: don't define a default xsite
config, but define it per cache that wants xsite repl.
And is it going to be relayed only to one site if the
Infinispan configuration lists a single site?
Yes
Not sure if this makes any sense, I just found it contrasting with
my
naive expectations of how such a configuration would look like.
thanks a lot, I hope this is proof enough that your video was pretty catchy :)
Thanks for the feedback !
[1] https://issues.jboss.org/browse/JGRP-1543
--
Bela Ban, JGroups lead (http://www.jgroups.org)
6:37 a.m.
On 15 Dec 2012, at 09:54, Bela Ban <bban(a)redhat.com> wrote:
>
> When pointing out the different IP addresses you say something about
> needing them to be different, but I didn't understand if you needed
> them different because they are in different places, or to separate
> otherwise local machines to have them appear as in different places.
The reason to separate mcast_addr/mcast_port was to simulate 3 sites on
the same box. Had I not used different addresses/ports for the 3 sites,
all nodes of the 3 sites would have found each other and formed a
cluster of 9.
Maybe for more impact next time, use AWS and set up servers in different availability
zones? Or better still, one cluster in our Boston lab, one on AWS, and one on Rackspace.
This would definitely have impact, as you can start a demo by showing the nodes in the AWS
and Rackspace consoles. :)
--
Manik Surtani
manik(a)jboss.org
twitter.com/maniksurtani
Platform Architect, JBoss Data Grid
http://red.ht/data-grid
6:46 a.m.
I don't do AWS anymore, killed my account last year
On 12/19/12 1:37 PM, Manik Surtani wrote:
On 15 Dec 2012, at 09:54, Bela Ban <bban(a)redhat.com> wrote:
>> When pointing out the different IP addresses you say something about
>> needing them to be different, but I didn't understand if you needed
>> them different because they are in different places, or to separate
>> otherwise local machines to have them appear as in different places.
> The reason to separate mcast_addr/mcast_port was to simulate 3 sites on
> the same box. Had I not used different addresses/ports for the 3 sites,
> all nodes of the 3 sites would have found each other and formed a
> cluster of 9.
Maybe for more impact next time, use AWS and set up servers in different availability
zones? Or better still, one cluster in our Boston lab, one on AWS, and one on Rackspace.
This would definitely have impact, as you can start a demo by showing the nodes in the AWS
and Rackspace consoles. :)
--
Manik Surtani
manik(a)jboss.org
twitter.com/maniksurtani
Platform Architect, JBoss Data Grid
http://red.ht/data-grid
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
--
Bela Ban, JGroups lead (http://www.jgroups.org)
2:22 a.m.
|
| 3# Since get operations are always local (site), they are as you say
| not meaningful for the benchmark; now since put operations are also
| not meaningful as it's async .. what is the benchmark measuring?
|
That's an assumption and the test is there also to confirm it. For example in my XS
perf test with async XS replication the reads are slowed down for some reasons as well
(investigating).
And writes have definitely some overhead for XS too, at least for the marshalling etc.
Radim
4174
days inactive
4179
days old
infinispan-dev@lists.jboss.org
8 comments
6 participants
participants (6)
-
Bela Ban -
Manik Surtani -
Radim Vansa -
Radoslav Husar -
Ray Tsang -
Sanne Grinovero