11:33 a.m.
Guys,
I have a topic branch with a fix for ISPN-825, to stagger remote GET calls. (See the JIRA
for details on this patch).
This should have an interesting effect on greatly reducing the pressure on the OOB thread
pool. This isn't a *real* fix for the problem that Radim reported (Pedro is working
on that with Bela), but reducing pressure on the OOB thread pool is a side effect of this
fix.
It should generally make things faster too, with less traffic on the network. I'd be
curious for you to give this branch a try, Radim - see how it impacts your tests.
https://github.com/maniksurtani/infinispan/tree/t_825
Cheers
Manik
--
Manik Surtani
manik(a)jboss.org
twitter.com/maniksurtani
Platform Architect, JBoss Data Grid
http://red.ht/data-grid
8:15 a.m.
Hi Manik,
so I have tried to compile this branch and issued a 20 minute stress test (preceded by 10
minute warmup) on 128 nodes, where each node has 10 stressor threads.
While in 5.2.0.CR3 the maximum OOB threadpool size was 553 with this configuration, with
t_825 it was 219. This looks good, but it's actually better :). When I looked on the
per-node maximum, in t_825 there was only one node with the 219 threads (as the max),
others were usually around 25, few around 40. On the contrary, in 5.2.0.CR3 all the nodes
had maximum around 500!
Glad to bring good news :)
Radim
----- Original Message -----
| From: "Manik Surtani" <msurtani(a)redhat.com>
| To: "infinispan -Dev List" <infinispan-dev(a)lists.jboss.org>, "Radim
Vansa" <rvansa(a)redhat.com>
| Sent: Tuesday, February 19, 2013 6:33:04 PM
| Subject: Staggering remote GET calls
|
| Guys,
|
| I have a topic branch with a fix for ISPN-825, to stagger remote GET
| calls. (See the JIRA for details on this patch).
|
| This should have an interesting effect on greatly reducing the
| pressure on the OOB thread pool. This isn't a *real* fix for the
| problem that Radim reported (Pedro is working on that with Bela),
| but reducing pressure on the OOB thread pool is a side effect of
| this fix.
|
| It should generally make things faster too, with less traffic on the
| network. I'd be curious for you to give this branch a try, Radim -
| see how it impacts your tests.
|
| https://github.com/maniksurtani/infinispan/tree/t_825
|
| Cheers
| Manik
| --
| Manik Surtani
| manik(a)jboss.org
| twitter.com/maniksurtani
|
| Platform Architect, JBoss Data Grid
| http://red.ht/data-grid
|
|
8:24 a.m.
Radim, just to be sure, you are testing embedded mode with RadarGun, right?
With HotRod most of the get operations should be initiated from the main
owner, so Manik's changes shouldn't make a big difference in the number of
active threads.
How about throughput, has it also improved compared to 5.2.0.CR3, or is it
the same?
On Wed, Feb 20, 2013 at 2:15 PM, Radim Vansa <rvansa(a)redhat.com> wrote:
Hi Manik,
so I have tried to compile this branch and issued a 20 minute stress test
(preceded by 10 minute warmup) on 128 nodes, where each node has 10
stressor threads.
While in 5.2.0.CR3 the maximum OOB threadpool size was 553 with this
configuration, with t_825 it was 219. This looks good, but it's actually
better :). When I looked on the per-node maximum, in t_825 there was only
one node with the 219 threads (as the max), others were usually around 25,
few around 40. On the contrary, in 5.2.0.CR3 all the nodes had maximum
around 500!
Glad to bring good news :)
Radim
----- Original Message -----
| From: "Manik Surtani" <msurtani(a)redhat.com>
| To: "infinispan -Dev List" <infinispan-dev(a)lists.jboss.org>,
"Radim
Vansa" <rvansa(a)redhat.com>
| Sent: Tuesday, February 19, 2013 6:33:04 PM
| Subject: Staggering remote GET calls
|
| Guys,
|
| I have a topic branch with a fix for ISPN-825, to stagger remote GET
| calls. (See the JIRA for details on this patch).
|
| This should have an interesting effect on greatly reducing the
| pressure on the OOB thread pool. This isn't a *real* fix for the
| problem that Radim reported (Pedro is working on that with Bela),
| but reducing pressure on the OOB thread pool is a side effect of
| this fix.
|
| It should generally make things faster too, with less traffic on the
| network. I'd be curious for you to give this branch a try, Radim -
| see how it impacts your tests.
|
| https://github.com/maniksurtani/infinispan/tree/t_825
|
| Cheers
| Manik
| --
| Manik Surtani
| manik(a)jboss.org
| twitter.com/maniksurtani
|
| Platform Architect, JBoss Data Grid
| http://red.ht/data-grid
|
|
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
8:38 a.m.
Ouch, call me dumbass... I haven't checked the test results. Something revoked my
cluster allocation and the test was prematurely stopped.
I'll rerun it (and check!), and show performance numbers as well.
Radim (the dumbass)
----- Original Message -----
| From: "Dan Berindei" <dan.berindei(a)gmail.com>
| To: "infinispan -Dev List" <infinispan-dev(a)lists.jboss.org>
| Cc: "Manik Surtani" <msurtani(a)redhat.com>
| Sent: Wednesday, February 20, 2013 3:24:46 PM
| Subject: Re: [infinispan-dev] Staggering remote GET calls
|
|
| Radim, just to be sure, you are testing embedded mode with RadarGun,
| right? With HotRod most of the get operations should be initiated
| from the main owner, so Manik's changes shouldn't make a big
| difference in the number of active threads.
|
| How about throughput, has it also improved compared to 5.2.0.CR3, or
| is it the same?
|
|
|
|
|
| On Wed, Feb 20, 2013 at 2:15 PM, Radim Vansa < rvansa(a)redhat.com >
| wrote:
|
|
| Hi Manik,
|
| so I have tried to compile this branch and issued a 20 minute stress
| test (preceded by 10 minute warmup) on 128 nodes, where each node
| has 10 stressor threads.
| While in 5.2.0.CR3 the maximum OOB threadpool size was 553 with this
| configuration, with t_825 it was 219. This looks good, but it's
| actually better :). When I looked on the per-node maximum, in t_825
| there was only one node with the 219 threads (as the max), others
| were usually around 25, few around 40. On the contrary, in 5.2.0.CR3
| all the nodes had maximum around 500!
|
| Glad to bring good news :)
|
| Radim
|
|
|
| ----- Original Message -----
| | From: "Manik Surtani" < msurtani(a)redhat.com >
| | To: "infinispan -Dev List" < infinispan-dev(a)lists.jboss.org >,
| | "Radim Vansa" < rvansa(a)redhat.com >
| | Sent: Tuesday, February 19, 2013 6:33:04 PM
| | Subject: Staggering remote GET calls
| |
| | Guys,
| |
| | I have a topic branch with a fix for ISPN-825, to stagger remote
| | GET
| | calls. (See the JIRA for details on this patch).
| |
| | This should have an interesting effect on greatly reducing the
| | pressure on the OOB thread pool. This isn't a *real* fix for the
| | problem that Radim reported (Pedro is working on that with Bela),
| | but reducing pressure on the OOB thread pool is a side effect of
| | this fix.
| |
| | It should generally make things faster too, with less traffic on
| | the
| | network. I'd be curious for you to give this branch a try, Radim -
| | see how it impacts your tests.
| |
| | https://github.com/maniksurtani/infinispan/tree/t_825
| |
| | Cheers
| | Manik
| | --
| | Manik Surtani
| | manik(a)jboss.org
| | twitter.com/maniksurtani
| |
| | Platform Architect, JBoss Data Grid
| | http://red.ht/data-grid
| |
| |
| _______________________________________________
| infinispan-dev mailing list
| infinispan-dev(a)lists.jboss.org
| https://lists.jboss.org/mailman/listinfo/infinispan-dev
|
|
| _______________________________________________
| infinispan-dev mailing list
| infinispan-dev(a)lists.jboss.org
| https://lists.jboss.org/mailman/listinfo/infinispan-dev
8:46 a.m.
So you are testing embedded mode with RadarGun?
Or remotely over hotrod?
On 20 Feb 2013, at 14:38, Radim Vansa wrote:
Ouch, call me dumbass... I haven't checked the test results.
Something revoked my cluster allocation and the test was prematurely stopped.
I'll rerun it (and check!), and show performance numbers as well.
Radim (the dumbass)
----- Original Message -----
| From: "Dan Berindei" <dan.berindei(a)gmail.com>
| To: "infinispan -Dev List" <infinispan-dev(a)lists.jboss.org>
| Cc: "Manik Surtani" <msurtani(a)redhat.com>
| Sent: Wednesday, February 20, 2013 3:24:46 PM
| Subject: Re: [infinispan-dev] Staggering remote GET calls
|
|
| Radim, just to be sure, you are testing embedded mode with RadarGun,
| right? With HotRod most of the get operations should be initiated
| from the main owner, so Manik's changes shouldn't make a big
| difference in the number of active threads.
|
| How about throughput, has it also improved compared to 5.2.0.CR3, or
| is it the same?
|
|
|
|
|
| On Wed, Feb 20, 2013 at 2:15 PM, Radim Vansa < rvansa(a)redhat.com >
| wrote:
|
|
| Hi Manik,
|
| so I have tried to compile this branch and issued a 20 minute stress
| test (preceded by 10 minute warmup) on 128 nodes, where each node
| has 10 stressor threads.
| While in 5.2.0.CR3 the maximum OOB threadpool size was 553 with this
| configuration, with t_825 it was 219. This looks good, but it's
| actually better :). When I looked on the per-node maximum, in t_825
| there was only one node with the 219 threads (as the max), others
| were usually around 25, few around 40. On the contrary, in 5.2.0.CR3
| all the nodes had maximum around 500!
|
| Glad to bring good news :)
|
| Radim
|
|
|
| ----- Original Message -----
| | From: "Manik Surtani" < msurtani(a)redhat.com >
| | To: "infinispan -Dev List" < infinispan-dev(a)lists.jboss.org >,
| | "Radim Vansa" < rvansa(a)redhat.com >
| | Sent: Tuesday, February 19, 2013 6:33:04 PM
| | Subject: Staggering remote GET calls
| |
| | Guys,
| |
| | I have a topic branch with a fix for ISPN-825, to stagger remote
| | GET
| | calls. (See the JIRA for details on this patch).
| |
| | This should have an interesting effect on greatly reducing the
| | pressure on the OOB thread pool. This isn't a *real* fix for the
| | problem that Radim reported (Pedro is working on that with Bela),
| | but reducing pressure on the OOB thread pool is a side effect of
| | this fix.
| |
| | It should generally make things faster too, with less traffic on
| | the
| | network. I'd be curious for you to give this branch a try, Radim -
| | see how it impacts your tests.
| |
| | https://github.com/maniksurtani/infinispan/tree/t_825
| |
| | Cheers
| | Manik
| | --
| | Manik Surtani
| | manik(a)jboss.org
| | twitter.com/maniksurtani
| |
| | Platform Architect, JBoss Data Grid
| | http://red.ht/data-grid
| |
| |
| _______________________________________________
| infinispan-dev mailing list
| infinispan-dev(a)lists.jboss.org
| https://lists.jboss.org/mailman/listinfo/infinispan-dev
|
|
| _______________________________________________
| infinispan-dev mailing list
| infinispan-dev(a)lists.jboss.org
| https://lists.jboss.org/mailman/listinfo/infinispan-dev
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Cheers,
--
Mircea Markus
Infinispan lead (www.infinispan.org)
8:57 a.m.
Only library (embeded) mode, as the load is constant (10 threads issuing requests all the
time) I believe that the results are more stable and comparable between runs.
Radim
----- Original Message -----
| From: "Mircea Markus" <mmarkus(a)redhat.com>
| To: "infinispan -Dev List" <infinispan-dev(a)lists.jboss.org>
| Sent: Wednesday, February 20, 2013 3:46:54 PM
| Subject: Re: [infinispan-dev] Staggering remote GET calls
|
|
| So you are testing embedded mode with RadarGun?
| Or remotely over hotrod?
|
|
|
|
| On 20 Feb 2013, at 14:38, Radim Vansa wrote:
|
|
|
| Ouch, call me dumbass... I haven't checked the test results.
| Something revoked my cluster allocation and the test was prematurely
| stopped.
|
| I'll rerun it (and check!), and show performance numbers as well.
|
| Radim (the dumbass)
|
| ----- Original Message -----
| | From: "Dan Berindei" < dan.berindei(a)gmail.com >
| | To: "infinispan -Dev List" < infinispan-dev(a)lists.jboss.org >
| | Cc: "Manik Surtani" < msurtani(a)redhat.com >
| | Sent: Wednesday, February 20, 2013 3:24:46 PM
| | Subject: Re: [infinispan-dev] Staggering remote GET calls
| |
| |
| | Radim, just to be sure, you are testing embedded mode with
| | RadarGun,
| | right? With HotRod most of the get operations should be initiated
| | from the main owner, so Manik's changes shouldn't make a big
| | difference in the number of active threads.
| |
| | How about throughput, has it also improved compared to 5.2.0.CR3,
| | or
| | is it the same?
| |
| |
| |
| |
| |
| | On Wed, Feb 20, 2013 at 2:15 PM, Radim Vansa < rvansa(a)redhat.com >
| | wrote:
| |
| |
| | Hi Manik,
| |
| | so I have tried to compile this branch and issued a 20 minute
| | stress
| | test (preceded by 10 minute warmup) on 128 nodes, where each node
| | has 10 stressor threads.
| | While in 5.2.0.CR3 the maximum OOB threadpool size was 553 with
| | this
| | configuration, with t_825 it was 219. This looks good, but it's
| | actually better :). When I looked on the per-node maximum, in t_825
| | there was only one node with the 219 threads (as the max), others
| | were usually around 25, few around 40. On the contrary, in
| | 5.2.0.CR3
| | all the nodes had maximum around 500!
| |
| | Glad to bring good news :)
| |
| | Radim
| |
| |
| |
| | ----- Original Message -----
| | | From: "Manik Surtani" < msurtani(a)redhat.com >
| | | To: "infinispan -Dev List" < infinispan-dev(a)lists.jboss.org >,
| | | "Radim Vansa" < rvansa(a)redhat.com >
| | | Sent: Tuesday, February 19, 2013 6:33:04 PM
| | | Subject: Staggering remote GET calls
| | |
| | | Guys,
| | |
| | | I have a topic branch with a fix for ISPN-825, to stagger remote
| | | GET
| | | calls. (See the JIRA for details on this patch).
| | |
| | | This should have an interesting effect on greatly reducing the
| | | pressure on the OOB thread pool. This isn't a *real* fix for the
| | | problem that Radim reported (Pedro is working on that with Bela),
| | | but reducing pressure on the OOB thread pool is a side effect of
| | | this fix.
| | |
| | | It should generally make things faster too, with less traffic on
| | | the
| | | network. I'd be curious for you to give this branch a try, Radim
| | | -
| | | see how it impacts your tests.
| | |
| | | https://github.com/maniksurtani/infinispan/tree/t_825
| | |
| | | Cheers
| | | Manik
| | | --
| | | Manik Surtani
| | | manik(a)jboss.org
| | | twitter.com/maniksurtani
| | |
| | | Platform Architect, JBoss Data Grid
| | | http://red.ht/data-grid
| | |
| | |
| | _______________________________________________
| | infinispan-dev mailing list
| | infinispan-dev(a)lists.jboss.org
| | https://lists.jboss.org/mailman/listinfo/infinispan-dev
| |
| |
| | _______________________________________________
| | infinispan-dev mailing list
| | infinispan-dev(a)lists.jboss.org
| | https://lists.jboss.org/mailman/listinfo/infinispan-dev
| _______________________________________________
| infinispan-dev mailing list
| infinispan-dev(a)lists.jboss.org
| https://lists.jboss.org/mailman/listinfo/infinispan-dev
|
|
|
|
|
|
|
| Cheers,
| --
| Mircea Markus
| Infinispan lead ( www.infinispan.org )
|
|
|
|
|
| _______________________________________________
| infinispan-dev mailing list
| infinispan-dev(a)lists.jboss.org
| https://lists.jboss.org/mailman/listinfo/infinispan-dev
9:08 a.m.
On 20 Feb 2013, at 14:57, Radim Vansa wrote:
Only library (embeded) mode, as the load is constant (10 threads
issuing requests all the time) I believe that the results are more stable and comparable
between runs.
Thanks. As Dan suggested, the staggering of requests should ONLY have
an (significant) impact over embedded client.
Cheers,
--
Mircea Markus
Infinispan lead (www.infinispan.org)
5:40 a.m.
Hi,
so I have re-ran (and checked this time!) both tests with the same setting.
The result is not as miraculous as the previous one but it is still 500 compared to 200,
which is good.
However, read performance has dropped by 25%, but write performance has increased by 57%!
See the charts (mainly the OOB maximum size distribution) in attachment.
Config: library mode, non-transactional distributed cache with 2 owners and 512 segments,
sync replication.
Test: 10m warmup, 20m test, 10 stressing threads per node.
Radim
----- Original Message -----
| From: "Radim Vansa" <rvansa(a)redhat.com>
| To: "infinispan -Dev List" <infinispan-dev(a)lists.jboss.org>
| Sent: Wednesday, February 20, 2013 3:38:49 PM
| Subject: Re: [infinispan-dev] Staggering remote GET calls
|
| Ouch, call me dumbass... I haven't checked the test results.
| Something revoked my cluster allocation and the test was prematurely
| stopped.
|
| I'll rerun it (and check!), and show performance numbers as well.
|
| Radim (the dumbass)
|
| ----- Original Message -----
| | From: "Dan Berindei" <dan.berindei(a)gmail.com>
| | To: "infinispan -Dev List" <infinispan-dev(a)lists.jboss.org>
| | Cc: "Manik Surtani" <msurtani(a)redhat.com>
| | Sent: Wednesday, February 20, 2013 3:24:46 PM
| | Subject: Re: [infinispan-dev] Staggering remote GET calls
| |
| |
| | Radim, just to be sure, you are testing embedded mode with
| | RadarGun,
| | right? With HotRod most of the get operations should be initiated
| | from the main owner, so Manik's changes shouldn't make a big
| | difference in the number of active threads.
| |
| | How about throughput, has it also improved compared to 5.2.0.CR3,
| | or
| | is it the same?
| |
| |
| |
| |
| |
| | On Wed, Feb 20, 2013 at 2:15 PM, Radim Vansa < rvansa(a)redhat.com >
| | wrote:
| |
| |
| | Hi Manik,
| |
| | so I have tried to compile this branch and issued a 20 minute
| | stress
| | test (preceded by 10 minute warmup) on 128 nodes, where each node
| | has 10 stressor threads.
| | While in 5.2.0.CR3 the maximum OOB threadpool size was 553 with
| | this
| | configuration, with t_825 it was 219. This looks good, but it's
| | actually better :). When I looked on the per-node maximum, in t_825
| | there was only one node with the 219 threads (as the max), others
| | were usually around 25, few around 40. On the contrary, in
| | 5.2.0.CR3
| | all the nodes had maximum around 500!
| |
| | Glad to bring good news :)
| |
| | Radim
| |
| |
| |
| | ----- Original Message -----
| | | From: "Manik Surtani" < msurtani(a)redhat.com >
| | | To: "infinispan -Dev List" < infinispan-dev(a)lists.jboss.org >,
| | | "Radim Vansa" < rvansa(a)redhat.com >
| | | Sent: Tuesday, February 19, 2013 6:33:04 PM
| | | Subject: Staggering remote GET calls
| | |
| | | Guys,
| | |
| | | I have a topic branch with a fix for ISPN-825, to stagger remote
| | | GET
| | | calls. (See the JIRA for details on this patch).
| | |
| | | This should have an interesting effect on greatly reducing the
| | | pressure on the OOB thread pool. This isn't a *real* fix for the
| | | problem that Radim reported (Pedro is working on that with Bela),
| | | but reducing pressure on the OOB thread pool is a side effect of
| | | this fix.
| | |
| | | It should generally make things faster too, with less traffic on
| | | the
| | | network. I'd be curious for you to give this branch a try, Radim
| | | -
| | | see how it impacts your tests.
| | |
| | | https://github.com/maniksurtani/infinispan/tree/t_825
| | |
| | | Cheers
| | | Manik
| | | --
| | | Manik Surtani
| | | manik(a)jboss.org
| | | twitter.com/maniksurtani
| | |
| | | Platform Architect, JBoss Data Grid
| | | http://red.ht/data-grid
| | |
| | |
| | _______________________________________________
| | infinispan-dev mailing list
| | infinispan-dev(a)lists.jboss.org
| | https://lists.jboss.org/mailman/listinfo/infinispan-dev
| |
| |
| | _______________________________________________
| | infinispan-dev mailing list
| | infinispan-dev(a)lists.jboss.org
| | https://lists.jboss.org/mailman/listinfo/infinispan-dev
| _______________________________________________
| infinispan-dev mailing list
| infinispan-dev(a)lists.jboss.org
| https://lists.jboss.org/mailman/listinfo/infinispan-dev
|
4:56 a.m.
I'm not surprised that read performance suffers a bit actually. Which is why we
broadcast the GETs originally. But once the staggering timeout becomes configurable, this
should be something people can tune.
- M
On 21 Feb 2013, at 11:40, Radim Vansa <rvansa(a)redhat.com> wrote:
Hi,
so I have re-ran (and checked this time!) both tests with the same setting.
The result is not as miraculous as the previous one but it is still 500 compared to 200,
which is good.
However, read performance has dropped by 25%, but write performance has increased by
57%!
See the charts (mainly the OOB maximum size distribution) in attachment.
Config: library mode, non-transactional distributed cache with 2 owners and 512 segments,
sync replication.
Test: 10m warmup, 20m test, 10 stressing threads per node.
Radim
----- Original Message -----
| From: "Radim Vansa" <rvansa(a)redhat.com>
| To: "infinispan -Dev List" <infinispan-dev(a)lists.jboss.org>
| Sent: Wednesday, February 20, 2013 3:38:49 PM
| Subject: Re: [infinispan-dev] Staggering remote GET calls
|
| Ouch, call me dumbass... I haven't checked the test results.
| Something revoked my cluster allocation and the test was prematurely
| stopped.
|
| I'll rerun it (and check!), and show performance numbers as well.
|
| Radim (the dumbass)
|
| ----- Original Message -----
| | From: "Dan Berindei" <dan.berindei(a)gmail.com>
| | To: "infinispan -Dev List" <infinispan-dev(a)lists.jboss.org>
| | Cc: "Manik Surtani" <msurtani(a)redhat.com>
| | Sent: Wednesday, February 20, 2013 3:24:46 PM
| | Subject: Re: [infinispan-dev] Staggering remote GET calls
| |
| |
| | Radim, just to be sure, you are testing embedded mode with
| | RadarGun,
| | right? With HotRod most of the get operations should be initiated
| | from the main owner, so Manik's changes shouldn't make a big
| | difference in the number of active threads.
| |
| | How about throughput, has it also improved compared to 5.2.0.CR3,
| | or
| | is it the same?
| |
| |
| |
| |
| |
| | On Wed, Feb 20, 2013 at 2:15 PM, Radim Vansa < rvansa(a)redhat.com >
| | wrote:
| |
| |
| | Hi Manik,
| |
| | so I have tried to compile this branch and issued a 20 minute
| | stress
| | test (preceded by 10 minute warmup) on 128 nodes, where each node
| | has 10 stressor threads.
| | While in 5.2.0.CR3 the maximum OOB threadpool size was 553 with
| | this
| | configuration, with t_825 it was 219. This looks good, but it's
| | actually better :). When I looked on the per-node maximum, in t_825
| | there was only one node with the 219 threads (as the max), others
| | were usually around 25, few around 40. On the contrary, in
| | 5.2.0.CR3
| | all the nodes had maximum around 500!
| |
| | Glad to bring good news :)
| |
| | Radim
| |
| |
| |
| | ----- Original Message -----
| | | From: "Manik Surtani" < msurtani(a)redhat.com >
| | | To: "infinispan -Dev List" < infinispan-dev(a)lists.jboss.org >,
| | | "Radim Vansa" < rvansa(a)redhat.com >
| | | Sent: Tuesday, February 19, 2013 6:33:04 PM
| | | Subject: Staggering remote GET calls
| | |
| | | Guys,
| | |
| | | I have a topic branch with a fix for ISPN-825, to stagger remote
| | | GET
| | | calls. (See the JIRA for details on this patch).
| | |
| | | This should have an interesting effect on greatly reducing the
| | | pressure on the OOB thread pool. This isn't a *real* fix for the
| | | problem that Radim reported (Pedro is working on that with Bela),
| | | but reducing pressure on the OOB thread pool is a side effect of
| | | this fix.
| | |
| | | It should generally make things faster too, with less traffic on
| | | the
| | | network. I'd be curious for you to give this branch a try, Radim
| | | -
| | | see how it impacts your tests.
| | |
| | | https://github.com/maniksurtani/infinispan/tree/t_825
| | |
| | | Cheers
| | | Manik
| | | --
| | | Manik Surtani
| | | manik(a)jboss.org
| | | twitter.com/maniksurtani
| | |
| | | Platform Architect, JBoss Data Grid
| | | http://red.ht/data-grid
| | |
| | |
| | _______________________________________________
| | infinispan-dev mailing list
| | infinispan-dev(a)lists.jboss.org
| | https://lists.jboss.org/mailman/listinfo/infinispan-dev
| |
| |
| | _______________________________________________
| | infinispan-dev mailing list
| | infinispan-dev(a)lists.jboss.org
| | https://lists.jboss.org/mailman/listinfo/infinispan-dev
| _______________________________________________
| infinispan-dev mailing list
| infinispan-dev(a)lists.jboss.org
| https://lists.jboss.org/mailman/listinfo/infinispan-dev
|
<t825.pdf>_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
--
Manik Surtani
manik(a)jboss.org
twitter.com/maniksurtani
Platform Architect, JBoss Data Grid
http://red.ht/data-grid
6:35 a.m.
On 26 Feb 2013, at 10:56, Manik Surtani wrote:
I'm not surprised that read performance suffers a bit actually.
Which is why we broadcast the GETs originally. But once the staggering timeout becomes
configurable, this should be something people can tune.
+1. Also the performance
increase for writes is huge. I think this should be the default, as read efficiency can be
improved by L1.
Cheers,
--
Mircea Markus
Infinispan lead (www.infinispan.org)
7:55 a.m.
Tune in real-time, of course ;)
Erik
From: infinispan-dev-bounces(a)lists.jboss.org
[mailto:infinispan-dev-bounces@lists.jboss.org] On Behalf Of Mircea Markus
Sent: Tuesday, February 26, 2013 7:36 AM
To: infinispan -Dev List
Subject: Re: [infinispan-dev] Staggering remote GET calls
On 26 Feb 2013, at 10:56, Manik Surtani wrote:
I'm not surprised that read performance suffers a bit actually. Which is
why we broadcast the GETs originally. But once the staggering timeout
becomes configurable, this should be something people can tune.
+1. Also the performance increase for writes is huge. I think this should be
the default, as read efficiency can be improved by L1.
Cheers,
--
Mircea Markus
Infinispan lead (www.infinispan.org)
8:12 a.m.
If you're really into self-tuning this parameter, I expect that a very
simple gradient-descent mechanism would actually work pretty well in
this case.
We have done similar work in the Cloud-TM project (applied to both
message batching and number of threads active per node), and if you're
interested I may send more references on this.
BTW, +1 for making this configurable: eager broadcasting read requests
seemed to me to be a little too aggressive for the normal/failure-free
behavior.
Cheers,
Paolo
On 2/26/13 1:55 PM, Erik Salter wrote:
Tune in real-time, of course ;)
Erik
*From:*infinispan-dev-bounces@lists.jboss.org
[mailto:infinispan-dev-bounces@lists.jboss.org] *On Behalf Of *Mircea
Markus
*Sent:* Tuesday, February 26, 2013 7:36 AM
*To:* infinispan -Dev List
*Subject:* Re: [infinispan-dev] Staggering remote GET calls
On 26 Feb 2013, at 10:56, Manik Surtani wrote:
I'm not surprised that read performance suffers a bit actually. Which
is why we broadcast the GETs originally. But once the staggering
timeout becomes configurable, this should be something people can tune.
+1. Also the performance increase for writes is huge. I think this
should be the default, as read efficiency can be improved by L1.
Cheers,
--
Mircea Markus
Infinispan lead (www.infinispan.org <http://www.infinispan.org>)
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
8:17 a.m.
On 26 Feb 2013, at 14:12, Paolo Romano <romano(a)inesc-id.pt> wrote:
If you're really into self-tuning this parameter, I expect that a
very simple gradient-descent mechanism would actually work pretty well in this case.
We have done similar work in the Cloud-TM project (applied to both message batching and
number of threads active per node), and if you're interested I may send more
references on this.
Yes, please do. :)
BTW, +1 for making this configurable: eager broadcasting read requests seemed to me to be
a little too aggressive for the normal/failure-free behavior.
Cheers,
Paolo
On 2/26/13 1:55 PM, Erik Salter wrote:
> Tune in real-time, of course ;)
>
> Erik
>
> From: infinispan-dev-bounces(a)lists.jboss.org
[mailto:infinispan-dev-bounces@lists.jboss.org] On Behalf Of Mircea Markus
> Sent: Tuesday, February 26, 2013 7:36 AM
> To: infinispan -Dev List
> Subject: Re: [infinispan-dev] Staggering remote GET calls
>
>
> On 26 Feb 2013, at 10:56, Manik Surtani wrote:
>
>
> I'm not surprised that read performance suffers a bit actually. Which is why we
broadcast the GETs originally. But once the staggering timeout becomes configurable, this
should be something people can tune.
>
> +1. Also the performance increase for writes is huge. I think this should be the
default, as read efficiency can be improved by L1.
>
> Cheers,
> --
> Mircea Markus
> Infinispan lead (www.infinispan.org)
>
>
>
>
>
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
--
Manik Surtani
manik(a)jboss.org
twitter.com/maniksurtani
Platform Architect, JBoss Data Grid
http://red.ht/data-grid
8:38 a.m.
On 2/26/13 2:17 PM, Manik Surtani wrote:
On 26 Feb 2013, at 14:12, Paolo Romano <romano(a)inesc-id.pt
<mailto:romano@inesc-id.pt>> wrote:
> If you're really into self-tuning this parameter, I expect that a
> very simple gradient-descent mechanism would actually work pretty
> well in this case.
>
> We have done similar work in the Cloud-TM project (applied to both
> message batching and number of threads active per node), and if
> you're interested I may send more references on this.
Yes, please do. :)
In attach 2 papers in pdf version:
- "paper_submitted.pdf" deals with optimization of the level of
parallelism in transactional memories (both centralized and distributed)
- "SASO12_paper84_PDFexpressOk.pdf" presents a mechanism for self-tuning
the batching (a.k.a. message packing) in total-order based protocols.
Diego Didona (who has already subscribed this mailing-list) will be glad
to provide more details ;)
9:22 a.m.
Excellent !
On 2/20/13 3:15 PM, Radim Vansa wrote:
Hi Manik,
so I have tried to compile this branch and issued a 20 minute stress test (preceded by 10
minute warmup) on 128 nodes, where each node has 10 stressor threads.
While in 5.2.0.CR3 the maximum OOB threadpool size was 553 with this configuration, with
t_825 it was 219. This looks good, but it's actually better :). When I looked on the
per-node maximum, in t_825 there was only one node with the 219 threads (as the max),
others were usually around 25, few around 40. On the contrary, in 5.2.0.CR3 all the nodes
had maximum around 500!
Glad to bring good news :)
Radim
----- Original Message -----
| From: "Manik Surtani" <msurtani(a)redhat.com>
| To: "infinispan -Dev List" <infinispan-dev(a)lists.jboss.org>,
"Radim Vansa" <rvansa(a)redhat.com>
| Sent: Tuesday, February 19, 2013 6:33:04 PM
| Subject: Staggering remote GET calls
|
| Guys,
|
| I have a topic branch with a fix for ISPN-825, to stagger remote GET
| calls. (See the JIRA for details on this patch).
|
| This should have an interesting effect on greatly reducing the
| pressure on the OOB thread pool. This isn't a *real* fix for the
| problem that Radim reported (Pedro is working on that with Bela),
| but reducing pressure on the OOB thread pool is a side effect of
| this fix.
|
| It should generally make things faster too, with less traffic on the
| network. I'd be curious for you to give this branch a try, Radim -
| see how it impacts your tests.
|
| https://github.com/maniksurtani/infinispan/tree/t_825
|
| Cheers
| Manik
| --
| Manik Surtani
| manik(a)jboss.org
| twitter.com/maniksurtani
|
| Platform Architect, JBoss Data Grid
| http://red.ht/data-grid
|
|
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
--
Bela Ban, JGroups lead (http://www.jgroups.org)
11:41 a.m.
Awesome. :) Mircea, are you going to finish up my patch, make the staggering timeout
configurable, etc?
Sent from my mobile phone
On 20 Feb 2013, at 15:15, Radim Vansa <rvansa(a)redhat.com> wrote:
Hi Manik,
so I have tried to compile this branch and issued a 20 minute stress test (preceded by 10
minute warmup) on 128 nodes, where each node has 10 stressor threads.
While in 5.2.0.CR3 the maximum OOB threadpool size was 553 with this configuration, with
t_825 it was 219. This looks good, but it's actually better :). When I looked on the
per-node maximum, in t_825 there was only one node with the 219 threads (as the max),
others were usually around 25, few around 40. On the contrary, in 5.2.0.CR3 all the nodes
had maximum around 500!
Glad to bring good news :)
Radim
----- Original Message -----
| From: "Manik Surtani" <msurtani(a)redhat.com>
| To: "infinispan -Dev List" <infinispan-dev(a)lists.jboss.org>,
"Radim Vansa" <rvansa(a)redhat.com>
| Sent: Tuesday, February 19, 2013 6:33:04 PM
| Subject: Staggering remote GET calls
|
| Guys,
|
| I have a topic branch with a fix for ISPN-825, to stagger remote GET
| calls. (See the JIRA for details on this patch).
|
| This should have an interesting effect on greatly reducing the
| pressure on the OOB thread pool. This isn't a *real* fix for the
| problem that Radim reported (Pedro is working on that with Bela),
| but reducing pressure on the OOB thread pool is a side effect of
| this fix.
|
| It should generally make things faster too, with less traffic on the
| network. I'd be curious for you to give this branch a try, Radim -
| see how it impacts your tests.
|
| https://github.com/maniksurtani/infinispan/tree/t_825
|
| Cheers
| Manik
| --
| Manik Surtani
| manik(a)jboss.org
| twitter.com/maniksurtani
|
| Platform Architect, JBoss Data Grid
| http://red.ht/data-grid
|
|
1:29 a.m.
Manik, as I have stated in later mails, the test was not executed properly, therefore
these good results.
Actually, I think there is a bug in the branch, as you set expected responses in
FutureCollator to dest.size() but still send only one message, if the first response is
not successful but without exception set (the objectFuture.get() is interrupted, I am not
sure why this happens, but it happens) the thread blocks infinitely. I have patched it and
hope I'll get results soon.
Radim
----- Original Message -----
| From: "Manik Surtani" <msurtani(a)redhat.com>
| To: "Radim Vansa" <rvansa(a)redhat.com>
| Cc: "infinispan -Dev List" <infinispan-dev(a)lists.jboss.org>
| Sent: Wednesday, February 20, 2013 6:41:49 PM
| Subject: Re: Staggering remote GET calls
|
| Awesome. :) Mircea, are you going to finish up my patch, make the
| staggering timeout configurable, etc?
|
| Sent from my mobile phone
|
| On 20 Feb 2013, at 15:15, Radim Vansa <rvansa(a)redhat.com> wrote:
|
| > Hi Manik,
| >
| > so I have tried to compile this branch and issued a 20 minute
| > stress test (preceded by 10 minute warmup) on 128 nodes, where
| > each node has 10 stressor threads.
| > While in 5.2.0.CR3 the maximum OOB threadpool size was 553 with
| > this configuration, with t_825 it was 219. This looks good, but
| > it's actually better :). When I looked on the per-node maximum, in
| > t_825 there was only one node with the 219 threads (as the max),
| > others were usually around 25, few around 40. On the contrary, in
| > 5.2.0.CR3 all the nodes had maximum around 500!
| >
| > Glad to bring good news :)
| >
| > Radim
| >
| > ----- Original Message -----
| > | From: "Manik Surtani" <msurtani(a)redhat.com>
| > | To: "infinispan -Dev List" <infinispan-dev(a)lists.jboss.org>,
| > | "Radim Vansa" <rvansa(a)redhat.com>
| > | Sent: Tuesday, February 19, 2013 6:33:04 PM
| > | Subject: Staggering remote GET calls
| > |
| > | Guys,
| > |
| > | I have a topic branch with a fix for ISPN-825, to stagger remote
| > | GET
| > | calls. (See the JIRA for details on this patch).
| > |
| > | This should have an interesting effect on greatly reducing the
| > | pressure on the OOB thread pool. This isn't a *real* fix for the
| > | problem that Radim reported (Pedro is working on that with Bela),
| > | but reducing pressure on the OOB thread pool is a side effect of
| > | this fix.
| > |
| > | It should generally make things faster too, with less traffic on
| > | the
| > | network. I'd be curious for you to give this branch a try, Radim
| > | -
| > | see how it impacts your tests.
| > |
| > | https://github.com/maniksurtani/infinispan/tree/t_825
| > |
| > | Cheers
| > | Manik
| > | --
| > | Manik Surtani
| > | manik(a)jboss.org
| > | twitter.com/maniksurtani
| > |
| > | Platform Architect, JBoss Data Grid
| > | http://red.ht/data-grid
| > |
| > |
|
4317
days inactive
4324
days old
infinispan-dev@lists.jboss.org
16 comments
7 participants
participants (7)
-
Bela Ban -
Dan Berindei -
Erik Salter -
Manik Surtani -
Mircea Markus -
Paolo Romano -
Radim Vansa