[JBoss JIRA] (JGRP-1564) TP: passing messages up in batches (part I)
by Bela Ban (JIRA)
[ https://issues.jboss.org/browse/JGRP-1564?page=com.atlassian.jira.plugin.... ]
Bela Ban edited comment on JGRP-1564 at 2/28/13 6:45 AM:
---------------------------------------------------------
The first part is done. A quick perf test showed:
h4. MPerf (fast.xml)
(requests/sec/node, 1000)
||Nodes||2||4||6||8||
|old|111|143|107|101|
|new|113|148|117|115|
h4. UnicastTestRpc (fast.xml)
||Node||2||
|old|111|
|new|111|
h4. UPerf (fast.xml)
(requests/sec/node)
||Node||4||8||
|old|6'818|5'352|
|new|7'607|6'211|
was (Author: belaban):
The first part is done. A quick perf test showed:
MPerf (fast.xml):
-----------------
(requests/sec/node, 1000)
||Nodes||2||4||6||8||
|old|111|143|107|101|
|new|113|148|117|115|
UnicastTestRpc (fast.xml):
--------------------------
||Node||2||
|old|111|
|new|111|
UPerf (fast.xml):
-----------------
(requests/sec/node)
||Node||4||8||
|old|6'818|5'352|
|new|7'607|6'211|
> TP: passing messages up in batches (part I)
> -------------------------------------------
>
> Key: JGRP-1564
> URL: https://issues.jboss.org/browse/JGRP-1564
> Project: JGroups
> Issue Type: Enhancement
> Reporter: Bela Ban
> Assignee: Bela Ban
> Fix For: 3.3
>
>
> When B receives a batch of 5 messages from A (unicast or multicast), then B uses the *same thread* to send the 5 messages up (this isn't the case for OOB messages).
> It would be more efficient to either have different threads passing the 5 messages up, or use a new *message batch event type* to pass all 5 messages up in one go.
> The advantage of different threads is that all 5 threads add their message to the window, but only 1 removes them and passes them up, rather than each thread adding and removing its own message (fewer lock acquisitions).
> We could try moving the unmarshalling of messages and message batches into TP.receive(). If a batch was received, that code could unmarshal the 5 messages and pass them to corresponding thread pools to send them up.
> The unmarshalling shouldn't take long, so TP.receive() should return quickly.
> This approach would allow us to send OOB messages in message batches, too (currently not allowed).
> The advantage of a message batch is that we pass *one* event up the stack, passing only *once* through all protocols from TP to UNICAST/2 and NAKACK/2, and not 5 times. Also, adding 5 messages to the window under the same lock is more eficient than acquiring the lock 5 times. Ditto for removal.
> The disadvantage is that we now need to handle a different event type (all protocols under UNICAST/NAKACK), e.g. ENCRYPT, SIZE, FRAG(2) (if placed under UNICAST/NAKACK), COMPRESS etc. However, we could add another up(Batch) method, which by default (in Protocol):
> - removes all messages for a given protocol P (by P.ID)
> and calls up(Event.MSG, msg) for all messages in the batch
> - calls up_prot.up(batch) if the batch is not empty
> This would allow for all current protocols to continue working and only the protocols which don't check for headers and/or need special processing (such as UNICAST and NAKACK) would have to implement up(Batch).
> This solution would be better than introducing another event type MSG_BATCH, as not every protocol overriding up(Event) calls super.up(Event).
> However, this solution is not symmetric, ie. messages are batched at the transport level, and should be unbatched at the transport level of the receiver(s) as well...
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 2 months
[JBoss JIRA] (JGRP-1564) TP: passing messages up in batches (part I)
by Bela Ban (JIRA)
[ https://issues.jboss.org/browse/JGRP-1564?page=com.atlassian.jira.plugin.... ]
Bela Ban edited comment on JGRP-1564 at 2/28/13 6:44 AM:
---------------------------------------------------------
With batching part II implemented, the results for MPerf were more or less the same. UPerf changed slightly:
h4. UPerf (fast.xml):
(requests/sec/node)
||Node||2||4||6||8||
|batch I|n/a|7'607|n/a|6'211|
|batch II|8'012 |{color:green}8'052{color}|7'377|{color:green}6'937{color}|
On the Red Hat cluster-XX lab:
||Node||2||4||6||8||
|batch II|11'376|15'958|15'932|14'925|
Compared to the home cluster, in the Red Hat lab, every process ran on a seperate physical box (no sharing of bandwidth etc).
was (Author: belaban):
With batch part II implemented, the results for MPerf were more or less the same. UPerf changed slightly:
h4. UPerf (fast.xml):
(requests/sec/node)
||Node||2||4||6||8||
|batch I|n/a|7'607|n/a|6'211|
|batch II|8012 |{color:green}8052{color}|7377|{color:green}6937{color}|
On the Red Hat cluster-XX lab:
||Node||2||4||6||8||
|batch II|11376|15958|15932|14925|
Compared to the home cluster, in the Red Hat lab, every process ran on a seperate physical box (no sharing of bandwidth etc).
> TP: passing messages up in batches (part I)
> -------------------------------------------
>
> Key: JGRP-1564
> URL: https://issues.jboss.org/browse/JGRP-1564
> Project: JGroups
> Issue Type: Enhancement
> Reporter: Bela Ban
> Assignee: Bela Ban
> Fix For: 3.3
>
>
> When B receives a batch of 5 messages from A (unicast or multicast), then B uses the *same thread* to send the 5 messages up (this isn't the case for OOB messages).
> It would be more efficient to either have different threads passing the 5 messages up, or use a new *message batch event type* to pass all 5 messages up in one go.
> The advantage of different threads is that all 5 threads add their message to the window, but only 1 removes them and passes them up, rather than each thread adding and removing its own message (fewer lock acquisitions).
> We could try moving the unmarshalling of messages and message batches into TP.receive(). If a batch was received, that code could unmarshal the 5 messages and pass them to corresponding thread pools to send them up.
> The unmarshalling shouldn't take long, so TP.receive() should return quickly.
> This approach would allow us to send OOB messages in message batches, too (currently not allowed).
> The advantage of a message batch is that we pass *one* event up the stack, passing only *once* through all protocols from TP to UNICAST/2 and NAKACK/2, and not 5 times. Also, adding 5 messages to the window under the same lock is more eficient than acquiring the lock 5 times. Ditto for removal.
> The disadvantage is that we now need to handle a different event type (all protocols under UNICAST/NAKACK), e.g. ENCRYPT, SIZE, FRAG(2) (if placed under UNICAST/NAKACK), COMPRESS etc. However, we could add another up(Batch) method, which by default (in Protocol):
> - removes all messages for a given protocol P (by P.ID)
> and calls up(Event.MSG, msg) for all messages in the batch
> - calls up_prot.up(batch) if the batch is not empty
> This would allow for all current protocols to continue working and only the protocols which don't check for headers and/or need special processing (such as UNICAST and NAKACK) would have to implement up(Batch).
> This solution would be better than introducing another event type MSG_BATCH, as not every protocol overriding up(Event) calls super.up(Event).
> However, this solution is not symmetric, ie. messages are batched at the transport level, and should be unbatched at the transport level of the receiver(s) as well...
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 2 months
[JBoss JIRA] (JGRP-1564) TP: passing messages up in batches (part I)
by Bela Ban (JIRA)
[ https://issues.jboss.org/browse/JGRP-1564?page=com.atlassian.jira.plugin.... ]
Bela Ban edited comment on JGRP-1564 at 2/28/13 6:43 AM:
---------------------------------------------------------
With batch part II implemented, the results for MPerf were more or less the same. UPerf changed slightly:
h4. UPerf (fast.xml):
(requests/sec/node)
||Node||2||4||6||8||
|batch I|n/a|7'607|n/a|6'211|
|batch II|8012 |{color:green}8052{color}|7377|{color:green}6937{color}|
On the Red Hat cluster-XX lab:
||Node||2||4||6||8||
|batch II|11376|15958|15932|14925|
Compared to the home cluster, in the Red Hat lab, every process ran on a seperate physical box (no sharing of bandwidth etc).
was (Author: belaban):
With batch part II implemented, the results for MPerf were more or less the same. UPerf changed slightly:
> TP: passing messages up in batches (part I)
> -------------------------------------------
>
> Key: JGRP-1564
> URL: https://issues.jboss.org/browse/JGRP-1564
> Project: JGroups
> Issue Type: Enhancement
> Reporter: Bela Ban
> Assignee: Bela Ban
> Fix For: 3.3
>
>
> When B receives a batch of 5 messages from A (unicast or multicast), then B uses the *same thread* to send the 5 messages up (this isn't the case for OOB messages).
> It would be more efficient to either have different threads passing the 5 messages up, or use a new *message batch event type* to pass all 5 messages up in one go.
> The advantage of different threads is that all 5 threads add their message to the window, but only 1 removes them and passes them up, rather than each thread adding and removing its own message (fewer lock acquisitions).
> We could try moving the unmarshalling of messages and message batches into TP.receive(). If a batch was received, that code could unmarshal the 5 messages and pass them to corresponding thread pools to send them up.
> The unmarshalling shouldn't take long, so TP.receive() should return quickly.
> This approach would allow us to send OOB messages in message batches, too (currently not allowed).
> The advantage of a message batch is that we pass *one* event up the stack, passing only *once* through all protocols from TP to UNICAST/2 and NAKACK/2, and not 5 times. Also, adding 5 messages to the window under the same lock is more eficient than acquiring the lock 5 times. Ditto for removal.
> The disadvantage is that we now need to handle a different event type (all protocols under UNICAST/NAKACK), e.g. ENCRYPT, SIZE, FRAG(2) (if placed under UNICAST/NAKACK), COMPRESS etc. However, we could add another up(Batch) method, which by default (in Protocol):
> - removes all messages for a given protocol P (by P.ID)
> and calls up(Event.MSG, msg) for all messages in the batch
> - calls up_prot.up(batch) if the batch is not empty
> This would allow for all current protocols to continue working and only the protocols which don't check for headers and/or need special processing (such as UNICAST and NAKACK) would have to implement up(Batch).
> This solution would be better than introducing another event type MSG_BATCH, as not every protocol overriding up(Event) calls super.up(Event).
> However, this solution is not symmetric, ie. messages are batched at the transport level, and should be unbatched at the transport level of the receiver(s) as well...
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 2 months
[JBoss JIRA] (JGRP-1564) TP: passing messages up in batches (part I)
by Bela Ban (JIRA)
[ https://issues.jboss.org/browse/JGRP-1564?page=com.atlassian.jira.plugin.... ]
Bela Ban commented on JGRP-1564:
--------------------------------
With batch part II implemented, the results for MPerf were more or less the same. UPerf changed slightly:
> TP: passing messages up in batches (part I)
> -------------------------------------------
>
> Key: JGRP-1564
> URL: https://issues.jboss.org/browse/JGRP-1564
> Project: JGroups
> Issue Type: Enhancement
> Reporter: Bela Ban
> Assignee: Bela Ban
> Fix For: 3.3
>
>
> When B receives a batch of 5 messages from A (unicast or multicast), then B uses the *same thread* to send the 5 messages up (this isn't the case for OOB messages).
> It would be more efficient to either have different threads passing the 5 messages up, or use a new *message batch event type* to pass all 5 messages up in one go.
> The advantage of different threads is that all 5 threads add their message to the window, but only 1 removes them and passes them up, rather than each thread adding and removing its own message (fewer lock acquisitions).
> We could try moving the unmarshalling of messages and message batches into TP.receive(). If a batch was received, that code could unmarshal the 5 messages and pass them to corresponding thread pools to send them up.
> The unmarshalling shouldn't take long, so TP.receive() should return quickly.
> This approach would allow us to send OOB messages in message batches, too (currently not allowed).
> The advantage of a message batch is that we pass *one* event up the stack, passing only *once* through all protocols from TP to UNICAST/2 and NAKACK/2, and not 5 times. Also, adding 5 messages to the window under the same lock is more eficient than acquiring the lock 5 times. Ditto for removal.
> The disadvantage is that we now need to handle a different event type (all protocols under UNICAST/NAKACK), e.g. ENCRYPT, SIZE, FRAG(2) (if placed under UNICAST/NAKACK), COMPRESS etc. However, we could add another up(Batch) method, which by default (in Protocol):
> - removes all messages for a given protocol P (by P.ID)
> and calls up(Event.MSG, msg) for all messages in the batch
> - calls up_prot.up(batch) if the batch is not empty
> This would allow for all current protocols to continue working and only the protocols which don't check for headers and/or need special processing (such as UNICAST and NAKACK) would have to implement up(Batch).
> This solution would be better than introducing another event type MSG_BATCH, as not every protocol overriding up(Event) calls super.up(Event).
> However, this solution is not symmetric, ie. messages are batched at the transport level, and should be unbatched at the transport level of the receiver(s) as well...
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 2 months
[JBoss JIRA] (AS7-6649) Enable checkstyle for clustering integration testsuite
by Radoslav Husar (JIRA)
Radoslav Husar created AS7-6649:
-----------------------------------
Summary: Enable checkstyle for clustering integration testsuite
Key: AS7-6649
URL: https://issues.jboss.org/browse/AS7-6649
Project: Application Server 7
Issue Type: Feature Request
Components: Clustering, Test Suite
Reporter: Radoslav Husar
Assignee: Radoslav Husar
Fix For: 8.0.0.Alpha1
Ideally this will make live easier by less rebasing due to formatting differences and easier PR reviews. Lets see how it goes.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 2 months
[JBoss JIRA] (AS7-6511) Investigate OSGi test failures after MSC upgrade to 1.1.1.Final
by Tomaz Cerar (JIRA)
[ https://issues.jboss.org/browse/AS7-6511?page=com.atlassian.jira.plugin.s... ]
Tomaz Cerar commented on AS7-6511:
----------------------------------
I have disabled this tests as part of AS7-6647
> Investigate OSGi test failures after MSC upgrade to 1.1.1.Final
> ---------------------------------------------------------------
>
> Key: AS7-6511
> URL: https://issues.jboss.org/browse/AS7-6511
> Project: Application Server 7
> Issue Type: Task
> Components: OSGi
> Affects Versions: 8.0.0.Alpha1
> Environment: Tested on (not able to reproduce)
> ---
> [blackhole][/home/opalka/git/jboss-as/trunk]>java -version
> java version "1.6.0_38"
> Java(TM) SE Runtime Environment (build 1.6.0_38-b05)
> Java HotSpot(TM) 64-Bit Server VM (build 20.13-b02, mixed mode)
> ---
> [blackhole][/home/opalka/git/jboss-as/trunk]>mvn -version
> Apache Maven 3.0.4 (r1232337; 2012-01-17 09:44:56+0100)
> Maven home: /home/opalka/java/maven/3.0.4
> Java version: 1.6.0_38, vendor: Sun Microsystems Inc.
> Java home: /home/opalka/java/jdk/orcl/jdk1.6.0_38/jre
> Default locale: en_US, platform encoding: UTF-8
> OS name: "linux", version: "3.7.6-201.fc18.x86_64", arch: "amd64", family: "unix"
> ---
> [blackhole][/home/opalka/git/jboss-as/trunk]>uname -a
> Linux blackhole 3.7.6-201.fc18.x86_64 #1 SMP Mon Feb 4 15:54:08 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
> Reporter: Richard Opalka
> Assignee: Thomas Diesler
> Priority: Blocker
> Fix For: 8.0.0.Alpha1
>
>
> I did my best to reproduce this issue but I'm not able to reproduce it locally:( It doesn't matter if I run whole test suite or just this test, the
> test is always passing for me. Since OSGi is not my domain, I'm assigning
> this issue to OSGi team. This bug is related to this PR:
> https://github.com/jbossas/jboss-as/pull/4052
> I'm temporarily disabling the failing test (the purpose of this JIRA).
> Failing test is:
> [INFO] Surefire report directory: /home/jenkins/jenkins-work/workspace/as7-param-pull/testsuite/integration/osgi/target/surefire-reports
> -------------------------------------------------------
> T E S T S
> -------------------------------------------------------
> ...
> Results :
> Failed tests: testFailStartLevel(org.jboss.as.test.integration.osgi.deployment.DeferredResolveTestCase): Bundle ACTIVE expected:<32> but was:<4>
> Tests run: 120, Failures: 1, Errors: 0, Skipped: 6
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 2 months
[JBoss JIRA] (AS7-6511) Investigate OSGi test failures after MSC upgrade to 1.1.1.Final
by Tomaz Cerar (JIRA)
[ https://issues.jboss.org/browse/AS7-6511?page=com.atlassian.jira.plugin.s... ]
Tomaz Cerar updated AS7-6511:
-----------------------------
Fix Version/s: 8.0.0.Alpha1
(was: 7.3.0.Alpha1)
> Investigate OSGi test failures after MSC upgrade to 1.1.1.Final
> ---------------------------------------------------------------
>
> Key: AS7-6511
> URL: https://issues.jboss.org/browse/AS7-6511
> Project: Application Server 7
> Issue Type: Task
> Components: OSGi
> Affects Versions: 8.0.0.Alpha1
> Environment: Tested on (not able to reproduce)
> ---
> [blackhole][/home/opalka/git/jboss-as/trunk]>java -version
> java version "1.6.0_38"
> Java(TM) SE Runtime Environment (build 1.6.0_38-b05)
> Java HotSpot(TM) 64-Bit Server VM (build 20.13-b02, mixed mode)
> ---
> [blackhole][/home/opalka/git/jboss-as/trunk]>mvn -version
> Apache Maven 3.0.4 (r1232337; 2012-01-17 09:44:56+0100)
> Maven home: /home/opalka/java/maven/3.0.4
> Java version: 1.6.0_38, vendor: Sun Microsystems Inc.
> Java home: /home/opalka/java/jdk/orcl/jdk1.6.0_38/jre
> Default locale: en_US, platform encoding: UTF-8
> OS name: "linux", version: "3.7.6-201.fc18.x86_64", arch: "amd64", family: "unix"
> ---
> [blackhole][/home/opalka/git/jboss-as/trunk]>uname -a
> Linux blackhole 3.7.6-201.fc18.x86_64 #1 SMP Mon Feb 4 15:54:08 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
> Reporter: Richard Opalka
> Assignee: Thomas Diesler
> Priority: Blocker
> Fix For: 8.0.0.Alpha1
>
>
> I did my best to reproduce this issue but I'm not able to reproduce it locally:( It doesn't matter if I run whole test suite or just this test, the
> test is always passing for me. Since OSGi is not my domain, I'm assigning
> this issue to OSGi team. This bug is related to this PR:
> https://github.com/jbossas/jboss-as/pull/4052
> I'm temporarily disabling the failing test (the purpose of this JIRA).
> Failing test is:
> [INFO] Surefire report directory: /home/jenkins/jenkins-work/workspace/as7-param-pull/testsuite/integration/osgi/target/surefire-reports
> -------------------------------------------------------
> T E S T S
> -------------------------------------------------------
> ...
> Results :
> Failed tests: testFailStartLevel(org.jboss.as.test.integration.osgi.deployment.DeferredResolveTestCase): Bundle ACTIVE expected:<32> but was:<4>
> Tests run: 120, Failures: 1, Errors: 0, Skipped: 6
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 2 months