[JBoss JIRA] Commented: (JGRP-244) long connect time in application that frequently closes and opens new channel
by Bela Ban (JIRA)
[ http://jira.jboss.com/jira/browse/JGRP-244?page=comments#action_12341047 ]
Bela Ban commented on JGRP-244:
-------------------------------
This is quite a tricky issue, as I recall that I removed the 'first' field for a reason ! Needs further investigation...
> long connect time in application that frequently closes and opens new channel
> -----------------------------------------------------------------------------
>
> Key: JGRP-244
> URL: http://jira.jboss.com/jira/browse/JGRP-244
> Project: JGroups
> Issue Type: Bug
> Affects Versions: 2.2.8, 2.2.9, 2.3, 2.2.9.1, 2.2.9.2
> Environment: SLES 9
> Reporter: Bruce Schuchardt
> Assigned To: Bela Ban
> Priority: Minor
> Fix For: 2.4
>
>
> I have an application that has one long-lived jgroups member and four other processes that often close their channel and create a new one. Most of the time these new connections are formed in a dozen milliseconds or so, but sometimes they're taking over 20 seconds. The apps are using TCPGOSSIP with multicast turned off.
> I turned on tracing and saw that the coordinator's UNICAST was having some trouble. It got a message from a departed member that it stored up and dispatched later when the departed member's address was reused by a new channel.
> a) A member left the view and UNICAST removed its connection for the member and added it to previous_members.
> b) Another message then arrived from the member, and UNICAST created a new connection for it. The message had seqno 4, and was put in the AckReceiverWindow and not passed up.
> c) A few seconds later, a process created a new channel and it got the same ID as the one the coordinator's UNICAST just dealt with.
> d) The new channel sent three UNICAST messages to the coordinator. On the third message, the coordinator's UNICAST removed #3 and the old #4 and passed them both up.
> e) The new channel sent message #4, a JOIN_REQ, and UNICAST discarded it
> The new channel eventually goes through discovery again and gets into the group, but it adds quite a bit to channel startup time, and I'm a little worried that there might be a case where a much higher seqno gets trapped in the receiver window like this.
> I fixed this for my needs by changing UNICAST.handleDataReceived to reject a previous_member message if the seqno is higher than the default initial seqno, but I suppose that might wreak havoc with some other algorithms.
> private void handleDataReceived(Object sender, long seqno, Message msg) {
> if(trace)
> log.trace(new StringBuffer().append(local_addr).append(" <-- DATA(").append(sender).append(": #").append(seqno));
> if(previous_members.contains(sender)) {
> // we don't want to see messages from departed members
> if (seqno > DEFAULT_FIRST_SEQNO) {
> if (trace)
> log.trace("discarding message " + seqno + " from previous member " + sender);
> return;
> }
> if(trace)
> log.trace("removed " + sender + " from previous_members as we received a message from it");
> previous_members.removeElement(sender);
> }
> Entry entry;
> synchronized(connections) {
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
19 years, 11 months
[JBoss JIRA] Commented: (JGRP-244) long connect time in application that frequently closes and opens new channel
by Bela Ban (JIRA)
[ http://jira.jboss.com/jira/browse/JGRP-244?page=comments#action_12341046 ]
Bela Ban commented on JGRP-244:
-------------------------------
A better solution is probably to re-introduce the 'first' field in UnicastHeader, from JGroups/doc/design/UNICAST.txt:
Author: Bela Ban
Date: Aug 10 2006
Version: $Id: UNICAST.txt,v 1.1 2006/08/10 12:09:18 belaban Exp $
Sending a unicast message to P
------------------------------
- When sending the first unicast message to a member P, we tag it with first=true
- The message is added to the AckSenderWindow, which keeps resending the message until
it is acked by P, or P leaves the group (or crashes)
View change
-----------
When a member P leaves, the connection for P is removed from the connection table
Receiving a unicast message from P
----------------------------------
- When receiving a message from P:
- If there is no entry for P in the connection table:
- If the message has first=true:
- Create a new entry for P in the connection table
- Add the message to the entry for P (AckReceiverWindow) and ack it back to P
- Else:
- Discard the message and do *not* ack it ! This is necessary, so P keeps retransmitting it !
- Else:
- Add the message to the entry for P (AckReceiverWindow) and ack it back to P
> long connect time in application that frequently closes and opens new channel
> -----------------------------------------------------------------------------
>
> Key: JGRP-244
> URL: http://jira.jboss.com/jira/browse/JGRP-244
> Project: JGroups
> Issue Type: Bug
> Affects Versions: 2.2.8, 2.2.9, 2.3, 2.2.9.1, 2.2.9.2
> Environment: SLES 9
> Reporter: Bruce Schuchardt
> Assigned To: Bela Ban
> Priority: Minor
> Fix For: 2.4
>
>
> I have an application that has one long-lived jgroups member and four other processes that often close their channel and create a new one. Most of the time these new connections are formed in a dozen milliseconds or so, but sometimes they're taking over 20 seconds. The apps are using TCPGOSSIP with multicast turned off.
> I turned on tracing and saw that the coordinator's UNICAST was having some trouble. It got a message from a departed member that it stored up and dispatched later when the departed member's address was reused by a new channel.
> a) A member left the view and UNICAST removed its connection for the member and added it to previous_members.
> b) Another message then arrived from the member, and UNICAST created a new connection for it. The message had seqno 4, and was put in the AckReceiverWindow and not passed up.
> c) A few seconds later, a process created a new channel and it got the same ID as the one the coordinator's UNICAST just dealt with.
> d) The new channel sent three UNICAST messages to the coordinator. On the third message, the coordinator's UNICAST removed #3 and the old #4 and passed them both up.
> e) The new channel sent message #4, a JOIN_REQ, and UNICAST discarded it
> The new channel eventually goes through discovery again and gets into the group, but it adds quite a bit to channel startup time, and I'm a little worried that there might be a case where a much higher seqno gets trapped in the receiver window like this.
> I fixed this for my needs by changing UNICAST.handleDataReceived to reject a previous_member message if the seqno is higher than the default initial seqno, but I suppose that might wreak havoc with some other algorithms.
> private void handleDataReceived(Object sender, long seqno, Message msg) {
> if(trace)
> log.trace(new StringBuffer().append(local_addr).append(" <-- DATA(").append(sender).append(": #").append(seqno));
> if(previous_members.contains(sender)) {
> // we don't want to see messages from departed members
> if (seqno > DEFAULT_FIRST_SEQNO) {
> if (trace)
> log.trace("discarding message " + seqno + " from previous member " + sender);
> return;
> }
> if(trace)
> log.trace("removed " + sender + " from previous_members as we received a message from it");
> previous_members.removeElement(sender);
> }
> Entry entry;
> synchronized(connections) {
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
19 years, 11 months
[JBoss JIRA] Updated: (JGRP-244) long connect time in application that frequently closes and opens new channel
by Bela Ban (JIRA)
[ http://jira.jboss.com/jira/browse/JGRP-244?page=all ]
Bela Ban updated JGRP-244:
--------------------------
Comment: was deleted
> long connect time in application that frequently closes and opens new channel
> -----------------------------------------------------------------------------
>
> Key: JGRP-244
> URL: http://jira.jboss.com/jira/browse/JGRP-244
> Project: JGroups
> Issue Type: Bug
> Affects Versions: 2.2.8, 2.2.9, 2.3, 2.2.9.1, 2.2.9.2
> Environment: SLES 9
> Reporter: Bruce Schuchardt
> Assigned To: Bela Ban
> Priority: Minor
> Fix For: 2.4
>
>
> I have an application that has one long-lived jgroups member and four other processes that often close their channel and create a new one. Most of the time these new connections are formed in a dozen milliseconds or so, but sometimes they're taking over 20 seconds. The apps are using TCPGOSSIP with multicast turned off.
> I turned on tracing and saw that the coordinator's UNICAST was having some trouble. It got a message from a departed member that it stored up and dispatched later when the departed member's address was reused by a new channel.
> a) A member left the view and UNICAST removed its connection for the member and added it to previous_members.
> b) Another message then arrived from the member, and UNICAST created a new connection for it. The message had seqno 4, and was put in the AckReceiverWindow and not passed up.
> c) A few seconds later, a process created a new channel and it got the same ID as the one the coordinator's UNICAST just dealt with.
> d) The new channel sent three UNICAST messages to the coordinator. On the third message, the coordinator's UNICAST removed #3 and the old #4 and passed them both up.
> e) The new channel sent message #4, a JOIN_REQ, and UNICAST discarded it
> The new channel eventually goes through discovery again and gets into the group, but it adds quite a bit to channel startup time, and I'm a little worried that there might be a case where a much higher seqno gets trapped in the receiver window like this.
> I fixed this for my needs by changing UNICAST.handleDataReceived to reject a previous_member message if the seqno is higher than the default initial seqno, but I suppose that might wreak havoc with some other algorithms.
> private void handleDataReceived(Object sender, long seqno, Message msg) {
> if(trace)
> log.trace(new StringBuffer().append(local_addr).append(" <-- DATA(").append(sender).append(": #").append(seqno));
> if(previous_members.contains(sender)) {
> // we don't want to see messages from departed members
> if (seqno > DEFAULT_FIRST_SEQNO) {
> if (trace)
> log.trace("discarding message " + seqno + " from previous member " + sender);
> return;
> }
> if(trace)
> log.trace("removed " + sender + " from previous_members as we received a message from it");
> previous_members.removeElement(sender);
> }
> Entry entry;
> synchronized(connections) {
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
19 years, 11 months
[JBoss JIRA] Commented: (JGRP-244) long connect time in application that frequently closes and opens new channel
by Bela Ban (JIRA)
[ http://jira.jboss.com/jira/browse/JGRP-244?page=comments#action_12341045 ]
Bela Ban commented on JGRP-244:
-------------------------------
A better solution is probably to re-introduce the 'first' field in UnicastHeader, from JGroups/doc/design/UNICAST.txt:
Author: Bela Ban
Date: Aug 10 2006
Version: $Id: UNICAST.txt,v 1.1 2006/08/10 12:09:18 belaban Exp $
Sending a unicast message to P
------------------------------
- When sending the first unicast message to a member P, we tag it with first=true
- The message is added to the AckSenderWindow, which keeps resending the message until
it is acked by P, or P leaves the group (or crashes)
View change
-----------
When a member P leaves, the connection for P is removed from the connection table
Receiving a unicast message from P
----------------------------------
- When receiving a message from P:
- If there is no entry for P in the connection table:
- If the message has first=true:
- Create a new entry for P in the connection table
- Add the message to the entry for P (AckReceiverWindow) and ack it back to P
- Else:
- Discard the message and do *not* ack it ! This is necessary, so P keeps retransmitting it !
- Else:
> long connect time in application that frequently closes and opens new channel
> -----------------------------------------------------------------------------
>
> Key: JGRP-244
> URL: http://jira.jboss.com/jira/browse/JGRP-244
> Project: JGroups
> Issue Type: Bug
> Affects Versions: 2.2.8, 2.2.9, 2.3, 2.2.9.1, 2.2.9.2
> Environment: SLES 9
> Reporter: Bruce Schuchardt
> Assigned To: Bela Ban
> Priority: Minor
> Fix For: 2.4
>
>
> I have an application that has one long-lived jgroups member and four other processes that often close their channel and create a new one. Most of the time these new connections are formed in a dozen milliseconds or so, but sometimes they're taking over 20 seconds. The apps are using TCPGOSSIP with multicast turned off.
> I turned on tracing and saw that the coordinator's UNICAST was having some trouble. It got a message from a departed member that it stored up and dispatched later when the departed member's address was reused by a new channel.
> a) A member left the view and UNICAST removed its connection for the member and added it to previous_members.
> b) Another message then arrived from the member, and UNICAST created a new connection for it. The message had seqno 4, and was put in the AckReceiverWindow and not passed up.
> c) A few seconds later, a process created a new channel and it got the same ID as the one the coordinator's UNICAST just dealt with.
> d) The new channel sent three UNICAST messages to the coordinator. On the third message, the coordinator's UNICAST removed #3 and the old #4 and passed them both up.
> e) The new channel sent message #4, a JOIN_REQ, and UNICAST discarded it
> The new channel eventually goes through discovery again and gets into the group, but it adds quite a bit to channel startup time, and I'm a little worried that there might be a case where a much higher seqno gets trapped in the receiver window like this.
> I fixed this for my needs by changing UNICAST.handleDataReceived to reject a previous_member message if the seqno is higher than the default initial seqno, but I suppose that might wreak havoc with some other algorithms.
> private void handleDataReceived(Object sender, long seqno, Message msg) {
> if(trace)
> log.trace(new StringBuffer().append(local_addr).append(" <-- DATA(").append(sender).append(": #").append(seqno));
> if(previous_members.contains(sender)) {
> // we don't want to see messages from departed members
> if (seqno > DEFAULT_FIRST_SEQNO) {
> if (trace)
> log.trace("discarding message " + seqno + " from previous member " + sender);
> return;
> }
> if(trace)
> log.trace("removed " + sender + " from previous_members as we received a message from it");
> previous_members.removeElement(sender);
> }
> Entry entry;
> synchronized(connections) {
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
19 years, 11 months
[JBoss JIRA] Created: (JBMICROCONT-100) LifecycleActions are not dispatched using the target ClassLoader
by Scott M Stark (JIRA)
LifecycleActions are not dispatched using the target ClassLoader
----------------------------------------------------------------
Key: JBMICROCONT-100
URL: http://jira.jboss.com/jira/browse/JBMICROCONT-100
Project: JBoss MicroContainer
Issue Type: Bug
Components: Kernel
Affects Versions: JBossMC_2_0_0M1
Reporter: Scott M Stark
For the pojoserver bootstrap work, I'm using a JMXKernel bean that uses a custom class loader which adds the jmx classes to the bean classpath:
<!-- Load the jboss jmx classes -->
<bean name="JMXClassLoader" class="org.jboss.vfs.classloading.VFSClassLoader">
<constructor factoryClass="org.jboss.vfs.classloading.VFSClassLoaderFactory"
factoryMethod="newClassLoader">
<parameter class="java.net.URL">${jboss.lib.url}</parameter>
<parameter>
<array class="[Ljava.lang.String;" elementClass="java.lang.String">
<value>dom4j.jar</value>
<value>jboss-j2se.jar</value>
<value>jboss-jmx.jar</value>
<value>jboss-mbeans.jar</value>
<value>jboss-system-jmx.jar</value>
</array>
</parameter>
<parameter><inject bean="MainDeployer" property="VFSFactory"/></parameter>
</constructor>
</bean>
<!-- A bean that setups a jboss-4.0.x type of jmx kernel to support the
the legacy mbean deployments handled by the SARDeployer
-->
<bean name="JMXKernel" class="org.jboss.system.server.jmx.JMXKernel">
<classloader><inject bean="JMXClassLoader"/></classloader>
<property name="serverImpl"><inject bean="org.jboss.system.server.Server"/></property>
</bean>
In the start method o the JMXKernel it is failing to create a MBeanServer instance because the thread context class loader is the ServerLoader NoAnnotationURLClassLoader rather than the JMXClassLoader instance.
I believe we need to be setting the bean ClassLoader somewhere along the dispatch path.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
19 years, 11 months
[JBoss JIRA] Created: (JBAS-3506) Maintain the server logs for the different testsuite clustering configs
by Brian Stansberry (JIRA)
Maintain the server logs for the different testsuite clustering configs
-----------------------------------------------------------------------
Key: JBAS-3506
URL: http://jira.jboss.com/jira/browse/JBAS-3506
Project: JBoss Application Server
Issue Type: Feature Request
Security Level: Public (Everyone can see)
Components: Test Suite
Reporter: Brian Stansberry
Assigned To: Brian Stansberry
Priority: Minor
Fix For: JBossAS-4.0.5.CR1, JBossAS-5.0.0.Beta
Previously the clustering tests worked by creating 2 configs "node0" and "node1" and launching 2 jboss instances, one using each.
To execute all the required tests, the build script would tear down and rebuild node0 and node1, altering various aspects of the config (e.g. use REPL_SYNC instead of REPL_ASYNC).
The problem with this is that when a version of node0/1 got torn down and replaced, it's server logs went with it. This makes subsequent debugging of any failures difficult, particularly with cruisecontrol runs that sometimes reveal problems we don't see on our local machines.
To fix this, I'm no longer using "node0/node1". Rather, each config will be given a name that somewhat describes what it is, e.g.:
cluster-UDP-0 and cluster-UDP-1
cluster-field-UDP-0/1 (configured for FIELD repl)
cluster-UDP-SYNC-0/1 (configured for REPL_SYNC)
cluster-field-UDP-SYNC-0/1
cluster-UDP-BR-0/1 (configured for BuddyReplication)
cluster-field-UDP-BR-0/1
Equivalent configs will be created for the tests running the TCP stack.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
19 years, 11 months
[JBoss JIRA] Created: (JBMICROCONT-101) Need better error reporting of circular dependencies
by Scott M Stark (JIRA)
Need better error reporting of circular dependencies
----------------------------------------------------
Key: JBMICROCONT-101
URL: http://jira.jboss.com/jira/browse/JBMICROCONT-101
Project: JBoss MicroContainer
Issue Type: Feature Request
Components: Dependency
Affects Versions: JBossMC_2_0_0M1
Reporter: Scott M Stark
Fix For: JBossMC_2_0_0M1
I switched around some dependencies and ended up producing a circular dependency:
<?xml version="1.0" encoding="UTF-8"?>
<!-- A ProfileServiceBootstrap bootstrap descriptor that uses the simple
VFSDeploymentScanner that goes directly to the MainDeployer rather than the
ProfileService.
$Id$
-->
<deployment xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="urn:jboss:bean-deployer:2.0 bean-deployer_2_0.xsd"
xmlns="urn:jboss:bean-deployer:2.0">
<!-- -->
<bean name="ProfileService" class="org.jboss.system.server.profileservice.ProfileServiceImpl">
<constructor>
<parameter>${jboss.server.name}</parameter>
</constructor>
<property name="profileRoot">${jboss.server.home.url}/profile</property>
</bean>
<!-- The MainDeployer -->
<bean name="MainDeployer" class="org.jboss.deployers.plugins.MainDeployerImpl">
<property name="VFSFactory"><inject bean="VFSFactory"/></property>
<property name="deployers">
<array class="[Lorg.jboss.deployers.spi.AspectDeployer;" elementClass="org.jboss.deployers.spi.AspectDeployer">
<inject bean="ClassLoadingDeployer"/>
<inject bean="BeanDeployer"/>
<inject bean="SARDeployer"/>
</array>
</property>
<supply>Deployers</supply>
</bean>
<!-- The factory for VFS instances -->
<bean name="VFSFactory" class="org.jboss.vfs.file.DefaultVFSFactory">
</bean>
<!-- Deployers. Declaration order defines the deployer chain order
-->
<bean name="ClassLoadingDeployer" class="org.jboss.deployers.plugins.classloading.ClassLoadingDeployer">
</bean>
<bean name="BeanDeployer" class="org.jboss.deployers.plugins.bean.BeanDeployer">
</bean>
<!-- Load the jboss jmx classes -->
<bean name="JMXClassLoader" class="org.jboss.vfs.classloading.VFSClassLoader">
<constructor factoryClass="org.jboss.vfs.classloading.VFSClassLoaderFactory"
factoryMethod="newClassLoader">
<parameter class="java.net.URL">${jboss.lib.url}</parameter>
<parameter>
<array class="[Ljava.lang.String;" elementClass="java.lang.String">
<value>dom4j.jar</value>
<value>jboss-j2se.jar</value>
<value>jboss-jmx.jar</value>
<value>jboss-mbeans.jar</value>
<value>jboss-system-jmx.jar</value>
</array>
</parameter>
<parameter><inject bean="MainDeployer" property="VFSFactory"/></parameter>
</constructor>
</bean>
<bean name="SARDeployer" class="org.jboss.deployment.SARDeployer">
<classloader><inject bean="JMXClassLoader"/></classloader>
<property name="useNamespaceAwareParser">true</property>
<property name="server"><inject bean="JMXKernel" property="mbeanServer" /></property>
<property name="serviceController"><inject bean="JMXKernel" property="serviceController" /></property>
</bean>
<!-- A bean that setups a jboss-4.0.x type of jmx kernel to support the
the legacy mbean deployments handled by the SARDeployer
-->
<bean name="JMXKernel" class="org.jboss.system.server.jmx.JMXKernel">
<classloader><inject bean="JMXClassLoader"/></classloader>
<property name="serverImpl"><inject bean="org.jboss.system.server.Server"/></property>
</bean>
<!-- Hot deployment service that directly uses the MainDeployer -->
<bean name="VFSDeploymentScanner" class="org.jboss.deployers.plugins.scanner.VFSDeploymentScannerImpl">
<property name="mainDeployer"><inject bean="MainDeployer"/></property>
<property name="VFSFactory"><inject bean="MainDeployer" property="VFSFactory" /></property>
<property name="URIList">
<list elementClass="java.net.URI"><value>${jboss.server.home.url}/deploy/</value></list>
</property>
<property name="scanPeriod"><value>500000</value></property>
<demand state="Instantiated">Deployers</demand>
</bean>
</deployment>
All that was reported was a NPE due to the target of the demand=Deployers not being Instantiated:
12:37:31,477 DEBUG [ServerImpl] Failed to start
java.lang.IllegalArgumentException: Null name
at org.jboss.dependency.plugins.AbstractController.getContext(AbstractController.java:101)
at org.jboss.kernel.plugins.dependency.AbstractKernelController.getContext(AbstractKernelController.java:94)
at org.jboss.kernel.plugins.deployment.AbstractKernelDeployer.internalValidate(AbstractKernelDeployer.java:229)
at org.jboss.kernel.plugins.deployment.AbstractKernelDeployer.validate(AbstractKernelDeployer.java:161)
at org.jboss.system.server.profileservice.ProfileServiceBootstrap.bootstrap(ProfileServiceBootstrap.java:202)
at org.jboss.kernel.plugins.bootstrap.AbstractBootstrap.run(AbstractBootstrap.java:89)
at org.jboss.system.server.profileservice.ServerImpl.doStart(ServerImpl.java:371)
at org.jboss.system.server.profileservice.ServerImpl.start(ServerImpl.java:323)
at org.jboss.Main.boot(Main.java:210)
at org.jboss.Main$1.run(Main.java:488)
at java.lang.Thread.run()V(Unknown Source)
What should be reported is the bean which supplies the demand and its state not matching and why.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
19 years, 11 months