3:47 p.m.
The problem with the hang when booting the all configuration
seems to be gone now, but I'm still seeing the clustering
take a long time to bootstrap.
It always seems to be stuck here:
"main" prio=1 tid=0x80b31a10 nid=0x44a4 in Object.wait()
[0x804d4000..0x804d6fb0]
at java.lang.Object.wait(Native Method)
- waiting on <0x98e62928> (a org.jgroups.util.Promise)
at org.jgroups.util.Promise.doWait(Promise.java:104)
at
org.jgroups.util.Promise._getResultWithTimeout(Promise.java:60)
at
org.jgroups.util.Promise.getResultWithTimeout(Promise.java:28)
- locked <0x98e62928> (a org.jgroups.util.Promise)
at org.jgroups.protocols.pbcast.FLUSH.startFlush(FLUSH.java:207)
It is doing this multiple times which leads to
a long total boot time:
15:37:50,931 INFO [ServerImpl] JBoss (Microcontainer) [5.0.0.Beta3
(build: SVNTag=JBoss_5_0_0_Beta3 date=200711081439)] Started in
5m:14s:880ms
Is there a problem with "flush" when there are no other members in the
cluster?
e.g.
15:34:14,466 WARN [JChannelFactory] Flush failed at 10.11.14.31:32795
DefaultPartition-JMS-CTRL
15:34:19,527 INFO [STDOUT]
-------------------------------------------------------
GMS: address is 10.11.14.31:32796
-------------------------------------------------------
15:34:43,545 WARN [JChannelFactory] Flush failed at 10.11.14.31:32796
DefaultPartition-JMS-DATA
--
xxxxxxxxxxxxxxxxxxxxxxxxxxxx
Adrian Brock
Chief Scientist
JBoss, a division of Red Hat
xxxxxxxxxxxxxxxxxxxxxxxxxxxx
4:06 p.m.
I am taking a look.
Adrian wrote:
The problem with the hang when booting the all configuration
seems to be gone now, but I'm still seeing the clustering
take a long time to bootstrap.
It always seems to be stuck here:
"main" prio=1 tid=0x80b31a10 nid=0x44a4 in Object.wait()
[0x804d4000..0x804d6fb0]
at java.lang.Object.wait(Native Method)
- waiting on <0x98e62928> (a org.jgroups.util.Promise)
at org.jgroups.util.Promise.doWait(Promise.java:104)
at
org.jgroups.util.Promise._getResultWithTimeout(Promise.java:60)
at
org.jgroups.util.Promise.getResultWithTimeout(Promise.java:28)
- locked <0x98e62928> (a org.jgroups.util.Promise)
at org.jgroups.protocols.pbcast.FLUSH.startFlush(FLUSH.java:207)
It is doing this multiple times which leads to
a long total boot time:
15:37:50,931 INFO [ServerImpl] JBoss (Microcontainer) [5.0.0.Beta3
(build: SVNTag=JBoss_5_0_0_Beta3 date=200711081439)] Started in
5m:14s:880ms
Is there a problem with "flush" when there are no other members in the
cluster?
e.g.
15:34:14,466 WARN [JChannelFactory] Flush failed at 10.11.14.31:32795
DefaultPartition-JMS-CTRL
15:34:19,527 INFO [STDOUT]
-------------------------------------------------------
GMS: address is 10.11.14.31:32796
-------------------------------------------------------
15:34:43,545 WARN [JChannelFactory] Flush failed at 10.11.14.31:32796
DefaultPartition-JMS-DATA
4:22 p.m.
Is your server bound to your VPN interface?
GMS: address is 10.11.14.31:32796
This sounds very similar to what the dev90 machine was experiencing when
doing hudson testsuite runs. There, multicast wasn't working properly
on the bound interface. FLUSH counts on getting back a response to its
own multicast, which it would never get. So, every startFlush would
hang for timeout * retries.
This gives me a chance to bring up something with the JGroups guys:
The behavior we see when FLUSH fails seems not so great -- long delay
and then a WARN. At least in some cases, like initial channel
connection, it would be better if it the connect() just failed. The
channel is not usable.
Adrian wrote:
The problem with the hang when booting the all configuration
seems to be gone now, but I'm still seeing the clustering
take a long time to bootstrap.
It always seems to be stuck here:
"main" prio=1 tid=0x80b31a10 nid=0x44a4 in Object.wait()
[0x804d4000..0x804d6fb0]
at java.lang.Object.wait(Native Method)
- waiting on <0x98e62928> (a org.jgroups.util.Promise)
at org.jgroups.util.Promise.doWait(Promise.java:104)
at
org.jgroups.util.Promise._getResultWithTimeout(Promise.java:60)
at
org.jgroups.util.Promise.getResultWithTimeout(Promise.java:28)
- locked <0x98e62928> (a org.jgroups.util.Promise)
at org.jgroups.protocols.pbcast.FLUSH.startFlush(FLUSH.java:207)
It is doing this multiple times which leads to
a long total boot time:
15:37:50,931 INFO [ServerImpl] JBoss (Microcontainer) [5.0.0.Beta3
(build: SVNTag=JBoss_5_0_0_Beta3 date=200711081439)] Started in
5m:14s:880ms
Is there a problem with "flush" when there are no other members in the
cluster?
e.g.
15:34:14,466 WARN [JChannelFactory] Flush failed at 10.11.14.31:32795
DefaultPartition-JMS-CTRL
15:34:19,527 INFO [STDOUT]
-------------------------------------------------------
GMS: address is 10.11.14.31:32796
-------------------------------------------------------
15:34:43,545 WARN [JChannelFactory] Flush failed at 10.11.14.31:32796
DefaultPartition-JMS-DATA
--
Brian Stansberry
Lead, AS Clustering
JBoss, a division of Red Hat
brian.stansberry(a)redhat.com
4:26 p.m.
Noted Brian. We over engineered FLUSH in 2.5 release and in 2.6 release
which is due next week these kind of issues should not happen.
I'll try AS-HEAD with JGroups HEAD before ship it as 2.6.
Brian Stansberry wrote:
Is your server bound to your VPN interface?
> GMS: address is 10.11.14.31:32796
This sounds very similar to what the dev90 machine was experiencing
when doing hudson testsuite runs. There, multicast wasn't working
properly on the bound interface. FLUSH counts on getting back a
response to its own multicast, which it would never get. So, every
startFlush would hang for timeout * retries.
This gives me a chance to bring up something with the JGroups guys:
The behavior we see when FLUSH fails seems not so great -- long delay
and then a WARN. At least in some cases, like initial channel
connection, it would be better if it the connect() just failed. The
channel is not usable.
5:43 p.m.
Note that FLUSH of 2.6 will get backported to 2.5
Vladimir Blagojevic wrote:
Noted Brian. We over engineered FLUSH in 2.5 release and in 2.6
release which is due next week these kind of issues should not happen.
I'll try AS-HEAD with JGroups HEAD before ship it as 2.6.
Brian Stansberry wrote:
> Is your server bound to your VPN interface?
>
> > GMS: address is 10.11.14.31:32796
>
> This sounds very similar to what the dev90 machine was experiencing
> when doing hudson testsuite runs. There, multicast wasn't working
> properly on the bound interface. FLUSH counts on getting back a
> response to its own multicast, which it would never get. So, every
> startFlush would hang for timeout * retries.
>
> This gives me a chance to bring up something with the JGroups guys:
>
> The behavior we see when FLUSH fails seems not so great -- long delay
> and then a WARN. At least in some cases, like initial channel
> connection, it would be better if it the connect() just failed. The
> channel is not usable.
_______________________________________________
jboss-development mailing list
jboss-development(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/jboss-development
--
Bela Ban
Lead JGroups / Clustering Team
JBoss - a division of Red Hat
4:35 p.m.
New subject: Not binding to localhost by default in jboss-head was: Re: Clustering bootstrap still taking a long time
On Thu, 2007-11-08 at 09:22 -0600, Brian Stansberry wrote:
Is your server bound to your VPN interface?
> GMS: address is 10.11.14.31:32796
Ok, then that reveals a different problem.
If I add "-b localhost", I see:
16:28:10,188 INFO [STDOUT]
-------------------------------------------------------
GMS: address is 127.0.0.1:32805
-------------------------------------------------------
16:28:12,523 INFO [STDOUT]
-------------------------------------------------------
GMS: address is 127.0.0.1:32807
-------------------------------------------------------
...
16:28:37,115 INFO [ServerImpl] JBoss (Microcontainer) [5.0.0.Beta3
(build: SVNTag=JBoss_5_0_0_Beta3 date=200711081439)] Started in
1m:42s:463ms
Not great, but better! :-)
However, "-b localhost" should be the default!
--
xxxxxxxxxxxxxxxxxxxxxxxxxxxx
Adrian Brock
Chief Scientist
JBoss, a division of Red Hat
xxxxxxxxxxxxxxxxxxxxxxxxxxxx
4:54 p.m.
New subject: Not binding to localhost by default in jboss-head was: Re: Clustering bootstrap still taking a long time
If this helps - I get a failure when running AS5 if my VPN is already
connected.
Adrian wrote:
On Thu, 2007-11-08 at 09:22 -0600, Brian Stansberry wrote:
> Is your server bound to your VPN interface?
>
> > GMS: address is 10.11.14.31:32796
Ok, then that reveals a different problem.
If I add "-b localhost", I see:
16:28:10,188 INFO [STDOUT]
-------------------------------------------------------
GMS: address is 127.0.0.1:32805
-------------------------------------------------------
16:28:12,523 INFO [STDOUT]
-------------------------------------------------------
GMS: address is 127.0.0.1:32807
-------------------------------------------------------
...
16:28:37,115 INFO [ServerImpl] JBoss (Microcontainer) [5.0.0.Beta3
(build: SVNTag=JBoss_5_0_0_Beta3 date=200711081439)] Started in
1m:42s:463ms
Not great, but better! :-)
However, "-b localhost" should be the default!
5:05 p.m.
New subject: Not binding to localhost by default in jboss-head was: Re: Clustering bootstrap still taking a long time
Try setting multicast path through localhost or a multicast capable
interface:
route add -net 224.0.0.0 netmask 240.0.0.0 lo
or
ip route add 224.0.0.0/4 dev lo
Ales Justin wrote, On 11/08/2007 05:54 PM (EEST):
If this helps - I get a failure when running AS5 if my VPN is
already
connected.
Adrian wrote:
> On Thu, 2007-11-08 at 09:22 -0600, Brian Stansberry wrote:
>> Is your server bound to your VPN interface?
>>
>> > GMS: address is 10.11.14.31:32796
>
> Ok, then that reveals a different problem.
> If I add "-b localhost", I see:
>
> 16:28:10,188 INFO [STDOUT]
> -------------------------------------------------------
> GMS: address is 127.0.0.1:32805
> -------------------------------------------------------
> 16:28:12,523 INFO [STDOUT]
> -------------------------------------------------------
> GMS: address is 127.0.0.1:32807
> -------------------------------------------------------
>
> ...
>
> 16:28:37,115 INFO [ServerImpl] JBoss (Microcontainer) [5.0.0.Beta3
> (build: SVNTag=JBoss_5_0_0_Beta3 date=200711081439)] Started in
> 1m:42s:463ms
>
> Not great, but better! :-)
>
>
> However, "-b localhost" should be the default!
_______________________________________________
jboss-development mailing list
jboss-development(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/jboss-development
5:44 p.m.
New subject: Not binding to localhost by default in jboss-head was: Re: Clustering bootstrap still taking a long time
Note that IP multicasting does *NOT* work over the VPN ! Even worse: on
my box, IP multicasting does *NOT* work over the interface which is used
by the VPN. Event if the VPN is NOT running !!
You have to de-install the VPN crap software to get this to work...
Cisco knows about this.
Ales Justin wrote:
If this helps - I get a failure when running AS5 if my VPN is already
connected.
Adrian wrote:
> On Thu, 2007-11-08 at 09:22 -0600, Brian Stansberry wrote:
>> Is your server bound to your VPN interface?
>>
>> > GMS: address is 10.11.14.31:32796
>
> Ok, then that reveals a different problem.
> If I add "-b localhost", I see:
>
> 16:28:10,188 INFO [STDOUT]
> -------------------------------------------------------
> GMS: address is 127.0.0.1:32805
> -------------------------------------------------------
> 16:28:12,523 INFO [STDOUT]
> -------------------------------------------------------
> GMS: address is 127.0.0.1:32807
> -------------------------------------------------------
>
> ...
>
> 16:28:37,115 INFO [ServerImpl] JBoss (Microcontainer) [5.0.0.Beta3
> (build: SVNTag=JBoss_5_0_0_Beta3 date=200711081439)] Started in
> 1m:42s:463ms
>
> Not great, but better! :-)
>
>
> However, "-b localhost" should be the default!
_______________________________________________
jboss-development mailing list
jboss-development(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/jboss-development
--
Bela Ban
Lead JGroups / Clustering Team
JBoss - a division of Red Hat
5:04 p.m.
New subject: Not binding to localhost by default in jboss-head was: Re: Clustering bootstrap still taking a long time
Adrian wrote:
On Thu, 2007-11-08 at 09:22 -0600, Brian Stansberry wrote:
> Is your server bound to your VPN interface?
>
> > GMS: address is 10.11.14.31:32796
Ok, then that reveals a different problem.
If I add "-b localhost", I see:
16:28:10,188 INFO [STDOUT]
-------------------------------------------------------
GMS: address is 127.0.0.1:32805
-------------------------------------------------------
16:28:12,523 INFO [STDOUT]
-------------------------------------------------------
GMS: address is 127.0.0.1:32807
-------------------------------------------------------
...
16:28:37,115 INFO [ServerImpl] JBoss (Microcontainer) [5.0.0.Beta3
(build: SVNTag=JBoss_5_0_0_Beta3 date=200711081439)] Started in
1m:42s:463ms
Not great, but better! :-)
However, "-b localhost" should be the default!
Yeah, it's not. If you don't specify -b or otherwise specify how
JGroups binds, JGroups picks the first non-loopback interface.
The only issue here is how much we care about whether we can still say
"go to two nodes and './run.sh -c all' and you'll have a cluster".
TBH,
I don't much care about that any more and have no objections to having
the AS set a localhost default. Would be interested in opinions of
others though.
(BTW, clustering JBM requires assigning a unique server peer id to each
node, so "go to two nodes and './run.sh -c all' and you'll have a
cluster" doesn't work anymore anyway. JBM guys might find a solution to
that though.)
--
Brian Stansberry
Lead, AS Clustering
JBoss, a division of Red Hat
brian.stansberry(a)redhat.com
5:29 p.m.
New subject: Not binding to localhost by default in jboss-head was: Re: Clustering bootstrap still taking a long time
> However, "-b localhost" should be the default!
Yes, that should be changed. The code in this regard should resemble that main() in
Branch_4_2.
Yeah, it's not. If you don't specify -b or otherwise specify
how
JGroups binds, JGroups picks the first non-loopback interface.
The only issue here is how much we care about whether we can still say
"go to two nodes and './run.sh -c all' and you'll have a cluster".
TBH,
I don't much care about that any more and have no objections to having
the AS set a localhost default. Would be interested in opinions of
others though.
You mean deep copy 'all' to all2 then:
run -c all -b x.y.z.1
run -c all2 -b x.y.z.2
That should work, like it does in AS 4.2, yes.
One glitch, is the "Post Office" of the new messaging. It prints out a
warning/stacktrace
when the 2nd instance in the cluster boots. I think that should work out of the box
without
problems.
(BTW, clustering JBM requires assigning a unique server peer id to each
node, so "go to two nodes and './run.sh -c all' and you'll have a
cluster" doesn't work anymore anyway. JBM guys might find a solution to
that though.)
5:24 p.m.
New subject: Not binding to localhost by default in jboss-head was: Re: Clustering bootstrap still taking a long time
I've fixed this in head.
http://jira.jboss.com/jira/browse/JBAS-4954
But I think the code in jboss-4.2 for the defaults for other
bind address is wrong.
It currently says:
// JBAS-4119, set the java.rmi.server.hostname if not set
if (System.getProperty("java.rmi.server.hostname") == null)
{
System.setProperty("java.rmi.server.hostname", "127.0.0.1");
}
This has two problems:
1) It doesn't set the other bind addresses
2) It doesn't necessarily agree with what the real bind address is.
In fact, it doesn't resolve the problem I'm seeing because
it doesn't set the jgroups.bind.addr property.
In jboss-head I've made the code:
// Fix up other bind addresses
String bindAddress =
System.getProperty(ServerConfig.SERVER_BIND_ADDRESS);
if (System.getProperty("java.rmi.server.hostname") == null)
System.setProperty("java.rmi.server.hostname", bindAddress);
if (System.getProperty("bind.address") == null)
System.setProperty("bind.address", bindAddress);
if (System.getProperty("jgroups.bind_addr") == null)
System.setProperty("jgroups.bind_addr", bindAddress);
Dimitris, is there a reason why you didn't fixup all
the bind addresses (see the -b handling code) in 4.2?
On Thu, 2007-11-08 at 16:35 +0100, Adrian wrote:
On Thu, 2007-11-08 at 09:22 -0600, Brian Stansberry wrote:
> Is your server bound to your VPN interface?
>
> > GMS: address is 10.11.14.31:32796
Ok, then that reveals a different problem.
If I add "-b localhost", I see:
16:28:10,188 INFO [STDOUT]
-------------------------------------------------------
GMS: address is 127.0.0.1:32805
-------------------------------------------------------
16:28:12,523 INFO [STDOUT]
-------------------------------------------------------
GMS: address is 127.0.0.1:32807
-------------------------------------------------------
...
16:28:37,115 INFO [ServerImpl] JBoss (Microcontainer) [5.0.0.Beta3
(build: SVNTag=JBoss_5_0_0_Beta3 date=200711081439)] Started in
1m:42s:463ms
Not great, but better! :-)
However, "-b localhost" should be the default!
--
xxxxxxxxxxxxxxxxxxxxxxxxxxxx
Adrian Brock
Chief Scientist
JBoss, a division of Red Hat
xxxxxxxxxxxxxxxxxxxxxxxxxxxx
5:36 p.m.
New subject: Not binding to localhost by default in jboss-head was: Re: Clustering bootstrap still taking a long time
Adrian wrote:
I've fixed this in head.
http://jira.jboss.com/jira/browse/JBAS-4954
But I think the code in jboss-4.2 for the defaults for other
bind address is wrong.
It currently says:
// JBAS-4119, set the java.rmi.server.hostname if not set
if (System.getProperty("java.rmi.server.hostname") == null)
{
System.setProperty("java.rmi.server.hostname",
"127.0.0.1");
}
This has two problems:
1) It doesn't set the other bind addresses
2) It doesn't necessarily agree with what the real bind address is.
In fact, it doesn't resolve the problem I'm seeing because
it doesn't set the jgroups.bind.addr property.
In jboss-head I've made the code:
// Fix up other bind addresses
String bindAddress =
System.getProperty(ServerConfig.SERVER_BIND_ADDRESS);
if (System.getProperty("java.rmi.server.hostname") == null)
System.setProperty("java.rmi.server.hostname", bindAddress);
if (System.getProperty("bind.address") == null)
System.setProperty("bind.address", bindAddress);
if (System.getProperty("jgroups.bind_addr") == null)
System.setProperty("jgroups.bind_addr", bindAddress);
Dimitris, is there a reason why you didn't fixup all
the bind addresses (see the -b handling code) in 4.2?
I don't remember changing the bind address myself in trunk? That was a TODO item :-)
They should be similar, yes.
5:38 p.m.
New subject: Not binding to localhost by default in jboss-head was: Re: Clustering bootstrap still taking a long time
Adrian wrote:
I've fixed this in head.
http://jira.jboss.com/jira/browse/JBAS-4954
But I think the code in jboss-4.2 for the defaults for other
bind address is wrong.
It currently says:
// JBAS-4119, set the java.rmi.server.hostname if not set
if (System.getProperty("java.rmi.server.hostname") == null)
{
System.setProperty("java.rmi.server.hostname",
"127.0.0.1");
}
This has two problems:
1) It doesn't set the other bind addresses
2) It doesn't necessarily agree with what the real bind address is.
In fact, it doesn't resolve the problem I'm seeing because
it doesn't set the jgroups.bind.addr property.
In jboss-head I've made the code:
// Fix up other bind addresses
String bindAddress =
System.getProperty(ServerConfig.SERVER_BIND_ADDRESS);
if (System.getProperty("java.rmi.server.hostname") == null)
System.setProperty("java.rmi.server.hostname", bindAddress);
if (System.getProperty("bind.address") == null)
System.setProperty("bind.address", bindAddress);
if (System.getProperty("jgroups.bind_addr") == null)
System.setProperty("jgroups.bind_addr", bindAddress);
Please don't do:
if (System.getProperty("bind.address") == null)
System.setProperty("bind.address", bindAddress);
That's a deprecated property replaced by jgroups.bind_addr; the only
reason it's mentioned at all in Main() is in case a user set it via -D.
Dimitris, is there a reason why you didn't fixup all
the bind addresses (see the -b handling code) in 4.2?
On Thu, 2007-11-08 at 16:35 +0100, Adrian wrote:
> On Thu, 2007-11-08 at 09:22 -0600, Brian Stansberry wrote:
>> Is your server bound to your VPN interface?
>>
>> > GMS: address is 10.11.14.31:32796
> Ok, then that reveals a different problem.
> If I add "-b localhost", I see:
>
> 16:28:10,188 INFO [STDOUT]
> -------------------------------------------------------
> GMS: address is 127.0.0.1:32805
> -------------------------------------------------------
> 16:28:12,523 INFO [STDOUT]
> -------------------------------------------------------
> GMS: address is 127.0.0.1:32807
> -------------------------------------------------------
>
> ...
>
> 16:28:37,115 INFO [ServerImpl] JBoss (Microcontainer) [5.0.0.Beta3
> (build: SVNTag=JBoss_5_0_0_Beta3 date=200711081439)] Started in
> 1m:42s:463ms
>
> Not great, but better! :-)
>
>
> However, "-b localhost" should be the default!
--
Brian Stansberry
Lead, AS Clustering
JBoss, a division of Red Hat
brian.stansberry(a)redhat.com
5:51 p.m.
New subject: Not binding to localhost by default in jboss-head was: Re: Clustering bootstrap still taking a long time
On Thu, 2007-11-08 at 10:38 -0600, Brian Stansberry wrote:
Please don't do:
> if (System.getProperty("bind.address") == null)
> System.setProperty("bind.address", bindAddress);
That's a deprecated property replaced by jgroups.bind_addr; the only
reason it's mentioned at all in Main() is in case a user set it via -D.
Ok, I've removed that one
(I don't really understand what problem it causes?)
and ported the "fix" to jboss-4.2
http://jira.jboss.com/jira/browse/JBAS-4955
--
xxxxxxxxxxxxxxxxxxxxxxxxxxxx
Adrian Brock
Chief Scientist
JBoss, a division of Red Hat
xxxxxxxxxxxxxxxxxxxxxxxxxxxx
6:25 p.m.
New subject: Not binding to localhost by default in jboss-head was: Re: Clustering bootstrap still taking a long time
Adrian wrote:
On Thu, 2007-11-08 at 10:38 -0600, Brian Stansberry wrote:
> Please don't do:
>
> > if (System.getProperty("bind.address") == null)
> > System.setProperty("bind.address", bindAddress);
>
> That's a deprecated property replaced by jgroups.bind_addr; the only
> reason it's mentioned at all in Main() is in case a user set it via -D.
>
Ok, I've removed that one
(I don't really understand what problem it causes?)
and ported the "fix" to jboss-4.2
http://jira.jboss.com/jira/browse/JBAS-4955
Thanks. It doesn't really cause any problem, it's just one less piece of
cruft hanging around.
--
Brian Stansberry
Lead, AS Clustering
JBoss, a division of Red Hat
brian.stansberry(a)redhat.com
6007
days inactive
6007
days old
jboss-development@lists.jboss.org
15 comments
7 participants
participants (7)
-
Adrian -
Aleksandar Kostadinov -
Ales Justin -
Bela Ban -
Brian Stansberry -
Dimitris Andreadis -
Vladimir Blagojevic