[JBoss JIRA] Commented: (JGRP-130) Problems with reincarnation
by Bela Ban (JIRA)
[ http://jira.jboss.com/jira/browse/JGRP-130?page=comments#action_12341057 ]
Bela Ban commented on JGRP-130:
-------------------------------
This should be fixed by simply (a) choosing different mcast_addr/mcast_port for each developer, or (b) (even simpler) choosing an ip_ttl of 1, so multicasts don't go beyond the local box. If every developer uses ip_ttl=1, this should fix your problem
> Problems with reincarnation
> ---------------------------
>
> Key: JGRP-130
> URL: http://jira.jboss.com/jira/browse/JGRP-130
> Project: JGroups
> Issue Type: Feature Request
> Affects Versions: 2.2.9
> Reporter: Bela Ban
> Assigned To: Bela Ban
> Fix For: 2.5
>
>
> Problems with reincarnation
> ===========================
> Author: Bela Ban
> Version: $Id$
> The identity of a JGroups member is always the IP address and a port. The port is usually chosen by the OS, unless
> bind_port is set (not set by default).
> Let's say a member's address is hostA:5000. When that member dies and is restarted, the OS will likely assign a
> higher port, say 5002. This depends on how many other processes requested a port in between the start and restart
> of the member.
> JGroups relies on the fact that the assignment of ports by the OS is always (not necessarily monotonically)
> *increasing* across a single machine. If this is not the case, then the following problems can occur:
> 1. Restart:
> When a member P crashes and then is restarted, if FD is used and P is restarted *before* it is excluded,
> then we have a new member *under the same old address* ! Since it lost all of its state (e.g. retransmission table),
> retransmission requests sent to the new P will fail.
> 2. Shunning:
> Regarding shunning: a member keeps its last N (default is 100) ports used, and makes sure it doesn't reuse one of
> those already-used ports when it is shunned. However, this is process-wide and *not* machine-wide, e.g. when we have
> processes P1 on A:5000 and P2 on A:5002 (on machine A), and both of them are shunned at the same time,
> when they rejoin, P1 does not use port 5000, but might use port 5002, and P2 doesn't use 5002, but might use 5000, so
> they could assume each other's identity !
> Both problems cannot be solved by remembering the last 100 ports: in case #1, this list is lost because we start a
> new process and in case #2, the list is process-wide, but not machine-wide.
> Again, these problems occur *only* when the OS reuses previously assigned ports.
> SOLUTION:
> A: Use temporary storage (per host) to store the last N addresses assigned on a given host. This makes sure we
> don't reuse previous addresses
> B: Use logical addresses, such as java.rmi.VMID or java.rmi.server.UID, which are unique over time for a given host.
> Then, it doesn't matter what ports we use because the ports are not used to determine a member's identity.
> The JIRA task for logical addresses is http://jira.jboss.com/jira/browse/JGRP-129.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
19 years, 11 months
[JBoss JIRA] Commented: (JGRP-130) Problems with reincarnation
by Ortwin Glück (JIRA)
[ http://jira.jboss.com/jira/browse/JGRP-130?page=comments#action_12341056 ]
Ortwin Glück commented on JGRP-130:
-----------------------------------
This bug is annoying our team very much. We have 10 developers running a local JBoss instance for development. They need to listen on their LAN address, as they need to be accessed via the Internet for testing. We also have an integration test machine in the same network, which is running 24/7. The developers may kill their JBoss instances every now and then because of the EJB3 redeployment memory leak. So we see this bug a lot!
The problem is everytime, to find the JGroups coordinator node (which is mostly the integration test machine as it's the one running the for longest time) and restart that. That's really, really annoying. Please fix this bug earlier than planned.
> Problems with reincarnation
> ---------------------------
>
> Key: JGRP-130
> URL: http://jira.jboss.com/jira/browse/JGRP-130
> Project: JGroups
> Issue Type: Feature Request
> Affects Versions: 2.2.9
> Reporter: Bela Ban
> Assigned To: Bela Ban
> Fix For: 2.5
>
>
> Problems with reincarnation
> ===========================
> Author: Bela Ban
> Version: $Id$
> The identity of a JGroups member is always the IP address and a port. The port is usually chosen by the OS, unless
> bind_port is set (not set by default).
> Let's say a member's address is hostA:5000. When that member dies and is restarted, the OS will likely assign a
> higher port, say 5002. This depends on how many other processes requested a port in between the start and restart
> of the member.
> JGroups relies on the fact that the assignment of ports by the OS is always (not necessarily monotonically)
> *increasing* across a single machine. If this is not the case, then the following problems can occur:
> 1. Restart:
> When a member P crashes and then is restarted, if FD is used and P is restarted *before* it is excluded,
> then we have a new member *under the same old address* ! Since it lost all of its state (e.g. retransmission table),
> retransmission requests sent to the new P will fail.
> 2. Shunning:
> Regarding shunning: a member keeps its last N (default is 100) ports used, and makes sure it doesn't reuse one of
> those already-used ports when it is shunned. However, this is process-wide and *not* machine-wide, e.g. when we have
> processes P1 on A:5000 and P2 on A:5002 (on machine A), and both of them are shunned at the same time,
> when they rejoin, P1 does not use port 5000, but might use port 5002, and P2 doesn't use 5002, but might use 5000, so
> they could assume each other's identity !
> Both problems cannot be solved by remembering the last 100 ports: in case #1, this list is lost because we start a
> new process and in case #2, the list is process-wide, but not machine-wide.
> Again, these problems occur *only* when the OS reuses previously assigned ports.
> SOLUTION:
> A: Use temporary storage (per host) to store the last N addresses assigned on a given host. This makes sure we
> don't reuse previous addresses
> B: Use logical addresses, such as java.rmi.VMID or java.rmi.server.UID, which are unique over time for a given host.
> Then, it doesn't matter what ports we use because the ports are not used to determine a member's identity.
> The JIRA task for logical addresses is http://jira.jboss.com/jira/browse/JGRP-129.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
19 years, 11 months
[JBoss JIRA] Commented: (JGRP-244) long connect time in application that frequently closes and opens new channel
by Bela Ban (JIRA)
[ http://jira.jboss.com/jira/browse/JGRP-244?page=comments#action_12341055 ]
Bela Ban commented on JGRP-244:
-------------------------------
Okay, I applied the patch anyway because it might affect UDP (small chance though because with UDP, we don't allocate the same Addresses, although this can be done using bind_port).
One change though: once we drop a message because it is from a previous member and its seqno is not 1, then we *must not* ack it ! Otherwise we run into http://jira.jboss.com/jira/browse/JGRP-126, where a discarded message is removed from the sender's table and never retransmitted, so the receiver blocks forever !
> long connect time in application that frequently closes and opens new channel
> -----------------------------------------------------------------------------
>
> Key: JGRP-244
> URL: http://jira.jboss.com/jira/browse/JGRP-244
> Project: JGroups
> Issue Type: Bug
> Affects Versions: 2.2.8, 2.2.9, 2.3, 2.2.9.1, 2.2.9.2
> Environment: SLES 9
> Reporter: Bruce Schuchardt
> Assigned To: Bela Ban
> Fix For: 2.4
>
>
> I have an application that has one long-lived jgroups member and four other processes that often close their channel and create a new one. Most of the time these new connections are formed in a dozen milliseconds or so, but sometimes they're taking over 20 seconds. The apps are using TCPGOSSIP with multicast turned off.
> I turned on tracing and saw that the coordinator's UNICAST was having some trouble. It got a message from a departed member that it stored up and dispatched later when the departed member's address was reused by a new channel.
> a) A member left the view and UNICAST removed its connection for the member and added it to previous_members.
> b) Another message then arrived from the member, and UNICAST created a new connection for it. The message had seqno 4, and was put in the AckReceiverWindow and not passed up.
> c) A few seconds later, a process created a new channel and it got the same ID as the one the coordinator's UNICAST just dealt with.
> d) The new channel sent three UNICAST messages to the coordinator. On the third message, the coordinator's UNICAST removed #3 and the old #4 and passed them both up.
> e) The new channel sent message #4, a JOIN_REQ, and UNICAST discarded it
> The new channel eventually goes through discovery again and gets into the group, but it adds quite a bit to channel startup time, and I'm a little worried that there might be a case where a much higher seqno gets trapped in the receiver window like this.
> I fixed this for my needs by changing UNICAST.handleDataReceived to reject a previous_member message if the seqno is higher than the default initial seqno, but I suppose that might wreak havoc with some other algorithms.
> private void handleDataReceived(Object sender, long seqno, Message msg) {
> if(trace)
> log.trace(new StringBuffer().append(local_addr).append(" <-- DATA(").append(sender).append(": #").append(seqno));
> if(previous_members.contains(sender)) {
> // we don't want to see messages from departed members
> if (seqno > DEFAULT_FIRST_SEQNO) {
> if (trace)
> log.trace("discarding message " + seqno + " from previous member " + sender);
> return;
> }
> if(trace)
> log.trace("removed " + sender + " from previous_members as we received a message from it");
> previous_members.removeElement(sender);
> }
> Entry entry;
> synchronized(connections) {
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
19 years, 11 months
[JBoss JIRA] Commented: (JGRP-244) long connect time in application that frequently closes and opens new channel
by Bela Ban (JIRA)
[ http://jira.jboss.com/jira/browse/JGRP-244?page=comments#action_12341051 ]
Bela Ban commented on JGRP-244:
-------------------------------
One question: why are you using UNICAST with TCP ? You don't need UNICAST if you run TCP as transport !
> long connect time in application that frequently closes and opens new channel
> -----------------------------------------------------------------------------
>
> Key: JGRP-244
> URL: http://jira.jboss.com/jira/browse/JGRP-244
> Project: JGroups
> Issue Type: Bug
> Affects Versions: 2.2.8, 2.2.9, 2.3, 2.2.9.1, 2.2.9.2
> Environment: SLES 9
> Reporter: Bruce Schuchardt
> Assigned To: Bela Ban
> Fix For: 2.4
>
>
> I have an application that has one long-lived jgroups member and four other processes that often close their channel and create a new one. Most of the time these new connections are formed in a dozen milliseconds or so, but sometimes they're taking over 20 seconds. The apps are using TCPGOSSIP with multicast turned off.
> I turned on tracing and saw that the coordinator's UNICAST was having some trouble. It got a message from a departed member that it stored up and dispatched later when the departed member's address was reused by a new channel.
> a) A member left the view and UNICAST removed its connection for the member and added it to previous_members.
> b) Another message then arrived from the member, and UNICAST created a new connection for it. The message had seqno 4, and was put in the AckReceiverWindow and not passed up.
> c) A few seconds later, a process created a new channel and it got the same ID as the one the coordinator's UNICAST just dealt with.
> d) The new channel sent three UNICAST messages to the coordinator. On the third message, the coordinator's UNICAST removed #3 and the old #4 and passed them both up.
> e) The new channel sent message #4, a JOIN_REQ, and UNICAST discarded it
> The new channel eventually goes through discovery again and gets into the group, but it adds quite a bit to channel startup time, and I'm a little worried that there might be a case where a much higher seqno gets trapped in the receiver window like this.
> I fixed this for my needs by changing UNICAST.handleDataReceived to reject a previous_member message if the seqno is higher than the default initial seqno, but I suppose that might wreak havoc with some other algorithms.
> private void handleDataReceived(Object sender, long seqno, Message msg) {
> if(trace)
> log.trace(new StringBuffer().append(local_addr).append(" <-- DATA(").append(sender).append(": #").append(seqno));
> if(previous_members.contains(sender)) {
> // we don't want to see messages from departed members
> if (seqno > DEFAULT_FIRST_SEQNO) {
> if (trace)
> log.trace("discarding message " + seqno + " from previous member " + sender);
> return;
> }
> if(trace)
> log.trace("removed " + sender + " from previous_members as we received a message from it");
> previous_members.removeElement(sender);
> }
> Entry entry;
> synchronized(connections) {
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
19 years, 11 months
[JBoss JIRA] Commented: (JGRP-244) long connect time in application that frequently closes and opens new channel
by Bela Ban (JIRA)
[ http://jira.jboss.com/jira/browse/JGRP-244?page=comments#action_12341050 ]
Bela Ban commented on JGRP-244:
-------------------------------
The diff to Bruce's solution is that I discard *all* initial messages (=messages who create a new connection) whose seqno != 1, whereas Bruce only discards them for messages from *previous* members.
> long connect time in application that frequently closes and opens new channel
> -----------------------------------------------------------------------------
>
> Key: JGRP-244
> URL: http://jira.jboss.com/jira/browse/JGRP-244
> Project: JGroups
> Issue Type: Bug
> Affects Versions: 2.2.8, 2.2.9, 2.3, 2.2.9.1, 2.2.9.2
> Environment: SLES 9
> Reporter: Bruce Schuchardt
> Assigned To: Bela Ban
> Fix For: 2.4
>
>
> I have an application that has one long-lived jgroups member and four other processes that often close their channel and create a new one. Most of the time these new connections are formed in a dozen milliseconds or so, but sometimes they're taking over 20 seconds. The apps are using TCPGOSSIP with multicast turned off.
> I turned on tracing and saw that the coordinator's UNICAST was having some trouble. It got a message from a departed member that it stored up and dispatched later when the departed member's address was reused by a new channel.
> a) A member left the view and UNICAST removed its connection for the member and added it to previous_members.
> b) Another message then arrived from the member, and UNICAST created a new connection for it. The message had seqno 4, and was put in the AckReceiverWindow and not passed up.
> c) A few seconds later, a process created a new channel and it got the same ID as the one the coordinator's UNICAST just dealt with.
> d) The new channel sent three UNICAST messages to the coordinator. On the third message, the coordinator's UNICAST removed #3 and the old #4 and passed them both up.
> e) The new channel sent message #4, a JOIN_REQ, and UNICAST discarded it
> The new channel eventually goes through discovery again and gets into the group, but it adds quite a bit to channel startup time, and I'm a little worried that there might be a case where a much higher seqno gets trapped in the receiver window like this.
> I fixed this for my needs by changing UNICAST.handleDataReceived to reject a previous_member message if the seqno is higher than the default initial seqno, but I suppose that might wreak havoc with some other algorithms.
> private void handleDataReceived(Object sender, long seqno, Message msg) {
> if(trace)
> log.trace(new StringBuffer().append(local_addr).append(" <-- DATA(").append(sender).append(": #").append(seqno));
> if(previous_members.contains(sender)) {
> // we don't want to see messages from departed members
> if (seqno > DEFAULT_FIRST_SEQNO) {
> if (trace)
> log.trace("discarding message " + seqno + " from previous member " + sender);
> return;
> }
> if(trace)
> log.trace("removed " + sender + " from previous_members as we received a message from it");
> previous_members.removeElement(sender);
> }
> Entry entry;
> synchronized(connections) {
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
19 years, 11 months
[JBoss JIRA] Updated: (JGRP-244) long connect time in application that frequently closes and opens new channel
by Bela Ban (JIRA)
[ http://jira.jboss.com/jira/browse/JGRP-244?page=all ]
Bela Ban updated JGRP-244:
--------------------------
Complexity: Medium
Priority: Major (was: Minor)
> long connect time in application that frequently closes and opens new channel
> -----------------------------------------------------------------------------
>
> Key: JGRP-244
> URL: http://jira.jboss.com/jira/browse/JGRP-244
> Project: JGroups
> Issue Type: Bug
> Affects Versions: 2.2.8, 2.2.9, 2.3, 2.2.9.1, 2.2.9.2
> Environment: SLES 9
> Reporter: Bruce Schuchardt
> Assigned To: Bela Ban
> Fix For: 2.4
>
>
> I have an application that has one long-lived jgroups member and four other processes that often close their channel and create a new one. Most of the time these new connections are formed in a dozen milliseconds or so, but sometimes they're taking over 20 seconds. The apps are using TCPGOSSIP with multicast turned off.
> I turned on tracing and saw that the coordinator's UNICAST was having some trouble. It got a message from a departed member that it stored up and dispatched later when the departed member's address was reused by a new channel.
> a) A member left the view and UNICAST removed its connection for the member and added it to previous_members.
> b) Another message then arrived from the member, and UNICAST created a new connection for it. The message had seqno 4, and was put in the AckReceiverWindow and not passed up.
> c) A few seconds later, a process created a new channel and it got the same ID as the one the coordinator's UNICAST just dealt with.
> d) The new channel sent three UNICAST messages to the coordinator. On the third message, the coordinator's UNICAST removed #3 and the old #4 and passed them both up.
> e) The new channel sent message #4, a JOIN_REQ, and UNICAST discarded it
> The new channel eventually goes through discovery again and gets into the group, but it adds quite a bit to channel startup time, and I'm a little worried that there might be a case where a much higher seqno gets trapped in the receiver window like this.
> I fixed this for my needs by changing UNICAST.handleDataReceived to reject a previous_member message if the seqno is higher than the default initial seqno, but I suppose that might wreak havoc with some other algorithms.
> private void handleDataReceived(Object sender, long seqno, Message msg) {
> if(trace)
> log.trace(new StringBuffer().append(local_addr).append(" <-- DATA(").append(sender).append(": #").append(seqno));
> if(previous_members.contains(sender)) {
> // we don't want to see messages from departed members
> if (seqno > DEFAULT_FIRST_SEQNO) {
> if (trace)
> log.trace("discarding message " + seqno + " from previous member " + sender);
> return;
> }
> if(trace)
> log.trace("removed " + sender + " from previous_members as we received a message from it");
> previous_members.removeElement(sender);
> }
> Entry entry;
> synchronized(connections) {
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
19 years, 11 months
[JBoss JIRA] Commented: (JGRP-244) long connect time in application that frequently closes and opens new channel
by Bela Ban (JIRA)
[ http://jira.jboss.com/jira/browse/JGRP-244?page=comments#action_12341049 ]
Bela Ban commented on JGRP-244:
-------------------------------
Okay, previous_members needs to be retained, because:
- FRAG sends a bunch of fragments, e.g. 1000, to P
- P crashes or leaves
- FRAG should stop sending those unicasts, but doesn't (need to change that too !)
- Once P is in previous_members, UNICAST simply discards FRAG's unicast packets to (non-existing) P
> long connect time in application that frequently closes and opens new channel
> -----------------------------------------------------------------------------
>
> Key: JGRP-244
> URL: http://jira.jboss.com/jira/browse/JGRP-244
> Project: JGroups
> Issue Type: Bug
> Affects Versions: 2.2.8, 2.2.9, 2.3, 2.2.9.1, 2.2.9.2
> Environment: SLES 9
> Reporter: Bruce Schuchardt
> Assigned To: Bela Ban
> Priority: Minor
> Fix For: 2.4
>
>
> I have an application that has one long-lived jgroups member and four other processes that often close their channel and create a new one. Most of the time these new connections are formed in a dozen milliseconds or so, but sometimes they're taking over 20 seconds. The apps are using TCPGOSSIP with multicast turned off.
> I turned on tracing and saw that the coordinator's UNICAST was having some trouble. It got a message from a departed member that it stored up and dispatched later when the departed member's address was reused by a new channel.
> a) A member left the view and UNICAST removed its connection for the member and added it to previous_members.
> b) Another message then arrived from the member, and UNICAST created a new connection for it. The message had seqno 4, and was put in the AckReceiverWindow and not passed up.
> c) A few seconds later, a process created a new channel and it got the same ID as the one the coordinator's UNICAST just dealt with.
> d) The new channel sent three UNICAST messages to the coordinator. On the third message, the coordinator's UNICAST removed #3 and the old #4 and passed them both up.
> e) The new channel sent message #4, a JOIN_REQ, and UNICAST discarded it
> The new channel eventually goes through discovery again and gets into the group, but it adds quite a bit to channel startup time, and I'm a little worried that there might be a case where a much higher seqno gets trapped in the receiver window like this.
> I fixed this for my needs by changing UNICAST.handleDataReceived to reject a previous_member message if the seqno is higher than the default initial seqno, but I suppose that might wreak havoc with some other algorithms.
> private void handleDataReceived(Object sender, long seqno, Message msg) {
> if(trace)
> log.trace(new StringBuffer().append(local_addr).append(" <-- DATA(").append(sender).append(": #").append(seqno));
> if(previous_members.contains(sender)) {
> // we don't want to see messages from departed members
> if (seqno > DEFAULT_FIRST_SEQNO) {
> if (trace)
> log.trace("discarding message " + seqno + " from previous member " + sender);
> return;
> }
> if(trace)
> log.trace("removed " + sender + " from previous_members as we received a message from it");
> previous_members.removeElement(sender);
> }
> Entry entry;
> synchronized(connections) {
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
19 years, 11 months
