[ https://issues.jboss.org/browse/JGRP-1426?page=com.atlassian.jira.plugin.... ] Bela Ban reassigned JGRP-1426: ------------------------------ Assignee: Vladimir Blagojevic (was: Bela Ban) Vladimir, can you take a look ? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/JGRP-1426?page=com.atlassian.jira.plugin.... ] Vladimir Blagojevic commented on JGRP-1426: ------------------------------------------- Can you repeat this scenario on some other network? We have seen many similar issues arise due to faulty network switches and/or transient issues on networks. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/JGRP-1426?page=com.atlassian.jira.plugin.... ] Vladimir Blagojevic commented on JGRP-1426: ------------------------------------------- Can I have a look at your test!? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/JGRP-1426?page=com.atlassian.jira.plugin.... ] David Hotham commented on JGRP-1426: ------------------------------------ (Ah, the comment about Scala won't have made any sense to you. I was referring to a previous exchange with Bela, and confused myself about who I was talking to... I've no idea whether you do Scala or not). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/JGRP-1426?page=com.atlassian.jira.plugin.... ] David Hotham commented on JGRP-1426: ------------------------------------ Ah, the formatting has made a real mess of that (especially the python code). Let me know if you can't figure out what I'm doing, and we'll find some other way of submitting this to you! -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/JGRP-1426?page=com.atlassian.jira.plugin.... ] David Hotham edited comment on JGRP-1426 at 2/22/12 7:29 AM: ------------------------------------------------------------- Ah, the formatting has made a real mess of that (especially the python code). Let me know if you can't figure out what I'm doing, and we'll find some other way of submitting this to you! Edit: figured out how to remove formatting was (Author: dimbleby): Ah, the formatting has made a real mess of that (especially the python code). Let me know if you can't figure out what I'm doing, and we'll find some other way of submitting this to you! -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/JGRP-1426?page=com.atlassian.jira.plugin.... ] David Hotham edited comment on JGRP-1426 at 2/22/12 7:30 AM: ------------------------------------------------------------- Hi, I decided that I should write a standalone repro for this. I can't see how to attach files to this issue so I'm just going to paste the code below. To repro: - I'm using loopback addresses 10.239.0.1 through 10.239.0.4 on my PC, and these are hardcoded in. You'll want to tweak to match whatever network you use - run instances 10.239.0.1 through 10.239.0.3 directly; these will just sit around - run instance 10.239.0.4 in a wrapper (I'll include the python code that I've been using) which - sits waiting for the file AOK.txt to be written - when it is written, kills instance 10.239.0.4 and starts a new one. Sooner or later you should see that D gets stuck outside the group, with one of the other members repeatedly logging "GMS flush by coordinator failed". I usually see it happen after about 10 or 15 loops. Here's the java code: {noformat} import org.jgroups.*; import org.jgroups.protocols.*; import org.jgroups.protocols.pbcast.*; import org.jgroups.stack.*; import java.io.File; import java.io.IOException; import java.net.InetAddress; import java.util.ArrayList; import java.util.List; public class Test extends ReceiverAdapter { String localAddress; Boolean sendMessage = false; public Test(String address) { localAddress = address; } // Parameter is an IP address, expected to be one of 10.239.0.1 - 4. public static void main(String[] args) throws Exception { Test test = new Test(args[0]); test.start(); } public void start() throws Exception { // Set up the stack, and channel. InetAddress inetAddr = InetAddress.getByName(localAddress); IpAddress member_a = new IpAddress("10.239.0.1", 60000); IpAddress member_b = new IpAddress("10.239.0.2", 60000); IpAddress member_c = new IpAddress("10.239.0.3", 60000); IpAddress member_d = new IpAddress("10.239.0.4", 60000); List<IpAddress> initialHosts = new ArrayList<IpAddress>(); initialHosts.add(member_a); initialHosts.add(member_b); initialHosts.add(member_c); initialHosts.add(member_d); JChannel ch=new JChannel(false); ProtocolStack stack=new ProtocolStack(); ch.setProtocolStack(stack); ch.setReceiver(this); stack.addProtocol((new TCP()) .setValue("bind_port", 60000) .setValue("bind_addr", inetAddr)); TCPPING tcpping = new TCPPING(); tcpping.setInitialHosts(initialHosts); stack.addProtocol(tcpping) .addProtocol(new MERGE3()) .addProtocol(new FD()) .addProtocol(new VERIFY_SUSPECT()) .addProtocol((new NAKACK2()) .setValue("use_mcast_xmit", false)) .addProtocol(new UNICAST2()) .addProtocol(new STABLE()) .addProtocol(new GMS()) .addProtocol(new UFC()) .addProtocol(new MFC()) .addProtocol(new FRAG2()) .addProtocol(new SEQUENCER()) .addProtocol(new FLUSH()); for (Protocol protocol: stack.getProtocols()) { protocol.setProtocolStack(stack); } stack.init(); // Connect to the cluster. ch.setName(localAddress); ch.connect("TestCluster"); // Loop round, waiting to be told to send a message. // Not thread safe - we might not send as many messages as we 'should' - but good enough. Boolean done = false; while (!done) { if (sendMessage) { sendMessage = false; ch.send(null, "This is a broadcast message from " + localAddress); } Thread.sleep(100); } } public void viewAccepted(View new_view) { System.out.println("view: " + new_view); // When D sees that the view has reached four members it creates the file AOK.txt, at which point we expect to // kill it. if (localAddress.equals("10.239.0.4") && (new_view.size() == 4)) { File f = new File("/tmp/AOK.txt"); try { f.createNewFile(); } catch (IOException e) { System.out.println("couldn't create AOK.txt"); } } // Changes of view trigger message sending. sendMessage = true; } public void receive(Message msg) { System.out.println("<< " + msg.getObject() + " [" + msg.getSrc() + "]"); } } {noformat} And the python wrapper: {noformat} import os import random import subprocess import time class TestWrapper: def __init__(self): self.subProc = None def startMemberD(self): try: self.subProc = subprocess.Popen("java -jar target/Test.jar 10.239.0.4") except OSError, e: print e def killMemberD(self): try: self.subProc.kill() except: os.kill(self.subProc.pid) self.subProc = None def start(self): counter = 0 while counter < 1000: counter += 1 print "Starting member D on loop %d" % counter self.startMemberD() # Wait for D to signal that all is well while not os.path.exists("c:\\downloads\\AOK.txt"): time.sleep(1) # Kill D, wait a bit. os.remove("/tmp/AOK.txt") time.sleep(2) self.killMemberD() delay = random.randrange(0, 10) print "Sleeping for %d seconds" % delay time.sleep(delay) wrapper = TestWrapper() wrapper.start() {noformat} was (Author: dimbleby): Hi, I decided that I should write a standalone repro for this. I can't see how to attach files to this issue so I'm just going to paste the code below. To repro: - I'm using loopback addresses 10.239.0.1 through 10.239.0.4 on my PC, and these are hardcoded in. You'll want to tweak to match whatever network you use - run instances 10.239.0.1 through 10.239.0.3 directly; these will just sit around - run instance 10.239.0.4 in a wrapper (I'll include the python code that I've been using) which - sits waiting for the file AOK.txt to be written - when it is written, kills instance 10.239.0.4 and starts a new one. Sooner or later you should see that D gets stuck outside the group, with one of the other members repeatedly logging "GMS flush by coordinator failed". I usually see it happen after about 10 or 15 loops. Here's the java code: {noformat} import org.jgroups.*; import org.jgroups.protocols.*; import org.jgroups.protocols.pbcast.*; import org.jgroups.stack.*; import java.io.File; import java.io.IOException; import java.net.InetAddress; import java.util.ArrayList; import java.util.List; public class Test extends ReceiverAdapter { String localAddress; Boolean sendMessage = false; public Test(String address) { localAddress = address; } // Parameter is an IP address, expected to be one of 10.239.0.1 - 4. public static void main(String[] args) throws Exception { Test test = new Test(args[0]); test.start(); } public void start() throws Exception { // Set up the stack, and channel. InetAddress inetAddr = InetAddress.getByName(localAddress); IpAddress member_a = new IpAddress("10.239.0.1", 60000); IpAddress member_b = new IpAddress("10.239.0.2", 60000); IpAddress member_c = new IpAddress("10.239.0.3", 60000); IpAddress member_d = new IpAddress("10.239.0.4", 60000); List<IpAddress> initialHosts = new ArrayList<IpAddress>(); initialHosts.add(member_a); initialHosts.add(member_b); initialHosts.add(member_c); initialHosts.add(member_d); JChannel ch=new JChannel(false); ProtocolStack stack=new ProtocolStack(); ch.setProtocolStack(stack); ch.setReceiver(this); stack.addProtocol((new TCP()) .setValue("bind_port", 60000) .setValue("bind_addr", inetAddr)); TCPPING tcpping = new TCPPING(); tcpping.setInitialHosts(initialHosts); stack.addProtocol(tcpping) .addProtocol(new MERGE3()) .addProtocol(new FD()) .addProtocol(new VERIFY_SUSPECT()) .addProtocol((new NAKACK2()) .setValue("use_mcast_xmit", false)) .addProtocol(new UNICAST2()) .addProtocol(new STABLE()) .addProtocol(new GMS()) .addProtocol(new UFC()) .addProtocol(new MFC()) .addProtocol(new FRAG2()) .addProtocol(new SEQUENCER()) .addProtocol(new FLUSH()); for (Protocol protocol: stack.getProtocols()) { protocol.setProtocolStack(stack); } stack.init(); // Connect to the cluster. ch.setName(localAddress); ch.connect("TestCluster"); // Loop round, waiting to be told to send a message. // Not thread safe - we might not send as many messages as we 'should' - but good enough. Boolean done = false; while (!done) { if (sendMessage) { sendMessage = false; ch.send(null, "This is a broadcast message from " + localAddress); } Thread.sleep(100); } } public void viewAccepted(View new_view) { System.out.println("view: " + new_view); // When D sees that the view has reached four members it creates the file AOK.txt, at which point we expect to // kill it. if (localAddress.equals("10.239.0.4") && (new_view.size() == 4)) { File f = new File("/tmp/AOK.txt"); try { f.createNewFile(); } catch (IOException e) { System.out.println("couldn't create AOK.txt"); } } // Changes of view trigger message sending. sendMessage = true; } public void receive(Message msg) { System.out.println("<< " + msg.getObject() + " [" + msg.getSrc() + "]"); } } {noformat} And the python wrapper: {noformat} import os import random import subprocess import time class TestWrapper: def __init__(self): self.subProc = None def startMemberD(self): try: self.subProc = subprocess.Popen("java -jar target/Test.jar 10.239.0.4") except OSError, e: print e def killMemberD(self): try: self.subProc.kill() except: os.kill(self.subProc.pid) self.subProc = None def start(self): counter = 0 while counter < 1000: counter += 1 print "Starting member D on loop %d" % counter self.startMemberD() # Wait for D to signal that all is well while not os.path.exists("c:\\downloads\\AOK.txt"): time.sleep(1) # Kill D, wait a bit. os.remove("/tmp/AOK.txt") time.sleep(2) self.killMemberD() delay = random.randrange(0, 10) print "Sleeping for %d seconds" % delay time.sleep(delay) wrapper = TestWrapper() wrapper.start() {noformat} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/JGRP-1426?page=com.atlassian.jira.plugin.... ] David Hotham commented on JGRP-1426: ------------------------------------ Hi, Have you had a chance to look at this? Does my repro help? When I initially described this problem in the mailing list thread "Member cannot join group due to stuck FLUSH" I actually reported two issues, though I've been concentrating on the second. I've now been taking another look at the first, in which the restarted D initially fails to join the group and forms a group [D]. Obviously if we don't hit partitioned groups in the first place, we're much less likely to hit this bug in merging groups. The formation of [D] happens because D pays attention to a delayed JOIN_RSP, even though it has already timed out the JOIN_REQ and embarked on a new round of discovery. This looks very much like [JGRP-687|https://issues.jboss.org/browse/JGRP-687], which seems to have been fixed at [aa1908f8|https://github.com/belaban/JGroups/commits/aa1908f8], but then (partially) undone at [e9c6e5e9|https://github.com/belaban/JGroups/commits/e9c6e5e9]. It's not obvious to me what that second commit was doing in ClientGmsImpl. I don't say that it was wrong - only that it's not obvious to me! I can confirm that undoing that change (in ClientGmsImpl only) significantly improves the behaviour of the system in the repro submitted above. Any idea why [e9c6e5e9|https://github.com/belaban/JGroups/commits/e9c6e5e9] made that change? Can the commented out code be safely reinstated? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/JGRP-1426?page=com.atlassian.jira.plugin.... ] David Hotham commented on JGRP-1426: ------------------------------------ Thanks, I appreciate the update. Enjoy your vacation! -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/JGRP-1426?page=com.atlassian.jira.plugin.... ] David Hotham commented on JGRP-1426: ------------------------------------ Hi, The python code looks to be failing trying to execute the java program. In my repro I'd built the code into a jar, so the command in that subprocess.Popen() line is appropriate. You should just need to change the string so that it contains the right command line to execute the java code in your environment. I can add a longer delay to the repro if you like, and I expect that it will make the problem be hit less frequently. I'm not sure what this would prove, though. I admit that my repro script is a little artificial - but that's because it's designed to demonstrate the window condition that causes this bug. Certainly if I don't try to it the bug, I'm less likely to hit it...! Anyway, probably the best thing would be for you to be able to reproduce this yourself. Let me know if you have any more trouble with the script. David PS I have submitted a pull request reinstating the JGRP-687 fix which, as above, definitely does make this bug (JGRP-1426) harder to hit. So far it has neither been accepted nor rejected. Do you have any insight as to what the intentions are for that pull request? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/JGRP-1426?page=com.atlassian.jira.plugin.... ] Vladimir Blagojevic commented on JGRP-1426: ------------------------------------------- David, Yes, I figured the part about starting the process and I adjusted the command line. Will look a bit more into this. However, unless you increase the delay you are basically hitting reincarnation issue where the same member who is not yet dead and removed from the view showing up as a new joiner. Give it a try with longer timeout and I would bet my money that the issue is not likely to pop up again! Where is your pull request? I do not see it here https://github.com/vblagoje/JGroups/pulls -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/JGRP-1426?page=com.atlassian.jira.plugin.... ] Vladimir Blagojevic commented on JGRP-1426: ------------------------------------------- Oops, I was looking at my JGroups fork. I'll ask Bela. Right, not the best word chosen (reincarnation), but what I meant is that you deliberately set the same address/port for recreated member D even though the previous D might not yet have been removed/killed/crashed. It could be other members [A,B,C] hold a stale connection object to D and keep communicating with D through it, failing and so on. Please give it a try with longer timeout! -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/JGRP-1426?page=com.atlassian.jira.plugin.... ] David Hotham updated JGRP-1426: ------------------------------- Attachment: Repro23March2012.zip Repro per update on 23rd March -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/JGRP-1426?page=com.atlassian.jira.plugin.... ] Vladimir Blagojevic commented on JGRP-1426: ------------------------------------------- David, have not yet tried the new version, looked through your code though. The important sleep is the one after you kill a member (activeMember.kill()), this is the sleep that should be 20 sec. Failure detection will take a minimum of 6 sec and you sleep randomly between 0 an 10 before you span the same member! This is bound to create issues! Can you change this sleep to 20 sec and the other sleep can be random between 0 an 10? Give it a try! -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/JGRP-1426?page=com.atlassian.jira.plugin.... ] Vladimir Blagojevic commented on JGRP-1426: ------------------------------------------- David, I stand corrected, your script launches a new member not the same one! However, I was unable to run your script: saudade:JGroups vladimir$ python test.py Starting loop 1 using member 10.239.0.4 [Errno 2] No such file or directory AOK - let things settle now Traceback (most recent call last): File "test.py", line 61, in <module> wrapper.start() File "test.py", line 51, in start activeMember.kill() File "test.py", line 26, in kill handle = win32api.OpenProcess(1, False, self.subProc.pid) NameError: global name 'win32api' is not defined I only changed the command line to cmdLine = "java -cp target/classes/ Test %s" % self.address because that is how I invoke the test class. Unable to run script I resorted to simulating this scenario. I opened five terminals. In first three I started members 10.239.0.1, 10.239.0.2, and 10.239.0.3. In remaining two terminals I launched and killed members 10.239.0.4 and 10.239.0.5 according to script. I tried to simulate the script and the timing - I accept that getting the right timing is really hard. Although I saw some temp errors cluster always recovered and continued to function normally. Yes, merges occurred as well. I tried even random scenarios, killing and launching 10.239.0.4 and 10.239.0.5 randomly. It still worked as expected! Find me on skype and lets talk this one further! Lets get to the bottom on of this one! Regards, Vladimir -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/JGRP-1426?page=com.atlassian.jira.plugin.... ] Vladimir Blagojevic commented on JGRP-1426: ------------------------------------------- Ok, lets go that route. I like your idea about final flush timeout, the second line of defense when things go wrong for whatever reason. I'll make this change on my fork of Jgroups and you can pick it up from there and give it a go! -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/JGRP-1426?page=com.atlassian.jira.plugin.... ] Vladimir Blagojevic commented on JGRP-1426: ------------------------------------------- I agree with Bela as total order already includes everything FLUSH provides. Can you run one of your night long tests David without SEQUENCER in stack? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/JGRP-1426?page=com.atlassian.jira.plugin.... ] Bela Ban updated JGRP-1426: --------------------------- Attachment: Hotham.java Attached Hotham.java which inserts DISCARD into the last 2 members of the cluster (192.168.1.14 and 192.168.1.15) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/JGRP-1426?page=com.atlassian.jira.plugin.... ] David Hotham commented on JGRP-1426: ------------------------------------ Just to confirm: this has reproduced very quickly for me without SEQUENCER. I should add that - apart from exposing me to JGRP-1443 - the fix that Vlad had made was working well. With that fix, I stopped seeing JGRP-1426, and my test runs generally last quite a bit longer before I run into JGRP-1443. So from my point of view, Vlad's fix is looking good. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/JGRP-1426?page=com.atlassian.jira.plugin.... ] Vladimir Blagojevic commented on JGRP-1426: ------------------------------------------- Bela, I looked into this one and noticed that we do not issue STOP_FLUSH on attempted flushing during GMS view installation. I added STOP_FLUSH if flushing fails. See https://github.com/vblagoje/JGroups/tree/t_1426_m for more details. I will investigate this one a bit further next week and, if you agree, integrate! -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/JGRP-1426?page=com.atlassian.jira.plugin.... ] David Hotham commented on JGRP-1426: ------------------------------------ If I can avoid FLUSH then that should certainly stop me from hitting any issues with it! Presumably views will only be part of the sequencing if I put SEQUENCER _beneath_ GMS in the stack? ie my stack will go: TCP TCPPING MERGE3 FD VERIFY_SUSPECT NAKACK2 UNICAST2 STABLE SEQUENCER GMS UFC MFC FRAG2 Does that look right? I'd be very happy to discuss this on the phone - though if you can confirm that the stack above looks OK, then that seems like it should resolve things from my point of view. (I'd still be happy to talk, but I don't know that I'd have much to say except "thanks!"). UK office hours would be best for me - drop me an email and let me know what works for you. Thanks! David -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/JGRP-1426?page=com.atlassian.jira.plugin.... ] David Hotham commented on JGRP-1426: ------------------------------------ So, good news: this stack looks to be working out well for me. Since it looks as though I can do without FLUSH, this particular ticket becomes a non-issue from my point of view. It sounds as though you're going to go ahead with Vlad's fix. I'll just re-confirm, then, that that fix was indeed working for me: with it I could no longer hit this issue. ie so far as I know, the fix is good! However... I intend to re-open JGRP-1443. - I've hit it both with and without the 1426 fix - I've now also hit it without FLUSH being in the stack I'll update with more details in that ticket: but I'm pretty sure that it should be regarded as an independent bug worthy of its own investigation. Thanks for all your help with this one. David -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/JGRP-1426?page=com.atlassian.jira.plugin.... ] Vladimir Blagojevic updated JGRP-1426: -------------------------------------- Git Pull Request: https://github.com/belaban/JGroups/pull/36 Ready for integration into master -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/JGRP-1426?page=com.atlassian.jira.plugin.... ] Bela Ban commented on JGRP-1426: -------------------------------- Applied the pull request, resolving this issue now. Thanks, Vladimir ! -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira