[ https://jira.jboss.org/browse/JGRP-1252?page=com.atlassian.jira.plugin.sy... ] Bela Ban updated JGRP-1252: --------------------------- Assignee: Vladimir Blagojevic (was: Bela Ban) Fix Version/s: 2.12 -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/browse/JGRP-1252?page=com.atlassian.jira.plugin.sy... ] Vladimir Blagojevic updated JGRP-1252: -------------------------------------- Attachment: JGRP-1252-output.txt Output of a sample run of test program with JGroups 2.10 -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/browse/JGRP-1252?page=com.atlassian.jira.plugin.sy... ] Vladimir Blagojevic updated JGRP-1252: -------------------------------------- Attachment: JGRP-1252-output2.txt It is just a matter of setting a proper binding interface! Here, in this test run I bound GR to a specific NIC and adjusted tcp.xml TCPGOSSIP protocol to point to GR on a new bind address. Again, everything worked as expected! -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/JGRP-1252?page=com.atlassian.jira.plugin.... ] Grahame Rogers updated JGRP-1252: --------------------------------- Attachment: JGroupsTest.zip HIghlights membership issue reported here -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/JGRP-1252?page=com.atlassian.jira.plugin.... ] ronald yang commented on JGRP-1252: ----------------------------------- @Grahame in your tester did you mix jgroups 2.11 and 2.10? -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/JGRP-1252?page=com.atlassian.jira.plugin.... ] Grahame Rogers updated JGRP-1252: --------------------------------- Attachment: JGroupsTest.zip -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/JGRP-1252?page=com.atlassian.jira.plugin.... ] Vladimir Blagojevic commented on JGRP-1252: ------------------------------------------- Grahame, I am taking a few days off this week but I will get this issue again next week. Until then if you get replicate this issue consistently see if you can find a cause. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/JGRP-1252?page=com.atlassian.jira.plugin.... ] Grahame Rogers commented on JGRP-1252: -------------------------------------- Hi, I think I have got to the bottom of why my test scenario is failing, I just need to make sure the same issue is true of my real project, but here's what I found. In my test app, when the gossip router is shut down, the RouterStubManager attempts to reconnect (startReconnecting). What I found was when this connection attempt fails, an exception is caught within this method and a warning is logged. However, the call to the logger never returns in this scenario, it just gets stuck. I dug a bit deeper and noticed that my test app does not have log4j on the classpath, so the JDKLogImpl is being used and if I add log4j the problem seems to go away (not fully tested yet). The weird thing is that the logging is generally fine and only goes wrong when startReconnecting comes into play. Therefore to reproduce, just make sure log4j is not on the classpath, hope this helps. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/JGRP-1252?page=com.atlassian.jira.plugin.... ] Grahame Rogers commented on JGRP-1252: -------------------------------------- Hi, I went back to the more manual test code I originally posted, as there was an issue with the more automated test harness where for some reason when using the jdklogger, threads were getting blocked - I will revisit this at some point as I appreciate this would be an easier way to replicate the issue. However, using the manual test I think I can see what the problem is, but would like to run it by you as obviously I don't fully understand the code. I start 4 chat clients (A,B,C and D). I then shut down D, then shut down the GR, restart GR and almost immediately start a new client (E). Now I believe my new client (E) goes and fetches a list of active clients from the GossipRouter via TCPGossip class and in my scenario, at this point A and B have re-connected but C has not (plausible timing issue). So TCPGossip then adds A and B's UUID, plus logical names to it's UUID cache (UUID.add). However, what happens next is the ClientGmsImpl class then attempts to join via the co-ordinator and the co-ordinator at this point is able to return a view which now also contains C, but my UUID cache does not contain this, so any subsequent toString() calls on this UUID results in the logical name not being returned, but instead something like 88d1f8ba-feb9-b998-2346-116e1344efcc which is generated from toStringLong when no cache entry exists. So at his point instead of my client logging out a view containing A,B,C and E, it logs A,B,88d1f8ba-feb9-b998-2346-116e1344efcc and E instead. Does this scenario sound correct to you? If so, then it almost seems that the issue is with relying on the discovery phase to tie up a logical name to a UUID instance, it's almost as if UUID (or a similar object) needs to contain a getLogicalName method so this caching of relationships is not required. Either that or perhaps there is another way for my code to derive a members logical name other than toString()? -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/JGRP-1252?page=com.atlassian.jira.plugin.... ] Grahame Rogers commented on JGRP-1252: -------------------------------------- Latest version of the automated test app uploaded. This forces the use of log4j for logging as there was a contention issue with the jdklogger. You should be able to run within a network and reproduce the issue quite quickly, although hopefully my post on the 31st of Dec may have helped clarify -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/JGRP-1252?page=com.atlassian.jira.plugin.... ] Grahame Rogers commented on JGRP-1252: -------------------------------------- Hi, I'm not sure what you mean "by the next discovery phase". Do you mean that if I wait long enough, the problem will resolve itself on the next sweep which occurs automatically? If so, I have waited minutes and the situation has not corrected itself - is there possible something wrong with the way I am configured? Example of my config is in the test projects I have uploaded here. If you mean, the next time a new node connects to GossipRouter then this will resolve itself, then this is correct, but I don't want to be waiting around for this scenario to occur as it could be a long period of time, or even never. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/JGRP-1252?page=com.atlassian.jira.plugin.... ] Grahame Rogers commented on JGRP-1252: -------------------------------------- Great, thanks. At the moment, I would say the scenario you describe is definitely not occurring. In the meantime I will perform some more testing, in the hope that I can add some more value to what I think is/isn't currently happening in terms of MERGE2 -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/JGRP-1252?page=com.atlassian.jira.plugin.... ] Grahame Rogers commented on JGRP-1252: -------------------------------------- reproduced in 2.12, please reopen -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/JGRP-1252?page=com.atlassian.jira.plugin.... ] Bela Ban reopened JGRP-1252: ---------------------------- -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/JGRP-1252?page=com.atlassian.jira.plugin.... ] Grahame Rogers commented on JGRP-1252: -------------------------------------- Hi. I tried what you described first and it did not reproduce the problem, so I had to go for the spawing solution which was far more of a pain to code to be honest -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/JGRP-1252?page=com.atlassian.jira.plugin.... ] Bela Ban commented on JGRP-1252: -------------------------------- I think the test may yield incorrect results because to determine the membership, you parse messages you get via the stdin of a forked child process. However, the might be garbled input, therefore resulting in incorrect membership... I did the following: #1 Start a GossipRouter #2 Start 4 chat processes #3 Kill last chat process (number 4) #4 Kill GR #5 Start GR #6 Start 4th chat #7 Check membership (probe.sh |grep view) #8 Go back to #3, rinse and repeat 10 times Step #7 *always* has the correct membership. Sometimes, nodes would show their UUID rather than their logical names. On the next drawing in the canvas (or a merge, whichever came first), the logical names would get displayed. So my manual test does exactly the same as your automatic test, and I could not reproduce your issue. I'm going to close this issue again. If you want to, we can have a phone call on this, but unless you provide me with a test that really shows the issue, I'm not going to re-open it. If you want to, we can have a call to dicuss this some more... -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/JGRP-1252?page=com.atlassian.jira.plugin.... ] Bela Ban commented on JGRP-1252: -------------------------------- Step #7 is optional; it is the equivalent of checkMembers() in your test. It just makes sure that the cluster has re-formed correctly after the restart of GR and the client. Again, I restarted GR and the client at least 10 times (with tcpchat.xml), and *never* ran into an issue. Note that - in your test - you wait for some time (20s) for the cluster to reform after the restart. However, MERGE2 is configured to merge between 10 and 30 seconds, so the merge may not yet have happened when you call checkMembers() in runTest(). Also, I ran this test on 2.12, maybe this is the difference... -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/JGRP-1252?page=com.atlassian.jira.plugin.... ] Bela Ban commented on JGRP-1252: -------------------------------- OK. Note that if you send a message (e.g. by drawing something in the canvas of Draw, or sending a chat message), and the physical address of the target(s) is not known, then JGroups will fetch that information from the GossipRouter. So if you see that the membership is correct, but some members don't have logical names, you can trigger this fetch... In a normal app, this would happen when a message is sent. JGroups does send messages regularly, e.g. stability or failure detection messages. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/JGRP-1252?page=com.atlassian.jira.plugin.... ] Bela Ban commented on JGRP-1252: -------------------------------- So, first of all, your view is *correct* (possibly after merging), even if it's duff, isn't it ? No, your config is correct. It's just the GossipRouter which maintains a table of members and their associated information (e.g. IP address, logical name etc). When GossipRouter is restarted, its table is empty until the existing members re-register. If the existing members 1-3 don't re-register *before* the new client is started, then you'll have a duff view, plus the cluster won't form correctly, until a merge happens. If the existing member re-register before the new client is started, then everything will be fine. You *could* reduce TCPGOSSIP.reconnect_interval (default is 10 secs), but then there's a lot of activity going on when the GossipRouter is down. In any case, if you *wait* for more than 10 second *after* restarting GR, but *before* starting client 4, the existing members should have re-registered and GR should have a fully populated table, so client 4 gets a correct view. Note that you can trigger the fetching of this information by calling probe.sh op=TCPGOSSIP.findInitialMembersAsString Alternatively, everyone could do this programatically: JChannel ch; Discovery discovery_prot=(Discovery)ch.getProtocolStack().findProtocol(TCPGOSSIP.class); discovery_prot.findInitialMembersAsString(); -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/JGRP-1252?page=com.atlassian.jira.plugin.... ] Bela Ban commented on JGRP-1252: -------------------------------- OK. Can you close this issue if the programmatic solution fixes your problem ? Note that you could always use something else than TCPGOSSIP, e.g. a FILE_PING over an NFS mounted (or otherwise shared) volume would not get you into this situation... -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/JGRP-1252?page=com.atlassian.jira.plugin.... ] Bela Ban commented on JGRP-1252: -------------------------------- Yes, that makes sense, as chances of not receiving the logical name are diminished with return_entire_cache=true. Depending on your version, you might also have to set ergonomics=false and/or max_rank=0 -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/JGRP-1252?page=com.atlassian.jira.plugin.... ] Grahame Rogers closed JGRP-1252. -------------------------------- provided with several viable workarounds -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira