[ https://jira.jboss.org/jira/browse/JBREM-1182?page=com.atlassian.jira.plu... ] Richard Achmatowicz reassigned JBREM-1182: ------------------------------------------ Assignee: Richard Achmatowicz -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/jira/browse/JBREM-1182?page=com.atlassian.jira.plu... ] Richard Achmatowicz commented on JBREM-1182: -------------------------------------------- It appears that the problems on Linux, in which JRunit-based tests fail because members cannot see each other, is related to the JGroups version being used, which is jgroups-2.2.8. I replaced jgroups-2.2.8 with jgroups-2.2.9.1 and the problem "went away". I believe we need to upgrade the jgroups jar from version 2.2.8 to 2.2.9.1 in $REMOTING_SRC/lib/jgroups/lib. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/jira/browse/JBREM-1182?page=com.atlassian.jira.plu... ] Richard Achmatowicz updated JBREM-1182: --------------------------------------- Attachment: jgroups.jar Attaching a JGroups 2.2.9.1 jgroups.jar file. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/jira/browse/JBREM-1182?page=com.atlassian.jira.plu... ] Richard Achmatowicz commented on JBREM-1182: -------------------------------------------- Going to do some testsuite runs locally to test out these changes. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/jira/browse/JBREM-1182?page=com.atlassian.jira.plu... ] Richard Achmatowicz commented on JBREM-1182: -------------------------------------------- 2. Configuring JRunit to use the loopback interface Sometimes, we don't have a network available and need to conduct testing locally. In such cases, we use the loopback interface. When jrunit.bind_addr is the loopback address 127.0.0.1, this means that the TestDriver, TestClient and TestServer will be bound to the loopback interface (lo on Linux). On Linux, there is no default routing for multicast packets which involves the loopback interface lo. This means that if we have two JGroups processes which communicate via the interface lo, multicast packets for these proceses will be sent out onto eth0 and will not arrive on the loopback interface. Members will not see each other. We need to do something additional when working with JRunit processes which use the loopback interfcace. The usual solution is to add in a route for multicast which involves lo. In order for multicast packets to be able to be passed from two processes using lo, we need to (again on Linux) add in the route: # route add -net 224.0.0.0 netmask 240.0.0.0 dev lo This has the effect of routing all multicast packets through the interface lo. You need to remember to delete this route after use, as it drives all multicast traffic down lo. Adding a new route on shared machines is not such a hot idea - so when running the testsuite say on Hudson, we shouldn't use the above approach. There is another way to achieve the same goal: instead of using routing to direct multicast packets onto lo, we can ask JGroups to send any multicast packets onto lo, in addition to another other interfaces it is sending them to (i.e. the jrunit.bind_addr interface). In JGroups 2.2.9 and above, we can define the properties: receive_on_all_interfaces=<true | false> receive_interfaces=<list of interfaces> send_on_all_interfaces=<true | false> send_interfaces=<list of interfaces> which allow controlling which interfaces JGroups would use for sending unicast and multicast packets, and which interfaces used for receiving unicast and multicast packets. So, instead of defining a new route for multicast onto lo, we can use the parameters: jrunit.receive_on_all_interfaces=true jrunit.send_interfaces=127.0.0.1 Assuming we use jrunit.bind_addr=127.0.0.1, this will cause JGroups to (i) listen for multicast and unicast on both 192.168.0.100 (say, for eth0) as well as 127.0.0.1 (ii) send all unicast and multicast packets onto the interface 127.0.0.1 So, we get JGroups style connectivity with no additional route. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/jira/browse/JBREM-1182?page=com.atlassian.jira.plu... ] Richard Achmatowicz commented on JBREM-1182: -------------------------------------------- Seems to be a bug in JGroups 2.2.9.1. I tried the same experiment in 2.4.6.GA and it worked... -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/jira/browse/JBREM-1182?page=com.atlassian.jira.plu... ] Richard Achmatowicz commented on JBREM-1182: -------------------------------------------- Probably doesn't work because people have been doing some serious non-compatible tinkering with the system properties passed to MessageBus between versions.... Remoting 2.2: JRunit version 1.0 Beta 2 (in tag remoting_binary_011207) Remoting 2.x: JRunit version 1.0 Beta 2.1 (in tag remoting_binary_081008) I now see what has been happening by looking at the history of org.jboss.jrunit.MessageBus. Lots of tinkeriung with the JGroups config, but JBREM-876 is definitely on the right track. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/jira/browse/JBREM-1182?page=com.atlassian.jira.plu... ] Richard Achmatowicz commented on JBREM-1182: -------------------------------------------- I have now found out why JRunit was not working with the settings: -Djrunit.bind_addr=127.0.0.1 -Djrunit.mcast_addr=228.15.2.3 -Djrunit.mcast_port=45565 -Djrunit.receive_on_all_interfaces=false -Djrunit.send_on_all_interfaces=false -Djrunit.send_interfaces=127.0.0.1 At the level of TestDriver communicating with the two ServerTestHarness instances, the JGroups groups were forming correctly, but the ServerTestHarness processes were not receiving the RunTestMessage multicast by TestDriver after it gets confirmation of Server startup. I couldn't understand how some messages were getting through and others were not. Eventually, I realised that this was the first message that TestDriver multicasts on the specified send interface, 127.0.0.1. The problem arises in JGroups: look at this section of output from UDP.createSockets: jrunit using bind_address: 127.0.0.1 jrunit using mcast_addr: 228.15.2.3 jrunit using mcast_port: 45566 jrunit using receive_on_all_interfaces=false jrunit using send_on_all_interfaces=false jrunit using send_interfaces=127.0.0.1 props: UDP(mcast_addr=228.15.2.3;mcast_port=45566;bind_addr=127.0.0.1;tos=8;ucast_recv_buf_size=80000;ucast_send_buf_size=150000;mcast_send_buf_size=150000;mcast_recv_buf_size=80000;loopback=false;discard_incompatible_packets=true;max_bundle_size=64000;max_bundle_timeout=30;use_incoming_packet_handler=true;use_outgoing_packet_handler=false;ip_ttl=2;down_thread=false;up_thread=false;enable_bundling=false;receive_on_all_interfaces=false;send_on_all_interfaces=false;send_interfaces=127.0.0.1;):PING(timeout=2000;num_initial_members=3;up_thread=false;down_thread=false):MERGE2(min_interval=10000;max_interval=20000;up_thread=false;down_thread=false):FD_SOCK(up_thread=false;down_thread=false):FD(timeout=10000;max_tries=5;shun=true;up_thread=false;down_thread=false):VERIFY_SUSPECT(timeout=1500;up_thread=false;down_thread=false):pbcast.NAKACK(max_xmit_size=8192;use_mcast_xmit=false;gc_lag=0;retransmit_timeout=300,600,1200,2400,4800;up_thread=false;down_thread=false):UNICAST(timeout=300,600,1200,2400;up_thread=false;down_thread=false):pbcast.STABLE(stability_delay=1000;desired_avg_gossip=50000;max_bytes=400000;up_thread=false;down_thread=false):VIEW_SYNC(avg_send_interval=60000;up_thread=false;down_thread=false):pbcast.GMS(join_timeout=3000;join_retry_timeout=2000;shun=false;view_bundling=false;print_local_addr=true;up_thread=false;down_thread=false):FC(max_credits=2000000;min_threshold=0.10;up_thread=false;down_thread=false):FRAG2(frag_size=60000;down_thread=false;up_thread=false):pbcast.STATE_TRANSFER(up_thread=false;down_thread=false) HELLO! local_addr=127.0.0.1:57464, mcast_addr=228.15.2.3:45566, bind_addr=/127.0.0.1, ttl=2 sock: bound to 127.0.0.1:57464, receive buffer size=55808, send buffer size=55808 mcast_recv_sock: bound to 127.0.0.1:45566, send buffer size=55808, receive buffer size=55808 1 mcast send sockets: 0:0:0:0:0:0:0:1%1:36720, send buffer size=55808, receive buffer size=55808 ------------------------------------------------------- GMS: address is 127.0.0.1:57464 ------------------------------------------------------- Note the mcast send socket used: it's an IPv6 version of localhost and not an IPv4 version, as specified in the system property -Djrunit.send_interfaces. It turns out that, if you don't start the MessageBus() with -Djava.net.preferIPv4Stack=true, this situation can arise, due to the way JGroups creates its multicast send sockets and the way NetworkInterface.getHostName() works when there is more than one IP address associated woith an interface (in this case, 127.0.0.1 and ::1). So, JGroups was sending multicasts on lo, but it was using an IPv6 stack and not an IPv4 stack. So the multicast receive in ServerTestHarness (whose processes by the way are started with -Djava.net.preferIPv4Stack=true) using his IPv4 stack would not receive the messages. The problem is fixed by adding a <sysprop name="java.net.preferIPv4Stack" value="true"/> to all of the junit targets which start test cases. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/jira/browse/JBREM-1182?page=com.atlassian.jira.plu... ] Richard Achmatowicz commented on JBREM-1182: -------------------------------------------- Unfortunately, forcing the TestDriver JVM to use the IPv4 stack (to fix the JGroups problem above) has the unwanted side-effect of causing IPv6 related tests to fail (as IPv6 protocol is not understood by the IPv4 stack). So, what I said earlier about addresses used for the test framework and addresses used by test cases being completely independent is not true - they must have a common IP version and this may lead to conflicts like the above. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/jira/browse/JBREM-1182?page=com.atlassian.jira.plu... ] Richard Achmatowicz updated JBREM-1182: --------------------------------------- Attachment: JBREM-1182.patch -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/jira/browse/JBREM-1182?page=com.atlassian.jira.plu... ] Richard Achmatowicz updated JBREM-1182: --------------------------------------- Attachment: (was: jgroups.jar) -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/jira/browse/JBREM-1182?page=com.atlassian.jira.plu... ] Ron Sigal updated JBREM-1182: ----------------------------- Fix Version/s: 2.2.3.SP3 (was: 2.2.3.SP2) Pushing to release 2.2.3.SP3. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/JBREM-1182?page=com.atlassian.jira.plugin... ] Ron Sigal updated JBREM-1182: ----------------------------- Issue Type: Task (was: Feature Request) Fix Version/s: 2.2.4.SP1 (was: 2.2.4) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira