[ https://jira.jboss.org/jira/browse/JGRP-976?page=com.atlassian.jira.plugi... ] Richard Achmatowicz commented on JGRP-976: ------------------------------------------ I have attached a small program BuggyServerSocketSender.java which reproduces ther error: When I run this program on Sun JDK 1.6.0_13, I can reproduce the error which is occurring in the JGroups testsuite (the Socket construction is the same as used by RouterStub). If I run the same program on OpenJDK 1.7, the error does not appear. By running the program with strace, which displays how the JDK calls the kernel (in particular, how the JDK makes calls to the socket function), you can see that in the Sun JDK case, an incorrect scope id is being used (lo). It is "incorrect", because I am passing a scoped link-local address to the socket call, which specifies the zone id of "%eth0". Here is the relevant section of the trace: [snip] [pid 3656] socket(PF_INET6, SOCK_STREAM, IPPROTO_IP) = 4 [pid 3656] bind(4, {sa_family=AF_INET6, sin6_port=htons(0), inet_pton(AF_INET6, "fe80::215:58ff:fec8:81a8", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=if_nametoindex("eth0")}, 28) = 0 [pid 3656] getsockname(4, {sa_family=AF_INET6, sin6_port=htons(43780), inet_pton(AF_INET6, "fe80::215:58ff:fec8:81a8", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=if_nametoindex("eth0")}, [28]) = 0 [pid 3656] connect(4, {sa_family=AF_INET6, sin6_port=htons(0), inet_pton(AF_INET6, "fe80::215:58ff:fec8:81a8", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=if_nametoindex("lo")}, 28) = -1 EINVAL (Invalid argument) exception connecting to host: /fe80:0:0:0:215:58ff:fec8:81a8%eth0, port: 0java.net.SocketException: Invalid argument or cannot assign requested address java.net.SocketException: Invalid argument or cannot assign requested address at java.net.PlainSocketImpl.socketConnect(Native Method) Note that the sin6_scope is set to "lo", and not "eth0". This is the root cause of not being able to connect. But because I specified %eth0, this should get picked up by thye JDK. By running the same program with strace and OpenJDK, you can see that the correct zoneid is being picked up. Here is the same relevant section of the trace: [snip] [pid 2889] socket(PF_INET6, SOCK_STREAM, IPPROTO_IP) = 5 [pid 2889] bind(5, {sa_family=AF_INET6, sin6_port=htons(0), inet_pton(AF_INET6, "fe80::215:58ff:fec8:81a8", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=if_nametoindex("eth0")}, 28) = 0 [pid 2889] getsockname(5, {sa_family=AF_INET6, sin6_port=htons(47388), inet_pton(AF_INET6, "fe80::215:58ff:fec8:81a8", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=if_nametoindex("eth0")}, [28]) = 0 [pid 2889] connect(5, {sa_family=AF_INET6, sin6_port=htons(0), inet_pton(AF_INET6, "fe80::215:58ff:fec8:81a8", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=if_nametoindex("eth0")}, 28) = -1 ECONNREFUSED (Connection refused) exception connecting to host: /fe80:0:0:0:215:58ff:fec8:81a8%eth0, port: 0java.net.ConnectException: Connection refused java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) Note that here we have a connection refused (which should be the case, as I don't have a server running). So, on Sun JDK 1.6.0_13 with link-local addresses, we have a problem it seems. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/jira/browse/JGRP-976?page=com.atlassian.jira.plugi... ] Richard Achmatowicz reassigned JGRP-976: ---------------------------------------- Assignee: Richard Achmatowicz (was: Bela Ban) -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/jira/browse/JGRP-976?page=com.atlassian.jira.plugi... ] Richard Achmatowicz commented on JGRP-976: ------------------------------------------ The bug I reported to Sun seems to be related to this one: http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6800096 The JVM decides which scope id to use for the link-local address based on looking at the routing table. If an incorrect route (to lo and not eth0) is specified, this gets picked up and used as the scoped id. The routing table on RHEL 5 (jawa01.englab.brq.redhat.com) includes a most specifc route which routes to lo: -bash-3.2$ /sbin/ifconfig eth0 eth0 Link encap:Ethernet HWaddr 00:15:17:2D:67:55 inet addr:10.34.32.153 Bcast:10.34.35.255 Mask:255.255.252.0 inet6 addr: 3ffe:ffff:100:f101::1/64 Scope:Global inet6 addr: fec0::f101:215:17ff:fe2d:6755/64 Scope:Site inet6 addr: fe80::215:17ff:fe2d:6755/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:662152465 errors:0 dropped:0 overruns:0 frame:0 TX packets:280481512 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:451032770922 (420.0 GiB) TX bytes:81942436544 (76.3 GiB) -bash-3.2$ /sbin/route -A inet6 Kernel IPv6 routing table Destination Next Hop Flags Metric Ref Use Iface 3ffe:ffff:100:f101::/64 * U 256 0 0 eth0 fe80::/64 * U 256 0 0 virbr0 fe80::/64 * U 256 0 0 xenbr0 fe80::/64 * U 256 0 0 vif0.0 fe80::/64 * U 256 0 0 peth0 fe80::/64 * U 256 0 0 eth0 fe80::/64 * U 256 0 0 eth2 fe80::/64 * U 256 0 0 eth1 fec0:0:0:f101::/64 * UA 256 465084 0 xenbr0 fec0:0:0:f101::/64 * UA 256 302002 0 eth0 */0 * U 256 0 0 eth0 localhost6.localdomain6/128 * U 0 3793 1 lo 3ffe:ffff:100:f101::/128 * U 0 0 2 lo 3ffe:ffff:100:f101::1/128 * U 0 90672 1 lo fe80::/128 * U 0 0 2 lo fe80::/128 * U 0 0 2 lo fe80::/128 * U 0 0 2 lo fe80::/128 * U 0 0 2 lo fe80::/128 * U 0 0 2 lo fe80::/128 * U 0 0 2 lo fe80::/128 * U 0 0 2 lo fe80::215:17ff:fe2d:6755/128 * U 0 5162 1 lo fe80::21a:a0ff:fe2c:3657/128 * U 0 14 1 lo fe80::21a:a0ff:fe2c:3658/128 * U 0 0 1 lo fec0:0:0:f101::/128 * U 0 0 2 lo fec0:0:0:f101::/128 * U 0 0 2 lo jawa01.englab.brq.redhat.com/128 * U 0 39 1 lo ff02::1/128 * U 0 90 1 lo ff02::1:ff00:1/128 ff02::1:ff00:1 UC 0 1 0 xenbr0 ff02::1:ff00:1/128 ff02::1:ff00:1 UC 0 1 0 eth0 ff00::/8 * U 256 0 0 virbr0 ff00::/8 * U 256 0 0 xenbr0 ff00::/8 * U 256 0 0 vif0.0 ff00::/8 * U 256 0 0 peth0 ff00::/8 * U 256 0 0 eth2 ff00::/8 * U 256 0 0 eth1 ff00::/8 * U 256 0 0 eth0 So, this would account for the bug turning up on RHEL. On the other hand, my Fedora 8 host at home does not have such a route: [nrla@localhost ~]$ /sbin/ifconfig eth0 eth0 Link encap:Ethernet HWaddr 00:15:58:C8:81:A8 inet addr:192.168.0.100 Bcast:192.168.0.255 Mask:255.255.255.0 inet6 addr: fe80::215:58ff:fec8:81a8/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:671185 errors:0 dropped:0 overruns:0 frame:0 TX packets:894851 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:100 RX bytes:552563003 (526.9 MiB) TX bytes:508468523 (484.9 MiB) Memory:ee000000-ee020000 [nrla@localhost ~]$ /sbin/route -A inet6 Kernel IPv6 routing table Destination Next Hop Flags Metric Ref Use Iface fe80::/64 * U 256 0 0 eth0 localhost6.localdomain6/128 * U 0 0 1 lo fe80::/128 * U 0 0 1 lo lenovo6/128 * U 0 0 1 lo ff00::/8 * U 256 0 0 eth0 So, I need to follow up with the RHEL/Fedora guys to see why this is occurring. This same basic error manifests itself in two different error emssages, depending on whch form you use to connect to a socket (i.e, directly from the constructor, or after construction using connect) -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/jira/browse/JGRP-976?page=com.atlassian.jira.plugi... ] Richard Achmatowicz commented on JGRP-976: ------------------------------------------ I've started an intersting discussion with Neil Horman on fedora-devel mailing list, concerning why there are static rules for routing traffic along lo instead of letting it go out onto eth0. It looks as though there are also some problems with IPv6 routing table entries which may be involved in this problem. Going to file a bug on at least one of the issues which seem to be appearing. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/jira/browse/JGRP-976?page=com.atlassian.jira.plugi... ] Richard Achmatowicz commented on JGRP-976: ------------------------------------------ Which host is that from? Here are the results from one of the recently configured hosts: bash-3.2$ hostname vmg17.mw.lab.eng.bos.redhat.com bash-3.2$ /sbin/ifconfig eth0 eth0 Link encap:Ethernet HWaddr 00:16:3E:70:DC:37 inet addr:10.16.89.221 Bcast:10.16.95.255 Mask:255.255.248.0 inet6 addr: fec0:0:a16:ffff::1/64 Scope:Site inet6 addr: fec0:0:a16:ffff::2/64 Scope:Site inet6 addr: fec0:0:a16:ffff::3/64 Scope:Site inet6 addr: fec0:0:a16:ffff::4/64 Scope:Site inet6 addr: fec0:0:a16:5f0:216:3eff:fe70:dc37/64 Scope:Site inet6 addr: fe80::216:3eff:fe70:dc37/64 Scope:Link // LINKLOCAL ADDRESS UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:4726660 errors:0 dropped:0 overruns:0 frame:0 TX packets:790052 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:1930786543 (1.7 GiB) TX bytes:361951626 (345.1 MiB) bash-3.2$ /sbin/route -A inet6 Kernel IPv6 routing table Destination Next Hop Flags Metric Ref Use Iface fe80::/64 * U 256 0 0 eth0 fec0:0:a16:5f0::/64 * UA 256 3042 0 eth0 fec0:0:a16:ffff::5/128 fec0:0:a16:ffff::5 UC 0 4637 9 eth0 fec0:0:a16:ffff::7/128 fec0:0:a16:ffff::7 UC 0 111 24 eth0 fec0:0:a16:ffff::/64 * U 256 0 0 eth0 */0 fe80::21e:beff:feba:d680 UGDA 1024 1 0 eth0 localhost6.localdomain6/128 * U 0 1235 3 lo fe80::216:3eff:fe70:dc37/128 * U 0 6770 1 lo fec0:0:a16:5f0:216:3eff:fe70:dc37/128 * U 0 0 1 lo // ROUTE THROUGH LO fec0:0:a16:ffff::1/128 * U 0 13337 1 lo fec0:0:a16:ffff::2/128 * U 0 0 1 lo fec0:0:a16:ffff::3/128 * U 0 3 1 lo fec0:0:a16:ffff::4/128 * U 0 1616 1 lo ff00::/8 * U 256 0 0 eth0 -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/jira/browse/JGRP-976?page=com.atlassian.jira.plugi... ] Bela Ban commented on JGRP-976: ------------------------------- Ok, so the issue is this (described in [1] at the bottom): A ServerSocket can bind to a scoped or unscoped IPv6 link-local address. A client socket can *not* connect to a scoped link-local IPv6 address ! E.g. sock.connect(InetAddress.getByName("ADDR%3", port) fails, whereas sock.connect(InetAddress.getByName("ADDR", port) succeeds. The solution adopted in JGroups was to remove the scope of a link-local IPv6 address in Socket.connect(), there's a utility method in Util which does this. [1] http://www.jboss.org/community/docs/DOC-10014 -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira