Guys I am seeing problems here again. Vladimir, you mentioned the problems you saw earlier had to do with nodes not being in the initial TCPPING discovery list. In one of my dist rehash tests, I see this as well, although I don't think it has to do with the initial discovery list. The test (will be in SVN soon) is o.i.distribution.RehashTest#testNoDataLoss() The test starts 3 caches, populates state. So far so good. Adds 3 more caches to the cluster. Now here, in the process of adding these, some of the original nodes cannot "see" some new nodes even though views have been installed, leading to dropped packets. (the next part of the test kills the 3 original nodes, but the test never gets this far). This is the sort of message we see in the logs. Wonder if the "no physical address" bit sheds any light. (Note that the JGroups address has been replaced with CACHE1..CACHE6 in the logs for readability) 2009-06-11 11:28:59,854 WARN [org.jgroups.protocols.TCP] (OOB-6,Infinispan-Cluster,CACHE3) no physical address for CACHE5, dropping message Also, just so you know, this works perfectly well with JGroups 2.7.0.GA. Any thoughts/ideas? Cheers -- Manik Surtani manik(a)jboss.org Lead, Infinispan Lead, JBoss Cache http://www.infinispan.org http://www.jbosscache.org _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

One more nugget of information. :-) Tried this with 2.8.0.Beta1/UDP to help pinpoint the issue. True enough, things work fine if I use UDP, in that I don't see the dropped messages and the tests pass. But it still is slooow when compared to 2.7.0. Cheers Manik

On 11 Jun 2009, at 12:31, Vladimir Blagojevic wrote: > On 6/11/09 1:16 PM, Manik Surtani wrote: >> One more nugget of information. :-) >> >> Tried this with 2.8.0.Beta1/UDP to help pinpoint the issue. >> >> True enough, things work fine if I use UDP, in that I don't see the >> dropped messages and the tests pass. But it still is slooow when >> compared to 2.7.0. >> >> Cheers >> Manik > Manik, > > We need to investigate this issue and it will take a few days to get > to the bottom of it. In the meantime do you want to switch test suite > to MPING/udp or do you plan to hold beta until this issue is fixed? I will look at getting the test suite to use MPING if I have time, otherwise it will have to be an open task for Mircea for next week. I'm still keen on cutting 4.0.0.BETA1 by tomorrow at latest, so I'll need to think about whether I want this to go with JGroups 2.7 or 2.8.

Cheers -- Manik Surtani manik(a)jboss.org Lead, Infinispan Lead, JBoss Cache http://www.infinispan.org http://www.jbosscache.org _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

It doesn't work on linux: 2009-06-11 11:28:40,026 WARN [org.jgroups.protocols.MPING] (pool-1-thread-4) could not bind to /228.10.10.15 (IPv4 address); make sure your mcast_addr is of the same type as the IP stack (IPv4 or IPv6). Will ignore mcast_addr, but this may lead to cross talking (see http://www.jboss.com/wiki/Edit.jsp?page=CrossTalking for details). Exception was: java.net.BindException: Cannot assign requested address Here is my config: <MPING bind_addr="127.0.0.1" break_on_coord_rsp="true" mcast_addr="230.8.8.8" mcast_port="17890" ip_ttl="2" num_initial_members="3"/> Any ideas?

The issue was the one pointed out. Cause was that "java.net.preferIpv4Stack" does not get propagated from property *parent\pom.xml* to the JVM running the tests. This is rather strange, as all other property do get propagated correctly. Anyone encountered anything similar before? The fix/hack was to enforce "java.net.preferIpv4Stack" programatical;y, in an @BeforeSuite. This sorted the problem. Bela Ban wrote: > What stack were you running on (IPv4 or IPv6) ? If it is IPv6 *only*, > then 230.8.8.8 is *not* a valid multicast address ! > > If you have a dual stack, use -Djava.net.preferIpv4Stack=true to get > IPv4 (then the mcast_addr is correct) or > -Djava.net.preferIPv6Addresses=true and change the mcast_addr to a > valid IPv6 address > > Mircea Markus wrote: >> It doesn't work on linux: >> 2009-06-11 11:28:40,026 WARN [org.jgroups.protocols.MPING] >> (pool-1-thread-4) could not bind to /228.10.10.15 (IPv4 address); >> make sure your mcast_addr is of the same type as the IP stack (IPv4 >> or IPv6). >> Will ignore mcast_addr, but this may lead to cross talking (see >> http://www.jboss.com/wiki/Edit.jsp?page=CrossTalking for details). >> Exception was: java.net.BindException: Cannot assign requested address >> >> Here is my config: >> <MPING bind_addr="127.0.0.1" break_on_coord_rsp="true" >> mcast_addr="230.8.8.8" mcast_port="17890" ip_ttl="2" >> num_initial_members="3"/> >> >> Any ideas? >

Bela Ban wrote: > For you information: in 2.8, we're attempting to find out whether > you're running an IPv6-only stack and accordingly provide a default > IPv6 multicast address. The opposite of course for IPv4-only stacks. > The problematic case is duplex stacks, which is unfortunately the > majority of all setups... Not sure that would work for parallel tests harness, as it parses the xml confing and changes the IP address in order to assure different clusters won't see each others. So if the address defaults to same value for all threads in the suite, clusters will conflict. > > In other words, even if you forget the system property above, you > should be fine in 2.8... hopefully... :-) > > Mircea Markus wrote: >> The issue was the one pointed out. >> Cause was that "java.net.preferIpv4Stack" does not get propagated >> from property *parent\pom.xml* to the JVM running the tests. >> This is rather strange, as all other property do get propagated >> correctly. Anyone encountered anything similar before? >> The fix/hack was to enforce "java.net.preferIpv4Stack" >> programatical;y, in an @BeforeSuite. This sorted the problem. >> >> >> >> Bela Ban wrote: >>> What stack were you running on (IPv4 or IPv6) ? If it is IPv6 >>> *only*, then 230.8.8.8 is *not* a valid multicast address ! >>> >>> If you have a dual stack, use -Djava.net.preferIpv4Stack=true to >>> get IPv4 (then the mcast_addr is correct) or >>> -Djava.net.preferIPv6Addresses=true and change the mcast_addr to a >>> valid IPv6 address >>> >>> Mircea Markus wrote: >>>> It doesn't work on linux: >>>> 2009-06-11 11:28:40,026 WARN [org.jgroups.protocols.MPING] >>>> (pool-1-thread-4) could not bind to /228.10.10.15 (IPv4 address); >>>> make sure your mcast_addr is of the same type as the IP stack >>>> (IPv4 or IPv6). >>>> Will ignore mcast_addr, but this may lead to cross talking (see >>>> http://www.jboss.com/wiki/Edit.jsp?page=CrossTalking for details). >>>> Exception was: java.net.BindException: Cannot assign requested >>>> address >>>> >>>> Here is my config: >>>> <MPING bind_addr="127.0.0.1" break_on_coord_rsp="true" >>>> mcast_addr="230.8.8.8" mcast_port="17890" ip_ttl="2" >>>> num_initial_members="3"/> >>>> >>>> Any ideas? >>> >> >

In the 2.8 version, you must list *all* servers in the cluster in TCPPING.initial_hosts ! We will not be able to send messages to any server not in this list because we don't have its physical address ! We've been discussing this on the jg-dev list recently, and an idea is to add new servers not in the list to the list dynamically. Question is when to remove them... While this is beeing discussed, I suggest you use MPING for the time being Manik Surtani wrote: > Guys I am seeing problems here again. > > Vladimir, you mentioned the problems you saw earlier had to do with > nodes not being in the initial TCPPING discovery list. > > In one of my dist rehash tests, I see this as well, although I don't > think it has to do with the initial discovery list. > > The test (will be in SVN soon) is > o.i.distribution.RehashTest#testNoDataLoss() > > The test starts 3 caches, populates state. So far so good. > Adds 3 more caches to the cluster. Now here, in the process of > adding these, some of the original nodes cannot "see" some new nodes > even though views have been installed, leading to dropped packets. > (the next part of the test kills the 3 original nodes, but the test > never gets this far). > > This is the sort of message we see in the logs. Wonder if the "no > physical address" bit sheds any light. (Note that the JGroups > address has been replaced with CACHE1..CACHE6 in the logs for > readability) > > 2009-06-11 11:28:59,854 WARN [org.jgroups.protocols.TCP] > (OOB-6,Infinispan-Cluster,CACHE3) no physical address for CACHE5, > dropping message > > Also, just so you know, this works perfectly well with JGroups 2.7.0.GA. > > Any thoughts/ideas? > > Cheers > -- > Manik Surtani > manik(a)jboss.org > Lead, Infinispan > Lead, JBoss Cache > http://www.infinispan.org > http://www.jbosscache.org > > > > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev