[infinispan-issues] [JBoss JIRA] (ISPN-2633) IllegalArgumentException: Address shall not be null and broken views upon JOIN
Michal Linhard (JIRA)
jira-events at lists.jboss.org
Thu Dec 13 12:16:18 EST 2012
[ https://issues.jboss.org/browse/ISPN-2633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michal Linhard updated ISPN-2633:
---------------------------------
Attachment: serverlogs.zip
configs.zip
serverlogs.zip - logs from 16 nodes
configs.zip - config of 16 nodes
this happened when I accidentally ran my test on infiniband interface in hyperion instead of ethernet
what is really mysterious is that jgroups packets travel BACK IN TIME ! :-)
{code}
node0001:
03:11:41,624 INFO [org.jboss.modules] JBoss Modules version 1.1.3.GA-redhat-1
...
03:11:46,653 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-5) ISPN000094: Received new cluster view: [node0015/default|87] [node0015/default, node0008/default, node0007/default, node0006/default, node0012/default, node0003/default, node0009/default, node0010/default, node0016/default, node0011/default, node0002/default, node0001/default, node0005/default, node0014/default, node0013/default, node0004/default, node0001/default]
03:11:47,026 ERROR [org.jgroups.JChannel] (MSC service thread 1-5) exception in channelConnected() callback: java.lang.IllegalStateException: JBAS010272: A node named node0001 already exists in this cluster. Perhaps there is already a server running on this host? If so, restart this server with a unique node name, via -Djboss.node.name=<node-name>
node0002:
03:11:56,615 INFO [org.jboss.modules] JBoss Modules version 1.1.3.GA-redhat-1
....
03:11:59,989 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-1) ISPN000094: Received new cluster view: [node0015/default|89] [node0015/default, node0008/default, node0007/default, node0006/default, node0012/default, node0003/default, node0009/default, node0010/default, node0016/default, node0011/default, node0002/default, node0001/default, node0005/default, node0014/default, node0013/default, node0004/default, node0002/default]
03:12:00,357 ERROR [org.jgroups.JChannel] (MSC service thread 1-1) exception in channelConnected() callback: java.lang.IllegalStateException: JBAS010272: A node named node0002 already exists in this cluster. Perhaps there is already a server running on this host? If so, restart this server with a unique node name, via -Djboss.node.name=<node-name>
{code}
node0002 gets in node0001's view before it starts!
before you ask:
1. time in hyperion machines is really well synchronized, diff is under 1s for sure
2. I REALLY checked that there weren't multiple server instances running on the network
My crazy theory is that something was wrong with infiniband network stack buffers and might have gotten traffic from previous run received in next run (is something like this possible ?)
cause this only happened when I used infiniband (by mistake, cause we don't test on infiniband right now)
> IllegalArgumentException: Address shall not be null and broken views upon JOIN
> ------------------------------------------------------------------------------
>
> Key: ISPN-2633
> URL: https://issues.jboss.org/browse/ISPN-2633
> Project: Infinispan
> Issue Type: Bug
> Environment: Windows 2008
> Reporter: Radoslav Husar
> Assignee: Dan Berindei
> Fix For: 5.2.0.CR1
>
> Attachments: configs.zip, serverlogs.zip
>
>
> Running our AS testsuite with master (which includes fix for ISPN-2572) I am instead seeing the following.
> The same node is present twice in the view, looks like something is wrong with JOINs as this happen upon joins.
> {noformat}
> 15:51:41,376 WARN [org.jgroups.protocols.TP$ProtocolAdapter] (OOB-20,null) dropping unicast message to wrong destination node-1/ejb; my local_addr is node-1/ejb
> 15:51:41,821 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (ServerService Thread Pool -- 19) ISPN000094: Received new cluster view: [node-0/ejb|12] [node-0/ejb, node-1/ejb, node-1/ejb]
> 15:51:41,825 WARN [org.jgroups.protocols.TCP] (FD_SOCK pinger,ejb,node-1/ejb) null: logical address cache didn't contain all physical address, sending up a discovery request
> 15:51:41,825 ERROR [org.jgroups.protocols.TCP] (FD_SOCK pinger,ejb,node-1/ejb) failed sending message to cluster (69 bytes): java.lang.NullPointerException, cause: null
> 15:51:42,185 ERROR [org.jgroups.JChannel] (ServerService Thread Pool -- 19) exception in channelConnected() callback: java.lang.IllegalStateException: JBAS010272: A node named node-1 already exists in this cluster. Perhaps there is already a server running on this host? If so, restart this server with a unique node name, via -Djboss.node.name=<node-name>
> at org.jboss.as.clustering.jgroups.subsystem.ChannelService.channelConnected(ChannelService.java:105)
> at org.jgroups.Channel.notifyChannelConnected(Channel.java:495)
> at org.jgroups.JChannel.connect(JChannel.java:286)
> at org.jgroups.JChannel.connect(JChannel.java:268)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.startJGroupsChannelIfNeeded(JGroupsTransport.java:207)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.start(JGroupsTransport.java:198)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.6.0_32]
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) [rt.jar:1.6.0_32]
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) [rt.jar:1.6.0_32]
> at java.lang.reflect.Method.invoke(Method.java:597) [rt.jar:1.6.0_32]
> at org.infinispan.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:203)
> at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:883)
> at org.infinispan.factories.AbstractComponentRegistry.invokeStartMethods(AbstractComponentRegistry.java:654)
> at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:643)
> at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:546)
> at org.infinispan.factories.GlobalComponentRegistry.start(GlobalComponentRegistry.java:225)
> at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:681)
> at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:653)
> at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:549)
> at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:563)
> at org.jboss.as.clustering.infinispan.DefaultEmbeddedCacheManager.getCache(DefaultEmbeddedCacheManager.java:107)
> at org.jboss.as.clustering.infinispan.DefaultEmbeddedCacheManager.getCache(DefaultEmbeddedCacheManager.java:98)
> at org.jboss.as.clustering.infinispan.subsystem.CacheService.start(CacheService.java:78)
> at org.jboss.as.clustering.msc.AsynchronousService$1.run(AsynchronousService.java:82)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_32]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_32]
> at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_32]
> at org.jboss.threads.JBossThread.run(JBossThread.java:122) [jboss-threads-2.0.0.GA.jar:2.0.0.GA]
> 15:51:42,204 ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool -- 19) MSC00001: Failed to start service jboss.infinispan.ejb.remote-connector-client-mappings: org.jboss.msc.service.StartException in service jboss.infinispan.ejb.remote-connector-client-mappings: org.infinispan.manager.EmbeddedCacheManagerStartupException: org.infinispan.CacheException: Unable to invoke method public void org.infinispan.remoting.transport.jgroups.JGroupsTransport.start() on object of type JGroupsTransport
> at org.jboss.as.clustering.msc.AsynchronousService$1.run(AsynchronousService.java:87)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_32]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_32]
> at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_32]
> at org.jboss.threads.JBossThread.run(JBossThread.java:122) [jboss-threads-2.0.0.GA.jar:2.0.0.GA]
> Caused by: org.infinispan.manager.EmbeddedCacheManagerStartupException: org.infinispan.CacheException: Unable to invoke method public void org.infinispan.remoting.transport.jgroups.JGroupsTransport.start() on object of type JGroupsTransport
> at org.infinispan.factories.GlobalComponentRegistry.start(GlobalComponentRegistry.java:247)
> at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:681)
> at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:653)
> at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:549)
> at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:563)
> at org.jboss.as.clustering.infinispan.DefaultEmbeddedCacheManager.getCache(DefaultEmbeddedCacheManager.java:107)
> at org.jboss.as.clustering.infinispan.DefaultEmbeddedCacheManager.getCache(DefaultEmbeddedCacheManager.java:98)
> at org.jboss.as.clustering.infinispan.subsystem.CacheService.start(CacheService.java:78)
> at org.jboss.as.clustering.msc.AsynchronousService$1.run(AsynchronousService.java:82)
> ... 4 more
> Caused by: org.infinispan.CacheException: Unable to invoke method public void org.infinispan.remoting.transport.jgroups.JGroupsTransport.start() on object of type JGroupsTransport
> at org.infinispan.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:205)
> at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:883)
> at org.infinispan.factories.AbstractComponentRegistry.invokeStartMethods(AbstractComponentRegistry.java:654)
> at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:643)
> at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:546)
> at org.infinispan.factories.GlobalComponentRegistry.start(GlobalComponentRegistry.java:225)
> ... 12 more
> Caused by: java.lang.IllegalArgumentException: Address shall not be null
> at org.infinispan.remoting.transport.jgroups.JGroupsAddress.<init>(JGroupsAddress.java:48)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.fromJGroupsAddress(JGroupsTransport.java:686)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.startJGroupsChannelIfNeeded(JGroupsTransport.java:227)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.start(JGroupsTransport.java:198)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.6.0_32]
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) [rt.jar:1.6.0_32]
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) [rt.jar:1.6.0_32]
> at java.lang.reflect.Method.invoke(Method.java:597) [rt.jar:1.6.0_32]
> at org.infinispan.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:203)
> ... 17 more
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
More information about the infinispan-issues
mailing list