[
https://issues.jboss.org/browse/ISPN-2633?page=com.atlassian.jira.plugin....
]
Michal Linhard updated ISPN-2633:
---------------------------------
Attachment: serverlogs.zip
configs.zip
serverlogs.zip - logs from 16 nodes
configs.zip - config of 16 nodes
this happened when I accidentally ran my test on infiniband interface in hyperion instead
of ethernet
what is really mysterious is that jgroups packets travel BACK IN TIME ! :-)
{code}
node0001:
03:11:41,624 INFO [org.jboss.modules] JBoss Modules version 1.1.3.GA-redhat-1
...
03:11:46,653 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC
service thread 1-5) ISPN000094: Received new cluster view: [node0015/default|87]
[node0015/default, node0008/default, node0007/default, node0006/default, node0012/default,
node0003/default, node0009/default, node0010/default, node0016/default, node0011/default,
node0002/default, node0001/default, node0005/default, node0014/default, node0013/default,
node0004/default, node0001/default]
03:11:47,026 ERROR [org.jgroups.JChannel] (MSC service thread 1-5) exception in
channelConnected() callback: java.lang.IllegalStateException: JBAS010272: A node named
node0001 already exists in this cluster. Perhaps there is already a server running on
this host? If so, restart this server with a unique node name, via
-Djboss.node.name=<node-name>
node0002:
03:11:56,615 INFO [org.jboss.modules] JBoss Modules version 1.1.3.GA-redhat-1
....
03:11:59,989 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC
service thread 1-1) ISPN000094: Received new cluster view: [node0015/default|89]
[node0015/default, node0008/default, node0007/default, node0006/default, node0012/default,
node0003/default, node0009/default, node0010/default, node0016/default, node0011/default,
node0002/default, node0001/default, node0005/default, node0014/default, node0013/default,
node0004/default, node0002/default]
03:12:00,357 ERROR [org.jgroups.JChannel] (MSC service thread 1-1) exception in
channelConnected() callback: java.lang.IllegalStateException: JBAS010272: A node named
node0002 already exists in this cluster. Perhaps there is already a server running on
this host? If so, restart this server with a unique node name, via
-Djboss.node.name=<node-name>
{code}
node0002 gets in node0001's view before it starts!
before you ask:
1. time in hyperion machines is really well synchronized, diff is under 1s for sure
2. I REALLY checked that there weren't multiple server instances running on the
network
My crazy theory is that something was wrong with infiniband network stack buffers and
might have gotten traffic from previous run received in next run (is something like this
possible ?)
cause this only happened when I used infiniband (by mistake, cause we don't test on
infiniband right now)
IllegalArgumentException: Address shall not be null and broken views
upon JOIN
------------------------------------------------------------------------------
Key: ISPN-2633
URL:
https://issues.jboss.org/browse/ISPN-2633
Project: Infinispan
Issue Type: Bug
Environment: Windows 2008
Reporter: Radoslav Husar
Assignee: Dan Berindei
Fix For: 5.2.0.CR1
Attachments: configs.zip, serverlogs.zip
Running our AS testsuite with master (which includes fix for ISPN-2572) I am instead
seeing the following.
The same node is present twice in the view, looks like something is wrong with JOINs as
this happen upon joins.
{noformat}
15:51:41,376 WARN [org.jgroups.protocols.TP$ProtocolAdapter] (OOB-20,null) dropping
unicast message to wrong destination node-1/ejb; my local_addr is node-1/ejb
15:51:41,821 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport]
(ServerService Thread Pool -- 19) ISPN000094: Received new cluster view: [node-0/ejb|12]
[node-0/ejb, node-1/ejb, node-1/ejb]
15:51:41,825 WARN [org.jgroups.protocols.TCP] (FD_SOCK pinger,ejb,node-1/ejb) null:
logical address cache didn't contain all physical address, sending up a discovery
request
15:51:41,825 ERROR [org.jgroups.protocols.TCP] (FD_SOCK pinger,ejb,node-1/ejb) failed
sending message to cluster (69 bytes): java.lang.NullPointerException, cause: null
15:51:42,185 ERROR [org.jgroups.JChannel] (ServerService Thread Pool -- 19) exception in
channelConnected() callback: java.lang.IllegalStateException: JBAS010272: A node named
node-1 already exists in this cluster. Perhaps there is already a server running on this
host? If so, restart this server with a unique node name, via
-Djboss.node.name=<node-name>
at
org.jboss.as.clustering.jgroups.subsystem.ChannelService.channelConnected(ChannelService.java:105)
at org.jgroups.Channel.notifyChannelConnected(Channel.java:495)
at org.jgroups.JChannel.connect(JChannel.java:286)
at org.jgroups.JChannel.connect(JChannel.java:268)
at
org.infinispan.remoting.transport.jgroups.JGroupsTransport.startJGroupsChannelIfNeeded(JGroupsTransport.java:207)
at
org.infinispan.remoting.transport.jgroups.JGroupsTransport.start(JGroupsTransport.java:198)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.6.0_32]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
[rt.jar:1.6.0_32]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
[rt.jar:1.6.0_32]
at java.lang.reflect.Method.invoke(Method.java:597) [rt.jar:1.6.0_32]
at org.infinispan.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:203)
at
org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:883)
at
org.infinispan.factories.AbstractComponentRegistry.invokeStartMethods(AbstractComponentRegistry.java:654)
at
org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:643)
at
org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:546)
at
org.infinispan.factories.GlobalComponentRegistry.start(GlobalComponentRegistry.java:225)
at
org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:681)
at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:653)
at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:549)
at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:563)
at
org.jboss.as.clustering.infinispan.DefaultEmbeddedCacheManager.getCache(DefaultEmbeddedCacheManager.java:107)
at
org.jboss.as.clustering.infinispan.DefaultEmbeddedCacheManager.getCache(DefaultEmbeddedCacheManager.java:98)
at
org.jboss.as.clustering.infinispan.subsystem.CacheService.start(CacheService.java:78)
at org.jboss.as.clustering.msc.AsynchronousService$1.run(AsynchronousService.java:82)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
[rt.jar:1.6.0_32]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
[rt.jar:1.6.0_32]
at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_32]
at org.jboss.threads.JBossThread.run(JBossThread.java:122)
[jboss-threads-2.0.0.GA.jar:2.0.0.GA]
15:51:42,204 ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool -- 19)
MSC00001: Failed to start service jboss.infinispan.ejb.remote-connector-client-mappings:
org.jboss.msc.service.StartException in service
jboss.infinispan.ejb.remote-connector-client-mappings:
org.infinispan.manager.EmbeddedCacheManagerStartupException:
org.infinispan.CacheException: Unable to invoke method public void
org.infinispan.remoting.transport.jgroups.JGroupsTransport.start() on object of type
JGroupsTransport
at org.jboss.as.clustering.msc.AsynchronousService$1.run(AsynchronousService.java:87)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
[rt.jar:1.6.0_32]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
[rt.jar:1.6.0_32]
at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_32]
at org.jboss.threads.JBossThread.run(JBossThread.java:122)
[jboss-threads-2.0.0.GA.jar:2.0.0.GA]
Caused by: org.infinispan.manager.EmbeddedCacheManagerStartupException:
org.infinispan.CacheException: Unable to invoke method public void
org.infinispan.remoting.transport.jgroups.JGroupsTransport.start() on object of type
JGroupsTransport
at
org.infinispan.factories.GlobalComponentRegistry.start(GlobalComponentRegistry.java:247)
at
org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:681)
at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:653)
at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:549)
at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:563)
at
org.jboss.as.clustering.infinispan.DefaultEmbeddedCacheManager.getCache(DefaultEmbeddedCacheManager.java:107)
at
org.jboss.as.clustering.infinispan.DefaultEmbeddedCacheManager.getCache(DefaultEmbeddedCacheManager.java:98)
at
org.jboss.as.clustering.infinispan.subsystem.CacheService.start(CacheService.java:78)
at org.jboss.as.clustering.msc.AsynchronousService$1.run(AsynchronousService.java:82)
... 4 more
Caused by: org.infinispan.CacheException: Unable to invoke method public void
org.infinispan.remoting.transport.jgroups.JGroupsTransport.start() on object of type
JGroupsTransport
at org.infinispan.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:205)
at
org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:883)
at
org.infinispan.factories.AbstractComponentRegistry.invokeStartMethods(AbstractComponentRegistry.java:654)
at
org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:643)
at
org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:546)
at
org.infinispan.factories.GlobalComponentRegistry.start(GlobalComponentRegistry.java:225)
... 12 more
Caused by: java.lang.IllegalArgumentException: Address shall not be null
at
org.infinispan.remoting.transport.jgroups.JGroupsAddress.<init>(JGroupsAddress.java:48)
at
org.infinispan.remoting.transport.jgroups.JGroupsTransport.fromJGroupsAddress(JGroupsTransport.java:686)
at
org.infinispan.remoting.transport.jgroups.JGroupsTransport.startJGroupsChannelIfNeeded(JGroupsTransport.java:227)
at
org.infinispan.remoting.transport.jgroups.JGroupsTransport.start(JGroupsTransport.java:198)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.6.0_32]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
[rt.jar:1.6.0_32]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
[rt.jar:1.6.0_32]
at java.lang.reflect.Method.invoke(Method.java:597) [rt.jar:1.6.0_32]
at org.infinispan.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:203)
... 17 more
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see:
http://www.atlassian.com/software/jira