JBOSS AS Infinispan nodes cannot interoperate with Plain JVM / Glassfish nodes ------------------------------------------------------------------------------ Key: ISPN-3783 URL: https://issues.jboss.org/browse/ISPN-3783 Project: Infinispan Issue Type: Bug Components: Configuration, Core, Marshalling, State Transfer Affects Versions: 5.2.1.Final Environment: JBOSS 7.2.0.FINAL / Glassfish 4 / Plain JVM Reporter: Andrew Scully Assignee: Galder Zamarreño I am able to start up an Infinispan (running over JGroups 3.2.7.Final) cluster, with nodes composed of plain JVMs and Glassfish instances. Equally, I am able to create an Infinispan cluster composed purely of JBOSS instances. My problem occurs when I include a JBOSS instance in a cluster of Plain JVMs / Glassfish instances. Infinispan running on JBOSS does not appear to be able to inter-operate with Infinispan running in a different environment. The JGroups configuration is the same on all nodes and I am able to send messages between all of the nodes to prove that the JGroups cluster is operational. As far as I can see, the Infinispan configuration is the same across all nodes. The following exception is logged in the JBOSS log: 2013-12-03 15:27:02,841 ERROR [org.jgroups.blocks.RequestCorrelator](OOB-18,shared=udp) failed unmarshalling buffer into return value: java.io.StreamCorruptedException: Unexpected byte found when reading an object: 0 at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:750) at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209) at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:37) at org.jboss.marshalling.ModularClassResolver.resolveClass(ModularClassResolver.java:99) at org.jboss.marshalling.river.RiverUnmarshaller.doReadClassDescriptor(RiverUnmarshaller.java:893) at org.jboss.marshalling.river.RiverUnmarshaller.doReadNewObject(RiverUnmarshaller.java:1205) at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:272) at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209) at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:37) at org.infinispan.remoting.responses.ExceptionResponse$Externalizer.readObject(ExceptionResponse.java:66) at org.infinispan.remoting.responses.ExceptionResponse$Externalizer.readObject(ExceptionResponse.java:58) at org.infinispan.marshall.jboss.ExternalizerTable$ExternalizerAdapter.readObject(ExternalizerTable.java:402) at org.infinispan.marshall.jboss.ExternalizerTable.readObject(ExternalizerTable.java:224) at org.infinispan.marshall.jboss.JBossMarshaller$ExternalizerTableProxy.readObject(JBossMarshaller.java:164) at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:351) at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209) at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:37) at org.infinispan.marshall.jboss.AbstractJBossMarshaller.objectFromObjectStream(AbstractJBossMarshaller.java:163) at org.infinispan.marshall.VersionAwareMarshaller.objectFromByteBuffer(VersionAwareMarshaller.java:116) at org.infinispan.marshall.AbstractDelegatingMarshaller.objectFromByteBuffer(AbstractDelegatingMarshaller.java:104) at org.infinispan.remoting.transport.jgroups.MarshallerAdapter.objectFromBuffer(MarshallerAdapter.java:50) at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:404) at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:249) at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:598) at org.jgroups.blocks.mux.MuxUpHandler.up(MuxUpHandler.java:130) at org.jgroups.JChannel.up(JChannel.java:707) at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1020) at org.jgroups.protocols.FRAG2.up(FRAG2.java:181) at org.jgroups.protocols.FlowControl.up(FlowControl.java:418) at org.jgroups.protocols.FlowControl.up(FlowControl.java:400) at org.jgroups.protocols.pbcast.GMS.up(GMS.java:896) at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:245) at org.jgroups.protocols.UNICAST2.handleDataReceived(UNICAST2.java:765) at org.jgroups.protocols.UNICAST2.up(UNICAST2.java:420) at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:606) at org.jgroups.protocols.BARRIER.up(BARRIER.java:102) at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:143) at org.jgroups.protocols.FD.up(FD.java:253) at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:288) at org.jgroups.protocols.MERGE2.up(MERGE2.java:205) at org.jgroups.protocols.Discovery.up(Discovery.java:359) at org.jgroups.protocols.TP$ProtocolAdapter.up(TP.java:2616) at org.jgroups.protocols.TP.passMessageUp(TP.java:1263) at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1825) at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1798) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:724) The following exception is logged on the plain JVM / Glassfish side: org.infinispan.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl .start() throws java.lang.Exception on object of type StateTransferManagerImpl at org.infinispan.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:205) at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:88 3) at org.infinispan.factories.AbstractComponentRegistry.invokeStartMethods(AbstractComponentRegistry.java:654) at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:643) at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:546) at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:199) at org.infinispan.CacheImpl.start(CacheImpl.java:559) at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:686) at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:649) at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:545) at com.saaconsultants.external.tester.DomainAccessTester.main(DomainAccessTester.java:137) Caused by: org.infinispan.CacheException: java.lang.IllegalArgumentException: Segment owner list cannot be null or empty at org.infinispan.util.Util.rewrapAsCacheException(Util.java:541) at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispat cher.java:186) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:515) at org.infinispan.topology.LocalTopologyManagerImpl.executeOnCoordinator(LocalTopologyManagerImpl.java:261) at org.infinispan.topology.LocalTopologyManagerImpl.join(LocalTopologyManagerImpl.java:101) at org.infinispan.statetransfer.StateTransferManagerImpl.start(StateTransferManagerImpl.java:117) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.infinispan.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:203) ... 10 more Caused by: java.lang.IllegalArgumentException: Segment owner list cannot be null or empty at org.infinispan.distribution.ch.DefaultConsistentHash.<init>(DefaultConsistentHash.java:78) at org.infinispan.distribution.ch.DefaultConsistentHash$Externalizer.readObject(DefaultConsistentHash.java:314) at org.infinispan.distribution.ch.DefaultConsistentHash$Externalizer.readObject(DefaultConsistentHash.java:294) at org.infinispan.marshall.jboss.ExternalizerTable$ExternalizerAdapter.readObject(ExternalizerTable.java:402) at org.infinispan.marshall.jboss.ExternalizerTable.readObject(ExternalizerTable.java:224) at org.infinispan.marshall.jboss.JBossMarshaller$ExternalizerTableProxy.readObject(JBossMarshaller.java:164) at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:351) at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209) at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:37) at org.infinispan.topology.CacheTopology$Externalizer.readObject(CacheTopology.java:132) at org.infinispan.topology.CacheTopology$Externalizer.readObject(CacheTopology.java:121) at org.infinispan.marshall.jboss.ExternalizerTable$ExternalizerAdapter.readObject(ExternalizerTable.java:402) at org.infinispan.marshall.jboss.ExternalizerTable.readObject(ExternalizerTable.java:224) at org.infinispan.marshall.jboss.JBossMarshaller$ExternalizerTableProxy.readObject(JBossMarshaller.java:164) at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:351) at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209) at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:37) at org.infinispan.remoting.responses.SuccessfulResponse$Externalizer.readObject(SuccessfulResponse.java:101) at org.infinispan.remoting.responses.SuccessfulResponse$Externalizer.readObject(SuccessfulResponse.java:86) at org.infinispan.marshall.jboss.ExternalizerTable$ExternalizerAdapter.readObject(ExternalizerTable.java:402) at org.infinispan.marshall.jboss.ExternalizerTable.readObject(ExternalizerTable.java:224) at org.infinispan.marshall.jboss.JBossMarshaller$ExternalizerTableProxy.readObject(JBossMarshaller.java:164) at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:351) at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209) at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:37) at org.infinispan.marshall.jboss.AbstractJBossMarshaller.objectFromObjectStream(AbstractJBossMarshaller.java:163 ) at org.infinispan.marshall.VersionAwareMarshaller.objectFromByteBuffer(VersionAwareMarshaller.java:116) at org.infinispan.marshall.AbstractDelegatingMarshaller.objectFromByteBuffer(AbstractDelegatingMarshaller.java:1 04) at org.infinispan.remoting.transport.jgroups.MarshallerAdapter.objectFromBuffer(MarshallerAdapter.java:50) at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:404) at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:249) at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:598) at org.jgroups.JChannel.up(JChannel.java:707) at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1020) at org.jgroups.protocols.FRAG2.up(FRAG2.java:181) at org.jgroups.protocols.FlowControl.up(FlowControl.java:418) at org.jgroups.protocols.FlowControl.up(FlowControl.java:400) at org.jgroups.protocols.pbcast.GMS.up(GMS.java:896) at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:245) at org.jgroups.protocols.UNICAST2.handleDataReceived(UNICAST2.java:765) at org.jgroups.protocols.UNICAST2.up(UNICAST2.java:420) at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:606) at org.jgroups.protocols.BARRIER.up(BARRIER.java:102) at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:143) at org.jgroups.protocols.FD.up(FD.java:253) at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:288) at org.jgroups.protocols.MERGE2.up(MERGE2.java:205) at org.jgroups.protocols.Discovery.up(Discovery.java:359) at org.jgroups.protocols.TP$ProtocolAdapter.up(TP.java:2616) at org.jgroups.protocols.TP.passMessageUp(TP.java:1263) at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1825) at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1798) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:724) I can supply my configuration if this would be useful.