6:33 p.m.
I'm (rarely) seeing this exception in one of my stress tests.. any
clue about what could be wrong?
I reported a similar one approx a year ago, in that case it was a
value type being unmarshalled as an instance of Class (was also never
resolved).
2014-08-05 00:22:29,521 WARN [CommandAwareRpcDispatcher]
(OOB-1,main-NodeD-22196) ISPN000220: Problems un-marshalling remote
command from byte buffer
java.lang.ClassCastException: java.lang.String cannot be cast to
org.infinispan.metadata.Metadata
at
org.infinispan.commands.write.PutKeyValueCommand.setParameters(PutKeyValueCommand.java:114)
at
org.infinispan.commands.RemoteCommandsFactory.fromStream(RemoteCommandsFactory.java:138)
at
org.infinispan.marshall.exts.ReplicableCommandExternalizer.readObject(ReplicableCommandExternalizer.java:85)
at
org.infinispan.marshall.exts.ReplicableCommandExternalizer.readObject(ReplicableCommandExternalizer.java:1)
at
org.infinispan.marshall.core.ExternalizerTable$ExternalizerAdapter.readObject(ExternalizerTable.java:409)
at org.infinispan.marshall.core.ExternalizerTable.readObject(ExternalizerTable.java:214)
at
org.infinispan.marshall.core.JBossMarshaller$ExternalizerTableProxy.readObject(JBossMarshaller.java:148)
at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:351)
at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)
at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:41)
at
org.infinispan.marshall.exts.ReplicableCommandExternalizer.readParameters(ReplicableCommandExternalizer.java:101)
at
org.infinispan.marshall.exts.CacheRpcCommandExternalizer.readObject(CacheRpcCommandExternalizer.java:153)
at
org.infinispan.marshall.exts.CacheRpcCommandExternalizer.readObject(CacheRpcCommandExternalizer.java:1)
at
org.infinispan.marshall.core.ExternalizerTable$ExternalizerAdapter.readObject(ExternalizerTable.java:409)
at org.infinispan.marshall.core.ExternalizerTable.readObject(ExternalizerTable.java:214)
at
org.infinispan.marshall.core.JBossMarshaller$ExternalizerTableProxy.readObject(JBossMarshaller.java:148)
at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:351)
at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)
at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:41)
at
org.infinispan.commons.marshall.jboss.AbstractJBossMarshaller.objectFromObjectStream(AbstractJBossMarshaller.java:135)
at
org.infinispan.marshall.core.VersionAwareMarshaller.objectFromByteBuffer(VersionAwareMarshaller.java:101)
at
org.infinispan.commons.marshall.AbstractDelegatingMarshaller.objectFromByteBuffer(AbstractDelegatingMarshaller.java:80)
at
org.infinispan.remoting.transport.jgroups.MarshallerAdapter.objectFromBuffer(MarshallerAdapter.java:28)
at
org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:204)
at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:460)
at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:377)
at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:250)
at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:674)
at org.jgroups.JChannel.up(JChannel.java:733)
at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1030)
at org.jgroups.protocols.RSVP.up(RSVP.java:190)
at org.jgroups.protocols.FRAG2.up(FRAG2.java:165)
at org.jgroups.protocols.FlowControl.up(FlowControl.java:390)
at org.jgroups.protocols.tom.TOA.up(TOA.java:121)
at org.jgroups.protocols.pbcast.GMS.up(GMS.java:1041)
at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:234)
at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1034)
at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:752)
at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:399)
at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:610)
at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:297)
at org.jgroups.protocols.Discovery.up(Discovery.java:245)
at org.jgroups.protocols.TP.passMessageUp(TP.java:1551)
at org.jgroups.protocols.TP$MyHandler.run(TP.java:1770)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
11:06 p.m.
It looks like the data was written by a different version of
PutKeyValueCommand than is trying to read it.
Make sure you're not mixing ISPN versions in the cluster and/or
accidentally clustering with another instance outside your test?
-Dennis
On 08/04/2014 06:33 PM, Sanne Grinovero wrote:
I'm (rarely) seeing this exception in one of my stress tests..
any
clue about what could be wrong?
I reported a similar one approx a year ago, in that case it was a
value type being unmarshalled as an instance of Class (was also never
resolved).
2014-08-05 00:22:29,521 WARN [CommandAwareRpcDispatcher]
(OOB-1,main-NodeD-22196) ISPN000220: Problems un-marshalling remote
command from byte buffer
java.lang.ClassCastException: java.lang.String cannot be cast to
org.infinispan.metadata.Metadata
at
org.infinispan.commands.write.PutKeyValueCommand.setParameters(PutKeyValueCommand.java:114)
at
org.infinispan.commands.RemoteCommandsFactory.fromStream(RemoteCommandsFactory.java:138)
at
org.infinispan.marshall.exts.ReplicableCommandExternalizer.readObject(ReplicableCommandExternalizer.java:85)
at
org.infinispan.marshall.exts.ReplicableCommandExternalizer.readObject(ReplicableCommandExternalizer.java:1)
at
org.infinispan.marshall.core.ExternalizerTable$ExternalizerAdapter.readObject(ExternalizerTable.java:409)
at org.infinispan.marshall.core.ExternalizerTable.readObject(ExternalizerTable.java:214)
at
org.infinispan.marshall.core.JBossMarshaller$ExternalizerTableProxy.readObject(JBossMarshaller.java:148)
at
org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:351)
at
org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)
at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:41)
at
org.infinispan.marshall.exts.ReplicableCommandExternalizer.readParameters(ReplicableCommandExternalizer.java:101)
at
org.infinispan.marshall.exts.CacheRpcCommandExternalizer.readObject(CacheRpcCommandExternalizer.java:153)
at
org.infinispan.marshall.exts.CacheRpcCommandExternalizer.readObject(CacheRpcCommandExternalizer.java:1)
at
org.infinispan.marshall.core.ExternalizerTable$ExternalizerAdapter.readObject(ExternalizerTable.java:409)
at org.infinispan.marshall.core.ExternalizerTable.readObject(ExternalizerTable.java:214)
at
org.infinispan.marshall.core.JBossMarshaller$ExternalizerTableProxy.readObject(JBossMarshaller.java:148)
at
org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:351)
at
org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)
at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:41)
at
org.infinispan.commons.marshall.jboss.AbstractJBossMarshaller.objectFromObjectStream(AbstractJBossMarshaller.java:135)
at
org.infinispan.marshall.core.VersionAwareMarshaller.objectFromByteBuffer(VersionAwareMarshaller.java:101)
at
org.infinispan.commons.marshall.AbstractDelegatingMarshaller.objectFromByteBuffer(AbstractDelegatingMarshaller.java:80)
at
org.infinispan.remoting.transport.jgroups.MarshallerAdapter.objectFromBuffer(MarshallerAdapter.java:28)
at
org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:204)
at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:460)
at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:377)
at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:250)
at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:674)
at org.jgroups.JChannel.up(JChannel.java:733)
at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1030)
at org.jgroups.protocols.RSVP.up(RSVP.java:190)
at org.jgroups.protocols.FRAG2.up(FRAG2.java:165)
at org.jgroups.protocols.FlowControl.up(FlowControl.java:390)
at org.jgroups.protocols.tom.TOA.up(TOA.java:121)
at org.jgroups.protocols.pbcast.GMS.up(GMS.java:1041)
at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:234)
at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1034)
at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:752)
at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:399)
at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:610)
at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:297)
at org.jgroups.protocols.Discovery.up(Discovery.java:245)
at org.jgroups.protocols.TP.passMessageUp(TP.java:1551)
at org.jgroups.protocols.TP$MyHandler.run(TP.java:1770)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
4:51 a.m.
Hi Dennis,
thanks for the ideas! but that's not possible. This is a single unit
test, run from a flat classpath, bound to localhost and no other JVMs
running on the same machine.
The only thing making "special" is that it's a rather long stress
test, it loops at high speed for at least 10 minutes before such
exceptions are thrown.
I suspect the stream is somehow corrupted under load, getting it to
invoke the wrong unmarshaller combination.
On 5 August 2014 05:06, Dennis Reed <dereed(a)redhat.com> wrote:
It looks like the data was written by a different version of
PutKeyValueCommand than is trying to read it.
Make sure you're not mixing ISPN versions in the cluster and/or
accidentally clustering with another instance outside your test?
-Dennis
On 08/04/2014 06:33 PM, Sanne Grinovero wrote:
> I'm (rarely) seeing this exception in one of my stress tests.. any
> clue about what could be wrong?
> I reported a similar one approx a year ago, in that case it was a
> value type being unmarshalled as an instance of Class (was also never
> resolved).
>
> 2014-08-05 00:22:29,521 WARN [CommandAwareRpcDispatcher]
> (OOB-1,main-NodeD-22196) ISPN000220: Problems un-marshalling remote
> command from byte buffer
> java.lang.ClassCastException: java.lang.String cannot be cast to
> org.infinispan.metadata.Metadata
> at
org.infinispan.commands.write.PutKeyValueCommand.setParameters(PutKeyValueCommand.java:114)
> at
org.infinispan.commands.RemoteCommandsFactory.fromStream(RemoteCommandsFactory.java:138)
> at
org.infinispan.marshall.exts.ReplicableCommandExternalizer.readObject(ReplicableCommandExternalizer.java:85)
> at
org.infinispan.marshall.exts.ReplicableCommandExternalizer.readObject(ReplicableCommandExternalizer.java:1)
> at
org.infinispan.marshall.core.ExternalizerTable$ExternalizerAdapter.readObject(ExternalizerTable.java:409)
> at
org.infinispan.marshall.core.ExternalizerTable.readObject(ExternalizerTable.java:214)
> at
org.infinispan.marshall.core.JBossMarshaller$ExternalizerTableProxy.readObject(JBossMarshaller.java:148)
> at
org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:351)
> at
org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)
> at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:41)
> at
org.infinispan.marshall.exts.ReplicableCommandExternalizer.readParameters(ReplicableCommandExternalizer.java:101)
> at
org.infinispan.marshall.exts.CacheRpcCommandExternalizer.readObject(CacheRpcCommandExternalizer.java:153)
> at
org.infinispan.marshall.exts.CacheRpcCommandExternalizer.readObject(CacheRpcCommandExternalizer.java:1)
> at
org.infinispan.marshall.core.ExternalizerTable$ExternalizerAdapter.readObject(ExternalizerTable.java:409)
> at
org.infinispan.marshall.core.ExternalizerTable.readObject(ExternalizerTable.java:214)
> at
org.infinispan.marshall.core.JBossMarshaller$ExternalizerTableProxy.readObject(JBossMarshaller.java:148)
> at
org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:351)
> at
org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)
> at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:41)
> at
org.infinispan.commons.marshall.jboss.AbstractJBossMarshaller.objectFromObjectStream(AbstractJBossMarshaller.java:135)
> at
org.infinispan.marshall.core.VersionAwareMarshaller.objectFromByteBuffer(VersionAwareMarshaller.java:101)
> at
org.infinispan.commons.marshall.AbstractDelegatingMarshaller.objectFromByteBuffer(AbstractDelegatingMarshaller.java:80)
> at
org.infinispan.remoting.transport.jgroups.MarshallerAdapter.objectFromBuffer(MarshallerAdapter.java:28)
> at
org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:204)
> at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:460)
> at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:377)
> at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:250)
> at
org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:674)
> at org.jgroups.JChannel.up(JChannel.java:733)
> at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1030)
> at org.jgroups.protocols.RSVP.up(RSVP.java:190)
> at org.jgroups.protocols.FRAG2.up(FRAG2.java:165)
> at org.jgroups.protocols.FlowControl.up(FlowControl.java:390)
> at org.jgroups.protocols.tom.TOA.up(TOA.java:121)
> at org.jgroups.protocols.pbcast.GMS.up(GMS.java:1041)
> at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:234)
> at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1034)
> at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:752)
> at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:399)
> at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:610)
> at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:297)
> at org.jgroups.protocols.Discovery.up(Discovery.java:245)
> at org.jgroups.protocols.TP.passMessageUp(TP.java:1551)
> at org.jgroups.protocols.TP$MyHandler.run(TP.java:1770)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
8:01 a.m.
On 05 Aug 2014, at 01:33, Sanne Grinovero <sanne(a)infinispan.org> wrote:
I'm (rarely) seeing this exception in one of my stress tests..
any
clue about what could be wrong?
Hmmm, it smells like a concurrency issue, e.g. buffer mixup, in either jboss marshalling,
jgroups or the externalizer layer in Infinispan.
I reported a similar one approx a year ago, in that case it was a
value type being unmarshalled as an instance of Class (was also never
resolved).
^ Do you have a JIRA for it?
Please definitely create one for this new CCE.
Cheers,
2014-08-05 00:22:29,521 WARN [CommandAwareRpcDispatcher]
(OOB-1,main-NodeD-22196) ISPN000220: Problems un-marshalling remote
command from byte buffer
java.lang.ClassCastException: java.lang.String cannot be cast to
org.infinispan.metadata.Metadata
at
org.infinispan.commands.write.PutKeyValueCommand.setParameters(PutKeyValueCommand.java:114)
at
org.infinispan.commands.RemoteCommandsFactory.fromStream(RemoteCommandsFactory.java:138)
at
org.infinispan.marshall.exts.ReplicableCommandExternalizer.readObject(ReplicableCommandExternalizer.java:85)
at
org.infinispan.marshall.exts.ReplicableCommandExternalizer.readObject(ReplicableCommandExternalizer.java:1)
at
org.infinispan.marshall.core.ExternalizerTable$ExternalizerAdapter.readObject(ExternalizerTable.java:409)
at org.infinispan.marshall.core.ExternalizerTable.readObject(ExternalizerTable.java:214)
at
org.infinispan.marshall.core.JBossMarshaller$ExternalizerTableProxy.readObject(JBossMarshaller.java:148)
at
org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:351)
at
org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)
at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:41)
at
org.infinispan.marshall.exts.ReplicableCommandExternalizer.readParameters(ReplicableCommandExternalizer.java:101)
at
org.infinispan.marshall.exts.CacheRpcCommandExternalizer.readObject(CacheRpcCommandExternalizer.java:153)
at
org.infinispan.marshall.exts.CacheRpcCommandExternalizer.readObject(CacheRpcCommandExternalizer.java:1)
at
org.infinispan.marshall.core.ExternalizerTable$ExternalizerAdapter.readObject(ExternalizerTable.java:409)
at org.infinispan.marshall.core.ExternalizerTable.readObject(ExternalizerTable.java:214)
at
org.infinispan.marshall.core.JBossMarshaller$ExternalizerTableProxy.readObject(JBossMarshaller.java:148)
at
org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:351)
at
org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)
at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:41)
at
org.infinispan.commons.marshall.jboss.AbstractJBossMarshaller.objectFromObjectStream(AbstractJBossMarshaller.java:135)
at
org.infinispan.marshall.core.VersionAwareMarshaller.objectFromByteBuffer(VersionAwareMarshaller.java:101)
at
org.infinispan.commons.marshall.AbstractDelegatingMarshaller.objectFromByteBuffer(AbstractDelegatingMarshaller.java:80)
at
org.infinispan.remoting.transport.jgroups.MarshallerAdapter.objectFromBuffer(MarshallerAdapter.java:28)
at
org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:204)
at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:460)
at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:377)
at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:250)
at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:674)
at org.jgroups.JChannel.up(JChannel.java:733)
at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1030)
at org.jgroups.protocols.RSVP.up(RSVP.java:190)
at org.jgroups.protocols.FRAG2.up(FRAG2.java:165)
at org.jgroups.protocols.FlowControl.up(FlowControl.java:390)
at org.jgroups.protocols.tom.TOA.up(TOA.java:121)
at org.jgroups.protocols.pbcast.GMS.up(GMS.java:1041)
at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:234)
at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1034)
at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:752)
at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:399)
at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:610)
at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:297)
at org.jgroups.protocols.Discovery.up(Discovery.java:245)
at org.jgroups.protocols.TP.passMessageUp(TP.java:1551)
at org.jgroups.protocols.TP$MyHandler.run(TP.java:1770)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
--
Galder Zamarreño
galder(a)redhat.com
twitter.com/galderz
9:02 a.m.
I've figured it out.
I'll not explain it yet, it's such a nice puzzler :-P
@Galder you were close: let me say JGroups is the only one not related.
On 5 August 2014 14:01, Galder Zamarreño <galder(a)redhat.com> wrote:
On 05 Aug 2014, at 01:33, Sanne Grinovero <sanne(a)infinispan.org> wrote:
> I'm (rarely) seeing this exception in one of my stress tests.. any
> clue about what could be wrong?
Hmmm, it smells like a concurrency issue, e.g. buffer mixup, in either jboss marshalling,
jgroups or the externalizer layer in Infinispan.
> I reported a similar one approx a year ago, in that case it was a
> value type being unmarshalled as an instance of Class (was also never
> resolved).
^ Do you have a JIRA for it?
Please definitely create one for this new CCE.
Cheers,
>
> 2014-08-05 00:22:29,521 WARN [CommandAwareRpcDispatcher]
> (OOB-1,main-NodeD-22196) ISPN000220: Problems un-marshalling remote
> command from byte buffer
> java.lang.ClassCastException: java.lang.String cannot be cast to
> org.infinispan.metadata.Metadata
> at
org.infinispan.commands.write.PutKeyValueCommand.setParameters(PutKeyValueCommand.java:114)
> at
org.infinispan.commands.RemoteCommandsFactory.fromStream(RemoteCommandsFactory.java:138)
> at
org.infinispan.marshall.exts.ReplicableCommandExternalizer.readObject(ReplicableCommandExternalizer.java:85)
> at
org.infinispan.marshall.exts.ReplicableCommandExternalizer.readObject(ReplicableCommandExternalizer.java:1)
> at
org.infinispan.marshall.core.ExternalizerTable$ExternalizerAdapter.readObject(ExternalizerTable.java:409)
> at
org.infinispan.marshall.core.ExternalizerTable.readObject(ExternalizerTable.java:214)
> at
org.infinispan.marshall.core.JBossMarshaller$ExternalizerTableProxy.readObject(JBossMarshaller.java:148)
> at
org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:351)
> at
org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)
> at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:41)
> at
org.infinispan.marshall.exts.ReplicableCommandExternalizer.readParameters(ReplicableCommandExternalizer.java:101)
> at
org.infinispan.marshall.exts.CacheRpcCommandExternalizer.readObject(CacheRpcCommandExternalizer.java:153)
> at
org.infinispan.marshall.exts.CacheRpcCommandExternalizer.readObject(CacheRpcCommandExternalizer.java:1)
> at
org.infinispan.marshall.core.ExternalizerTable$ExternalizerAdapter.readObject(ExternalizerTable.java:409)
> at
org.infinispan.marshall.core.ExternalizerTable.readObject(ExternalizerTable.java:214)
> at
org.infinispan.marshall.core.JBossMarshaller$ExternalizerTableProxy.readObject(JBossMarshaller.java:148)
> at
org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:351)
> at
org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)
> at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:41)
> at
org.infinispan.commons.marshall.jboss.AbstractJBossMarshaller.objectFromObjectStream(AbstractJBossMarshaller.java:135)
> at
org.infinispan.marshall.core.VersionAwareMarshaller.objectFromByteBuffer(VersionAwareMarshaller.java:101)
> at
org.infinispan.commons.marshall.AbstractDelegatingMarshaller.objectFromByteBuffer(AbstractDelegatingMarshaller.java:80)
> at
org.infinispan.remoting.transport.jgroups.MarshallerAdapter.objectFromBuffer(MarshallerAdapter.java:28)
> at
org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:204)
> at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:460)
> at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:377)
> at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:250)
> at
org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:674)
> at org.jgroups.JChannel.up(JChannel.java:733)
> at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1030)
> at org.jgroups.protocols.RSVP.up(RSVP.java:190)
> at org.jgroups.protocols.FRAG2.up(FRAG2.java:165)
> at org.jgroups.protocols.FlowControl.up(FlowControl.java:390)
> at org.jgroups.protocols.tom.TOA.up(TOA.java:121)
> at org.jgroups.protocols.pbcast.GMS.up(GMS.java:1041)
> at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:234)
> at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1034)
> at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:752)
> at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:399)
> at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:610)
> at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:297)
> at org.jgroups.protocols.Discovery.up(Discovery.java:245)
> at org.jgroups.protocols.TP.passMessageUp(TP.java:1551)
> at org.jgroups.protocols.TP$MyHandler.run(TP.java:1770)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
--
Galder Zamarreño
galder(a)redhat.com
twitter.com/galderz
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
1:04 p.m.
Sanne, you really need to say more, I was hoping to see more information in
the PR/JIRA but both descriptions are very terse :)
If I understand correctly, the problem is that you're keeping a reference
to a cache value and modifying it without a put(), which could break
marshalling for a concurrent put()?
Cheers
Dan
On Tue, Aug 5, 2014 at 5:02 PM, Sanne Grinovero <sanne(a)infinispan.org>
wrote:
I've figured it out.
I'll not explain it yet, it's such a nice puzzler :-P
@Galder you were close: let me say JGroups is the only one not related.
On 5 August 2014 14:01, Galder Zamarreño <galder(a)redhat.com> wrote:
>
> On 05 Aug 2014, at 01:33, Sanne Grinovero <sanne(a)infinispan.org> wrote:
>
>> I'm (rarely) seeing this exception in one of my stress tests.. any
>> clue about what could be wrong?
>
> Hmmm, it smells like a concurrency issue, e.g. buffer mixup, in either
jboss marshalling, jgroups or the externalizer layer in Infinispan.
>
>> I reported a similar one approx a year ago, in that case it was a
>> value type being unmarshalled as an instance of Class (was also never
>> resolved).
>
> ^ Do you have a JIRA for it?
>
> Please definitely create one for this new CCE.
>
> Cheers,
>
>>
>> 2014-08-05 00:22:29,521 WARN [CommandAwareRpcDispatcher]
>> (OOB-1,main-NodeD-22196) ISPN000220: Problems un-marshalling remote
>> command from byte buffer
>> java.lang.ClassCastException: java.lang.String cannot be cast to
>> org.infinispan.metadata.Metadata
>> at
org.infinispan.commands.write.PutKeyValueCommand.setParameters(PutKeyValueCommand.java:114)
>> at
org.infinispan.commands.RemoteCommandsFactory.fromStream(RemoteCommandsFactory.java:138)
>> at
org.infinispan.marshall.exts.ReplicableCommandExternalizer.readObject(ReplicableCommandExternalizer.java:85)
>> at
org.infinispan.marshall.exts.ReplicableCommandExternalizer.readObject(ReplicableCommandExternalizer.java:1)
>> at
org.infinispan.marshall.core.ExternalizerTable$ExternalizerAdapter.readObject(ExternalizerTable.java:409)
>> at
org.infinispan.marshall.core.ExternalizerTable.readObject(ExternalizerTable.java:214)
>> at
org.infinispan.marshall.core.JBossMarshaller$ExternalizerTableProxy.readObject(JBossMarshaller.java:148)
>> at
org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:351)
>> at
org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)
>> at
org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:41)
>> at
org.infinispan.marshall.exts.ReplicableCommandExternalizer.readParameters(ReplicableCommandExternalizer.java:101)
>> at
org.infinispan.marshall.exts.CacheRpcCommandExternalizer.readObject(CacheRpcCommandExternalizer.java:153)
>> at
org.infinispan.marshall.exts.CacheRpcCommandExternalizer.readObject(CacheRpcCommandExternalizer.java:1)
>> at
org.infinispan.marshall.core.ExternalizerTable$ExternalizerAdapter.readObject(ExternalizerTable.java:409)
>> at
org.infinispan.marshall.core.ExternalizerTable.readObject(ExternalizerTable.java:214)
>> at
org.infinispan.marshall.core.JBossMarshaller$ExternalizerTableProxy.readObject(JBossMarshaller.java:148)
>> at
org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:351)
>> at
org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)
>> at
org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:41)
>> at
org.infinispan.commons.marshall.jboss.AbstractJBossMarshaller.objectFromObjectStream(AbstractJBossMarshaller.java:135)
>> at
org.infinispan.marshall.core.VersionAwareMarshaller.objectFromByteBuffer(VersionAwareMarshaller.java:101)
>> at
org.infinispan.commons.marshall.AbstractDelegatingMarshaller.objectFromByteBuffer(AbstractDelegatingMarshaller.java:80)
>> at
org.infinispan.remoting.transport.jgroups.MarshallerAdapter.objectFromBuffer(MarshallerAdapter.java:28)
>> at
org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:204)
>> at
org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:460)
>> at
org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:377)
>> at
org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:250)
>> at
org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:674)
>> at org.jgroups.JChannel.up(JChannel.java:733)
>> at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1030)
>> at org.jgroups.protocols.RSVP.up(RSVP.java:190)
>> at org.jgroups.protocols.FRAG2.up(FRAG2.java:165)
>> at org.jgroups.protocols.FlowControl.up(FlowControl.java:390)
>> at org.jgroups.protocols.tom.TOA.up(TOA.java:121)
>> at org.jgroups.protocols.pbcast.GMS.up(GMS.java:1041)
>> at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:234)
>> at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1034)
>> at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:752)
>> at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:399)
>> at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:610)
>> at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:297)
>> at org.jgroups.protocols.Discovery.up(Discovery.java:245)
>> at org.jgroups.protocols.TP.passMessageUp(TP.java:1551)
>> at org.jgroups.protocols.TP$MyHandler.run(TP.java:1770)
>> at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:745)
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev(a)lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
>
> --
> Galder Zamarreño
> galder(a)redhat.com
> twitter.com/galderz
>
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
1:26 p.m.
On 6 August 2014 19:04, Dan Berindei <dan.berindei(a)gmail.com> wrote:
Sanne, you really need to say more, I was hoping to see more
information in
the PR/JIRA but both descriptions are very terse :)
hum ok if I need to spoil the fun :)
If I understand correctly, the problem is that you're keeping a
reference to
a cache value and modifying it without a put(), which could break
marshalling for a concurrent put()?
Exactly, my problem was that I would write a concurrent structure, but
without preventing this structure from being manipulated from
different threads.
The marshaller would first write the size, then iterate on the
elements and write each of them, but there could be a mismatch between
the written size and the actually written entries, causing such weird
exceptions.
I guess the idea of writing a concurrent map while keeping a reference
to it might be a dumb mistake, but the exception and error happening
in such a different area makes it "interesting" to correlate to the
actual cause.
Cheers,
Sanne
4276
days inactive
4278
days old
infinispan-dev@lists.jboss.org
6 comments
4 participants
participants (4)
-
Dan Berindei -
Dennis Reed -
Galder Zamarreño -
Sanne Grinovero