<div dir="ltr">Sanne, you really need to say more, I was hoping to see more information in the PR/JIRA but both descriptions are very terse :)<div><br></div><div>If I understand correctly, the problem is that you're keeping a reference to a cache value and modifying it without a put(), which could break marshalling for a concurrent put()?</div>
<div><br></div><div>Cheers</div><div>Dan</div><div><br></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Tue, Aug 5, 2014 at 5:02 PM, Sanne Grinovero <span dir="ltr"><<a href="mailto:sanne@infinispan.org" target="_blank">sanne@infinispan.org</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">I've figured it out.<br>
I'll not explain it yet, it's such a nice puzzler :-P<br>
@Galder you were close: let me say JGroups is the only one not related.<br>
<div class="HOEnZb"><div class="h5"><br>
On 5 August 2014 14:01, Galder Zamarreño <<a href="mailto:galder@redhat.com">galder@redhat.com</a>> wrote:<br>
><br>
> On 05 Aug 2014, at 01:33, Sanne Grinovero <<a href="mailto:sanne@infinispan.org">sanne@infinispan.org</a>> wrote:<br>
><br>
>> I'm (rarely) seeing this exception in one of my stress tests.. any<br>
>> clue about what could be wrong?<br>
><br>
> Hmmm, it smells like a concurrency issue, e.g. buffer mixup, in either jboss marshalling, jgroups or the externalizer layer in Infinispan.<br>
><br>
>> I reported a similar one approx a year ago, in that case it was a<br>
>> value type being unmarshalled as an instance of Class (was also never<br>
>> resolved).<br>
><br>
> ^ Do you have a JIRA for it?<br>
><br>
> Please definitely create one for this new CCE.<br>
><br>
> Cheers,<br>
><br>
>><br>
>> 2014-08-05 00:22:29,521 WARN [CommandAwareRpcDispatcher]<br>
>> (OOB-1,main-NodeD-22196) ISPN000220: Problems un-marshalling remote<br>
>> command from byte buffer<br>
>> java.lang.ClassCastException: java.lang.String cannot be cast to<br>
>> org.infinispan.metadata.Metadata<br>
>> at org.infinispan.commands.write.PutKeyValueCommand.setParameters(PutKeyValueCommand.java:114)<br>
>> at org.infinispan.commands.RemoteCommandsFactory.fromStream(RemoteCommandsFactory.java:138)<br>
>> at org.infinispan.marshall.exts.ReplicableCommandExternalizer.readObject(ReplicableCommandExternalizer.java:85)<br>
>> at org.infinispan.marshall.exts.ReplicableCommandExternalizer.readObject(ReplicableCommandExternalizer.java:1)<br>
>> at org.infinispan.marshall.core.ExternalizerTable$ExternalizerAdapter.readObject(ExternalizerTable.java:409)<br>
>> at org.infinispan.marshall.core.ExternalizerTable.readObject(ExternalizerTable.java:214)<br>
>> at org.infinispan.marshall.core.JBossMarshaller$ExternalizerTableProxy.readObject(JBossMarshaller.java:148)<br>
>> at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:351)<br>
>> at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)<br>
>> at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:41)<br>
>> at org.infinispan.marshall.exts.ReplicableCommandExternalizer.readParameters(ReplicableCommandExternalizer.java:101)<br>
>> at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.readObject(CacheRpcCommandExternalizer.java:153)<br>
>> at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.readObject(CacheRpcCommandExternalizer.java:1)<br>
>> at org.infinispan.marshall.core.ExternalizerTable$ExternalizerAdapter.readObject(ExternalizerTable.java:409)<br>
>> at org.infinispan.marshall.core.ExternalizerTable.readObject(ExternalizerTable.java:214)<br>
>> at org.infinispan.marshall.core.JBossMarshaller$ExternalizerTableProxy.readObject(JBossMarshaller.java:148)<br>
>> at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:351)<br>
>> at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)<br>
>> at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:41)<br>
>> at org.infinispan.commons.marshall.jboss.AbstractJBossMarshaller.objectFromObjectStream(AbstractJBossMarshaller.java:135)<br>
>> at org.infinispan.marshall.core.VersionAwareMarshaller.objectFromByteBuffer(VersionAwareMarshaller.java:101)<br>
>> at org.infinispan.commons.marshall.AbstractDelegatingMarshaller.objectFromByteBuffer(AbstractDelegatingMarshaller.java:80)<br>
>> at org.infinispan.remoting.transport.jgroups.MarshallerAdapter.objectFromBuffer(MarshallerAdapter.java:28)<br>
>> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:204)<br>
>> at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:460)<br>
>> at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:377)<br>
>> at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:250)<br>
>> at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:674)<br>
>> at org.jgroups.JChannel.up(JChannel.java:733)<br>
>> at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1030)<br>
>> at org.jgroups.protocols.RSVP.up(RSVP.java:190)<br>
>> at org.jgroups.protocols.FRAG2.up(FRAG2.java:165)<br>
>> at org.jgroups.protocols.FlowControl.up(FlowControl.java:390)<br>
>> at org.jgroups.protocols.tom.TOA.up(TOA.java:121)<br>
>> at org.jgroups.protocols.pbcast.GMS.up(GMS.java:1041)<br>
>> at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:234)<br>
>> at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1034)<br>
>> at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:752)<br>
>> at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:399)<br>
>> at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:610)<br>
>> at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:297)<br>
>> at org.jgroups.protocols.Discovery.up(Discovery.java:245)<br>
>> at org.jgroups.protocols.TP.passMessageUp(TP.java:1551)<br>
>> at <a href="http://org.jgroups.protocols.TP" target="_blank">org.jgroups.protocols.TP</a>$MyHandler.run(TP.java:1770)<br>
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)<br>
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)<br>
>> at java.lang.Thread.run(Thread.java:745)<br>
>> _______________________________________________<br>
>> infinispan-dev mailing list<br>
>> <a href="mailto:infinispan-dev@lists.jboss.org">infinispan-dev@lists.jboss.org</a><br>
>> <a href="https://lists.jboss.org/mailman/listinfo/infinispan-dev" target="_blank">https://lists.jboss.org/mailman/listinfo/infinispan-dev</a><br>
><br>
><br>
> --<br>
> Galder Zamarreño<br>
> <a href="mailto:galder@redhat.com">galder@redhat.com</a><br>
> <a href="http://twitter.com/galderz" target="_blank">twitter.com/galderz</a><br>
><br>
><br>
> _______________________________________________<br>
> infinispan-dev mailing list<br>
> <a href="mailto:infinispan-dev@lists.jboss.org">infinispan-dev@lists.jboss.org</a><br>
> <a href="https://lists.jboss.org/mailman/listinfo/infinispan-dev" target="_blank">https://lists.jboss.org/mailman/listinfo/infinispan-dev</a><br>
<br>
_______________________________________________<br>
infinispan-dev mailing list<br>
<a href="mailto:infinispan-dev@lists.jboss.org">infinispan-dev@lists.jboss.org</a><br>
<a href="https://lists.jboss.org/mailman/listinfo/infinispan-dev" target="_blank">https://lists.jboss.org/mailman/listinfo/infinispan-dev</a></div></div></blockquote></div><br></div>