I don't think it's an externalizer issue, as I also see some
exceptions on the node that generates state:
2011-06-09 18:16:18,250 ERROR
[org.infinispan.remoting.transport.jgroups.JGroupsTransport]
(STREAMING_STATE_TRANSFER-sender-1,Infinispan-Cluster,NodeA-57902)
ISPN00095: Caught while responding to state transfer request
org.infinispan.statetransfer.StateTransferException:
java.util.concurrent.TimeoutException:
STREAMING_STATE_TRANSFER-sender-1,Infinispan-Cluster,NodeA-57902 could
not obtain exclusive processing lock after 10 seconds. Locks in
question are java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock@a35c90[Read
locks = 1] and
java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock@111fb7f[Unlocked]
at
org.infinispan.statetransfer.StateTransferManagerImpl.generateState(StateTransferManagerImpl.java:177)
at
org.infinispan.remoting.InboundInvocationHandlerImpl.generateState(InboundInvocationHandlerImpl.java:248)
at
org.infinispan.remoting.transport.jgroups.JGroupsTransport.getState(JGroupsTransport.java:585)
at
org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.handleUpEvent(MessageDispatcher.java:690)
at
org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:771)
at org.jgroups.JChannel.up(JChannel.java:1484)
at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1074)
at
org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER$StateProviderHandler.process(STREAMING_STATE_TRANSFER.java:651)
at
org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER$StateProviderThreadSpawner$1.run(STREAMING_STATE_TRANSFER.java:580)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
Caused by: java.util.concurrent.TimeoutException:
STREAMING_STATE_TRANSFER-sender-1,Infinispan-Cluster,NodeA-57902 could
not obtain exclusive processing lock after 10 seconds. Locks in
question are java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock@a35c90[Read
locks = 1] and
java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock@111fb7f[Unlocked]
at
org.infinispan.remoting.transport.jgroups.JGroupsDistSync.acquireProcessingLock(JGroupsDistSync.java:100)
at
org.infinispan.statetransfer.StateTransferManagerImpl.generateTransactionLog(StateTransferManagerImpl.java:204)
at
org.infinispan.statetransfer.StateTransferManagerImpl.generateState(StateTransferManagerImpl.java:167)
... 11 more
I guess we could write an error marker in the stream to prevent the
EOFException on the receiving side, but the end result would be the
same.
Dan
On Thu, Jun 9, 2011 at 5:58 PM, Sanne Grinovero <sanne(a)infinispan.org> wrote:
Hello all,
if I happen to look at the console while the tests are running, I see
this exception popup very often:
2011-06-09 15:32:18,092 ERROR [JGroupsTransport]
(Incoming-1,Infinispan-Cluster,NodeB-32230) ISPN00096: Caught while
requesting or applying state
org.infinispan.statetransfer.StateTransferException:
java.io.EOFException: Read past end of file
at
org.infinispan.statetransfer.StateTransferManagerImpl.applyState(StateTransferManagerImpl.java:333)
at
org.infinispan.remoting.InboundInvocationHandlerImpl.applyState(InboundInvocationHandlerImpl.java:230)
at
org.infinispan.remoting.transport.jgroups.JGroupsTransport.setState(JGroupsTransport.java:602)
at
org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.handleUpEvent(MessageDispatcher.java:711)
at
org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:771)
at org.jgroups.JChannel.up(JChannel.java:1441)
at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1074)
at
org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER.connectToStateProvider(STREAMING_STATE_TRANSFER.java:523)
at
org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER.handleStateRsp(STREAMING_STATE_TRANSFER.java:462)
at
org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER.up(STREAMING_STATE_TRANSFER.java:223)
at org.jgroups.protocols.FRAG2.up(FRAG2.java:189)
at org.jgroups.protocols.FC.up(FC.java:479)
at org.jgroups.protocols.pbcast.GMS.up(GMS.java:891)
at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:246)
at org.jgroups.protocols.UNICAST.handleDataReceived(UNICAST.java:613)
at org.jgroups.protocols.UNICAST.up(UNICAST.java:294)
at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:703)
at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:133)
at org.jgroups.protocols.FD.up(FD.java:275)
at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:275)
at org.jgroups.protocols.MERGE2.up(MERGE2.java:209)
at org.jgroups.protocols.Discovery.up(Discovery.java:291)
at org.jgroups.protocols.TP.passMessageUp(TP.java:1102)
at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1658)
at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1640)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.EOFException: Read past end of file
at org.jboss.marshalling.SimpleDataInput.eofOnRead(SimpleDataInput.java:126)
at
org.jboss.marshalling.SimpleDataInput.readUnsignedByteDirect(SimpleDataInput.java:263)
at
org.jboss.marshalling.SimpleDataInput.readUnsignedByte(SimpleDataInput.java:224)
at
org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)
at
org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:37)
at
org.infinispan.marshall.jboss.GenericJBossMarshaller.objectFromObjectStream(GenericJBossMarshaller.java:192)
at
org.infinispan.marshall.VersionAwareMarshaller.objectFromObjectStream(VersionAwareMarshaller.java:190)
at
org.infinispan.statetransfer.StateTransferManagerImpl.processCommitLog(StateTransferManagerImpl.java:230)
at
org.infinispan.statetransfer.StateTransferManagerImpl.applyTransactionLog(StateTransferManagerImpl.java:252)
at
org.infinispan.statetransfer.StateTransferManagerImpl.applyState(StateTransferManagerImpl.java:322)
... 27 more
But I'm not sure if it's an issue, as it seems tests are not failing.
I consider a "Read past end of file" quite suspiciously looking; would
it be possible to think that some internal Externalizer is writing
less bytes than what it's attempting to read?
Is there something clever I could do to understand which object the
marshaller is trying to read when something like this is happening?
I've found debugging this quite hard.
Also, it doesn't look like our externalizers have a good test
coverage; They are likely implicitly tested as I assume that nothing
would work if they aren't, but still it looks like we have no explicit
tests for them?
Cheers,
Sanne
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev