11:44 a.m.
I think I know what is going on here and it has nothing to do with
Galder's externalizers.
At the moment when state producer generates state transfer it cannot
lock processing lock which is being held by some other reader thread
which got bogged down in container due to lock or whatever. State
producer thread in turn times out waiting for this lock and generates
garbage stream to state receiver. That is why you see these
java.io.EOFException: Read past end of file on receiver.
I think we can do the following. Reduce timeout on this processing lock
or increase state transfer timeout. Reader lock will be released and
state will be generated. However, this will in turn lead to other
problems like failed tx and what not. I initially thought that this is
related to FLUSH being removed but the same thing happens with FLUSH in
stack. It would be great if we can push this together and figure out
what is the best course of action.
Vladimir
On 11-06-09 11:24 AM, Dan Berindei wrote:
I don't think it's an externalizer issue, as I also see some
exceptions on the node that generates state:
2011-06-09 18:16:18,250 ERROR
[org.infinispan.remoting.transport.jgroups.JGroupsTransport]
(STREAMING_STATE_TRANSFER-sender-1,Infinispan-Cluster,NodeA-57902)
ISPN00095: Caught while responding to state transfer request
org.infinispan.statetransfer.StateTransferException:
java.util.concurrent.TimeoutException:
STREAMING_STATE_TRANSFER-sender-1,Infinispan-Cluster,NodeA-57902 could
not obtain exclusive processing lock after 10 seconds. Locks in
question are java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock@a35c90[Read
locks = 1] and
java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock@111fb7f[Unlocked]
at
org.infinispan.statetransfer.StateTransferManagerImpl.generateState(StateTransferManagerImpl.java:177)
at
org.infinispan.remoting.InboundInvocationHandlerImpl.generateState(InboundInvocationHandlerImpl.java:248)
at
org.infinispan.remoting.transport.jgroups.JGroupsTransport.getState(JGroupsTransport.java:585)
at
org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.handleUpEvent(MessageDispatcher.java:690)
at
org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:771)
at org.jgroups.JChannel.up(JChannel.java:1484)
at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1074)
at
org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER$StateProviderHandler.process(STREAMING_STATE_TRANSFER.java:651)
at
org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER$StateProviderThreadSpawner$1.run(STREAMING_STATE_TRANSFER.java:580)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
Caused by: java.util.concurrent.TimeoutException:
STREAMING_STATE_TRANSFER-sender-1,Infinispan-Cluster,NodeA-57902 could
not obtain exclusive processing lock after 10 seconds. Locks in
question are java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock@a35c90[Read
locks = 1] and
java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock@111fb7f[Unlocked]
at
org.infinispan.remoting.transport.jgroups.JGroupsDistSync.acquireProcessingLock(JGroupsDistSync.java:100)
at
org.infinispan.statetransfer.StateTransferManagerImpl.generateTransactionLog(StateTransferManagerImpl.java:204)
at
org.infinispan.statetransfer.StateTransferManagerImpl.generateState(StateTransferManagerImpl.java:167)
... 11 more
I guess we could write an error marker in the stream to prevent the
EOFException on the receiving side, but the end result would be the
same.
Dan
On Thu, Jun 9, 2011 at 5:58 PM, Sanne Grinovero<sanne(a)infinispan.org> wrote:
> Hello all,
> if I happen to look at the console while the tests are running, I see
> this exception popup very often:
>
> 2011-06-09 15:32:18,092 ERROR [JGroupsTransport]
> (Incoming-1,Infinispan-Cluster,NodeB-32230) ISPN00096: Caught while
> requesting or applying state
> org.infinispan.statetransfer.StateTransferException:
> java.io.EOFException: Read past end of file
> at
org.infinispan.statetransfer.StateTransferManagerImpl.applyState(StateTransferManagerImpl.java:333)
> at
org.infinispan.remoting.InboundInvocationHandlerImpl.applyState(InboundInvocationHandlerImpl.java:230)
> at
org.infinispan.remoting.transport.jgroups.JGroupsTransport.setState(JGroupsTransport.java:602)
> at
org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.handleUpEvent(MessageDispatcher.java:711)
> at
org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:771)
> at org.jgroups.JChannel.up(JChannel.java:1441)
> at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1074)
> at
org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER.connectToStateProvider(STREAMING_STATE_TRANSFER.java:523)
> at
org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER.handleStateRsp(STREAMING_STATE_TRANSFER.java:462)
> at
org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER.up(STREAMING_STATE_TRANSFER.java:223)
> at org.jgroups.protocols.FRAG2.up(FRAG2.java:189)
> at org.jgroups.protocols.FC.up(FC.java:479)
> at org.jgroups.protocols.pbcast.GMS.up(GMS.java:891)
> at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:246)
> at org.jgroups.protocols.UNICAST.handleDataReceived(UNICAST.java:613)
> at org.jgroups.protocols.UNICAST.up(UNICAST.java:294)
> at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:703)
> at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:133)
> at org.jgroups.protocols.FD.up(FD.java:275)
> at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:275)
> at org.jgroups.protocols.MERGE2.up(MERGE2.java:209)
> at org.jgroups.protocols.Discovery.up(Discovery.java:291)
> at org.jgroups.protocols.TP.passMessageUp(TP.java:1102)
> at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1658)
> at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1640)
> at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: java.io.EOFException: Read past end of file
> at org.jboss.marshalling.SimpleDataInput.eofOnRead(SimpleDataInput.java:126)
> at
org.jboss.marshalling.SimpleDataInput.readUnsignedByteDirect(SimpleDataInput.java:263)
> at
org.jboss.marshalling.SimpleDataInput.readUnsignedByte(SimpleDataInput.java:224)
> at
org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)
> at
org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:37)
> at
org.infinispan.marshall.jboss.GenericJBossMarshaller.objectFromObjectStream(GenericJBossMarshaller.java:192)
> at
org.infinispan.marshall.VersionAwareMarshaller.objectFromObjectStream(VersionAwareMarshaller.java:190)
> at
org.infinispan.statetransfer.StateTransferManagerImpl.processCommitLog(StateTransferManagerImpl.java:230)
> at
org.infinispan.statetransfer.StateTransferManagerImpl.applyTransactionLog(StateTransferManagerImpl.java:252)
> at
org.infinispan.statetransfer.StateTransferManagerImpl.applyState(StateTransferManagerImpl.java:322)
> ... 27 more
>
> But I'm not sure if it's an issue, as it seems tests are not failing.
> I consider a "Read past end of file" quite suspiciously looking; would
> it be possible to think that some internal Externalizer is writing
> less bytes than what it's attempting to read?
> Is there something clever I could do to understand which object the
> marshaller is trying to read when something like this is happening?
> I've found debugging this quite hard.
>
> Also, it doesn't look like our externalizers have a good test
> coverage; They are likely implicitly tested as I assume that nothing
> would work if they aren't, but still it looks like we have no explicit
> tests for them?
>
> Cheers,
> Sanne
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev