[ https://jira.jboss.org/jira/browse/ISPN-335?page=com.atlassian.jira.plugi... ] Manik Surtani updated ISPN-335: ------------------------------- Fix Version/s: 4.0.0.GA -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/jira/browse/ISPN-335?page=com.atlassian.jira.plugi... ] Work on ISPN-335 started by Galder Zamarreno. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/jira/browse/ISPN-335?page=com.atlassian.jira.plugi... ] Galder Zamarreno updated ISPN-335: ---------------------------------- Summary: EOFException when fetchPersistentState (was: fetchPersistentState generate an exception) Description: In a clustered replicated cache, when I flag the fetchPersistentState attribute to true for a FileStore, Infinispan generate an exception. 2010-01-20 15:33:27,935 9829 ERROR [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-1,Infinispan-Cluster,eq-65253:) Caught while requesting or applying state org.infinispan.statetransfer.StateTransferException: java.io.EOFException: Read past end of file at org.infinispan.statetransfer.StateTransferManagerImpl.assertDelimited(StateTransferManagerImpl.java:382) at org.infinispan.statetransfer.StateTransferManagerImpl.applyState(StateTransferManagerImpl.java:307) at org.infinispan.remoting.InboundInvocationHandlerImpl.applyState(InboundInvocationHandlerImpl.java:73) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.setState(JGroupsTransport.java:564) at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.handleUpEvent(MessageDispatcher.java:655) at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:715) at org.jgroups.JChannel.up(JChannel.java:1413) at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:829) at org.jgroups.protocols.pbcast.FLUSH.up(FLUSH.java:489) at org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER.connectToStateProvider(STREAMING_STATE_TRANSFER.java:526) at org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER.handleStateRsp(STREAMING_STATE_TRANSFER.java:465) at org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER.up(STREAMING_STATE_TRANSFER.java:230) at org.jgroups.protocols.FRAG2.up(FRAG2.java:188) at org.jgroups.protocols.FC.up(FC.java:470) at org.jgroups.protocols.pbcast.GMS.up(GMS.java:890) at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:236) at org.jgroups.protocols.UNICAST.handleDataReceived(UNICAST.java:588) at org.jgroups.protocols.UNICAST.up(UNICAST.java:275) at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:692) at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:132) at org.jgroups.protocols.FD.up(FD.java:259) at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:269) at org.jgroups.stack.Protocol.up(Protocol.java:340) at org.jgroups.protocols.Discovery.up(Discovery.java:277) at org.jgroups.protocols.TP.passMessageUp(TP.java:953) at org.jgroups.protocols.TP.access$100(TP.java:53) at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1457) at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1439) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:636) Caused by: java.io.EOFException: Read past end of file at org.jboss.marshalling.AbstractUnmarshaller.eofOnRead(AbstractUnmarshaller.java:184) at org.jboss.marshalling.AbstractUnmarshaller.readUnsignedByteDirect(AbstractUnmarshaller.java:312) at org.jboss.marshalling.AbstractUnmarshaller.readUnsignedByte(AbstractUnmarshaller.java:280) at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:207) at org.jboss.marshalling.AbstractUnmarshaller.readObject(AbstractUnmarshaller.java:85) at org.infinispan.marshall.jboss.JBossMarshaller.objectFromObjectStream(JBossMarshaller.java:207) at org.infinispan.marshall.VersionAwareMarshaller.objectFromObjectStream(VersionAwareMarshaller.java:171) at org.infinispan.statetransfer.StateTransferManagerImpl.assertDelimited(StateTransferManagerImpl.java:380) ... 30 more Unrelated: Also, if I set the preload attribute to true, I expect the store to fetch the persistant state before loading the data from the store but that's not the case. was: In a clustered replicated cache, when I flag the fetchPersistentState attribute to true for a FileStore, Infinispan generate an exception. Also, if I set the preload attribute to true, I expect the store to fetch the persistant state before loading the data from the store but that's not the case. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/jira/browse/ISPN-335?page=com.atlassian.jira.plugi... ] Galder Zamarreno commented on ISPN-335: --------------------------------------- To make it clearer: 146 if (log.isDebugEnabled()) log.debug("Generating state. Can provide? {0}", canProvideState); 147 oo = marshaller.startObjectOutput(out, false); 148 marshaller.objectToObjectStream(canProvideState, oo); 150 if (canProvideState) { 113 151 delimit(oo); 152 if (transientState) generateInMemoryState(oo); 153 delimit(oo); 154 if (persistentState) generatePersistentState(oo); 155 delimit(oo); 156 generateTransactionLog(oo); 157 158 if (log.isDebugEnabled()) log.debug("State generated, closing object stream"); 159 } else { 160 if (log.isDebugEnabled()) log.debug("Not providing state!"); 161 } I can't see how a single thread could print line 160 and then line 154 without having first printed line 146!! -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/jira/browse/ISPN-335?page=com.atlassian.jira.plugi... ] Galder Zamarreno commented on ISPN-335: --------------------------------------- For future reference, here's a chat between David and myself: <galderz> hi dmlloyd <dmlloyd> hey galderz, how's it going <galderz> not too bad dmlloyd and urself? <galderz> do u have spare few mins? <dmlloyd> getting along <dmlloyd> sure thing <galderz> a user has reported a EOFException during a concurrent infinispan state transfer, let me explain <galderz> two nodes start in paralell and request state from node A <galderz> node A starts serving one of them and writes partially to the stream <galderz> at that point, the other node requests from this same node A <galderz> node A starts serving the other node with another thread, <galderz> now, the 1st request continuous to write to the stream but for some reason, the other side that was waiting to read stuff finishes with an EOFEx like you see here: https://jira.jboss.org/jira/browse/ISPN-335 <galderz> i mean, i have not called finish on the marshaller, how is it possible that the 2nd node sees an EOFException? <dmlloyd> are you using the same marshaller from more than one thread? <galderz> yup, i think so <dmlloyd> a marshaller can only be used on one stream at a time, from one thread at a time <dmlloyd> if you have two concurrent streams, you need two marshallers <galderz> hmmmm <galderz> didn't realise that <dmlloyd> the marshaller has an internal state that is updated as things are written, and it's all sequential <galderz> hmmm, <galderz> one sec <galderz> actually, we're already doing that <galderz> the org.jboss.marshalling.Marshaller used underneath is stored in a ThreadLocal <galderz> so each thread should be using a different org.jboss.marshalling.Marshaller <dmlloyd> and one thread won't be handling two streams at once? <dmlloyd> can you say the same about the Unmarshaller side? <galderz> yeah, same for unmarshaller <galderz> let me add some identity hashcodes to confirm what I'm sayting but i'm pretty positive <dmlloyd> EOFException can be caused by a premature EOF in the underlying stream as well <dmlloyd> like a framing error or lost packet <dmlloyd> are you sure the thread name is the same? maybe it really is the same thread and something non-obvious is happening <galderz> defo it's different, have a diff thread name <galderz> in fact, had issues with JGroups cos it was naming diff threads with same name <dmlloyd> ok, just making sure <galderz> added some numbering to them now and can for sure be sure they're diff <galderz> i'm adding identity hash code info on the o.j.m.M used when writing on the reading side <galderz> so that I can see whether each thread is really using a diff one <galderz> gimme a few mins <galderz> hmmm, dmlloyd could some flush calls on the underlying jbmarshaller be causing issues? <dmlloyd> flush should be harmless <galderz> dmlloyd, i think ur initial suspicion of 2 thread 2 marshallers might be in the right direction <galderz> gimme a sec to confirm what I see in logs <galderz> no, false alarm <galderz> i suspect it might be related to: "and one thread won't be handling two streams at once?" <galderz> checking... <galderz> dmlloyd, think I found the problen <galderz> http://pastebin.mozilla.org/698472 <galderz> the same reader calls start input twice before calling fiunish <dmlloyd> start implicitly calls finish if the stream is already open <galderz> i remember dealing with a similar scenario in the past where I added reentrancy capabilities, to make sure a new one was returned if twice was called <dmlloyd> so that'd be where the EOF comes from <galderz> ahhhhh <galderz> that makes sense now :) <galderz> man, you're good at this! :) <dmlloyd> :D -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/jira/browse/ISPN-335?page=com.atlassian.jira.plugi... ] Galder Zamarreno resolved ISPN-335. ----------------------------------- Resolution: Done -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira