[jbosscache-dev] Re: Integrating steaming transfer in JBC

[Brian Stansberry]

Right now the cache loaders are actually using a
ByteArrayInput/OutputStream to deal with the byte[]. So I wouldn't think
it would be a big deal to refactor a bit to deal with an externally
provided stream.  Really, the whole idea is to make the byte[] go away
-- it serves no purpose, and once its created most (probably all) of the
benefit of streaming state transfer is lost.

As you say the tricky part is how does the cache loader know the
boundaries of what to read?  Probably the simplest solution is the
StateTransferGenerator ensures the persistent state is the last stuff
written; the receiving cache loader then just reads until there is no
more.  The StateTransferGenerator would write some kid of marker to the
stream to identify that the persistent state is about to start.
StateTransferIntegrator monitors for that, and passes the stream to the
cache loader once its read.

Hmm. Perhaps the CacheLoader API needs to use ObjectInput/OutputStream.
Have to think about that a bit.

Manik Surtani wrote:
> Well then this is the ideal time to do it, while the
> interfaces are still malleable.
> 
> I suspect the quick and dirty impl for most cache loader
> impls will be to wait till the stream closes to capture the
> byte[] and process this rather than a more efficient on-the-fly
> approach. 
> 
>> Yep.  The thread kind of morphed into a design discussion :)
>> 
>> jbosscache-dev-bounces at lists.jboss.org wrote:
>>> (This should be on the jbc dev mail list, btw - or on the design
>>> forum if it warrants it) 
>>> 
>>> Vladimir/Brian:
>>> 
>>> Does this relate to the need for something like:
>>> 
>>> void storeEntireState(OutputStream s); InputStream
>>> loadEntireState(); void storeState(OutputStream s, Fqn subtree);
>>> InputStream loadState(Fqn subtree); 
>>> 
>>> In the CacheLoader interface?
>>> 
>>> Firstly, is this feasible when it comes to cache loaders - e.g., a
>>> JDBC cache loader - what sort of stream markers will we have to let
>>> the cache loader know when it can start chunking up data from the
>>> stream, parsing it and pushing this state down to storage?
>  Or will
>>> it just be an API-level kludge where the CL impl attempts to keep
>>> reading from the stream until the stream closes, yanks out a byte[]
>>> and passes that into storeEntireState(byte[] b)?  (May be the case
>>> for an initial impl, just so we have the interface methods in place
>>> for 
>>> 2.0.0)
>>> 
>>> Cheers,
>>> 
>>>> I should have sent a copy to Manik and Bela as well.  See below.
>>>> 
>>>> Main point is: we need additional methods in CacheLoader to
>>>> accommodate streaming state transfer.
>>>> How should we proceed?
>>>> 
>>>> Vladimir
>>>> 
>>>> -----Original Message-----
>>>> From: Vladimir Blagojevic
>>>> Sent: Tuesday, August 15, 2006 11:41 AM
>>>> To: Brian Stansberry
>>>> Subject: RE: Integrating steaming transfer in JBC
>>>> 
>>>> Oyyyy. Not only that but it also affects CacheLoader API:
>>>> 
>>>> byte[] loadEntireState() throws Exception; void
>>>> storeEntireState(byte[] state) throws Exception; byte[] loadState
>>>> (Fqn subtree) throws Exception; void storeState(byte[] state, Fqn
>>>> subtree) throws Exception; 
>>>> 
>>>> These should have streaming counterparts if we really want large
>>>> state OOME to go away.... 
>>>> 
>>>> 
>>>> 
>>>> -----Original Message-----
>>>> From: Brian Stansberry
>>>> Sent: Tuesday, August 15, 2006 11:27 AM
>>>> To: Vladimir Blagojevic
>>>> Subject: RE: Integrating steaming transfer in JBC
>>>> 
>>>> Yep, plus there are different types of state that will be handled
>>>> by different subsystems; that will be interesting for streaming
>>>> state transfer. 
>>>> 
>>>> Also, for buddy replication, the state transfer is a push rather
>>>> than a pull.  Not sure what to do about that :(
>>>> 
>>>> Vladimir Blagojevic wrote:
>>>>> Ok I see. So it is not only integration of state transfer but
>>>>> removing
>>>> 
>>>>> rpc mechanism for partial state integration. I am looking at the
>>>>> code and learning how is this done. Then I'll ping you tomorrow/
>>>>> Thursday to
>>>> 
>>>>> have a discussion with you.
>>>>> 
>>>>> -----Original Message-----
>>>>> From: Brian Stansberry
>>>>> Sent: Monday, August 14, 2006 3:02 PM
>>>>> To: Vladimir Blagojevic
>>>>> Subject: RE: Integrating steaming transfer in JBC
>>>>> 
>>>>> Just look at the JBCACHE issues and filter for the ones assigned
>>>>> to you :) 
>>>>> 
>>>>> Vladimir Blagojevic wrote:
>>>>>> Hey,
>>>>>> 
>>>>>> What do you mean by looking at the non  streaming state transfer
>>>>>> tasks? Which ones are those? 
>>>>>> 
>>>>>> Vladimir
>>>>>> 
>>>>>> 
>>>>>> -----Original Message-----
>>>>>> From: "Brian Stansberry" <brian.stansberry at jboss.com>
>>>>>> Date: Mon, 14 Aug 2006 12:24:35
>>>>>> To:<vladimirb at rogers.blackberry.net>
>>>>>> Subject: RE: Integrating steaming transfer in JBC
>>>>>> 
>>>>>> Sure, just let me know.  But, I'd start with looking at the
>>>>>> non-streaming state transfer tasks.  That will get you familiar
>>>>>> with the use cases and existing code.
>>>>>> 
>>> 
>>> _______________________________________________
>>> jbosscache-dev mailing list
>>> jbosscache-dev at lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/jbosscache-dev
>> 
>> 
>> 
>> Brian Stansberry
>> Lead, AS Clustering
>> JBoss, a division of Red Hat
>> Ph: 510-396-3864
>> skype: bstansberry



Brian Stansberry
Lead, AS Clustering
JBoss, a division of Red Hat
Ph: 510-396-3864
skype: bstansberry