[infinispan-dev] ISPN-78 and Large Object Support/Streaming API

[Manik Surtani]

And another spin on this - we'd also need to think about such streaming functionality for HotRod.  :)  

Galder, care to create a related JIRA for streaming over HotRod?

On 18 May 2010, at 16:29, Philippe Van Dyck wrote:

> 
> 
> On Tue, May 18, 2010 at 5:11 PM, Manik Surtani <manik at jboss.org> wrote:
> 
> On 18 May 2010, at 16:02, Philippe Van Dyck wrote:
> 
>> It may sound a bit extreme but what about using streams as the base interface and converting values to streams asap ?
> 
> It's certainly worth thinking about.
> 
> How would his affect listeners though, where listeners expect objects, not streams?
> 
> 
> Welcome to stream plumbing ;-)
>  
> You are not supposed to read or write to the stream... producers and consumers do ! (read cache stores & cache users - not you !)
> 
> Now, the question is ... are your listeners stream consumers or even producers ?
> 
> How much consumers do you have ? Each of them will need a copy of the stream... that's a lot of plumbing. 
> 
> Think about some kind of spider tube, with one input and multiple outputs... Nothing complicated really.
>  
> The real question with streams, after they are all connected, is to find leaks and seriously think about what procedure to apply when the system is overheating ... how do you collect all the leaks (exceptions) ?
> 
> Funny isn't it ?
> 
> phil
> 
>> The conversion could be seen as the marshaling operation and the result as a stream of bytes ?
>> 
>> The 'base' interface would be as simple as:
>> OutputStream<byte[]> get(K)
>> put(K, InputStream<byte[]>)
>> 
>> the Map<K,V> facade simply 'marshalls' values to byte arrays and calls the 'base' interface.
>> 
>> This way, you get rid of any buffering from the very beginning and give streams to cache stores... (You actually replace byte arrays by piped(in/out)putstreams in your futures)... 
>> 
>> Btw, I can assure you that some cache store maintainers will be super happy to receive and produce streams ;-)
>> 
>> my 2 cents...
>> 
>> phil
>> 
>> 
>> On Tue, May 18, 2010 at 3:59 PM, Manik Surtani <manik at jboss.org> wrote:
>> I have put together a brief design for ISPN-78.  Please take a look, it is on the wiki:
>> 
>>        https://community.jboss.org/wiki/LargeObjectSupport
>> 
>> I have also deferred ISPN-78 to 5.0.0 rather than 4.1.0 as I'd rather not hold up 4.1.0 for new features at this stage.
>> 
>> Please have a look at the designs and let me know what you think - or comment on the wiki page.
>> 
>> Cheers
>> Manik
>> --
>> Manik Surtani
>> manik at jboss.org
>> Lead, Infinispan
>> Lead, JBoss Cache
>> http://www.infinispan.org
>> http://www.jbosscache.org
>> 
>> 
>> 
>> 
>> 
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>> 
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> 
> 
> --
> Manik Surtani
> manik at jboss.org
> Lead, Infinispan
> Lead, JBoss Cache
> http://www.infinispan.org
> http://www.jbosscache.org
> 
> 
> 
> 
> 
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> 
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev

--
Manik Surtani
manik at jboss.org
Lead, Infinispan
Lead, JBoss Cache
http://www.infinispan.org
http://www.jbosscache.org




-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/infinispan-dev/attachments/20100519/999ca22a/attachment-0001.html