[infinispan-dev] ISPN-78 and Large Object Support/Streaming API

[Philippe Van Dyck]

On Tue, May 18, 2010 at 5:11 PM, Manik Surtani <manik at jboss.org> wrote:

>
> On 18 May 2010, at 16:02, Philippe Van Dyck wrote:
>
> It may sound a bit extreme but what about using streams as the base
> interface and converting values to streams asap ?
>
>
> It's certainly worth thinking about.
>
> How would his affect listeners though, where listeners expect objects, not
> streams?
>
>
Welcome to stream plumbing ;-)

You are not supposed to read or write to the stream... producers and
consumers do ! (read cache stores & cache users - not you !)

Now, the question is ... are your listeners stream consumers or even
producers ?

How much consumers do you have ? Each of them will need a copy of the
stream... that's a lot of plumbing.

Think about some kind of spider tube, with one input and multiple outputs...
Nothing complicated really.

The real question with streams, after they are all connected, is to find
leaks and seriously think about what procedure to apply when the system is
overheating ... how do you collect all the leaks (exceptions) ?

Funny isn't it ?

phil

The conversion could be seen as the marshaling operation and the result as a
> stream of bytes ?
>
> The 'base' interface would be as simple as:
> OutputStream<byte[]> get(K)
> put(K, InputStream<byte[]>)
>
> the Map<K,V> facade simply 'marshalls' values to byte arrays and calls the
> 'base' interface.
>
> This way, you get rid of any buffering from the very beginning and give
> streams to cache stores... (You actually replace byte arrays by
> piped(in/out)putstreams in your futures)...
>
> Btw, I can assure you that some cache store maintainers will be super happy
> to receive and produce streams ;-)
>
> my 2 cents...
>
> phil
>
>
> On Tue, May 18, 2010 at 3:59 PM, Manik Surtani <manik at jboss.org> wrote:
>
>> I have put together a brief design for ISPN-78.  Please take a look, it is
>> on the wiki:
>>
>>        https://community.jboss.org/wiki/LargeObjectSupport
>>
>> I have also deferred ISPN-78 to 5.0.0 rather than 4.1.0 as I'd rather not
>> hold up 4.1.0 for new features at this stage.
>>
>> Please have a look at the designs and let me know what you think - or
>> comment on the wiki page.
>>
>> Cheers
>> Manik
>> --
>> Manik Surtani
>> manik at jboss.org
>> Lead, Infinispan
>> Lead, JBoss Cache
>> http://www.infinispan.org
>> http://www.jbosscache.org
>>
>>
>>
>>
>>
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
>
> --
> Manik Surtani
> manik at jboss.org
> Lead, Infinispan
> Lead, JBoss Cache
> http://www.infinispan.org
> http://www.jbosscache.org
>
>
>
>
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/infinispan-dev/attachments/20100518/f596a582/attachment.html