[infinispan-dev] Compressing Marshaller Wrapper

Thu Feb 25 12:14:04 EST 2010

On Thu, Feb 25, 2010 at 5:50 PM, Manik Surtani <manik at jboss.org> wrote:

>
> On 25 Feb 2010, at 16:49, Manik Surtani wrote:
>
> RE: JIRA, why don't you change it then to say something like "Add
> compression to CloudCacheStore"
>
> done

>
> The patch looks good, although I'd recommend using 3 spaces instead of
> tabs.  ;)
>
>
Do you have a formatting configuration to apply to eclipse ?

>
> Apart from that, feel free to commit this to trunk yourself, so that your
> name is attached to the commit.
>
>
>
np

> Just noticed that you're not on the committers list.  If you fill out the
> contributors form on
>
> https://www.jboss.org/contribute
>
>
done

> and send me your jboss.org username I'll add you as a committer.
>
>
jboss username : pvdyck

cheers

phil

> Cheers
> Manik
>
>
>
>
>
> Cheers
> Manik
>
> On 25 Feb 2010, at 16:14, Philippe Van Dyck wrote:
>
> Ok, tested and working. Here is the patch. Could be improved slightly by
> adding the compression level.
>
> Cherry on top, I suffix ".gz" to S3 names, so you can download and
> automagically uncompress them ;-)
>
> Manik, what do we do with the JIRA issue ? The patch attached is a bit
> unrelated...
>
>
>
> phil
>
> On Thu, Feb 25, 2010 at 3:33 PM, Philippe Van Dyck <pvdyck at gmail.com>wrote:
>
>> Ok, I need a solution before the end of the day so I will choose the
>> easiest way.
>>
>> I plan to update CloudCacheStoreConfig and add a "compress" option.
>> Then I only need to change two simple methods "writeToBlob" and
>> "readFromBlob" (thanks Adrian... next time make them protected ;-))
>>
>> I will post a patch later.
>>
>> phil
>>
>>
>> On Thu, Feb 25, 2010 at 3:21 PM, Manik Surtani <manik at jboss.org> wrote:
>>
>>>
>>> On 25 Feb 2010, at 14:17, Philippe Van Dyck wrote:
>>>
>>> Another solution is to include a "compression flag" in the version
>>> itself.
>>>
>>>
>>> That would work as well...  then you could just handle the compression in
>>> the VAM
>>>
>>>
>>> WDYT ?
>>>
>>> phil
>>>
>>> On Thu, Feb 25, 2010 at 3:15 PM, Philippe Van Dyck <pvdyck at gmail.com>wrote:
>>>
>>>> For my purpose, the cachestore delegator is enough...
>>>>
>>>> Well, I'll take a second look at VersionAwareMarshaller...
>>>>
>>>> phil
>>>>
>>>>
>>>> On Thu, Feb 25, 2010 at 3:10 PM, Manik Surtani <manik at jboss.org> wrote:
>>>>
>>>>> I think it makes more sense in the marshaller, simply because the
>>>>> compressed stream is useful elsewhere as well - such as the network, etc.
>>>>>
>>>>> On 25 Feb 2010, at 13:40, Philippe Van Dyck wrote:
>>>>>
>>>>> Sounds like a better idea, closer to the real purpose since the
>>>>> Marshaller is used in other places... but how do I configure this ?
>>>>>
>>>>> Anyway, the marshaller is done, I am testing it, but the code is very
>>>>> easy to adapt to a cachestore...
>>>>>
>>>>> phil
>>>>>
>>>>>
>>>>> On Thu, Feb 25, 2010 at 2:33 PM, Mircea Markus <
>>>>> mircea.markus at jboss.com> wrote:
>>>>>
>>>>>> Can't we handle this through a delegating CacheStore, that would
>>>>>> compress the data and pass it on to the actual store for further processing
>>>>>> (the actual store will take then the binary data and marshall it (ni
>>>>>> marshalling I assume as it receives byte[]).?
>>>>>>
>>>>>> On 25 Feb 2010, at 14:06, Philippe Van Dyck wrote:
>>>>>>
>>>>>> Done
>>>>>>
>>>>>> https://jira.jboss.org/jira/browse/ISPN-357
>>>>>>
>>>>>> Phil
>>>>>>
>>>>>> On Thu, Feb 25, 2010 at 12:22 PM, Manik Surtani <manik at jboss.org>wrote:
>>>>>>
>>>>>>>
>>>>>>> On 25 Feb 2010, at 11:17, Philippe Van Dyck wrote:
>>>>>>>
>>>>>>> Thanks Manik but it is actually related to the compression of the
>>>>>>> stored and (https) transferred ones on the S3 side.
>>>>>>> The S3 bill is 95% lower here... and https transfer time is lower too
>>>>>>> (you pay for booth).
>>>>>>>
>>>>>>>
>>>>>>> Makes sense.  Create a JIRA for this, I'm sure such a feature would
>>>>>>> be generically useful to others as well - if you're happy to contribute your
>>>>>>> work back in.  :)
>>>>>>>
>>>>>>> The Marshaller interface is exposing a lot of methods and it looks
>>>>>>> like the call each other...
>>>>>>>
>>>>>>>
>>>>>>> The original Marshaller interface is legacy, from JBoss Cache, which
>>>>>>> was built as an extension to JGroups' Marshaller interface.  It may look a
>>>>>>> little confusing as a result, but have a look at the AbstractMarshaller
>>>>>>> which should help simplify things.  Further, look at the
>>>>>>> VersionAwareMarshaller, which is the entry point into the marshalling
>>>>>>> system.
>>>>>>>
>>>>>>> HTH
>>>>>>> Manik
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> phil
>>>>>>>
>>>>>>> On Thu, Feb 25, 2010 at 12:11 PM, Manik Surtani <manik at jboss.org>wrote:
>>>>>>>
>>>>>>>> If it is compression for transmission you are concerned about, you
>>>>>>>> can do this by adding the JGroups COMPRESS protocol to your JGroups cfg.
>>>>>>>>
>>>>>>>> On 25 Feb 2010, at 11:02, philippe van dyck wrote:
>>>>>>>>
>>>>>>>> > Hi All,
>>>>>>>> >
>>>>>>>> > Currently, I compress all data before sending it to the cache.
>>>>>>>> Once compressed, I gain 95% of the JSonized qi4j objects.
>>>>>>>> >
>>>>>>>> > I did some profiling during the load tests and compression is
>>>>>>>> taking roughly 80% of the cpu time.
>>>>>>>> > So I would like to compress only the data sent to the store, not
>>>>>>>> in memory.
>>>>>>>> >
>>>>>>>> > Looks like the Marshaller is my friend here, and I plan to write a
>>>>>>>> compressing wrapper around it.
>>>>>>>> >
>>>>>>>> > Now, when I look at it, I see two ways to wrap the compression
>>>>>>>> process.
>>>>>>>> >
>>>>>>>> > One way is with the ObjectInput / ObjectOutput but I am bothered
>>>>>>>> by the reentrant flag.
>>>>>>>> > The other is the ByteBuffer stuff, no concurrency problem here,
>>>>>>>> but it looks like more work.
>>>>>>>> >
>>>>>>>> > WDYT ?
>>>>>>>> >
>>>>>>>> > Cheers,
>>>>>>>> >
>>>>>>>> > phil
>>>>>>>> > _______________________________________________
>>>>>>>> > infinispan-dev mailing list
>>>>>>>> > infinispan-dev at lists.jboss.org
>>>>>>>> > https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>>>>>>>
>>>>>>>> --
>>>>>>>> Manik Surtani
>>>>>>>> manik at jboss.org
>>>>>>>> Lead, Infinispan
>>>>>>>> Lead, JBoss Cache
>>>>>>>> http://www.infinispan.org
>>>>>>>> http://www.jbosscache.org
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> infinispan-dev mailing list
>>>>>>>> infinispan-dev at lists.jboss.org
>>>>>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> infinispan-dev mailing list
>>>>>>> infinispan-dev at lists.jboss.org
>>>>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>>>>>>
>>>>>>>
>>>>>>>  --
>>>>>>> Manik Surtani
>>>>>>> manik at jboss.org
>>>>>>> Lead, Infinispan
>>>>>>> Lead, JBoss Cache
>>>>>>> http://www.infinispan.org
>>>>>>> http://www.jbosscache.org
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> infinispan-dev mailing list
>>>>>>> infinispan-dev at lists.jboss.org
>>>>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> infinispan-dev mailing list
>>>>>> infinispan-dev at lists.jboss.org
>>>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>>>>>
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> infinispan-dev mailing list
>>>>>> infinispan-dev at lists.jboss.org
>>>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> infinispan-dev mailing list
>>>>> infinispan-dev at lists.jboss.org
>>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>>>>
>>>>>
>>>>>  --
>>>>> Manik Surtani
>>>>> manik at jboss.org
>>>>> Lead, Infinispan
>>>>> Lead, JBoss Cache
>>>>> http://www.infinispan.org
>>>>> http://www.jbosscache.org
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> infinispan-dev mailing list
>>>>> infinispan-dev at lists.jboss.org
>>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>>>>
>>>>
>>>>
>>> _______________________________________________
>>> infinispan-dev mailing list
>>> infinispan-dev at lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>>
>>>
>>>  --
>>> Manik Surtani
>>> manik at jboss.org
>>> Lead, Infinispan
>>> Lead, JBoss Cache
>>> http://www.infinispan.org
>>> http://www.jbosscache.org
>>>
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> infinispan-dev mailing list
>>> infinispan-dev at lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>>
>>
>>
> <cloud.patch>_______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
>
> --
> Manik Surtani
> manik at jboss.org
> Lead, Infinispan
> Lead, JBoss Cache
> http://www.infinispan.org
> http://www.jbosscache.org
>
>
>
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
>
> --
> Manik Surtani
> manik at jboss.org
> Lead, Infinispan
> Lead, JBoss Cache
> http://www.infinispan.org
> http://www.jbosscache.org
>
>
>
>
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/infinispan-dev/attachments/20100225/e8d0e0ce/attachment-0002.html 