[infinispan-dev] Compressing Marshaller Wrapper

Galder Zamarreno galder at redhat.com
Fri Feb 26 05:18:55 EST 2010 

On Thu, 25 Feb 2010 18:14:04 +0100, Philippe Van Dyck <pvdyck at gmail.com>  
wrote:

> On Thu, Feb 25, 2010 at 5:50 PM, Manik Surtani <manik at jboss.org> wrote:
>
>>
>> On 25 Feb 2010, at 16:49, Manik Surtani wrote:
>>
>> RE: JIRA, why don't you change it then to say something like "Add
>> compression to CloudCacheStore"
>>
>> done
>
>
>>
>> The patch looks good, although I'd recommend using 3 spaces instead of
>> tabs.  ;)
>>
>>
> Do you have a formatting configuration to apply to eclipse ?

Maybe the quick start guide for developers can help? ;)

http://community.jboss.org/wiki/InfinispanQuickStartGuideforDevelopers

>
>
>>
>> Apart from that, feel free to commit this to trunk yourself, so that  
>> your
>> name is attached to the commit.
>>
>>
>>
> np
>
>
>> Just noticed that you're not on the committers list.  If you fill out  
>> the
>> contributors form on
>>
>> https://www.jboss.org/contribute
>>
>>
> done
>
>
>> and send me your jboss.org username I'll add you as a committer.
>>
>>
> jboss username : pvdyck
>
>
> cheers
>
> phil
>
>
>
>> Cheers
>> Manik
>>
>>
>>
>>
>>
>> Cheers
>> Manik
>>
>> On 25 Feb 2010, at 16:14, Philippe Van Dyck wrote:
>>
>> Ok, tested and working. Here is the patch. Could be improved slightly by
>> adding the compression level.
>>
>> Cherry on top, I suffix ".gz" to S3 names, so you can download and
>> automagically uncompress them ;-)
>>
>> Manik, what do we do with the JIRA issue ? The patch attached is a bit
>> unrelated...
>>
>>
>>
>> phil
>>
>> On Thu, Feb 25, 2010 at 3:33 PM, Philippe Van Dyck  
>> <pvdyck at gmail.com>wrote:
>>
>>> Ok, I need a solution before the end of the day so I will choose the
>>> easiest way.
>>>
>>> I plan to update CloudCacheStoreConfig and add a "compress" option.
>>> Then I only need to change two simple methods "writeToBlob" and
>>> "readFromBlob" (thanks Adrian... next time make them protected ;-))
>>>
>>> I will post a patch later.
>>>
>>> phil
>>>
>>>
>>> On Thu, Feb 25, 2010 at 3:21 PM, Manik Surtani <manik at jboss.org> wrote:
>>>
>>>>
>>>> On 25 Feb 2010, at 14:17, Philippe Van Dyck wrote:
>>>>
>>>> Another solution is to include a "compression flag" in the version
>>>> itself.
>>>>
>>>>
>>>> That would work as well...  then you could just handle the  
>>>> compression in
>>>> the VAM
>>>>
>>>>
>>>> WDYT ?
>>>>
>>>> phil
>>>>
>>>> On Thu, Feb 25, 2010 at 3:15 PM, Philippe Van Dyck  
>>>> <pvdyck at gmail.com>wrote:
>>>>
>>>>> For my purpose, the cachestore delegator is enough...
>>>>>
>>>>> Well, I'll take a second look at VersionAwareMarshaller...
>>>>>
>>>>> phil
>>>>>
>>>>>
>>>>> On Thu, Feb 25, 2010 at 3:10 PM, Manik Surtani <manik at jboss.org>  
>>>>> wrote:
>>>>>
>>>>>> I think it makes more sense in the marshaller, simply because the
>>>>>> compressed stream is useful elsewhere as well - such as the  
>>>>>> network, etc.
>>>>>>
>>>>>> On 25 Feb 2010, at 13:40, Philippe Van Dyck wrote:
>>>>>>
>>>>>> Sounds like a better idea, closer to the real purpose since the
>>>>>> Marshaller is used in other places... but how do I configure this ?
>>>>>>
>>>>>> Anyway, the marshaller is done, I am testing it, but the code is  
>>>>>> very
>>>>>> easy to adapt to a cachestore...
>>>>>>
>>>>>> phil
>>>>>>
>>>>>>
>>>>>> On Thu, Feb 25, 2010 at 2:33 PM, Mircea Markus <
>>>>>> mircea.markus at jboss.com> wrote:
>>>>>>
>>>>>>> Can't we handle this through a delegating CacheStore, that would
>>>>>>> compress the data and pass it on to the actual store for further  
>>>>>>> processing
>>>>>>> (the actual store will take then the binary data and marshall it  
>>>>>>> (ni
>>>>>>> marshalling I assume as it receives byte[]).?
>>>>>>>
>>>>>>> On 25 Feb 2010, at 14:06, Philippe Van Dyck wrote:
>>>>>>>
>>>>>>> Done
>>>>>>>
>>>>>>> https://jira.jboss.org/jira/browse/ISPN-357
>>>>>>>
>>>>>>> Phil
>>>>>>>
>>>>>>> On Thu, Feb 25, 2010 at 12:22 PM, Manik Surtani  
>>>>>>> <manik at jboss.org>wrote:
>>>>>>>
>>>>>>>>
>>>>>>>> On 25 Feb 2010, at 11:17, Philippe Van Dyck wrote:
>>>>>>>>
>>>>>>>> Thanks Manik but it is actually related to the compression of the
>>>>>>>> stored and (https) transferred ones on the S3 side.
>>>>>>>> The S3 bill is 95% lower here... and https transfer time is lower  
>>>>>>>> too
>>>>>>>> (you pay for booth).
>>>>>>>>
>>>>>>>>
>>>>>>>> Makes sense.  Create a JIRA for this, I'm sure such a feature  
>>>>>>>> would
>>>>>>>> be generically useful to others as well - if you're happy to  
>>>>>>>> contribute your
>>>>>>>> work back in.  :)
>>>>>>>>
>>>>>>>> The Marshaller interface is exposing a lot of methods and it looks
>>>>>>>> like the call each other...
>>>>>>>>
>>>>>>>>
>>>>>>>> The original Marshaller interface is legacy, from JBoss Cache,  
>>>>>>>> which
>>>>>>>> was built as an extension to JGroups' Marshaller interface.  It  
>>>>>>>> may look a
>>>>>>>> little confusing as a result, but have a look at the  
>>>>>>>> AbstractMarshaller
>>>>>>>> which should help simplify things.  Further, look at the
>>>>>>>> VersionAwareMarshaller, which is the entry point into the  
>>>>>>>> marshalling
>>>>>>>> system.
>>>>>>>>
>>>>>>>> HTH
>>>>>>>> Manik
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> phil
>>>>>>>>
>>>>>>>> On Thu, Feb 25, 2010 at 12:11 PM, Manik Surtani  
>>>>>>>> <manik at jboss.org>wrote:
>>>>>>>>
>>>>>>>>> If it is compression for transmission you are concerned about,  
>>>>>>>>> you
>>>>>>>>> can do this by adding the JGroups COMPRESS protocol to your  
>>>>>>>>> JGroups cfg.
>>>>>>>>>
>>>>>>>>> On 25 Feb 2010, at 11:02, philippe van dyck wrote:
>>>>>>>>>
>>>>>>>>> > Hi All,
>>>>>>>>> >
>>>>>>>>> > Currently, I compress all data before sending it to the cache.
>>>>>>>>> Once compressed, I gain 95% of the JSonized qi4j objects.
>>>>>>>>> >
>>>>>>>>> > I did some profiling during the load tests and compression is
>>>>>>>>> taking roughly 80% of the cpu time.
>>>>>>>>> > So I would like to compress only the data sent to the store,  
>>>>>>>>> not
>>>>>>>>> in memory.
>>>>>>>>> >
>>>>>>>>> > Looks like the Marshaller is my friend here, and I plan to  
>>>>>>>>> write a
>>>>>>>>> compressing wrapper around it.
>>>>>>>>> >
>>>>>>>>> > Now, when I look at it, I see two ways to wrap the compression
>>>>>>>>> process.
>>>>>>>>> >
>>>>>>>>> > One way is with the ObjectInput / ObjectOutput but I am  
>>>>>>>>> bothered
>>>>>>>>> by the reentrant flag.
>>>>>>>>> > The other is the ByteBuffer stuff, no concurrency problem here,
>>>>>>>>> but it looks like more work.
>>>>>>>>> >
>>>>>>>>> > WDYT ?
>>>>>>>>> >
>>>>>>>>> > Cheers,
>>>>>>>>> >
>>>>>>>>> > phil
>>>>>>>>> > _______________________________________________
>>>>>>>>> > infinispan-dev mailing list
>>>>>>>>> > infinispan-dev at lists.jboss.org
>>>>>>>>> > https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Manik Surtani
>>>>>>>>> manik at jboss.org
>>>>>>>>> Lead, Infinispan
>>>>>>>>> Lead, JBoss Cache
>>>>>>>>> http://www.infinispan.org
>>>>>>>>> http://www.jbosscache.org
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> infinispan-dev mailing list
>>>>>>>>> infinispan-dev at lists.jboss.org
>>>>>>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> infinispan-dev mailing list
>>>>>>>> infinispan-dev at lists.jboss.org
>>>>>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>>>>>>>
>>>>>>>>
>>>>>>>>  --
>>>>>>>> Manik Surtani
>>>>>>>> manik at jboss.org
>>>>>>>> Lead, Infinispan
>>>>>>>> Lead, JBoss Cache
>>>>>>>> http://www.infinispan.org
>>>>>>>> http://www.jbosscache.org
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> infinispan-dev mailing list
>>>>>>>> infinispan-dev at lists.jboss.org
>>>>>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> infinispan-dev mailing list
>>>>>>> infinispan-dev at lists.jboss.org
>>>>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> infinispan-dev mailing list
>>>>>>> infinispan-dev at lists.jboss.org
>>>>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> infinispan-dev mailing list
>>>>>> infinispan-dev at lists.jboss.org
>>>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>>>>>
>>>>>>
>>>>>>  --
>>>>>> Manik Surtani
>>>>>> manik at jboss.org
>>>>>> Lead, Infinispan
>>>>>> Lead, JBoss Cache
>>>>>> http://www.infinispan.org
>>>>>> http://www.jbosscache.org
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> infinispan-dev mailing list
>>>>>> infinispan-dev at lists.jboss.org
>>>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>>>>>
>>>>>
>>>>>
>>>> _______________________________________________
>>>> infinispan-dev mailing list
>>>> infinispan-dev at lists.jboss.org
>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>>>
>>>>
>>>>  --
>>>> Manik Surtani
>>>> manik at jboss.org
>>>> Lead, Infinispan
>>>> Lead, JBoss Cache
>>>> http://www.infinispan.org
>>>> http://www.jbosscache.org
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> infinispan-dev mailing list
>>>> infinispan-dev at lists.jboss.org
>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>>>
>>>
>>>
>> <cloud.patch>_______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>
>>
>> --
>> Manik Surtani
>> manik at jboss.org
>> Lead, Infinispan
>> Lead, JBoss Cache
>> http://www.infinispan.org
>> http://www.jbosscache.org
>>
>>
>>
>>
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>
>>
>> --
>> Manik Surtani
>> manik at jboss.org
>> Lead, Infinispan
>> Lead, JBoss Cache
>> http://www.infinispan.org
>> http://www.jbosscache.org
>>
>>
>>
>>
>>
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>


-- 
Galder Zamarreño
Sr. Software Engineer
Infinispan, JBoss Cache

More information about the infinispan-dev mailing list