Re: [infinispan-dev] Compressing Marshaller Wrapper

Ok, I need a solution before the end of the day so I will choose the easiest way. I plan to update CloudCacheStoreConfig and add a "compress" option. Then I only need to change two simple methods "writeToBlob" and "readFromBlob" (thanks Adrian... next time make them protected ;-)) I will post a patch later. phil On Thu, Feb 25, 2010 at 3:21 PM, Manik Surtani <manik(a)jboss.org&gt; wrote: > > On 25 Feb 2010, at 14:17, Philippe Van Dyck wrote: > > Another solution is to include a "compression flag" in the version itself. > > > That would work as well... then you could just handle the compression in > the VAM > > > WDYT ? > > phil > > On Thu, Feb 25, 2010 at 3:15 PM, Philippe Van Dyck <pvdyck(a)gmail.com&gt;wrote: > >> For my purpose, the cachestore delegator is enough... >> >> Well, I'll take a second look at VersionAwareMarshaller... >> >> phil >> >> >> On Thu, Feb 25, 2010 at 3:10 PM, Manik Surtani <manik(a)jboss.org&gt; wrote: >> >>> I think it makes more sense in the marshaller, simply because the >>> compressed stream is useful elsewhere as well - such as the network, etc. >>> >>> On 25 Feb 2010, at 13:40, Philippe Van Dyck wrote: >>> >>> Sounds like a better idea, closer to the real purpose since the >>> Marshaller is used in other places... but how do I configure this ? >>> >>> Anyway, the marshaller is done, I am testing it, but the code is very >>> easy to adapt to a cachestore... >>> >>> phil >>> >>> >>> On Thu, Feb 25, 2010 at 2:33 PM, Mircea Markus <mircea.markus(a)jboss.com >>> > wrote: >>> >>>> Can't we handle this through a delegating CacheStore, that would >>>> compress the data and pass it on to the actual store for further processing >>>> (the actual store will take then the binary data and marshall it (ni >>>> marshalling I assume as it receives byte[]).? >>>> >>>> On 25 Feb 2010, at 14:06, Philippe Van Dyck wrote: >>>> >>>> Done >>>> >>>> https://jira.jboss.org/jira/browse/ISPN-357 >>>> >>>> Phil >>>> >>>> On Thu, Feb 25, 2010 at 12:22 PM, Manik Surtani <manik(a)jboss.org&gt;wrote: >>>> >>>>> >>>>> On 25 Feb 2010, at 11:17, Philippe Van Dyck wrote: >>>>> >>>>> Thanks Manik but it is actually related to the compression of the >>>>> stored and (https) transferred ones on the S3 side. >>>>> The S3 bill is 95% lower here... and https transfer time is lower too >>>>> (you pay for booth). >>>>> >>>>> >>>>> Makes sense. Create a JIRA for this, I'm sure such a feature would be >>>>> generically useful to others as well - if you're happy to contribute your >>>>> work back in. :) >>>>> >>>>> The Marshaller interface is exposing a lot of methods and it looks >>>>> like the call each other... >>>>> >>>>> >>>>> The original Marshaller interface is legacy, from JBoss Cache, which >>>>> was built as an extension to JGroups' Marshaller interface. It may look a >>>>> little confusing as a result, but have a look at the AbstractMarshaller >>>>> which should help simplify things. Further, look at the >>>>> VersionAwareMarshaller, which is the entry point into the marshalling >>>>> system. >>>>> >>>>> HTH >>>>> Manik >>>>> >>>>> >>>>> >>>>> >>>>> phil >>>>> >>>>> On Thu, Feb 25, 2010 at 12:11 PM, Manik Surtani <manik(a)jboss.org&gt;wrote: >>>>> >>>>>> If it is compression for transmission you are concerned about, you >>>>>> can do this by adding the JGroups COMPRESS protocol to your JGroups cfg. >>>>>> >>>>>> On 25 Feb 2010, at 11:02, philippe van dyck wrote: >>>>>> >>>>>> > Hi All, >>>>>> > >>>>>> > Currently, I compress all data before sending it to the cache. Once >>>>>> compressed, I gain 95% of the JSonized qi4j objects. >>>>>> > >>>>>> > I did some profiling during the load tests and compression is >>>>>> taking roughly 80% of the cpu time. >>>>>> > So I would like to compress only the data sent to the store, not in >>>>>> memory. >>>>>> > >>>>>> > Looks like the Marshaller is my friend here, and I plan to write a >>>>>> compressing wrapper around it. >>>>>> > >>>>>> > Now, when I look at it, I see two ways to wrap the compression >>>>>> process. >>>>>> > >>>>>> > One way is with the ObjectInput / ObjectOutput but I am bothered by >>>>>> the reentrant flag. >>>>>> > The other is the ByteBuffer stuff, no concurrency problem here, but >>>>>> it looks like more work. >>>>>> > >>>>>> > WDYT ? >>>>>> > >>>>>> > Cheers, >>>>>> > >>>>>> > phil >>>>>> > _______________________________________________ >>>>>> > infinispan-dev mailing list >>>>>> > infinispan-dev(a)lists.jboss.org >>>>>> > https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>>>> >>>>>> -- >>>>>> Manik Surtani >>>>>> manik(a)jboss.org >>>>>> Lead, Infinispan >>>>>> Lead, JBoss Cache >>>>>> http://www.infinispan.org >>>>>> http://www.jbosscache.org >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> infinispan-dev mailing list >>>>>> infinispan-dev(a)lists.jboss.org >>>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>>>> >>>>> >>>>> _______________________________________________ >>>>> infinispan-dev mailing list >>>>> infinispan-dev(a)lists.jboss.org >>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>>> >>>>> >>>>> -- >>>>> Manik Surtani >>>>> manik(a)jboss.org >>>>> Lead, Infinispan >>>>> Lead, JBoss Cache >>>>> http://www.infinispan.org >>>>> http://www.jbosscache.org >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> infinispan-dev mailing list >>>>> infinispan-dev(a)lists.jboss.org >>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>>> >>>> >>>> _______________________________________________ >>>> infinispan-dev mailing list >>>> infinispan-dev(a)lists.jboss.org >>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>> >>>> >>>> >>>> _______________________________________________ >>>> infinispan-dev mailing list >>>> infinispan-dev(a)lists.jboss.org >>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>> >>> >>> _______________________________________________ >>> infinispan-dev mailing list >>> infinispan-dev(a)lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>> >>> >>> -- >>> Manik Surtani >>> manik(a)jboss.org >>> Lead, Infinispan >>> Lead, JBoss Cache >>> http://www.infinispan.org >>> http://www.jbosscache.org >>> >>> >>> >>> >>> >>> _______________________________________________ >>> infinispan-dev mailing list >>> infinispan-dev(a)lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>> >> >> > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev > > > -- > Manik Surtani > manik(a)jboss.org > Lead, Infinispan > Lead, JBoss Cache > http://www.infinispan.org > http://www.jbosscache.org > > > > > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev >