If it is compression for transmission you are concerned about, you can do this by adding the JGroups COMPRESS protocol to your JGroups cfg. On 25 Feb 2010, at 11:02, philippe van dyck wrote: > Hi All, > > Currently, I compress all data before sending it to the cache. Once compressed, I gain 95% of the JSonized qi4j objects. > > I did some profiling during the load tests and compression is taking roughly 80% of the cpu time. > So I would like to compress only the data sent to the store, not in memory. > > Looks like the Marshaller is my friend here, and I plan to write a compressing wrapper around it. > > Now, when I look at it, I see two ways to wrap the compression process. > > One way is with the ObjectInput / ObjectOutput but I am bothered by the reentrant flag. > The other is the ByteBuffer stuff, no concurrency problem here, but it looks like more work. > > WDYT ? > > Cheers, > > phil > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev -- Manik Surtani manik(a)jboss.org Lead, Infinispan Lead, JBoss Cache http://www.infinispan.org http://www.jbosscache.org _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

On 25 Feb 2010, at 11:17, Philippe Van Dyck wrote: Thanks Manik but it is actually related to the compression of the stored and (https) transferred ones on the S3 side. The S3 bill is 95% lower here... and https transfer time is lower too (you pay for booth). Makes sense. Create a JIRA for this, I'm sure such a feature would be generically useful to others as well - if you're happy to contribute your work back in. :) The Marshaller interface is exposing a lot of methods and it looks like the call each other... The original Marshaller interface is legacy, from JBoss Cache, which was built as an extension to JGroups' Marshaller interface. It may look a little confusing as a result, but have a look at the AbstractMarshaller which should help simplify things. Further, look at the VersionAwareMarshaller, which is the entry point into the marshalling system. HTH Manik phil On Thu, Feb 25, 2010 at 12:11 PM, Manik Surtani <manik(a)jboss.org&gt; wrote: > If it is compression for transmission you are concerned about, you can do > this by adding the JGroups COMPRESS protocol to your JGroups cfg. > > On 25 Feb 2010, at 11:02, philippe van dyck wrote: > > > Hi All, > > > > Currently, I compress all data before sending it to the cache. Once > compressed, I gain 95% of the JSonized qi4j objects. > > > > I did some profiling during the load tests and compression is taking > roughly 80% of the cpu time. > > So I would like to compress only the data sent to the store, not in > memory. > > > > Looks like the Marshaller is my friend here, and I plan to write a > compressing wrapper around it. > > > > Now, when I look at it, I see two ways to wrap the compression process. > > > > One way is with the ObjectInput / ObjectOutput but I am bothered by the > reentrant flag. > > The other is the ByteBuffer stuff, no concurrency problem here, but it > looks like more work. > > > > WDYT ? > > > > Cheers, > > > > phil > > _______________________________________________ > > infinispan-dev mailing list > > infinispan-dev(a)lists.jboss.org > > https://lists.jboss.org/mailman/listinfo/infinispan-dev > > -- > Manik Surtani > manik(a)jboss.org > Lead, Infinispan > Lead, JBoss Cache > http://www.infinispan.org > http://www.jbosscache.org > > > > > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev > _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev -- Manik Surtani manik(a)jboss.org Lead, Infinispan Lead, JBoss Cache http://www.infinispan.org http://www.jbosscache.org _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

Can't we handle this through a delegating CacheStore, that would compress the data and pass it on to the actual store for further processing (the actual store will take then the binary data and marshall it (ni marshalling I assume as it receives byte[]).? On 25 Feb 2010, at 14:06, Philippe Van Dyck wrote: Done https://jira.jboss.org/jira/browse/ISPN-357 Phil On Thu, Feb 25, 2010 at 12:22 PM, Manik Surtani <manik(a)jboss.org&gt; wrote: > > On 25 Feb 2010, at 11:17, Philippe Van Dyck wrote: > > Thanks Manik but it is actually related to the compression of the stored > and (https) transferred ones on the S3 side. > The S3 bill is 95% lower here... and https transfer time is lower too (you > pay for booth). > > > Makes sense. Create a JIRA for this, I'm sure such a feature would be > generically useful to others as well - if you're happy to contribute your > work back in. :) > > The Marshaller interface is exposing a lot of methods and it looks like > the call each other... > > > The original Marshaller interface is legacy, from JBoss Cache, which was > built as an extension to JGroups' Marshaller interface. It may look a > little confusing as a result, but have a look at the AbstractMarshaller > which should help simplify things. Further, look at the > VersionAwareMarshaller, which is the entry point into the marshalling > system. > > HTH > Manik > > > > > phil > > On Thu, Feb 25, 2010 at 12:11 PM, Manik Surtani <manik(a)jboss.org&gt; wrote: > >> If it is compression for transmission you are concerned about, you can do >> this by adding the JGroups COMPRESS protocol to your JGroups cfg. >> >> On 25 Feb 2010, at 11:02, philippe van dyck wrote: >> >> > Hi All, >> > >> > Currently, I compress all data before sending it to the cache. Once >> compressed, I gain 95% of the JSonized qi4j objects. >> > >> > I did some profiling during the load tests and compression is taking >> roughly 80% of the cpu time. >> > So I would like to compress only the data sent to the store, not in >> memory. >> > >> > Looks like the Marshaller is my friend here, and I plan to write a >> compressing wrapper around it. >> > >> > Now, when I look at it, I see two ways to wrap the compression process. >> > >> > One way is with the ObjectInput / ObjectOutput but I am bothered by the >> reentrant flag. >> > The other is the ByteBuffer stuff, no concurrency problem here, but it >> looks like more work. >> > >> > WDYT ? >> > >> > Cheers, >> > >> > phil >> > _______________________________________________ >> > infinispan-dev mailing list >> > infinispan-dev(a)lists.jboss.org >> > https://lists.jboss.org/mailman/listinfo/infinispan-dev >> >> -- >> Manik Surtani >> manik(a)jboss.org >> Lead, Infinispan >> Lead, JBoss Cache >> http://www.infinispan.org >> http://www.jbosscache.org >> >> >> >> >> >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev >> > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev > > > -- > Manik Surtani > manik(a)jboss.org > Lead, Infinispan > Lead, JBoss Cache > http://www.infinispan.org > http://www.jbosscache.org > > > > > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev > _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

Sounds like a better idea, closer to the real purpose since the Marshaller is used in other places... but how do I configure this ? Anyway, the marshaller is done, I am testing it, but the code is very easy to adapt to a cachestore... phil On Thu, Feb 25, 2010 at 2:33 PM, Mircea Markus <mircea.markus(a)jboss.com&gt; wrote: Can't we handle this through a delegating CacheStore, that would compress the data and pass it on to the actual store for further processing (the actual store will take then the binary data and marshall it (ni marshalling I assume as it receives byte[]).? On 25 Feb 2010, at 14:06, Philippe Van Dyck wrote: > Done > > https://jira.jboss.org/jira/browse/ISPN-357 > > Phil > > On Thu, Feb 25, 2010 at 12:22 PM, Manik Surtani <manik(a)jboss.org&gt; wrote: > > On 25 Feb 2010, at 11:17, Philippe Van Dyck wrote: > >> Thanks Manik but it is actually related to the compression of the stored and (https) transferred ones on the S3 side. >> The S3 bill is 95% lower here... and https transfer time is lower too (you pay for booth). > > Makes sense. Create a JIRA for this, I'm sure such a feature would be generically useful to others as well - if you're happy to contribute your work back in. :) > >> The Marshaller interface is exposing a lot of methods and it looks like the call each other... > > The original Marshaller interface is legacy, from JBoss Cache, which was built as an extension to JGroups' Marshaller interface. It may look a little confusing as a result, but have a look at the AbstractMarshaller which should help simplify things. Further, look at the VersionAwareMarshaller, which is the entry point into the marshalling system. > > HTH > Manik > > > >> >> phil >> >> On Thu, Feb 25, 2010 at 12:11 PM, Manik Surtani <manik(a)jboss.org&gt; wrote: >> If it is compression for transmission you are concerned about, you can do this by adding the JGroups COMPRESS protocol to your JGroups cfg. >> >> On 25 Feb 2010, at 11:02, philippe van dyck wrote: >> >> > Hi All, >> > >> > Currently, I compress all data before sending it to the cache. Once compressed, I gain 95% of the JSonized qi4j objects. >> > >> > I did some profiling during the load tests and compression is taking roughly 80% of the cpu time. >> > So I would like to compress only the data sent to the store, not in memory. >> > >> > Looks like the Marshaller is my friend here, and I plan to write a compressing wrapper around it. >> > >> > Now, when I look at it, I see two ways to wrap the compression process. >> > >> > One way is with the ObjectInput / ObjectOutput but I am bothered by the reentrant flag. >> > The other is the ByteBuffer stuff, no concurrency problem here, but it looks like more work. >> > >> > WDYT ? >> > >> > Cheers, >> > >> > phil >> > _______________________________________________ >> > infinispan-dev mailing list >> > infinispan-dev(a)lists.jboss.org >> > https://lists.jboss.org/mailman/listinfo/infinispan-dev >> >> -- >> Manik Surtani >> manik(a)jboss.org >> Lead, Infinispan >> Lead, JBoss Cache >> http://www.infinispan.org >> http://www.jbosscache.org >> >> >> >> >> >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev >> >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev > > > -- > Manik Surtani > manik(a)jboss.org > Lead, Infinispan > Lead, JBoss Cache > http://www.infinispan.org > http://www.jbosscache.org > > > > > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

For my purpose, the cachestore delegator is enough... Well, I'll take a second look at VersionAwareMarshaller... phil On Thu, Feb 25, 2010 at 3:10 PM, Manik Surtani <manik(a)jboss.org&gt; wrote: > I think it makes more sense in the marshaller, simply because the > compressed stream is useful elsewhere as well - such as the network, etc. > > On 25 Feb 2010, at 13:40, Philippe Van Dyck wrote: > > Sounds like a better idea, closer to the real purpose since the Marshaller > is used in other places... but how do I configure this ? > > Anyway, the marshaller is done, I am testing it, but the code is very easy > to adapt to a cachestore... > > phil > > > On Thu, Feb 25, 2010 at 2:33 PM, Mircea Markus <mircea.markus(a)jboss.com&gt;wrote: > >> Can't we handle this through a delegating CacheStore, that would compress >> the data and pass it on to the actual store for further processing (the >> actual store will take then the binary data and marshall it (ni marshalling >> I assume as it receives byte[]).? >> >> On 25 Feb 2010, at 14:06, Philippe Van Dyck wrote: >> >> Done >> >> https://jira.jboss.org/jira/browse/ISPN-357 >> >> Phil >> >> On Thu, Feb 25, 2010 at 12:22 PM, Manik Surtani <manik(a)jboss.org&gt; wrote: >> >>> >>> On 25 Feb 2010, at 11:17, Philippe Van Dyck wrote: >>> >>> Thanks Manik but it is actually related to the compression of the stored >>> and (https) transferred ones on the S3 side. >>> The S3 bill is 95% lower here... and https transfer time is lower too >>> (you pay for booth). >>> >>> >>> Makes sense. Create a JIRA for this, I'm sure such a feature would be >>> generically useful to others as well - if you're happy to contribute your >>> work back in. :) >>> >>> The Marshaller interface is exposing a lot of methods and it looks like >>> the call each other... >>> >>> >>> The original Marshaller interface is legacy, from JBoss Cache, which was >>> built as an extension to JGroups' Marshaller interface. It may look a >>> little confusing as a result, but have a look at the AbstractMarshaller >>> which should help simplify things. Further, look at the >>> VersionAwareMarshaller, which is the entry point into the marshalling >>> system. >>> >>> HTH >>> Manik >>> >>> >>> >>> >>> phil >>> >>> On Thu, Feb 25, 2010 at 12:11 PM, Manik Surtani <manik(a)jboss.org&gt;wrote: >>> >>>> If it is compression for transmission you are concerned about, you can >>>> do this by adding the JGroups COMPRESS protocol to your JGroups cfg. >>>> >>>> On 25 Feb 2010, at 11:02, philippe van dyck wrote: >>>> >>>> > Hi All, >>>> > >>>> > Currently, I compress all data before sending it to the cache. Once >>>> compressed, I gain 95% of the JSonized qi4j objects. >>>> > >>>> > I did some profiling during the load tests and compression is taking >>>> roughly 80% of the cpu time. >>>> > So I would like to compress only the data sent to the store, not in >>>> memory. >>>> > >>>> > Looks like the Marshaller is my friend here, and I plan to write a >>>> compressing wrapper around it. >>>> > >>>> > Now, when I look at it, I see two ways to wrap the compression >>>> process. >>>> > >>>> > One way is with the ObjectInput / ObjectOutput but I am bothered by >>>> the reentrant flag. >>>> > The other is the ByteBuffer stuff, no concurrency problem here, but >>>> it looks like more work. >>>> > >>>> > WDYT ? >>>> > >>>> > Cheers, >>>> > >>>> > phil >>>> > _______________________________________________ >>>> > infinispan-dev mailing list >>>> > infinispan-dev(a)lists.jboss.org >>>> > https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>> >>>> -- >>>> Manik Surtani >>>> manik(a)jboss.org >>>> Lead, Infinispan >>>> Lead, JBoss Cache >>>> http://www.infinispan.org >>>> http://www.jbosscache.org >>>> >>>> >>>> >>>> >>>> >>>> _______________________________________________ >>>> infinispan-dev mailing list >>>> infinispan-dev(a)lists.jboss.org >>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>> >>> >>> _______________________________________________ >>> infinispan-dev mailing list >>> infinispan-dev(a)lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>> >>> >>> -- >>> Manik Surtani >>> manik(a)jboss.org >>> Lead, Infinispan >>> Lead, JBoss Cache >>> http://www.infinispan.org >>> http://www.jbosscache.org >>> >>> >>> >>> >>> >>> _______________________________________________ >>> infinispan-dev mailing list >>> infinispan-dev(a)lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>> >> >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev >> >> >> >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev >> > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev > > > -- > Manik Surtani > manik(a)jboss.org > Lead, Infinispan > Lead, JBoss Cache > http://www.infinispan.org > http://www.jbosscache.org > > > > > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev >

On 25 Feb 2010, at 14:17, Philippe Van Dyck wrote: Another solution is to include a "compression flag" in the version itself. That would work as well... then you could just handle the compression in the VAM WDYT ? phil On Thu, Feb 25, 2010 at 3:15 PM, Philippe Van Dyck <pvdyck(a)gmail.com&gt;wrote: > For my purpose, the cachestore delegator is enough... > > Well, I'll take a second look at VersionAwareMarshaller... > > phil > > > On Thu, Feb 25, 2010 at 3:10 PM, Manik Surtani <manik(a)jboss.org&gt; wrote: > >> I think it makes more sense in the marshaller, simply because the >> compressed stream is useful elsewhere as well - such as the network, etc. >> >> On 25 Feb 2010, at 13:40, Philippe Van Dyck wrote: >> >> Sounds like a better idea, closer to the real purpose since the >> Marshaller is used in other places... but how do I configure this ? >> >> Anyway, the marshaller is done, I am testing it, but the code is very >> easy to adapt to a cachestore... >> >> phil >> >> >> On Thu, Feb 25, 2010 at 2:33 PM, Mircea Markus <mircea.markus(a)jboss.com&gt;wrote: >> >>> Can't we handle this through a delegating CacheStore, that would >>> compress the data and pass it on to the actual store for further processing >>> (the actual store will take then the binary data and marshall it (ni >>> marshalling I assume as it receives byte[]).? >>> >>> On 25 Feb 2010, at 14:06, Philippe Van Dyck wrote: >>> >>> Done >>> >>> https://jira.jboss.org/jira/browse/ISPN-357 >>> >>> Phil >>> >>> On Thu, Feb 25, 2010 at 12:22 PM, Manik Surtani <manik(a)jboss.org&gt;wrote: >>> >>>> >>>> On 25 Feb 2010, at 11:17, Philippe Van Dyck wrote: >>>> >>>> Thanks Manik but it is actually related to the compression of the >>>> stored and (https) transferred ones on the S3 side. >>>> The S3 bill is 95% lower here... and https transfer time is lower too >>>> (you pay for booth). >>>> >>>> >>>> Makes sense. Create a JIRA for this, I'm sure such a feature would be >>>> generically useful to others as well - if you're happy to contribute your >>>> work back in. :) >>>> >>>> The Marshaller interface is exposing a lot of methods and it looks like >>>> the call each other... >>>> >>>> >>>> The original Marshaller interface is legacy, from JBoss Cache, which >>>> was built as an extension to JGroups' Marshaller interface. It may look a >>>> little confusing as a result, but have a look at the AbstractMarshaller >>>> which should help simplify things. Further, look at the >>>> VersionAwareMarshaller, which is the entry point into the marshalling >>>> system. >>>> >>>> HTH >>>> Manik >>>> >>>> >>>> >>>> >>>> phil >>>> >>>> On Thu, Feb 25, 2010 at 12:11 PM, Manik Surtani <manik(a)jboss.org&gt;wrote: >>>> >>>>> If it is compression for transmission you are concerned about, you can >>>>> do this by adding the JGroups COMPRESS protocol to your JGroups cfg. >>>>> >>>>> On 25 Feb 2010, at 11:02, philippe van dyck wrote: >>>>> >>>>> > Hi All, >>>>> > >>>>> > Currently, I compress all data before sending it to the cache. Once >>>>> compressed, I gain 95% of the JSonized qi4j objects. >>>>> > >>>>> > I did some profiling during the load tests and compression is taking >>>>> roughly 80% of the cpu time. >>>>> > So I would like to compress only the data sent to the store, not in >>>>> memory. >>>>> > >>>>> > Looks like the Marshaller is my friend here, and I plan to write a >>>>> compressing wrapper around it. >>>>> > >>>>> > Now, when I look at it, I see two ways to wrap the compression >>>>> process. >>>>> > >>>>> > One way is with the ObjectInput / ObjectOutput but I am bothered by >>>>> the reentrant flag. >>>>> > The other is the ByteBuffer stuff, no concurrency problem here, but >>>>> it looks like more work. >>>>> > >>>>> > WDYT ? >>>>> > >>>>> > Cheers, >>>>> > >>>>> > phil >>>>> > _______________________________________________ >>>>> > infinispan-dev mailing list >>>>> > infinispan-dev(a)lists.jboss.org >>>>> > https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>>> >>>>> -- >>>>> Manik Surtani >>>>> manik(a)jboss.org >>>>> Lead, Infinispan >>>>> Lead, JBoss Cache >>>>> http://www.infinispan.org >>>>> http://www.jbosscache.org >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> infinispan-dev mailing list >>>>> infinispan-dev(a)lists.jboss.org >>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>>> >>>> >>>> _______________________________________________ >>>> infinispan-dev mailing list >>>> infinispan-dev(a)lists.jboss.org >>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>> >>>> >>>> -- >>>> Manik Surtani >>>> manik(a)jboss.org >>>> Lead, Infinispan >>>> Lead, JBoss Cache >>>> http://www.infinispan.org >>>> http://www.jbosscache.org >>>> >>>> >>>> >>>> >>>> >>>> _______________________________________________ >>>> infinispan-dev mailing list >>>> infinispan-dev(a)lists.jboss.org >>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>> >>> >>> _______________________________________________ >>> infinispan-dev mailing list >>> infinispan-dev(a)lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>> >>> >>> >>> _______________________________________________ >>> infinispan-dev mailing list >>> infinispan-dev(a)lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>> >> >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev >> >> >> -- >> Manik Surtani >> manik(a)jboss.org >> Lead, Infinispan >> Lead, JBoss Cache >> http://www.infinispan.org >> http://www.jbosscache.org >> >> >> >> >> >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev >> > > _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev -- Manik Surtani manik(a)jboss.org Lead, Infinispan Lead, JBoss Cache http://www.infinispan.org http://www.jbosscache.org _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

Ok, tested and working. Here is the patch. Could be improved slightly by adding the compression level. Cherry on top, I suffix ".gz" to S3 names, so you can download and automagically uncompress them ;-) Manik, what do we do with the JIRA issue ? The patch attached is a bit unrelated... phil On Thu, Feb 25, 2010 at 3:33 PM, Philippe Van Dyck <pvdyck(a)gmail.com&gt; wrote: Ok, I need a solution before the end of the day so I will choose the easiest way. I plan to update CloudCacheStoreConfig and add a "compress" option. Then I only need to change two simple methods "writeToBlob" and "readFromBlob" (thanks Adrian... next time make them protected ;-)) I will post a patch later. phil On Thu, Feb 25, 2010 at 3:21 PM, Manik Surtani <manik(a)jboss.org&gt; wrote: On 25 Feb 2010, at 14:17, Philippe Van Dyck wrote: > Another solution is to include a "compression flag" in the version itself. That would work as well... then you could just handle the compression in the VAM > > WDYT ? > > phil > > On Thu, Feb 25, 2010 at 3:15 PM, Philippe Van Dyck <pvdyck(a)gmail.com&gt; wrote: > For my purpose, the cachestore delegator is enough... > > Well, I'll take a second look at VersionAwareMarshaller... > > phil > > > On Thu, Feb 25, 2010 at 3:10 PM, Manik Surtani <manik(a)jboss.org&gt; wrote: > I think it makes more sense in the marshaller, simply because the compressed stream is useful elsewhere as well - such as the network, etc. > > On 25 Feb 2010, at 13:40, Philippe Van Dyck wrote: > >> Sounds like a better idea, closer to the real purpose since the Marshaller is used in other places... but how do I configure this ? >> >> Anyway, the marshaller is done, I am testing it, but the code is very easy to adapt to a cachestore... >> >> phil >> >> >> On Thu, Feb 25, 2010 at 2:33 PM, Mircea Markus <mircea.markus(a)jboss.com&gt; wrote: >> Can't we handle this through a delegating CacheStore, that would compress the data and pass it on to the actual store for further processing (the actual store will take then the binary data and marshall it (ni marshalling I assume as it receives byte[]).? >> >> On 25 Feb 2010, at 14:06, Philippe Van Dyck wrote: >> >>> Done >>> >>> https://jira.jboss.org/jira/browse/ISPN-357 >>> >>> Phil >>> >>> On Thu, Feb 25, 2010 at 12:22 PM, Manik Surtani <manik(a)jboss.org&gt; wrote: >>> >>> On 25 Feb 2010, at 11:17, Philippe Van Dyck wrote: >>> >>>> Thanks Manik but it is actually related to the compression of the stored and (https) transferred ones on the S3 side. >>>> The S3 bill is 95% lower here... and https transfer time is lower too (you pay for booth). >>> >>> Makes sense. Create a JIRA for this, I'm sure such a feature would be generically useful to others as well - if you're happy to contribute your work back in. :) >>> >>>> The Marshaller interface is exposing a lot of methods and it looks like the call each other... >>> >>> The original Marshaller interface is legacy, from JBoss Cache, which was built as an extension to JGroups' Marshaller interface. It may look a little confusing as a result, but have a look at the AbstractMarshaller which should help simplify things. Further, look at the VersionAwareMarshaller, which is the entry point into the marshalling system. >>> >>> HTH >>> Manik >>> >>> >>> >>>> >>>> phil >>>> >>>> On Thu, Feb 25, 2010 at 12:11 PM, Manik Surtani <manik(a)jboss.org&gt; wrote: >>>> If it is compression for transmission you are concerned about, you can do this by adding the JGroups COMPRESS protocol to your JGroups cfg. >>>> >>>> On 25 Feb 2010, at 11:02, philippe van dyck wrote: >>>> >>>> > Hi All, >>>> > >>>> > Currently, I compress all data before sending it to the cache. Once compressed, I gain 95% of the JSonized qi4j objects. >>>> > >>>> > I did some profiling during the load tests and compression is taking roughly 80% of the cpu time. >>>> > So I would like to compress only the data sent to the store, not in memory. >>>> > >>>> > Looks like the Marshaller is my friend here, and I plan to write a compressing wrapper around it. >>>> > >>>> > Now, when I look at it, I see two ways to wrap the compression process. >>>> > >>>> > One way is with the ObjectInput / ObjectOutput but I am bothered by the reentrant flag. >>>> > The other is the ByteBuffer stuff, no concurrency problem here, but it looks like more work. >>>> > >>>> > WDYT ? >>>> > >>>> > Cheers, >>>> > >>>> > phil >>>> > _______________________________________________ >>>> > infinispan-dev mailing list >>>> > infinispan-dev(a)lists.jboss.org >>>> > https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>> >>>> -- >>>> Manik Surtani >>>> manik(a)jboss.org >>>> Lead, Infinispan >>>> Lead, JBoss Cache >>>> http://www.infinispan.org >>>> http://www.jbosscache.org >>>> >>>> >>>> >>>> >>>> >>>> _______________________________________________ >>>> infinispan-dev mailing list >>>> infinispan-dev(a)lists.jboss.org >>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>> >>>> _______________________________________________ >>>> infinispan-dev mailing list >>>> infinispan-dev(a)lists.jboss.org >>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>> >>> >>> -- >>> Manik Surtani >>> manik(a)jboss.org >>> Lead, Infinispan >>> Lead, JBoss Cache >>> http://www.infinispan.org >>> http://www.jbosscache.org >>> >>> >>> >>> >>> >>> _______________________________________________ >>> infinispan-dev mailing list >>> infinispan-dev(a)lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>> >>> _______________________________________________ >>> infinispan-dev mailing list >>> infinispan-dev(a)lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >> >> >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev >> >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev > > -- > Manik Surtani > manik(a)jboss.org > Lead, Infinispan > Lead, JBoss Cache > http://www.infinispan.org > http://www.jbosscache.org > > > > > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev > > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev -- Manik Surtani manik(a)jboss.org Lead, Infinispan Lead, JBoss Cache http://www.infinispan.org http://www.jbosscache.org _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev <cloud.patch>_______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

On 25 Feb 2010, at 16:49, Manik Surtani wrote: RE: JIRA, why don't you change it then to say something like "Add compression to CloudCacheStore" done

The patch looks good, although I'd recommend using 3 spaces instead of tabs. ;)

Apart from that, feel free to commit this to trunk yourself, so that your name is attached to the commit.

Just noticed that you're not on the committers list. If you fill out the contributors form on https://www.jboss.org/contribute

and send me your jboss.org username I'll add you as a committer.

Cheers Manik Cheers Manik On 25 Feb 2010, at 16:14, Philippe Van Dyck wrote: Ok, tested and working. Here is the patch. Could be improved slightly by adding the compression level. Cherry on top, I suffix ".gz" to S3 names, so you can download and automagically uncompress them ;-) Manik, what do we do with the JIRA issue ? The patch attached is a bit unrelated... phil On Thu, Feb 25, 2010 at 3:33 PM, Philippe Van Dyck <pvdyck(a)gmail.com&gt;wrote: > Ok, I need a solution before the end of the day so I will choose the > easiest way. > > I plan to update CloudCacheStoreConfig and add a "compress" option. > Then I only need to change two simple methods "writeToBlob" and > "readFromBlob" (thanks Adrian... next time make them protected ;-)) > > I will post a patch later. > > phil > > > On Thu, Feb 25, 2010 at 3:21 PM, Manik Surtani <manik(a)jboss.org&gt; wrote: > >> >> On 25 Feb 2010, at 14:17, Philippe Van Dyck wrote: >> >> Another solution is to include a "compression flag" in the version >> itself. >> >> >> That would work as well... then you could just handle the compression in >> the VAM >> >> >> WDYT ? >> >> phil >> >> On Thu, Feb 25, 2010 at 3:15 PM, Philippe Van Dyck <pvdyck(a)gmail.com&gt;wrote: >> >>> For my purpose, the cachestore delegator is enough... >>> >>> Well, I'll take a second look at VersionAwareMarshaller... >>> >>> phil >>> >>> >>> On Thu, Feb 25, 2010 at 3:10 PM, Manik Surtani <manik(a)jboss.org&gt; wrote: >>> >>>> I think it makes more sense in the marshaller, simply because the >>>> compressed stream is useful elsewhere as well - such as the network, etc. >>>> >>>> On 25 Feb 2010, at 13:40, Philippe Van Dyck wrote: >>>> >>>> Sounds like a better idea, closer to the real purpose since the >>>> Marshaller is used in other places... but how do I configure this ? >>>> >>>> Anyway, the marshaller is done, I am testing it, but the code is very >>>> easy to adapt to a cachestore... >>>> >>>> phil >>>> >>>> >>>> On Thu, Feb 25, 2010 at 2:33 PM, Mircea Markus < >>>> mircea.markus(a)jboss.com&gt; wrote: >>>> >>>>> Can't we handle this through a delegating CacheStore, that would >>>>> compress the data and pass it on to the actual store for further processing >>>>> (the actual store will take then the binary data and marshall it (ni >>>>> marshalling I assume as it receives byte[]).? >>>>> >>>>> On 25 Feb 2010, at 14:06, Philippe Van Dyck wrote: >>>>> >>>>> Done >>>>> >>>>> https://jira.jboss.org/jira/browse/ISPN-357 >>>>> >>>>> Phil >>>>> >>>>> On Thu, Feb 25, 2010 at 12:22 PM, Manik Surtani <manik(a)jboss.org&gt;wrote: >>>>> >>>>>> >>>>>> On 25 Feb 2010, at 11:17, Philippe Van Dyck wrote: >>>>>> >>>>>> Thanks Manik but it is actually related to the compression of the >>>>>> stored and (https) transferred ones on the S3 side. >>>>>> The S3 bill is 95% lower here... and https transfer time is lower too >>>>>> (you pay for booth). >>>>>> >>>>>> >>>>>> Makes sense. Create a JIRA for this, I'm sure such a feature would >>>>>> be generically useful to others as well - if you're happy to contribute your >>>>>> work back in. :) >>>>>> >>>>>> The Marshaller interface is exposing a lot of methods and it looks >>>>>> like the call each other... >>>>>> >>>>>> >>>>>> The original Marshaller interface is legacy, from JBoss Cache, which >>>>>> was built as an extension to JGroups' Marshaller interface. It may look a >>>>>> little confusing as a result, but have a look at the AbstractMarshaller >>>>>> which should help simplify things. Further, look at the >>>>>> VersionAwareMarshaller, which is the entry point into the marshalling >>>>>> system. >>>>>> >>>>>> HTH >>>>>> Manik >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> phil >>>>>> >>>>>> On Thu, Feb 25, 2010 at 12:11 PM, Manik Surtani <manik(a)jboss.org&gt;wrote: >>>>>> >>>>>>> If it is compression for transmission you are concerned about, you >>>>>>> can do this by adding the JGroups COMPRESS protocol to your JGroups cfg. >>>>>>> >>>>>>> On 25 Feb 2010, at 11:02, philippe van dyck wrote: >>>>>>> >>>>>>> > Hi All, >>>>>>> > >>>>>>> > Currently, I compress all data before sending it to the cache. >>>>>>> Once compressed, I gain 95% of the JSonized qi4j objects. >>>>>>> > >>>>>>> > I did some profiling during the load tests and compression is >>>>>>> taking roughly 80% of the cpu time. >>>>>>> > So I would like to compress only the data sent to the store, not >>>>>>> in memory. >>>>>>> > >>>>>>> > Looks like the Marshaller is my friend here, and I plan to write a >>>>>>> compressing wrapper around it. >>>>>>> > >>>>>>> > Now, when I look at it, I see two ways to wrap the compression >>>>>>> process. >>>>>>> > >>>>>>> > One way is with the ObjectInput / ObjectOutput but I am bothered >>>>>>> by the reentrant flag. >>>>>>> > The other is the ByteBuffer stuff, no concurrency problem here, >>>>>>> but it looks like more work. >>>>>>> > >>>>>>> > WDYT ? >>>>>>> > >>>>>>> > Cheers, >>>>>>> > >>>>>>> > phil >>>>>>> > _______________________________________________ >>>>>>> > infinispan-dev mailing list >>>>>>> > infinispan-dev(a)lists.jboss.org >>>>>>> > https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>>>>> >>>>>>> -- >>>>>>> Manik Surtani >>>>>>> manik(a)jboss.org >>>>>>> Lead, Infinispan >>>>>>> Lead, JBoss Cache >>>>>>> http://www.infinispan.org >>>>>>> http://www.jbosscache.org >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> infinispan-dev mailing list >>>>>>> infinispan-dev(a)lists.jboss.org >>>>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> infinispan-dev mailing list >>>>>> infinispan-dev(a)lists.jboss.org >>>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>>>> >>>>>> >>>>>> -- >>>>>> Manik Surtani >>>>>> manik(a)jboss.org >>>>>> Lead, Infinispan >>>>>> Lead, JBoss Cache >>>>>> http://www.infinispan.org >>>>>> http://www.jbosscache.org >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> infinispan-dev mailing list >>>>>> infinispan-dev(a)lists.jboss.org >>>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>>>> >>>>> >>>>> _______________________________________________ >>>>> infinispan-dev mailing list >>>>> infinispan-dev(a)lists.jboss.org >>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> infinispan-dev mailing list >>>>> infinispan-dev(a)lists.jboss.org >>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>>> >>>> >>>> _______________________________________________ >>>> infinispan-dev mailing list >>>> infinispan-dev(a)lists.jboss.org >>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>> >>>> >>>> -- >>>> Manik Surtani >>>> manik(a)jboss.org >>>> Lead, Infinispan >>>> Lead, JBoss Cache >>>> http://www.infinispan.org >>>> http://www.jbosscache.org >>>> >>>> >>>> >>>> >>>> >>>> _______________________________________________ >>>> infinispan-dev mailing list >>>> infinispan-dev(a)lists.jboss.org >>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>> >>> >>> >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev >> >> >> -- >> Manik Surtani >> manik(a)jboss.org >> Lead, Infinispan >> Lead, JBoss Cache >> http://www.infinispan.org >> http://www.jbosscache.org >> >> >> >> >> >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev >> > > <cloud.patch>_______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev -- Manik Surtani manik(a)jboss.org Lead, Infinispan Lead, JBoss Cache http://www.infinispan.org http://www.jbosscache.org _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev -- Manik Surtani manik(a)jboss.org Lead, Infinispan Lead, JBoss Cache http://www.infinispan.org http://www.jbosscache.org _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

On Thu, Feb 25, 2010 at 5:50 PM, Manik Surtani <manik(a)jboss.org&gt; wrote: > > On 25 Feb 2010, at 16:49, Manik Surtani wrote: > > RE: JIRA, why don't you change it then to say something like "Add > compression to CloudCacheStore" > > done > > The patch looks good, although I'd recommend using 3 spaces instead of > tabs. ;) > > Do you have a formatting configuration to apply to eclipse ?

> > Apart from that, feel free to commit this to trunk yourself, so that > your > name is attached to the commit. > > > np > Just noticed that you're not on the committers list. If you fill out > the > contributors form on > > https://www.jboss.org/contribute > > done > and send me your jboss.org username I'll add you as a committer. > > jboss username : pvdyck cheers phil > Cheers > Manik > > > > > > Cheers > Manik > > On 25 Feb 2010, at 16:14, Philippe Van Dyck wrote: > > Ok, tested and working. Here is the patch. Could be improved slightly by > adding the compression level. > > Cherry on top, I suffix ".gz" to S3 names, so you can download and > automagically uncompress them ;-) > > Manik, what do we do with the JIRA issue ? The patch attached is a bit > unrelated... > > > > phil > > On Thu, Feb 25, 2010 at 3:33 PM, Philippe Van Dyck > <pvdyck(a)gmail.com&gt;wrote: > >> Ok, I need a solution before the end of the day so I will choose the >> easiest way. >> >> I plan to update CloudCacheStoreConfig and add a "compress" option. >> Then I only need to change two simple methods "writeToBlob" and >> "readFromBlob" (thanks Adrian... next time make them protected ;-)) >> >> I will post a patch later. >> >> phil >> >> >> On Thu, Feb 25, 2010 at 3:21 PM, Manik Surtani <manik(a)jboss.org&gt; wrote: >> >>> >>> On 25 Feb 2010, at 14:17, Philippe Van Dyck wrote: >>> >>> Another solution is to include a "compression flag" in the version >>> itself. >>> >>> >>> That would work as well... then you could just handle the >>> compression in >>> the VAM >>> >>> >>> WDYT ? >>> >>> phil >>> >>> On Thu, Feb 25, 2010 at 3:15 PM, Philippe Van Dyck >>> <pvdyck(a)gmail.com&gt;wrote: >>> >>>> For my purpose, the cachestore delegator is enough... >>>> >>>> Well, I'll take a second look at VersionAwareMarshaller... >>>> >>>> phil >>>> >>>> >>>> On Thu, Feb 25, 2010 at 3:10 PM, Manik Surtani <manik(a)jboss.org&gt; >>>> wrote: >>>> >>>>> I think it makes more sense in the marshaller, simply because the >>>>> compressed stream is useful elsewhere as well - such as the >>>>> network, etc. >>>>> >>>>> On 25 Feb 2010, at 13:40, Philippe Van Dyck wrote: >>>>> >>>>> Sounds like a better idea, closer to the real purpose since the >>>>> Marshaller is used in other places... but how do I configure this ? >>>>> >>>>> Anyway, the marshaller is done, I am testing it, but the code is >>>>> very >>>>> easy to adapt to a cachestore... >>>>> >>>>> phil >>>>> >>>>> >>>>> On Thu, Feb 25, 2010 at 2:33 PM, Mircea Markus < >>>>> mircea.markus(a)jboss.com&gt; wrote: >>>>> >>>>>> Can't we handle this through a delegating CacheStore, that would >>>>>> compress the data and pass it on to the actual store for further >>>>>> processing >>>>>> (the actual store will take then the binary data and marshall it >>>>>> (ni >>>>>> marshalling I assume as it receives byte[]).? >>>>>> >>>>>> On 25 Feb 2010, at 14:06, Philippe Van Dyck wrote: >>>>>> >>>>>> Done >>>>>> >>>>>> https://jira.jboss.org/jira/browse/ISPN-357 >>>>>> >>>>>> Phil >>>>>> >>>>>> On Thu, Feb 25, 2010 at 12:22 PM, Manik Surtani >>>>>> <manik(a)jboss.org&gt;wrote: >>>>>> >>>>>>> >>>>>>> On 25 Feb 2010, at 11:17, Philippe Van Dyck wrote: >>>>>>> >>>>>>> Thanks Manik but it is actually related to the compression of the >>>>>>> stored and (https) transferred ones on the S3 side. >>>>>>> The S3 bill is 95% lower here... and https transfer time is lower >>>>>>> too >>>>>>> (you pay for booth). >>>>>>> >>>>>>> >>>>>>> Makes sense. Create a JIRA for this, I'm sure such a feature >>>>>>> would >>>>>>> be generically useful to others as well - if you're happy to >>>>>>> contribute your >>>>>>> work back in. :) >>>>>>> >>>>>>> The Marshaller interface is exposing a lot of methods and it looks >>>>>>> like the call each other... >>>>>>> >>>>>>> >>>>>>> The original Marshaller interface is legacy, from JBoss Cache, >>>>>>> which >>>>>>> was built as an extension to JGroups' Marshaller interface. It >>>>>>> may look a >>>>>>> little confusing as a result, but have a look at the >>>>>>> AbstractMarshaller >>>>>>> which should help simplify things. Further, look at the >>>>>>> VersionAwareMarshaller, which is the entry point into the >>>>>>> marshalling >>>>>>> system. >>>>>>> >>>>>>> HTH >>>>>>> Manik >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> phil >>>>>>> >>>>>>> On Thu, Feb 25, 2010 at 12:11 PM, Manik Surtani >>>>>>> <manik(a)jboss.org&gt;wrote: >>>>>>> >>>>>>>> If it is compression for transmission you are concerned about, >>>>>>>> you >>>>>>>> can do this by adding the JGroups COMPRESS protocol to your >>>>>>>> JGroups cfg. >>>>>>>> >>>>>>>> On 25 Feb 2010, at 11:02, philippe van dyck wrote: >>>>>>>> >>>>>>>> > Hi All, >>>>>>>> > >>>>>>>> > Currently, I compress all data before sending it to the cache. >>>>>>>> Once compressed, I gain 95% of the JSonized qi4j objects. >>>>>>>> > >>>>>>>> > I did some profiling during the load tests and compression is >>>>>>>> taking roughly 80% of the cpu time. >>>>>>>> > So I would like to compress only the data sent to the store, >>>>>>>> not >>>>>>>> in memory. >>>>>>>> > >>>>>>>> > Looks like the Marshaller is my friend here, and I plan to >>>>>>>> write a >>>>>>>> compressing wrapper around it. >>>>>>>> > >>>>>>>> > Now, when I look at it, I see two ways to wrap the compression >>>>>>>> process. >>>>>>>> > >>>>>>>> > One way is with the ObjectInput / ObjectOutput but I am >>>>>>>> bothered >>>>>>>> by the reentrant flag. >>>>>>>> > The other is the ByteBuffer stuff, no concurrency problem here, >>>>>>>> but it looks like more work. >>>>>>>> > >>>>>>>> > WDYT ? >>>>>>>> > >>>>>>>> > Cheers, >>>>>>>> > >>>>>>>> > phil >>>>>>>> > _______________________________________________ >>>>>>>> > infinispan-dev mailing list >>>>>>>> > infinispan-dev(a)lists.jboss.org >>>>>>>> > https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>>>>>> >>>>>>>> -- >>>>>>>> Manik Surtani >>>>>>>> manik(a)jboss.org >>>>>>>> Lead, Infinispan >>>>>>>> Lead, JBoss Cache >>>>>>>> http://www.infinispan.org >>>>>>>> http://www.jbosscache.org >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> infinispan-dev mailing list >>>>>>>> infinispan-dev(a)lists.jboss.org >>>>>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> infinispan-dev mailing list >>>>>>> infinispan-dev(a)lists.jboss.org >>>>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Manik Surtani >>>>>>> manik(a)jboss.org >>>>>>> Lead, Infinispan >>>>>>> Lead, JBoss Cache >>>>>>> http://www.infinispan.org >>>>>>> http://www.jbosscache.org >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> infinispan-dev mailing list >>>>>>> infinispan-dev(a)lists.jboss.org >>>>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> infinispan-dev mailing list >>>>>> infinispan-dev(a)lists.jboss.org >>>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>>>> >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> infinispan-dev mailing list >>>>>> infinispan-dev(a)lists.jboss.org >>>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>>>> >>>>> >>>>> _______________________________________________ >>>>> infinispan-dev mailing list >>>>> infinispan-dev(a)lists.jboss.org >>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>>> >>>>> >>>>> -- >>>>> Manik Surtani >>>>> manik(a)jboss.org >>>>> Lead, Infinispan >>>>> Lead, JBoss Cache >>>>> http://www.infinispan.org >>>>> http://www.jbosscache.org >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> infinispan-dev mailing list >>>>> infinispan-dev(a)lists.jboss.org >>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>>> >>>> >>>> >>> _______________________________________________ >>> infinispan-dev mailing list >>> infinispan-dev(a)lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>> >>> >>> -- >>> Manik Surtani >>> manik(a)jboss.org >>> Lead, Infinispan >>> Lead, JBoss Cache >>> http://www.infinispan.org >>> http://www.jbosscache.org >>> >>> >>> >>> >>> >>> _______________________________________________ >>> infinispan-dev mailing list >>> infinispan-dev(a)lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>> >> >> > <cloud.patch>_______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev > > > -- > Manik Surtani > manik(a)jboss.org > Lead, Infinispan > Lead, JBoss Cache > http://www.infinispan.org > http://www.jbosscache.org > > > > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev > > > -- > Manik Surtani > manik(a)jboss.org > Lead, Infinispan > Lead, JBoss Cache > http://www.infinispan.org > http://www.jbosscache.org > > > > > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev >

Hi All, Currently, I compress all data before sending it to the cache. Once compressed, I gain 95% of the JSonized qi4j objects. I did some profiling during the load tests and compression is taking roughly 80% of the cpu time. So I would like to compress only the data sent to the store, not in memory. Looks like the Marshaller is my friend here, and I plan to write a compressing wrapper around it. Now, when I look at it, I see two ways to wrap the compression process. One way is with the ObjectInput / ObjectOutput but I am bothered by the reentrant flag.

The other is the ByteBuffer stuff, no concurrency problem here, but it looks like more work. WDYT ? Cheers, phil _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

Thanks for the reentrant scenario Galder. https://jira.jboss.org/jira/browse/ISPN-357 is now closed. If the Marshaller is used for something else than storing cache entries, I don't think it is a good idea to implement compression at this level.

Compression is cpu intensive, and it may be a good idea to "prepare" entries in memory (with a low priority thread), like adding a "compressed" flag to a cache entry. This way, they are ready for storage or transfer... they consume less memory, but they cost much more to use (decompression time). In fact, it is a very old tradeoff and IMO if compression should be integrated in Infinispan, it is at a higher level -- and another discussion. > From my point of view, S3 entries are now compressed and cost less to > transfer and store, it was my initial goal. cheers, phil Le 26 févr. 2010 à 11:16, Galder Zamarreno a écrit : > On Thu, 25 Feb 2010 12:02:34 +0100, philippe van dyck <pvdyck(a)gmail.com&gt; > wrote: > >> Hi All, >> >> Currently, I compress all data before sending it to the cache. Once >> compressed, I gain 95% of the JSonized qi4j objects. >> >> I did some profiling during the load tests and compression is taking >> roughly 80% of the cpu time. >> So I would like to compress only the data sent to the store, not in >> memory. >> >> Looks like the Marshaller is my friend here, and I plan to write a >> compressing wrapper around it. >> >> Now, when I look at it, I see two ways to wrap the compression process. >> >> One way is with the ObjectInput / ObjectOutput but I am bothered by the >> reentrant flag. > > As a side note, the reentrant flag is used to signal the marshaller > whether several ObjectOutput/ObjectInput as open without a close, i.e. > -- > marshaller.startObjectOutput(x, false) > marshaller.startObjectOutput(x, true) -> is reentrant, so mark it as > such > -- > marshaller.startObjectOutput(x, false) > marshaller.finishObjectOutput() > marshaller.startObjectOutput(x, false) -> not reentrant > marshaller.finishObjectOutput() > -- > > Why do we use this? To enable marshaller implementations to return a > different ObjectOutput if the call is reentrant. If you look at > org.infinispan.marshall.jboss.JBossMarshaller you see that the > ObjectOutput (or org.jboss.marshalling.Marshaller) is a ThreadLocal, but > JBossMarshaller does not allow for the same > org.jboss.marshalling.Marshaller to be opened twice. So, by using the > reentrant flag, we can make sure that the 2nd time that > startObjectOutput > is called, a different one is provided. > > For an example of reentrancy, see the javadoc: > > * <p>On the other hand, when a call is reentrant, i.e. > startObjectOutput/startObjectOutput(reentrant)...finishObjectOutput/finishObjectOutput, > * the Marshaller implementation might treat it differently. An > example > of reentrancy would be marshalling of {@link MarshalledValue}. > * When sending or storing a MarshalledValue, a call to > startObjectOutput() would occur so that the stream is open and > * following, a 2nd call could occur so that MarshalledValue's raw > byte > array version is calculated and sent accross. > * This enables lazy deserialization on the receiver side which is > performance gain. The Marshaller implementation could decide > * that it needs a separate ObjectOutput or similar for the 2nd call > since it's aim is only to get the raw byte array version > * and the close finish with it.</p> > > The second reentrant call is the one to create the MarshalledValue form > of > the in memory data. The first call would be the stream opened to send > the > put or get or whichever op you're sending around. > > As a side note, using ThreadLocal is a much cleaner solution to having > to > maintain a pool of org.jboss.marshalling.Marshaller instances. > > Hope this clarifies further what the reentrant stuff does. > > Cheers, > >> The other is the ByteBuffer stuff, no concurrency problem here, but it >> looks like more work. >> >> WDYT ? >> >> Cheers, >> >> phil >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev > > > -- > Galder Zamarreño > Sr. Software Engineer > Infinispan, JBoss Cache > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev