Ok, tested and working. Here is the patch. Could be improved slightly by adding the compression level. <div><br></div><div>Cherry on top, I suffix ".gz" to S3 names, so you can download and automagically uncompress them ;-)</div>
<div><br></div><div>Manik, what do we do with the JIRA issue ? The patch attached is a bit unrelated...</div><div><br></div><div><br></div><div><br></div><div>phil<br><br><div class="gmail_quote">On Thu, Feb 25, 2010 at 3:33 PM, Philippe Van Dyck <span dir="ltr"><<a href="mailto:pvdyck@gmail.com">pvdyck@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">Ok, I need a solution before the end of the day so I will choose the easiest way.<div><br><div>I plan to update CloudCacheStoreConfig and add a "compress" option.</div>
<div>Then I only need to change two simple methods "writeToBlob" and "readFromBlob" (thanks Adrian... next time make them protected ;-))</div>
<div><br></div><div>I will post a patch later.</div><div><br></div><div>phil<div><div></div><div class="h5"><br><br><div class="gmail_quote">On Thu, Feb 25, 2010 at 3:21 PM, Manik Surtani <span dir="ltr"><<a href="mailto:manik@jboss.org" target="_blank">manik@jboss.org</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word"><br><div><div><div>On 25 Feb 2010, at 14:17, Philippe Van Dyck wrote:</div>
<br><blockquote type="cite">Another solution is to include a "compression flag" in the version itself.</blockquote><div><br></div></div><div>That would work as well... then you could just handle the compression in the VAM</div>
<div><div></div><div><br><blockquote type="cite"><div><br></div><div>WDYT ?</div><div><br></div><div>phil <br><br><div class="gmail_quote">On Thu, Feb 25, 2010 at 3:15 PM, Philippe Van Dyck <span dir="ltr"><<a href="mailto:pvdyck@gmail.com" target="_blank">pvdyck@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">For my purpose, the cachestore delegator is enough...<div><br></div><div>Well, I'll take a second look at VersionAwareMarshaller...</div>
<div><br></div><div>phil<div><div></div><div><br><br><div class="gmail_quote">On Thu, Feb 25, 2010 at 3:10 PM, Manik Surtani <span dir="ltr"><<a href="mailto:manik@jboss.org" target="_blank">manik@jboss.org</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word">I think it makes more sense in the marshaller, simply because the compressed stream is useful elsewhere as well - such as the network, etc.<div>
<div></div><div><div><br><div><div>On 25 Feb 2010, at 13:40, Philippe Van Dyck wrote:</div><br><blockquote type="cite">Sounds like a better idea, closer to the real purpose since the Marshaller is used in other places... but how do I configure this ?<div>
<br></div><div>Anyway, the marshaller is done, I am testing it, but the code is very easy to adapt to a cachestore...</div>
<div><br></div><div>phil</div><div><br><br><div class="gmail_quote">On Thu, Feb 25, 2010 at 2:33 PM, Mircea Markus <span dir="ltr"><<a href="mailto:mircea.markus@jboss.com" target="_blank">mircea.markus@jboss.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word">Can't we handle this through a delegating CacheStore, that would compress the data and pass it on to the actual store for further processing (the actual store will take then the binary data and marshall it (ni marshalling I assume as it receives byte[]).?<div>
<div></div><div><br><div><div>On 25 Feb 2010, at 14:06, Philippe Van Dyck wrote:</div><br><blockquote type="cite">Done<div><br></div><div><a href="https://jira.jboss.org/jira/browse/ISPN-357" target="_blank">https://jira.jboss.org/jira/browse/ISPN-357</a></div>
<div><br></div><div>Phil<br><br><div class="gmail_quote">On Thu, Feb 25, 2010 at 12:22 PM, Manik Surtani <span dir="ltr"><<a href="mailto:manik@jboss.org" target="_blank">manik@jboss.org</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word"><br><div><div><div>On 25 Feb 2010, at 11:17, Philippe Van Dyck wrote:</div>
<br><blockquote type="cite">Thanks Manik but it is actually related to the compression of the stored and (https) transferred ones on the S3 side.<div>The S3 bill is 95% lower here... and https transfer time is lower too (you pay for booth).</div>
</blockquote><div><br></div></div><div>Makes sense. Create a JIRA for this, I'm sure such a feature would be generically useful to others as well - if you're happy to contribute your work back in. :)</div><div>
<br><blockquote type="cite"><div>The Marshaller interface is exposing a lot of methods and it looks like the call each other... </div></blockquote><div><br></div></div><div>The original Marshaller interface is legacy, from JBoss Cache, which was built as an extension to JGroups' Marshaller interface. It may look a little confusing as a result, but have a look at the AbstractMarshaller which should help simplify things. Further, look at the VersionAwareMarshaller, which is the entry point into the marshalling system.</div>
<div><br></div><div>HTH</div><div>Manik</div><div><div></div><div><div><br></div><div><br></div><br><blockquote type="cite"><div><br></div><div>phil <br><br><div class="gmail_quote">On Thu, Feb 25, 2010 at 12:11 PM, Manik Surtani <span dir="ltr"><<a href="mailto:manik@jboss.org" target="_blank">manik@jboss.org</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">If it is compression for transmission you are concerned about, you can do this by adding the JGroups COMPRESS protocol to your JGroups cfg.<br>
<div><div></div><div><br>
On 25 Feb 2010, at 11:02, philippe van dyck wrote:<br>
<br>
> Hi All,<br>
><br>
> Currently, I compress all data before sending it to the cache. Once compressed, I gain 95% of the JSonized qi4j objects.<br>
><br>
> I did some profiling during the load tests and compression is taking roughly 80% of the cpu time.<br>
> So I would like to compress only the data sent to the store, not in memory.<br>
><br>
> Looks like the Marshaller is my friend here, and I plan to write a compressing wrapper around it.<br>
><br>
> Now, when I look at it, I see two ways to wrap the compression process.<br>
><br>
> One way is with the ObjectInput / ObjectOutput but I am bothered by the reentrant flag.<br>
> The other is the ByteBuffer stuff, no concurrency problem here, but it looks like more work.<br>
><br>
> WDYT ?<br>
><br>
> Cheers,<br>
><br>
> phil<br>
</div></div>> _______________________________________________<br>
> infinispan-dev mailing list<br>
> <a href="mailto:infinispan-dev@lists.jboss.org" target="_blank">infinispan-dev@lists.jboss.org</a><br>
> <a href="https://lists.jboss.org/mailman/listinfo/infinispan-dev" target="_blank">https://lists.jboss.org/mailman/listinfo/infinispan-dev</a><br>
<br>
--<br>
Manik Surtani<br>
<a href="mailto:manik@jboss.org" target="_blank">manik@jboss.org</a><br>
Lead, Infinispan<br>
Lead, JBoss Cache<br>
<a href="http://www.infinispan.org/" target="_blank">http://www.infinispan.org</a><br>
<a href="http://www.jbosscache.org/" target="_blank">http://www.jbosscache.org</a><br>
<br>
<br>
<br>
<br>
<br>
_______________________________________________<br>
infinispan-dev mailing list<br>
<a href="mailto:infinispan-dev@lists.jboss.org" target="_blank">infinispan-dev@lists.jboss.org</a><br>
<a href="https://lists.jboss.org/mailman/listinfo/infinispan-dev" target="_blank">https://lists.jboss.org/mailman/listinfo/infinispan-dev</a><br>
</blockquote></div><br></div>
_______________________________________________<br>infinispan-dev mailing list<br><a href="mailto:infinispan-dev@lists.jboss.org" target="_blank">infinispan-dev@lists.jboss.org</a><br><a href="https://lists.jboss.org/mailman/listinfo/infinispan-dev" target="_blank">https://lists.jboss.org/mailman/listinfo/infinispan-dev</a></blockquote>
</div></div></div><div><div></div><div><br><div>
<span style="border-collapse:separate;color:rgb(0, 0, 0);font-family:Helvetica;font-size:12px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:auto;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><span style="border-collapse:separate;color:rgb(0, 0, 0);font-family:Helvetica;font-size:12px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><div style="word-wrap:break-word">
<div>--</div><div>Manik Surtani</div><div><a href="mailto:manik@jboss.org" target="_blank">manik@jboss.org</a></div><div>Lead, Infinispan</div><div>Lead, JBoss Cache</div><div><a href="http://www.infinispan.org/" target="_blank">http://www.infinispan.org</a></div>
<div><a href="http://www.jbosscache.org/" target="_blank">http://www.jbosscache.org</a></div><div><br></div></div></span><br></span><br>
</div>
<br></div></div></div><br>_______________________________________________<br>
infinispan-dev mailing list<br>
<a href="mailto:infinispan-dev@lists.jboss.org" target="_blank">infinispan-dev@lists.jboss.org</a><br>
<a href="https://lists.jboss.org/mailman/listinfo/infinispan-dev" target="_blank">https://lists.jboss.org/mailman/listinfo/infinispan-dev</a><br></blockquote></div><br></div>
_______________________________________________<br>infinispan-dev mailing list<br><a href="mailto:infinispan-dev@lists.jboss.org" target="_blank">infinispan-dev@lists.jboss.org</a><br><a href="https://lists.jboss.org/mailman/listinfo/infinispan-dev" target="_blank">https://lists.jboss.org/mailman/listinfo/infinispan-dev</a></blockquote>
</div><br></div></div></div><br>_______________________________________________<br>
infinispan-dev mailing list<br>
<a href="mailto:infinispan-dev@lists.jboss.org" target="_blank">infinispan-dev@lists.jboss.org</a><br>
<a href="https://lists.jboss.org/mailman/listinfo/infinispan-dev" target="_blank">https://lists.jboss.org/mailman/listinfo/infinispan-dev</a><br></blockquote></div><br></div>
_______________________________________________<br>infinispan-dev mailing list<br><a href="mailto:infinispan-dev@lists.jboss.org" target="_blank">infinispan-dev@lists.jboss.org</a><br><a href="https://lists.jboss.org/mailman/listinfo/infinispan-dev" target="_blank">https://lists.jboss.org/mailman/listinfo/infinispan-dev</a></blockquote>
</div><br><div>
<span style="border-collapse:separate;color:rgb(0, 0, 0);font-family:Helvetica;font-size:12px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:auto;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><span style="border-collapse:separate;color:rgb(0, 0, 0);font-family:Helvetica;font-size:12px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><div style="word-wrap:break-word">
<div>--</div><div>Manik Surtani</div><div><a href="mailto:manik@jboss.org" target="_blank">manik@jboss.org</a></div><div>Lead, Infinispan</div><div>Lead, JBoss Cache</div><div><a href="http://www.infinispan.org/" target="_blank">http://www.infinispan.org</a></div>
<div><a href="http://www.jbosscache.org/" target="_blank">http://www.jbosscache.org</a></div><div><br></div></div></span><br></span><br>
</div>
<br></div></div></div></div><br>_______________________________________________<br>
infinispan-dev mailing list<br>
<a href="mailto:infinispan-dev@lists.jboss.org" target="_blank">infinispan-dev@lists.jboss.org</a><br>
<a href="https://lists.jboss.org/mailman/listinfo/infinispan-dev" target="_blank">https://lists.jboss.org/mailman/listinfo/infinispan-dev</a><br></blockquote></div><br></div></div></div>
</blockquote></div><br></div>
_______________________________________________<br>infinispan-dev mailing list<br><a href="mailto:infinispan-dev@lists.jboss.org" target="_blank">infinispan-dev@lists.jboss.org</a><br><a href="https://lists.jboss.org/mailman/listinfo/infinispan-dev" target="_blank">https://lists.jboss.org/mailman/listinfo/infinispan-dev</a></blockquote>
</div></div></div><div><div></div><div><br><div>
<span style="border-collapse:separate;color:rgb(0, 0, 0);font-family:Helvetica;font-size:12px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:auto;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><span style="border-collapse:separate;color:rgb(0, 0, 0);font-family:Helvetica;font-size:12px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><div style="word-wrap:break-word">
<div>--</div><div>Manik Surtani</div><div><a href="mailto:manik@jboss.org" target="_blank">manik@jboss.org</a></div><div>Lead, Infinispan</div><div>Lead, JBoss Cache</div><div><a href="http://www.infinispan.org" target="_blank">http://www.infinispan.org</a></div>
<div><a href="http://www.jbosscache.org" target="_blank">http://www.jbosscache.org</a></div><div><br></div></div></span><br></span><br>
</div>
<br></div></div></div><br>_______________________________________________<br>
infinispan-dev mailing list<br>
<a href="mailto:infinispan-dev@lists.jboss.org" target="_blank">infinispan-dev@lists.jboss.org</a><br>
<a href="https://lists.jboss.org/mailman/listinfo/infinispan-dev" target="_blank">https://lists.jboss.org/mailman/listinfo/infinispan-dev</a><br></blockquote></div><br></div></div></div></div>
</blockquote></div><br></div>