<html><head></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">Hi Adrian and thanks for your answer.<div><br></div><div>I dug into the source of Infinispan S3 cache store and, no offense, but it looks more like a proof of concept than something I could use in production.</div><div><br></div><div>First of all, in order to achieve a minimum of efficiency, we need to use concurrency in this specific cache store.</div><div>Since it is using the JCloud's Map interface and not the Future<> asynchronous one... well, you know that, you wrote a big part of JClouds ;-)</div><div><br></div><div>The CacheStore interface does not offer an asynchronous solution, but a workaround is available.</div><div><br></div><div>I just modified the S3 cache store, and every write operation is now asynchronous and the resulting future is stored in a ThreadLocal queue.</div><div>After each transaction (S3CacheStore.applyModifications) I empty the queue and wait for each Future to finish, in order to catch errors (and allow rollbacks... or else the whole transaction mechanism is useless).</div><div><br></div><div>The drawback is obvious, if you don't use a transaction manager to update the cache, exceptions will die silently (but come on, nobody does that ;-).</div><div>The solution is working and I updated 1000 entries in ... 20 seconds (for me it means 'mission accomplished').</div><div><br></div><div>Secondly, there are still a couple of very strange things happening in the S3 cache store, but the most intriguing one is definitely <span class="Apple-style-span" style="color: rgb(245, 230, 187); font-family: Monaco; font-size: 11px; "><font class="Apple-style-span" color="#000000">JCloudsBucket </font><span class="Apple-style-span" style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: medium; "> <span class="Apple-style-span" style="color: rgb(226, 227, 213); font-family: Monaco; font-size: 11px; "><font class="Apple-style-span" color="#000000">public Set<Bucket> values()</font><font class="Apple-style-span" color="#000000" face="Helvetica"><span class="Apple-style-span" style="font-size: medium;">.</span></font></span></span></span></div><div><br></div><div>Is it really serious ? Must we be able to load *ALL* of our data in order to rehash on some cluster join/leave operation ?</div><div><br></div><div>I plan to store a couple of 10's of GB on S3 so... well you see the problem.</div><div>It seems especially problematic since I was planning to use Amazon's EC2 autoscale feature to add Infinispan instances to my 'previously' working cluster.</div><div><br></div><div>I am quite sure I misunderstood something, or maybe all the rest.</div><div><br></div><div>Any help most welcome.</div><div><br></div><div>Philippe </div><div><br></div><div><br><div><div>Le 2 déc. 2009 à 18:19, Adrian Cole a écrit :</div><br class="Apple-interchange-newline"><blockquote type="cite"><div>Hi, phillipe.<br><br>Apologies about the differences in trunk not being in infinispan, yet.<br> Obviously that version would help narrow down what's going on.<br><br>when you mentioned this: "it is abysmal (when using httpclient<br>directly I had a min of 100/sec using 20 connections)."<br><br>Are you saying you are comparing to code that is writing to S3 via<br>httpclient apis?<br><br>S3 tends to have a 80ms minimum overhead on PUT commands as measured<br>by <a href="http://hostedftp.com">hostedftp.com</a> [1]<br><br>Our normal perf tests from trunk do about 100 concurrent puts in 2-3<br>seconds untuned to s3 without the use of nio.<br><br>Cheers,<br>-Adrian<br><br><br>[1] <a href="http://hostedftp.wordpress.com/2009/06/30/hostedftp-amazon-aws-s3-performance-report-how-fast-is-the-cloud/">http://hostedftp.wordpress.com/2009/06/30/hostedftp-amazon-aws-s3-performance-report-how-fast-is-the-cloud/</a><br><br>On Wed, Dec 2, 2009 at 6:06 AM, Bela Ban <<a href="mailto:bban@redhat.com">bban@redhat.com</a>> wrote:<br><blockquote type="cite">OK, then someone from the Infinispan team needs to help you.<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">If the option to use write-behind (instead of write-through) for a cache<br></blockquote><blockquote type="cite">loader still exists, that might be a perf boost.<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">The basic issue with the S3 cache loader is that it needs to send slow<br></blockquote><blockquote type="cite">and bulky SOAP messages to S3, and that's always slow. I don't know the<br></blockquote><blockquote type="cite">current S3 cache loader impl, but I suggest take a look and see what<br></blockquote><blockquote type="cite">properties they support. E.g. they might have an option to batch updates<br></blockquote><blockquote type="cite">and write them to S3 in collected form.<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">philippe van dyck wrote:<br></blockquote><blockquote type="cite"><blockquote type="cite">Thanks for your help Bela.<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">Indeed, when I replace the S3 cache store with the disk one, the performance problem disappears (takes less than a second to store my 100 'put' updates when I commit the transaction).<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">Here is my config :<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><?xml version="1.0" encoding="UTF-8"?><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><infinispan xmlns:xsi="<a href="http://www.w3.org/2001/XMLSchema-instance">http://www.w3.org/2001/XMLSchema-instance</a>"<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"> xmlns="urn:infinispan:config:4.0"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"> <global><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"> <transport<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"> transportClass="org.infinispan.remoting.transport.jgroups.JGroupsTransport"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"> <properties><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"> <property name="configurationFile" value="jgroups.xml" /><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"> </properties><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"> </transport><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"> </global><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"> <default><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"> <transaction<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"> transactionManagerLookupClass="org.infinispan.transaction.lookup.DummyTransactionManagerLookup" /><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"> <clustering mode="distribution"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"> <l1 enabled="true" lifespan="100000" /><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"> <hash numOwners="2" rehashRpcTimeout="120000" /><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"> </clustering><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"> <loaders passivation="false" shared="true" preload="false"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"> <loader class="org.infinispan.loaders.s3.S3CacheStore"<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"> fetchPersistentState="false" ignoreModifications="false"<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"> purgeOnStartup="false"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"> <properties><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"> <property name="awsAccessKey" value="xxx" /><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"> <property name="awsSecretKey" value="xxx" /><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"> <property name="bucketPrefix" value="store" /><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"> </properties><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"> <async enabled="true"/><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"> </loader><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"> </loaders><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"> <unsafe unreliableReturnValues="true" /><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"> </default><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"></infinispan><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">And the log :<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">INFO (14:54:37): JGroupsTransport - Starting JGroups Channel<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">INFO (14:54:38): JChannel - JGroups version: 2.8.0.CR5<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">-------------------------------------------------------------------<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">GMS: address=sakapuss.local-16157, cluster=Infinispan-Cluster, physical address=192.168.1.136:7800<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">-------------------------------------------------------------------<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">INFO (14:54:49): JGroupsTransport - Received new cluster view: [sakapuss.local-16157|0] [sakapuss.local-16157]<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">INFO (14:54:49): JGroupsTransport - Cache local address is sakapuss.local-16157, physical address is 192.168.1.136:7800<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">INFO (14:54:49): GlobalComponentRegistry - Infinispan version: Infinispan 'Starobrno' 4.0.0.CR2<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">INFO (14:54:49): AsyncStore - Async cache loader starting org.infinispan.loaders.decorators.AsyncStore@7254d7ac<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">WARN (14:54:51): utureCommandConnectionPool - org.jclouds.http.httpnio.pool.HttpNioFutureCommandConnectionPool@fcd4eca1 - saturated connection pool<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">INFO (14:54:52): ComponentRegistry - Infinispan version: Infinispan 'Starobrno' 4.0.0.CR2<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">please note the HttpNioFutureCommandConnectionPool@fcd4eca1 - saturated connection pool (??)<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">Philippe<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">Le 2 déc. 2009 à 14:28, Bela Ban a écrit :<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">Just to narrow down the issue: when you disable the S3 cache store, I<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">assume the performance problem goes away, correct ?<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">Just trying to pin the blame on the S3 cache loader, then I don't even<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">need to see whether it is a JGroups problem... :-)<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">philippe van dyck wrote:<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">Hi Infinispan mailing list,<br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">a couple of days ago, I succeeded in writing an entity store for qi4j (<a href="http://www.qi4j.org/">http://www.qi4j.org/</a>) using Infinispan, the S3 store and the S3_PING JGroups clustering configuration.<br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">It works like a charm, discovers new EC2 instances, synchronizes and process transactions perfectly... you did an amazing job.<br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">But I have a serious performance problems.<br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">When I write an update (<1k) to the cache, it takes around 500 ms to be stored on S3.<br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">The best result I achieved was around 10 cache writes per second... it is abysmal (when using httpclient directly I had a min of 100/sec using 20 connections).<br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">When I commit a JTA transaction made of 100 cache writes, it takes around 30 seconds (cpu<5%) and the first write ends on S3 after at least 5 seconds of 'idle' time (SSL negotiation??).<br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">I disabled the store asynchronism and work without JTA transactions, no effect on performance.<br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">I also modified the jClouds configuration, multiplying by 10 all worker threads, connections and the rest... no improvement!<br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">When I (load) test my web app (wicket based+qi4j+...infinispan) the cpu stays idle (<5%) and ... JTA transactions fails (timeouts) and I cannot acquire locks before the 10 seconds timeout.<br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">Is there something fishy in the jclouds configuration ? in the httpnio use of jclouds ? in the version of jclouds (the trunk one with the blob store seems to be so different) ?<br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">Am I missing something ?<br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">Any pointer to any doc/help/experience is welcome ;-)<br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">Philippe<br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">_______________________________________________<br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">infinispan-dev mailing list<br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><a href="mailto:infinispan-dev@lists.jboss.org">infinispan-dev@lists.jboss.org</a><br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><a href="https://lists.jboss.org/mailman/listinfo/infinispan-dev">https://lists.jboss.org/mailman/listinfo/infinispan-dev</a><br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">--<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">Bela Ban<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">Lead JGroups / Clustering Team<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">JBoss<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">_______________________________________________<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">infinispan-dev mailing list<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><a href="mailto:infinispan-dev@lists.jboss.org">infinispan-dev@lists.jboss.org</a><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><a href="https://lists.jboss.org/mailman/listinfo/infinispan-dev">https://lists.jboss.org/mailman/listinfo/infinispan-dev</a><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">------------------------------------------------------------------------<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">_______________________________________________<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">infinispan-dev mailing list<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><a href="mailto:infinispan-dev@lists.jboss.org">infinispan-dev@lists.jboss.org</a><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><a href="https://lists.jboss.org/mailman/listinfo/infinispan-dev">https://lists.jboss.org/mailman/listinfo/infinispan-dev</a><br></blockquote></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">--<br></blockquote><blockquote type="cite">Bela Ban<br></blockquote><blockquote type="cite">Lead JGroups / Clustering Team<br></blockquote><blockquote type="cite">JBoss<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">_______________________________________________<br></blockquote><blockquote type="cite">infinispan-dev mailing list<br></blockquote><blockquote type="cite"><a href="mailto:infinispan-dev@lists.jboss.org">infinispan-dev@lists.jboss.org</a><br></blockquote><blockquote type="cite"><a href="https://lists.jboss.org/mailman/listinfo/infinispan-dev">https://lists.jboss.org/mailman/listinfo/infinispan-dev</a><br></blockquote><blockquote type="cite"><br></blockquote><br>_______________________________________________<br>infinispan-dev mailing list<br><a href="mailto:infinispan-dev@lists.jboss.org">infinispan-dev@lists.jboss.org</a><br>https://lists.jboss.org/mailman/listinfo/infinispan-dev<br></div></blockquote></div><br></div></body></html>